注:我是新的Flutter和面临一个死胡同后,一些故障排除的图像捕捉与相机的问题,任何见解/帮助,可以指出我的任何方向是非常感谢。
我正面临着一个奇怪的问题与我的Flutter应用程序拍摄的照片。图像看起来很好,元数据看起来也不错,但是使用机器学习模型识别坐标的处理是关闭的。(图1)。处理在一定程度上是成功的(能够成功识别物品),但边界框坐标相差甚远(图2和3)
关于应用程序:应用程序将图像>发布到后端,以便使用经过训练的机器学习模型进行处理并识别项目(如果需要,后端位于AWS上)。图像存储在s3 bucket中,使用lambda进行处理,我已经根据下面所示的冰箱图像训练了机器学习模型。识别将用边界框识别相应的项目。示例(图1)
问题:然而,无论我使用什么设备(ios/ android),每当我使用Flutter应用程序拍摄图像并上传时,图像在s3中显示正常,但边界框的处理非常不正常。
示例:(图2和图3)请忽略相对较低的分辨率,我尝试了多个高分辨率和类似质量的图像,但结果仍然显示相同。
在发布到s3之前,我没有在我的代码中对图像做任何事情。从边界框,似乎图像比例/尺寸是关闭的不知何故,但图像显示在s3的OK。我的图像捕获和发布代码在底部共享。
我为排除故障和排除潜在问题以找到根本原因所做的工作:
- 我尝试了两个不同的库,一个是通过相机包https://pub.dev/packages/camera(^0.10.5+3),另一个是图像拾取器库https://pub.dev/packages/image_picker(^1.0.4),两个图像拍摄返回类似的结果。但是,当我使用图像选择器画廊源加载从图像采取的React应用程序或网络摄像头,结果显示正常(即。类似于图1)。
- 我尝试使用其他(React应用程序/网络摄像头)进行类似的后端处理,(所有条件保持不变,照明等,物品位置,不同设备等)。所有返回的结果都是相似的。边界框未关闭。
- 通过直接/ API网关上传,两者返回相同的结果。我还尝试使用图像拾取器上传使用其他应用程序(React)拍摄的图像,图像将被处理并确定边界框,如图1所示。
- 也尝试了不同分辨率的图像,并排除了分辨率引起的问题,因为超高分辨率仍然显示边界框坐标偏离。(虽然能够成功识别)
我的图像捕获和上传代码:
使用普通相机库(相机已初始化ok)
图像捕获,然后到DisplayPictureScreen
,显示图片并在按下上传按钮后发布到s3。
FloatingActionButton(
// Provide an onPressed callback.
onPressed: () async {
// Take the Picture in a try / catch block. If anything goes wrong,
// catch the error.
try {
// Ensure that the camera is initialized.
await _initializeControllerFuture;
// Attempt to take a picture and get the file `image`
// where it was saved.
final image = await cameraController.takePicture();
if (!mounted) return;
// If the picture was taken, display it on a new screen.
await Navigator.of(context).push(
MaterialPageRoute(
builder: (context) => DisplayPictureScreen(
// Pass the automatically generated path to
// the DisplayPictureScreen widget.
imagePath: image.path,
),
),
);
} catch (e) {
// If an error occurs, log the error to the console.
debugPrint(e.toString());
}
},
child: const Icon(Icons.camera_alt),
),
显示图片和上传画面
class DisplayPictureScreen extends StatelessWidget {
final String imagePath;
const DisplayPictureScreen({super.key, required this.imagePath});
@override
Widget build(BuildContext context) {
return Scaffold(
appBar: AppBar(title: const Text('Upload and detect for ingredients')),
resizeToAvoidBottomInset: false,
body: SingleChildScrollView(
child: Column(
mainAxisAlignment: MainAxisAlignment.center,
children: [
Image.file(File(imagePath)),
ElevatedButton(
child: const Text('Upload'),
onPressed: () async {
// HTTP POST request to perform file upload
File imageFile = File(imagePath);
const region = 'us-west-2';
const signer = AWSSigV4Signer();
final scope = AWSCredentialScope(
region: region, service: AWSService.s3);
const bucketName = 'upload-bucket-name';
const host = '$bucketName.s3.$region.amazonaws.com';
final serviceConfiguration = S3ServiceConfiguration();
// Create a file and write some contents to it. Then, open it
// for reading.
final Directory directory =
await getApplicationDocumentsDirectory();
final filename = File('${directory.path}/testImage.jpg');
print(filename.path);
final uint8List = imageFile.readAsBytesSync();
final file = filename..writeAsBytesSync(uint8List);
final contents = file.openRead();
const key = '/testImage.jpg';
// Create a PUT request to the path of the file.
print('creating upload requests...');
final uploadRequest = AWSStreamedHttpRequest.raw(
method: AWSHttpMethod.put,
host: host,
// uri: Uri.parse('http://$host'),
path: key,
body: contents,
headers: {
AWSHeaders.host: host,
AWSHeaders.contentType: 'image/jpeg',
},
);
// Sign and send the upload request
print('signing and sending upload requests...');
final signedUploadRequest = await signer.sign(
uploadRequest,
credentialScope: scope,
serviceConfiguration: serviceConfiguration,
);
final uploadResponse = await signedUploadRequest.send();
}
),
],
),
),
);
}
}
除了使用AWS dart SDK https://pub.dev/packages/aws_signature_v4,我还尝试了另一种方式,通过API网关上传,并以用户身份登录Firebase auth(最初认为这可能是API网关更改任何元数据的问题)
class DisplayPictureScreen extends StatelessWidget {
final String imagePath;
const DisplayPictureScreen({super.key, required this.imagePath});
@override
Widget build(BuildContext context) {
return Scaffold(
appBar: AppBar(title: const Text('Upload and detect for ingredients')),
// The image is stored as a file on the device. Use the `Image.file`
// constructor with the given path to display the image.
resizeToAvoidBottomInset: false,
body: SingleChildScrollView(
child: Column(
mainAxisAlignment: MainAxisAlignment.center,
children: [
Image.file(File(imagePath)),
ElevatedButton(
child: const Text('Upload'),
onPressed: () async {
final user = FirebaseAuth.instance.currentUser;
final idToken = await user?.getIdTokenResult();
// HTTP POST request to perform file upload
File imageFile = File(imagePath);
// http upload - agw
final http.Response response;
try {
response = await http.post(
Uri.parse(
"https://my-api-gateway-url.amazonaws.com/Prod/"),
// Send authorization headers to the backend.
headers: {
HttpHeaders.authorizationHeader:
idToken!.token.toString(),
HttpHeaders.contentTypeHeader: 'image/jpeg',
},
// body: blob);
body: imageFile.readAsBytesSync());
final List result = json.decode(response.body);
final int length = result.length;
debugPrint(result.toString());
if (length == 0) {
Navigator.push(
context,
MaterialPageRoute(
builder: (context) => const NoResultScreen(),
),
);
} else {
await Navigator.of(context).push(
MaterialPageRoute(
builder: (context) => ListIngredientsPage(
resultData: result,
resultLength: length,
),
),
);
}
} on Exception catch (_) {
rethrow;
}
}
),
],
),
),
);
}
}
1条答案
按热度按时间1mrurvl11#
就像@Wh1t3rabbit所引导的那样,事实证明这是由于图像文件的元数据。不知道为什么默认情况下,flutter应用程序创建了一个包含
Image Orientation = Rotated 90 CW
元数据的图像。因此,它会干扰ML处理后端,因为处理过程会检索图像并基于坐标计算边界框。我这样做是为了解决这个问题:
获取镜像文件的EXIF:
删除镜像文件的EXIF
(来源:https://medium.com/swlh/removing-metadata-exif-data-from-images-in-flutter-fbefd615c554)
希望这对那些再次面临类似无法解释的情况的人有所帮助。