ios dart 相机在处理过程中具有奇怪的图像尺寸/宽高比,即使它看起来像正常图像

aemubtdh  于 2023-10-21  发布在  iOS
关注(0)|答案(1)|浏览(117)

注:我是新的Flutter和面临一个死胡同后,一些故障排除的图像捕捉与相机的问题,任何见解/帮助,可以指出我的任何方向是非常感谢。
我正面临着一个奇怪的问题与我的Flutter应用程序拍摄的照片。图像看起来很好,元数据看起来也不错,但是使用机器学习模型识别坐标的处理是关闭的。(图1)。处理在一定程度上是成功的(能够成功识别物品),但边界框坐标相差甚远(图2和3)
关于应用程序:应用程序将图像>发布到后端,以便使用经过训练的机器学习模型进行处理并识别项目(如果需要,后端位于AWS上)。图像存储在s3 bucket中,使用lambda进行处理,我已经根据下面所示的冰箱图像训练了机器学习模型。识别将用边界框识别相应的项目。示例(图1)

问题:然而,无论我使用什么设备(ios/ android),每当我使用Flutter应用程序拍摄图像并上传时,图像在s3中显示正常,但边界框的处理非常不正常。
示例:(图2和图3)请忽略相对较低的分辨率,我尝试了多个高分辨率和类似质量的图像,但结果仍然显示相同。

在发布到s3之前,我没有在我的代码中对图像做任何事情。从边界框,似乎图像比例/尺寸是关闭的不知何故,但图像显示在s3的OK。我的图像捕获和发布代码在底部共享。
我为排除故障和排除潜在问题以找到根本原因所做的工作:

  • 我尝试了两个不同的库,一个是通过相机包https://pub.dev/packages/camera(^0.10.5+3),另一个是图像拾取器库https://pub.dev/packages/image_picker(^1.0.4),两个图像拍摄返回类似的结果。但是,当我使用图像选择器画廊源加载从图像采取的React应用程序或网络摄像头,结果显示正常(即。类似于图1)。
  • 我尝试使用其他(React应用程序/网络摄像头)进行类似的后端处理,(所有条件保持不变,照明等,物品位置,不同设备等)。所有返回的结果都是相似的。边界框未关闭。
  • 通过直接/ API网关上传,两者返回相同的结果。我还尝试使用图像拾取器上传使用其他应用程序(React)拍摄的图像,图像将被处理并确定边界框,如图1所示。
  • 也尝试了不同分辨率的图像,并排除了分辨率引起的问题,因为超高分辨率仍然显示边界框坐标偏离。(虽然能够成功识别)

我的图像捕获和上传代码:
使用普通相机库(相机已初始化ok)

图像捕获,然后到DisplayPictureScreen,显示图片并在按下上传按钮后发布到s3。

FloatingActionButton(
        // Provide an onPressed callback.
        onPressed: () async {
          // Take the Picture in a try / catch block. If anything goes wrong,
          // catch the error.
          try {
            // Ensure that the camera is initialized.
            await _initializeControllerFuture;

            // Attempt to take a picture and get the file `image`
            // where it was saved.
            final image = await cameraController.takePicture();

            if (!mounted) return;

            // If the picture was taken, display it on a new screen.
            await Navigator.of(context).push(
              MaterialPageRoute(
                builder: (context) => DisplayPictureScreen(
                  // Pass the automatically generated path to
                  // the DisplayPictureScreen widget.
                  imagePath: image.path,
                ),
              ),
            );
          } catch (e) {
            // If an error occurs, log the error to the console.
            debugPrint(e.toString());
          }
        },
        child: const Icon(Icons.camera_alt),
      ),

显示图片和上传画面

class DisplayPictureScreen extends StatelessWidget {
  final String imagePath;

  const DisplayPictureScreen({super.key, required this.imagePath});

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: const Text('Upload and detect for ingredients')),
      resizeToAvoidBottomInset: false,
      body: SingleChildScrollView(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: [
            Image.file(File(imagePath)),
            ElevatedButton(
                child: const Text('Upload'),
                onPressed: () async {
                  // HTTP POST request to perform file upload
                  File imageFile = File(imagePath);
                  const region = 'us-west-2';
                  const signer = AWSSigV4Signer();
                  final scope = AWSCredentialScope(
                      region: region, service: AWSService.s3);
                  const bucketName = 'upload-bucket-name';
                  const host = '$bucketName.s3.$region.amazonaws.com';
                  final serviceConfiguration = S3ServiceConfiguration();

// Create a file and write some contents to it. Then, open it
// for reading.
                  final Directory directory =
                      await getApplicationDocumentsDirectory();
                  final filename = File('${directory.path}/testImage.jpg');
                  print(filename.path);
                  final uint8List = imageFile.readAsBytesSync();
                  final file = filename..writeAsBytesSync(uint8List);
                  final contents = file.openRead();
                  const key = '/testImage.jpg';

// Create a PUT request to the path of the file.
                  print('creating upload requests...');
                  final uploadRequest = AWSStreamedHttpRequest.raw(
                    method: AWSHttpMethod.put,
                    host: host,
                    // uri: Uri.parse('http://$host'),
                    path: key,
                    body: contents,
                    headers: {
                      AWSHeaders.host: host,
                      AWSHeaders.contentType: 'image/jpeg',
                    },
                  );

// Sign and send the upload request
                  print('signing and sending upload requests...');
                  final signedUploadRequest = await signer.sign(
                    uploadRequest,
                    credentialScope: scope,
                    serviceConfiguration: serviceConfiguration,
                  );
                  final uploadResponse = await signedUploadRequest.send();
                }
                ),
          ],
        ),
      ),
    );
  }
}

除了使用AWS dart SDK https://pub.dev/packages/aws_signature_v4,我还尝试了另一种方式,通过API网关上传,并以用户身份登录Firebase auth(最初认为这可能是API网关更改任何元数据的问题)

class DisplayPictureScreen extends StatelessWidget {
  final String imagePath;

  const DisplayPictureScreen({super.key, required this.imagePath});

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: const Text('Upload and detect for ingredients')),
      // The image is stored as a file on the device. Use the `Image.file`
      // constructor with the given path to display the image.
      resizeToAvoidBottomInset: false,
      body: SingleChildScrollView(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: [
            Image.file(File(imagePath)),
            ElevatedButton(
                child: const Text('Upload'),
                onPressed: () async {
                  final user = FirebaseAuth.instance.currentUser;
                  final idToken = await user?.getIdTokenResult();

                  // HTTP POST request to perform file upload
                  File imageFile = File(imagePath);

                  // http upload - agw
                  final http.Response response;
                  try {
                    response = await http.post(
                        Uri.parse(
                            "https://my-api-gateway-url.amazonaws.com/Prod/"),
                        // Send authorization headers to the backend.
                        headers: {
                          HttpHeaders.authorizationHeader:
                              idToken!.token.toString(),
                          HttpHeaders.contentTypeHeader: 'image/jpeg',
                        },
                        // body: blob);
                        body: imageFile.readAsBytesSync());

                    final List result = json.decode(response.body);
                    final int length = result.length;
                    debugPrint(result.toString());

                    if (length == 0) {
                      Navigator.push(
                        context,
                        MaterialPageRoute(
                          builder: (context) => const NoResultScreen(),
                        ),
                      );
                    } else {
                      await Navigator.of(context).push(
                        MaterialPageRoute(
                          builder: (context) => ListIngredientsPage(
                            resultData: result,
                            resultLength: length,
                          ),
                        ),
                      );
                    }
                  } on Exception catch (_) {
                    rethrow;
                  } 
                }
                ),
          ],
        ),
      ),
    );
  }
}
1mrurvl1

1mrurvl11#

就像@Wh1t3rabbit所引导的那样,事实证明这是由于图像文件的元数据。不知道为什么默认情况下,flutter应用程序创建了一个包含Image Orientation = Rotated 90 CW元数据的图像。因此,它会干扰ML处理后端,因为处理过程会检索图像并基于坐标计算边界框。
我这样做是为了解决这个问题:

获取镜像文件的EXIF:

final fileBytes = cleanFilename.readAsBytesSync();
final data = await readExifFromBytes(fileBytes);
debugPrint('image metadata: ');
for (final e in data.entries) {
    debugPrint('${e.key} = ${e.value}');
}

删除镜像文件的EXIF

(来源:https://medium.com/swlh/removing-metadata-exif-data-from-images-in-flutter-fbefd615c554

final Directory directory =
await getApplicationDocumentsDirectory();
final filename = File('${directory.path}/testImage.jpg');
final targetPath = '${directory.path}/cleanImage.jpg';
await FlutterImageCompress.compressAndGetFile(
      filename.path, targetPath);

debugPrint(targetPath);
final cleanFilename = File(targetPath);

希望这对那些再次面临类似无法解释的情况的人有所帮助。

相关问题