ios 如何从AVCapturePhoto对象将深度数据(AVDepthData)写入照片文件?

ltqd579y  于 2023-06-25  发布在  iOS
关注(0)|答案(1)|浏览(111)

我已经实现了类似于Apple's example的解决方案,用iPhone激光雷达摄像头捕获深度数据。主要代码片段如下:
1.设置深度格式

let device = AVCaptureDevice.default(.builtInLiDARDepthCamera, for: .video, position: .back)!
let vidFormatsWithDepth = device.formats.filter { format in
    format.formatDescription.dimensions.width == 1920 &&
    format.formatDescription.mediaSubType.rawValue == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange &&
    !format.isVideoBinned &&
    !format.supportedDepthDataFormats.isEmpty &&
    format.supportedDepthDataFormats.contains { $0.formatDescription.mediaSubType.rawValue == kCVPixelFormatType_DepthFloat16 }
}

if let format = vidFormatsWithDepth.first {
    let depthFormats = format.supportedDepthDataFormats.filter { $0.formatDescription.mediaSubType.rawValue == kCVPixelFormatType_DepthFloat16 }
    try! device.lockForConfiguration()
    device.activeFormat = format
    device.activeDepthDataFormat = depthFormats.last
    device.unlockForConfiguration()
}

1.照片输出

func setUpPhotoOutput() {
    photoOutput = AVCapturePhotoOutput()
    photoOutput.maxPhotoQualityPrioritization = .quality
    self.captureSession.addOutput(photoOutput)
    photoOutput.isDepthDataDeliveryEnabled = photoOutput.isDepthDataDeliverySupported
}

1.拍摄照片

var format: [String: Any] = [:]
if photoOutput.availablePhotoPixelFormatTypes.contains(kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) {
    format[kCVPixelBufferPixelFormatTypeKey as String] = kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
}
let settings = AVCapturePhotoSettings(format: format)
settings.isDepthDataDeliveryEnabled = photoOutput.isDepthDataDeliveryEnabled
settings.isDepthDataFiltered = photoOutput.isDepthDataDeliveryEnabled
settings.embedsDepthDataInPhoto = photoOutput.isDepthDataDeliveryEnabled
photoOutput.capturePhoto(with: settings , delegate: self)

1.处理捕获的照片数据

func createPhotoFile(
    photo: AVCapturePhoto
) {
    let customizer = PhotoDataCustomizer()
    var mainImageData = photo.fileDataRepresentation(with: customizer)!
    // note mainImageData should have embeded depth data, but...
    let imageSource = CGImageSourceCreateWithData(mainImageData as CFData, nil)!
    let depthDataDict = CGImageSourceCopyAuxiliaryDataInfoAtIndex(
        imageSource,
        0,
        kCGImageAuxiliaryDataTypeDepth
    )
    let disparityDataDict = CGImageSourceCopyAuxiliaryDataInfoAtIndex(
        imageSource,
        0,
        kCGImageAuxiliaryDataTypeDisparity
    )
    print("depthDataDict", depthDataDict ?? "nil")
    print("disparityDataDict", disparityDataDict ?? "nil")
    // ... both depthDataDict and disparityDataDict come out as nil
}

class PhotoDataCustomizer: NSObject, AVCapturePhotoFileDataRepresentationCustomizer {
    func replacementDepthData(for photo: AVCapturePhoto) -> AVDepthData? {
        let depthData = photo.depthData?.converting(toDepthDataType: kCVPixelFormatType_DepthFloat16)
        return depthData
    }
}

AVCapturePhoto的photo.depthData是存在的(不是nil),我希望它是嵌入的,如果settings.embedsDepthDataInPhoto = true,但两个深度数据变体(kCGImageAuxiliaryDataTypeDepth,kCGImageAuxiliaryDataTypeDisparity)从CGImageSource中出来为nil。
如何正确读取照片文件中的深度数据...还是首先正确地写入深度数据?

wnrlj8wa

wnrlj8wa1#

我能够通过手动添加它来保存深度数据,绕过var mainImageData = photo.fileDataRepresentation(with: customizer)!,如下所示:

func createPhotoFile(
    photo: AVCapturePhoto
) {
    let cgMainImage = photo.cgImageRepresentation()!
    let mainImageData = NSMutableData()
    let imageDest = CGImageDestinationCreateWithData(mainImageData, imageFormat as CFString, 1, nil)!
    var options: [String: Any] = photo.metadata
    options[kCGImageDestinationLossyCompressionQuality as String] = 0.8
    options[kCGImagePropertyOrientation as String] = NSNumber(value: orientation.rawValue)
    CGImageDestinationAddImage(imageDest, cgMainImage, options as CFDictionary)

    var depthDataVariant: NSString?
    if let depthData = photo.depthData?.converting(toDepthDataType: kCVPixelFormatType_DepthFloat16)
        .applyingExifOrientation(orientation),
        let depthDataDict = depthData.dictionaryRepresentation(forAuxiliaryDataType: &depthDataVariant),
        let depthDataVariant = depthDataVariant
    {
        CGImageDestinationAddAuxiliaryDataInfo(imageDest, depthDataVariant, depthDataDict as CFDictionary)
    }
    CGImageDestinationFinalize(imageDest)

    // final test ...
    let imageSource = CGImageSourceCreateWithData(mainImageData as CFData, nil)!
    let depthDataDict = CGImageSourceCopyAuxiliaryDataInfoAtIndex(
        imageSource,
        0,
        kCGImageAuxiliaryDataTypeDepth
    )
    print("depthDataDict", depthDataDict ?? "nil")
    // ... now the depthDataDict exists!
}

相关问题