Android NDK/Camera2 Create OpenGL Surface from `ImageReader` Surface / Frame Processing

fkvaft9z  于 2023-05-12  发布在  Android
关注(0)|答案(1)|浏览(168)

我试图创建一个应用程序,从相机流帧,画一个红色的框到帧使用Skia,并最终显示在屏幕上(预览)。
我已经设法设置了Camera 2并将其连接到显示相机预览的SurfaceView,但我不知道如何正确设置中间帧处理部分。
我尝试创建一个ImageReader(Java):

// create image reader
imageReader = ImageReader.newInstance(width, height, ImageFormat.YUV_420_888, 3)
imageReader.setOnImageAvailableListener({ reader ->
  val image = reader.acquireLatestImage()
  Log.d(TAG, "Frame: ${image.width} x ${image.height}")
  nativeOnDraw(image)
  image.close()
}, null)

创建一个OpenGL/Skia曲面(C++):

EGLDisplay eglDisplay = eglGetDisplay(EGL_DEFAULT_DISPLAY);
if (eglDisplay == EGL_NO_DISPLAY) {
    throw std::runtime_error("Failed to get EGL Display! " + std::to_string(glGetError()));
}

EGLint major, minor;
if (!eglInitialize(eglDisplay, &major, &minor)) {
    throw std::runtime_error("Failed to initialize EGL! " + std::to_string(glGetError()));
}

EGLint att[] = {EGL_RENDERABLE_TYPE, EGL_OPENGL_ES2_BIT,
                EGL_SURFACE_TYPE, EGL_WINDOW_BIT,
                EGL_RECORDABLE_ANDROID, 1,
                EGL_ALPHA_SIZE, 8,
                EGL_BLUE_SIZE, 8,
                EGL_GREEN_SIZE, 8,
                EGL_RED_SIZE, 8,
                EGL_NONE};

EGLint numConfigs;
EGLConfig eglConfig;
if (!eglChooseConfig(eglDisplay, att, &eglConfig, 1, &numConfigs) ||
    numConfigs == 0) {
    throw std::runtime_error("Failed to choose a GL Config! " + std::to_string(glGetError()));
}

EGLint contextAttribs[] = {EGL_CONTEXT_CLIENT_VERSION, 2, EGL_NONE};
EGLContext eglContext = eglCreateContext(eglDisplay,
                                         eglConfig,
                                         EGL_NO_CONTEXT,
                                         contextAttribs);
if (eglContext == EGL_NO_CONTEXT) {
    throw std::runtime_error("Failed to create a GL Context! " + std::to_string(glGetError()));
}

EGLSurface eglSurface = eglCreateWindowSurface(eglDisplay, eglConfig, nativeWindow, nullptr);
if (!eglMakeCurrent(eglDisplay, eglSurface, eglSurface, eglContext)) {
    throw std::runtime_error("Failed to set current surface! " + std::to_string(glGetError()));
}

GLint buffer;
glGetIntegerv(GL_FRAMEBUFFER_BINDING, &buffer);
GLint stencil;
glGetIntegerv(GL_STENCIL_BITS, &stencil);
GLint samples;
glGetIntegerv(GL_SAMPLES, &samples);

// Create the Skia backend context
auto backendInterface = GrGLMakeNativeInterface();
auto grContext = GrDirectContext::MakeGL(backendInterface);
if (grContext == nullptr) {
    throw std::runtime_error("Failed to create Skia Context from GL Context! " + std::to_string(glGetError()));
}

auto maxSamples = grContext->maxSurfaceSampleCountForColorType(kRGBA_8888_SkColorType);
if (samples > maxSamples)
    samples = maxSamples;

GrGLFramebufferInfo fbInfo;
fbInfo.fFBOID = buffer;
fbInfo.fFormat = 0x8058; // GR_GL_RGBA8

auto renderTarget = GrBackendRenderTarget(width, height, samples, stencil, fbInfo);

struct RenderContext {
    EGLDisplay display;
    EGLSurface surface;
};
auto ctx = new RenderContext({eglDisplay, eglSurface});

auto surface = SkSurface::MakeFromBackendRenderTarget(
        grContext.get(), renderTarget, kBottomLeft_GrSurfaceOrigin,
        kRGBA_8888_SkColorType, nullptr, nullptr,
        [](void *addr) {
            auto ctx = reinterpret_cast<RenderContext *>(addr);
            eglDestroySurface(ctx->display, ctx->surface);
            delete ctx;
        },
        reinterpret_cast<void *>(ctx));

// `surface` can now be used for drawing

..并最终将其传递给Camera(Java):

val captureRequestBuilder = captureSession.device.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW)
captureRequestBuilder.addTarget(imageReader.surface)
val captureRequest = captureRequestBuilder.build()
captureSession.setRepeatingRequest(captureRequest, null, cameraHandler)

但这只会崩溃,并出现以下错误:

D  eglCreateContext: 0xb40000735bdf0790: maj 3 min 0 rcv 3
D  eglMakeCurrent: 0xb40000735bdf0790: ver 3 0 (tinfo 0x75819f0100) (first time)
I  Re-rendering camera page with active camera. Device: "back (0)" (1280x720 @ 30fps)
D  onSurfaceCreated()
D  onSurfaceChanged()
D  Camera 0 successfully opened
E  [ImageReader-640x480f23m3-1403-0](id:57b00000002,api:1,p:1403,c:1403) connect: already connected (cur=1 req=4)
W  Stream configuration failed due to: endConfigure:648: Camera 0: Unsupported set of inputs/outputs provided
E  Session 0: Failed to create capture session; configuration failed

让我感到震惊的是ImageReader ... already connected错误--从现有的ImageReader表面创建OpenGL/Skia表面是不可能的吗?
有没有其他方法来做到这一点,不需要昂贵的CPU缓冲区副本的图像每帧?

h5qlskok

h5qlskok1#

我不清楚你在这里想做什么,但是你是在使用ImageReader.getSurface中的Surface来创建eglSurface吗?这很好,但这意味着您正在设置GPU -> ImageReader的渲染路径。
然后,您尝试将相机连接到ImageReader(Camera -> ImageReader),这会出错,因为您已经从GPU为ImageReader提供了数据。
如果需要Camera -> GPU,请使用SurfaceTexture将相机帧转换为GL纹理(通过构造函数从它创建Surface)。
但也许你需要澄清你试图创建的管道是什么。

相关问题