ios 在MTKView上渲染MTLTexture未保持纵横比

of1yzvn4  于 2023-03-05  发布在  iOS
关注(0)|答案(4)|浏览(143)

我有一个1080x1920像素的纹理。我试图在MTKView上渲染它,这不是相同的长宽比。(即iPad/iPhone X全屏)。
这是我如何渲染MTKView的纹理:

private func render(_ texture: MTLTexture, withCommandBuffer commandBuffer: MTLCommandBuffer, device: MTLDevice) {
    guard let currentRenderPassDescriptor = metalView?.currentRenderPassDescriptor,
            let currentDrawable = metalView?.currentDrawable,
            let renderPipelineState = renderPipelineState,
            let encoder = commandBuffer.makeRenderCommandEncoder(descriptor: currentRenderPassDescriptor) else {
                semaphore.signal()
                return
        }

    encoder.pushDebugGroup("RenderFrame")
    encoder.setRenderPipelineState(renderPipelineState)
    encoder.setFragmentTexture(texture, index: 0)
    encoder.drawPrimitives(type: .triangleStrip, vertexStart: 0, vertexCount: 4, instanceCount: 1)
    encoder.popDebugGroup()
    encoder.endEncoding()

    // Called after the command buffer is scheduled
    commandBuffer.addScheduledHandler { [weak self] _ in
        guard let strongSelf = self else {
            return
        }
        strongSelf.didRender(texture: texture)
        strongSelf.semaphore.signal()
    }

    commandBuffer.present(currentDrawable)
    commandBuffer.commit()
}

我希望纹理像.scaleAspectFill一样在UIView上渲染,我正在学习Metal,所以我不确定应该在哪里寻找它(.metal文件、管道、视图本身、编码器等)。
谢谢!
编辑:下面是着色器代码:

#include <metal_stdlib> using namespace metal;

typedef struct {
    float4 renderedCoordinate [[position]];
    float2 textureCoordinate; } TextureMappingVertex;

vertex TextureMappingVertex mapTexture(unsigned int vertex_id [[ vertex_id ]]) {
    float4x4 renderedCoordinates = float4x4(float4( -1.0, -1.0, 0.0, 1.0 ),
                                            float4(  1.0, -1.0, 0.0, 1.0 ),
                                            float4( -1.0,  1.0, 0.0, 1.0 ),
                                            float4(  1.0,  1.0, 0.0, 1.0 ));

    float4x2 textureCoordinates = float4x2(float2( 0.0, 1.0 ),
                                           float2( 1.0, 1.0 ),
                                           float2( 0.0, 0.0 ),
                                           float2( 1.0, 0.0 ));
    TextureMappingVertex outVertex;
    outVertex.renderedCoordinate = renderedCoordinates[vertex_id];
    outVertex.textureCoordinate = textureCoordinates[vertex_id];

    return outVertex; }

fragment half4 displayTexture(TextureMappingVertex mappingVertex [[ stage_in ]],texture2d<float, access::sample> texture [[ texture(0) ]]) {
    constexpr sampler s(address::clamp_to_edge, filter::linear);

    return half4(texture.sample(s, mappingVertex.textureCoordinate));
}
0h4hbjxa

0h4hbjxa1#

处理“金属”纹理或“一般金属”时,需要从以下几个常规事项开始:

  • 您应该考虑 * 点 * 和 * 像素 * 之间的差异,请参阅文档here。UIView子类的frame属性(MTKView是其中之一)总是以 * 点 * 为单位提供视图的宽度和高度。
  • 从点到实际像素的Map是通过contentScaleFactor选项控制的。MTKView自动选择一个纹理,其适合的纵横比与设备的实际像素相匹配。例如,iPhone X上MTKView的底层纹理的分辨率为2436 x 1125(实际显示大小,以像素为单位)。这在下面进行了记录:* “MTKView类自动支持本机屏幕缩放。默认情况下,始终保证视图当前可绘制对象的大小与视图本身的大小匹配。"*
  • 如本文所述,.scaleAspectFill选项“缩放内容以填充视图的大小。内容的某些部分可能会被剪切以填充视图的边界”。您希望模拟此行为。
  • 用金属渲染只不过是“绘制”到解析纹理,这是由MTKView自动设置的。然而,你仍然有完全的控制权,可以自己手动创建纹理并在renderPassDescriptor中设置它们。但你现在不需要关心这个。你应该关心的唯一一件事是什么,要在解析纹理中渲染解析纹理中的1080 x1920像素纹理的位置和部分(可能具有不同的纵横比)。我们希望完全填充(“scaleAspectFill”)解析纹理,所以我们在片段着色器中保留renderedCoordinates不变。它们在整个解析纹理上定义了一个矩形,这意味着在解析纹理中的每一个像素都会调用片段着色器。2接下来,我们将简单地改变纹理坐标。
  • 让我们定义长宽比为ratio = width / height,解析纹理为r_tex,你想要渲染的纹理为tex

1.因此,假设解析纹理没有相同的纵横比,则有两种可能的情况:
1.您要渲染的纹理的纵横比比比您的解析纹理(金属渲染的纹理)的纵横比,这意味着您要渲染的纹理比解析纹理的宽度大。在这种情况下,我们保留坐标的y值不变。纹理坐标的x值将被更改:

x_left  = 0 + ((tex.width - r_tex.width) / 2.0)
x_right = tex_width - ((tex.width - r_tex_width) / 2.0)

这些值必须规格化,因为纹理采样需要0到1范围内的坐标:

x_left  = x_left / tex.width
x_right = x_right / tex.width

我们有了新的纹理坐标:

topLeft = float2(x_left,0)
topRight = float2(x_right,0)
bottomLeft = float2(x_left,1)
bottomRight = float2(x_right,1)

这将产生这样的效果:纹理的顶部或底部不会被剪切,但左侧和右侧的一些外部部分将被剪切,即不可见。
1.你想要渲染的纹理的长宽比比比你的解析纹理的长宽比小**。这个过程和第一个场景一样,但是这次我们要改变y坐标
这应该渲染你的纹理,使解析纹理被完全填充,并且你的纹理的纵横比在x轴上保持不变。保持y轴的工作原理类似。另外,你必须检查纹理的哪一边更大/更小,并将其纳入你的计算中。这将像使用scaleAspectFill时一样裁剪你的纹理的一部分。请注意,上面的解决方案是未经测试的。但我希望它是有帮助的。一定要访问金属最佳实践文档不时,这是非常有帮助的,以获得基本概念的权利。

bmp9r5qi

bmp9r5qi2#

所以顶点着色器直接决定了源纹理被拉伸到视口的尺寸。你渲染的是一个充满视口的四边形,因为它的坐标在水平和垂直方向上都在规格化设备坐标系的端点([-1,1])。
你在同一个范围内对角Map源纹理,这是因为你为纹理坐标指定了纹理坐标空间的极值([0,1])。
有很多方法可以实现你想要的。你可以通过缓冲区将顶点坐标传递给着色器,而不是硬编码。这样,你就可以在应用代码中计算适当的值。你可以在渲染目标中计算所需的目标坐标,用NDC表示。所以,从概念上讲,类似于left_ndc = (left_pixel / target_width) * 2 - 1等。
或者(可能更简单),您可以保持着色器不变,并更改绘制操作的视口,以仅针对渲染目标的适当部分。

k4aesqcs

k4aesqcs3#

1awuesterose的答案是有效的,我唯一要补充的是使用abs(tex.width - r_tex.width)
这是我的代码的样子如果有用的话。

#ifndef BridgingHeader_h
#define BridgingHeader_h
#include <simd/simd.h>

struct Vertex {
    vector_float2 position;
};

#endif /* BridgingHeader_h */
#include <metal_stdlib>
using namespace metal;
#include <BridgingHeader.h>

struct VertexOut {
    float4 position [[position]];
    float2 texture;
};

vertex VertexOut textureVertexAspectFill(const device Vertex *vertexArray [[buffer(0)]],
                                         unsigned int vid [[vertex_id]],
                                         texture2d<float> texture [[ texture(0) ]],
                                         texture2d<float> resolveTexture [[ texture(1) ]]) {
    float textureWidth = texture.get_width();
    float textureHeight = texture.get_height();
    
    float textureAspectRatio = textureWidth / textureHeight;
    
    float resolveWidth = resolveTexture.get_width();
    float resolveHeight = resolveTexture.get_height();
    
    float resolveAspectRatio = resolveWidth / resolveHeight;
    
    bool isTextureRatioGreater = textureAspectRatio > resolveAspectRatio;
    
    float2 topLeft;
    float2 topRight;
    float2 bottomLeft;
    float2 bottomRight;
    
    if (isTextureRatioGreater) {
        float left = abs(textureWidth - resolveWidth) / 2.0;
        float right = textureWidth - left;
        
        float normalizedLeft = left / textureWidth;
        float normalizedRight = right / textureWidth;
        
        topLeft = float2(normalizedLeft, 0);
        topRight = float2(normalizedRight, 0);
        bottomLeft = float2(normalizedLeft, 1);
        bottomRight = float2(normalizedRight, 1);
    } else {
        float top = abs(textureHeight - resolveHeight) / 2.0;
        float bottom = textureHeight - top;
        
        float normalizedTop = top / textureHeight;
        float normalizedBottom = bottom / textureHeight;
        
        topLeft = float2(0, normalizedTop);
        topRight = float2(1, normalizedTop);
        bottomLeft = float2(0, normalizedBottom);
        bottomRight = float2(1, normalizedBottom);
    }
    
    float4x2 textureCoordinates = float4x2(topLeft,
                                           bottomLeft,
                                           bottomRight,
                                           topRight);

    VertexOut output;
    output.position = float4(vertexArray[vid].position, 0, 1);
    output.texture = textureCoordinates[vid];

    return output;
}

fragment half4 textureFragmentAspectFill(VertexOut input [[stage_in]],
                               texture2d<float> texture [[ texture(0) ]]) {
    constexpr sampler defaultSampler;
    float4 color = texture.sample(defaultSampler, input.texture);
    return half4(color.r, color.g, color.b, 1);
}
// ...

    private let vertexBuffer: any MTLBuffer
    private let indexBuffer: any MTLBuffer
    var indices: [UInt16] = [
        0, 1, 2,
        2, 3, 0
    ]

    // ...

    let vertices = [
        Vertex(position: [-1, 1]), // Top left
        Vertex(position: [-1, -1]), // Bottom left
        Vertex(position: [1, -1]), // Bottom right
        Vertex(position: [1, 1]) // Top right
    ]
        
    vertexBuffer = context.device.makeBuffer(
        bytes: vertices,
        length: vertices.count * MemoryLayout<Vertex>.stride,
        options: []
    )!
        
    indexBuffer = context.device.makeBuffer(
        bytes: indices,
        length: indices.count * MemoryLayout<UInt16>.size
    )!

    // ...

    public func draw(in view: MTKView) {
        guard let pixelBuffer else { return }
        guard let textureCache,
              let commandBuffer = context.commandQueue.makeCommandBuffer(),
              let renderPassDescriptor = view.currentRenderPassDescriptor
        else {
            assertionFailure("Can't perform draw")
            return
        }
        
        renderPassDescriptor.colorAttachments[0].clearColor = MTLClearColorMake(0, 0, 0, 1.0)
        renderPassDescriptor.colorAttachments[0].loadAction = .clear
        renderPassDescriptor.colorAttachments[0].storeAction = .store
        guard let encoder = commandBuffer.makeRenderCommandEncoder(
            descriptor: renderPassDescriptor
        ) else {
            assertionFailure("Could not create encoder")
            return
        }
        guard let cvTexture = getCVTexture(from: pixelBuffer, and: textureCache),
              let inputTexture = CVMetalTextureGetTexture(cvTexture)
        else {
            assertionFailure("Failed to create metal textures")
            return
        }
        
        encoder.setRenderPipelineState(pipelineState)
        autoreleasepool {
            guard let drawable = view.currentDrawable else { return }
            
            encoder.setVertexBuffer(vertexBuffer, offset: 0, index: 0)
            encoder.setVertexTexture(inputTexture, index: 0)
            encoder.setVertexTexture(drawable.texture, index: 1)
            encoder.setFragmentTexture(inputTexture, index: 0)
            
            encoder.drawIndexedPrimitives(
                type: .triangle,
                indexCount: indices.count,
                indexType: .uint16,
                indexBuffer: indexBuffer,
                indexBufferOffset: 0
            )
            encoder.endEncoding()
            
            commandBuffer.present(drawable)
            commandBuffer.commit()
        }
    }
ppcbkaq5

ppcbkaq54#

另一方面,我决定在渲染器中使用CPU进行纹理坐标计算,并将矩阵注入着色器函数中,而不是对每个像素进行相同的计算。

着色器

#include <metal_stdlib>
using namespace metal;
#include <BridgingHeader.h>

struct VertexOut {
    float4 position [[position]];
    float2 texture;
};

vertex VertexOut textureWithCustomCoordinatesVertex(const device Vertex *vertexArray [[buffer(0)]],
                                                    unsigned int vid [[vertex_id]],
                                                    constant float4x2 &textureCoordinates [[ buffer(1) ]]
                                                    ) {
    VertexOut output;
    output.position = float4(vertexArray[vid].position, 0, 1);
    output.texture = textureCoordinates[vid];

    return output;
}

fragment half4 textureWithCustomCoordinatesFragment(VertexOut input [[stage_in]],
                                                    texture2d<float> texture [[ texture(0) ]],
                                                    sampler sampler2d [[ sampler(0) ]]
                                                    ) {
    float4 color = texture.sample(sampler2d, input.texture);
    return half4(color.r, color.g, color.b, 1);
}

渲染器

final class MyRenderer: NSObject, MTKViewDelegate {
    // ... 

    private func getTextureAspectFillCoordinates(
        inputTexture: MTLTexture,
        drawableTexture: MTLTexture
    ) -> simd_float4x2 {
        // 1. Get ratios for both textures
        let textureWidth: simd_float1 = .init(inputTexture.width)
        let textureHeight: simd_float1 = .init(inputTexture.height)
        let textureAspectRatio: simd_float1 = textureWidth / textureHeight
        
        let drawableWidth: simd_float1 = .init(drawableTexture.width)
        let drawableHeight: simd_float1 = .init(drawableTexture.height)
        let drawableAspectRatio: simd_float1 = drawableWidth / drawableHeight
        
        // 2. Declare output points
        let topLeft: simd_float2
        let topRight: simd_float2
        let bottomLeft: simd_float2
        let bottomRight: simd_float2
        
        // 3. Check if texture's ratio is greater than drawable's
        if textureAspectRatio > drawableAspectRatio {
            // We need to draw whole height and clip width
            let left: simd_float1 = abs(textureWidth - drawableWidth) / 2.0
            let right = textureWidth - left
            
            let normalizedLeft = left / textureWidth
            let normalizedRight = right / textureWidth
            
            topLeft = .init(normalizedLeft, 0)
            topRight = .init(normalizedRight, 0)
            bottomLeft = .init(normalizedLeft, 1)
            bottomRight = .init(normalizedRight, 1)
        } else {
            // We need to draw whole width and clip height
            let top: simd_float1 = abs(textureHeight - drawableHeight) / 2.0
            let bottom = textureHeight - top
            
            let normalizedTop = top / textureHeight
            let normalizedBottom = bottom / textureHeight
        
            topLeft = .init(0, normalizedTop)
            topRight = .init(1, normalizedTop)
            bottomLeft = .init(0, normalizedBottom)
            bottomRight = .init(1, normalizedBottom)
        }
        
        return .init(columns: (topLeft, bottomLeft, bottomRight, topRight))
    }

    // ...

    func draw(in view: MTKView) {
        guard let pixelBuffer else { return }
        guard let textureCache,
              let commandBuffer = context.commandQueue.makeCommandBuffer(),
              let renderPassDescriptor = view.currentRenderPassDescriptor
        else {
            assertionFailure("Can't perform draw")
            return
        }
        
        renderPassDescriptor.colorAttachments[0].clearColor = MTLClearColorMake(0, 0, 0, 1.0)
        renderPassDescriptor.colorAttachments[0].loadAction = .clear
        renderPassDescriptor.colorAttachments[0].storeAction = .store
        guard let encoder = commandBuffer.makeRenderCommandEncoder(
            descriptor: renderPassDescriptor
        ) else {
            assertionFailure("Could not create encoder")
            return
        }
        guard let cvTexture = getCVTexture(from: pixelBuffer, and: textureCache),
              let inputTexture = CVMetalTextureGetTexture(cvTexture)
        else {
            assertionFailure("Failed to create metal textures")
            return
        }
        
        encoder.setRenderPipelineState(pipelineState)
        autoreleasepool {
            guard let drawable = view.currentDrawable else { return }
            
            var textureCoordinates = getTextureAspectFillCoordinates(
                inputTexture: inputTexture,
                drawableTexture: drawable.texture
            )
            
            encoder.setVertexBuffer(vertexBuffer, offset: 0, index: 0)
            encoder.setVertexBytes(
                &textureCoordinates,
                length: MemoryLayout<simd_float4x2>.size,
                index: 1
            )
            encoder.setFragmentTexture(inputTexture, index: 0)
            encoder.setFragmentSamplerState(samplerState, index: 0)
            
            encoder.drawIndexedPrimitives(
                type: .triangle,
                indexCount: indices.count,
                indexType: .uint16,
                indexBuffer: indexBuffer,
                indexBufferOffset: 0
            )
            encoder.endEncoding()
            
            commandBuffer.present(drawable)
            commandBuffer.commit()
        }
    }
}

相关问题