(MTL S01E05) Primitives pt.2: Textures

In the last episode, we covered MTLBuffer-s. Since Metal revolves around the GPU, and GPUs are fundamentally tied to graphics, the next essential topic is textures, which we’ll explore in this episode.

What is `MTLTexture`

At its core, a texture is similar to a buffer — it’s a block of memory. However, it includes additional metadata that allows it to be accessed as an image, optimizing operations compared to using a plain buffer. Textures also enable features like working with normalized values and sampling without needing extra implementations. Unlike buffers, Metal handles all type mappings between the CPU and GPU automatically, raising errors if any mismatches occur.

Look inside

You can use Xcode’s Frame Capture tool to inspect the contents of your buffers. I provided a brief explanation on how to use it in the previous episode, so in this episode, we’ll focus on textures specifically.

By double-clicking on a texture, you can view its preview, where you can:

• Inspect the value of individual pixels

• Check the texture’s pixel format and dimensions

• Adjust preview settings, such as remapping channels or changing the color space

You can also inspect a texture’s content in the debugger, but be aware that the texture may contain outdated data if it hasn’t been properly synchronized between the CPU and GPU.

Creating texture

The easiest case

To create a basic texture, you only need a single call:

let texture = device.makeTexture(                           // (1)
    descriptor: MTLTextureDescriptor.texture2DDescriptor(   // (2)
        pixelFormat: .rgba8Unorm,                           // (3)
        width: 640,                                         // (4)
        height: 480,                                        // (5)
        mipmapped: false))                                  // (6)

1. A texture is created on the device.
2. The texture descriptor is created. With this approach, the texture will have shared storage mode and will be read-only. If you need to modify the texture, manually create a descriptor and configure the necessary parameters.
3. The pixel format of the texture. In this case, it’s a 4-channel image with 8 bits per channel, normalized to a [0, 1] float range in shaders (though it’s stored as an integer [0, 255] in memory).
4. The width of the texture (maximum values depend on the chipset).
5. The height of the texture (with the same limitations as width).
6. A flag to specify whether the texture is mipmapped.

A mipmap is a pyramid of images where the first (zero) level contains the original image, and each subsequent level is scaled down to half the size of the previous one, with the final level being a 1x1 pixel. While some standards and platforms require textures to have power-of-two dimensions, Metal allows textures of any size.

By en:User:Mulad, based on a NASA image — Created by en:User:Mulad based on File:ISS from Atlantis — Sts101–714–016.jpg, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=1140741

Mipmaps are commonly used for trilinear interpolation, but fundamentally, they are just a pyramid. You can also use them for techniques like pyramid blending, dynamic blurring, and more.

By BillyBob CornCob — Own work, CC0, https://commons.wikimedia.org/w/index.php?curid=76760547

Note that mipmaps aren’t generated automatically — you need to create or populate them using one of the blit operations (which we’ll cover in a later episode or you can refer to the documentation).

IOSurface

You can also bind a texture to an IOSurface, which can be useful for alternative rendering of UI elements, video processing, and other tasks. It’s your responsibility to configure the descriptor so that it’s compatible with the surface plane.

device.makeTexture(
    descriptor: MTLTextureDescriptor.texture2DDescriptor(
        pixelFormat: .rgba8Unorm,
        width: 640, height: 480,
        mipmapped: false),
    iosurface: surface,
    plane: 0)

MTLTextureDescriptor

If you need a more customized texture, such as for writing or rendering into it (and yes, these are distinct operations), you’ll need to manually configure the texture descriptor. For example, let’s say we need a lookup table (LUT) for color mapping. A cube texture is ideal for this purpose, as it can be initialized and used exclusively on the GPU.

let descriptor = MTLTextureDescriptor()                         // (1)
descriptor.textureType = .type3D                                // (2)
descriptor.pixelFormat = .rgba32Float                           // (3)
descriptor.width = 64                                           // \
descriptor.height = 64                                          //  (4)
descriptor.depth = 64                                           // /
descriptor.mipmapLevelCount = 0                                 // (5)
descriptor.sampleCount = 1                                      // (6)
descriptor.storageMode = .private                               // (7)
descriptor.usage = [.shaderRead, .shaderWrite, .renderTarget]   // (8)

1. Creating a blank instance of `MTLTextureDescriptor`.
2. Setting the texture type to 3D (other options include 1D, 2D, or arrays of these types).
3. Using 4-channel float pixels, which might be necessary for precise color mapping.
4. Defining the texture dimensions, including depth, as it’s a 3D texture.
5. Mipmapping is unnecessary for the LUT, and you don’t need to create mipmaps down to 1x1 pixels.
6. A sample count of 1 per pixel is sufficient — higher values are for multisampling, which is only applicable to 2D textures.
7. The texture is stored on the GPU with no CPU access.
8. The texture can be read and written from shaders, or used as a render target, which is needed for programmatic initialisation.

There are additional fields available, but they are more specialized and will be covered in future episodes.

Initialisation

The most common use of textures is loading an image and applying it to a surface with shaders. But how do you upload an image if `makeTexture` doesn’t handle that directly?

From CGImage

In the easiest way, you can let MetalKit handle this by using MTKTextureLoader.

func loadTexture(device: MTLDevice, image: CGImage) -> MTLTexture? {
    let textureLoader = MTKTextureLoader(device: device)                                            // (1)
    let usage: MTLTextureUsage = [
        MTLTextureUsage.shaderRead,
        MTLTextureUsage.shaderWrite,
        MTLTextureUsage.renderTarget]
    let textureLoaderOptions = [
        MTKTextureLoader.Option.origin: MTKTextureLoader.Origin.bottomLeft,                         // (2)
        MTKTextureLoader.Option.textureUsage: NSNumber(value: usage.rawValue),                      // (3)
        MTKTextureLoader.Option.textureStorageMode: NSNumber(value: MTLStorageMode.shared.rawValue),// (4)
        MTKTextureLoader.Option.SRGB: false                                                         // (5)
    ] as [MTKTextureLoader.Option: Any]

    var texture: MTLTexture?
    do {
        texture = try textureLoader.newTexture(cgImage: image, options: textureLoaderOptions)       // (6)
    } catch {
        print("Unable to load texture. Error info: \(error)")
    }

    return texture
}

1. Create an instance of MTKTextureLoader.
2. Set the origin of the image — this can be useful if the image is vertically flipped.
3. Specify texture usage options (refer to the previous section for details).
4. Set the storage mode (as discussed earlier).
5. Indicate whether the texture uses an sRGB (gamma-corrected) pixel format.
6. Create the texture from a given CGImage with the specified options.

From Metal Buffer

Alternatively, you can upload your raw image buffer to an MTLBuffer and then blit it to your texture:

func upload(imageBuffer: vImage_Buffer, to texture: MTLTexture, commandBuffer: MTLCommandBuffer) throws {
    guard let buffer = texture.device.makeBuffer(                               // (1)
        bytes: imageBuffer.data,
        length: imageBuffer.rowBytes * Int(imageBuffer.height)),
          let blitEncoder = commandBuffer.makeBlitCommandEncoder()              // (2)
    else { throw fatalError("Unabel create buffer or blit encoder") }
    
    blitEncoder.copy(                                                           // (3)
        from: buffer,                                                           // (4)
        sourceOffset: 0,                                                        // (5)
        sourceBytesPerRow: imageBuffer.rowBytes,                                // (6)
        sourceBytesPerImage: imageBuffer.rowBytes * Int(imageBuffer.height),    // (7)
        sourceSize: MTLSize(width: Int(imageBuffer.width),                      // (8)
                            height: Int(imageBuffer.height),
                            depth: 1),
        to: texture,                                                            // (9)
        destinationSlice: 0,                                                    // (10)
        destinationLevel: 0,                                                    // (11)
        destinationOrigin: MTLOrigin(x: 0, y: 0, z: 0))                         // (12)
    blitEncoder.endEncoding()                                                   // (13)
}

In the function we assume that uploading is happeining in an existing command buffer from a vImage_Buffer (not neccessary, but for convenience) to a given existing texture.

1. Create a buffer. (You likely have your own device instance. If not, you can retrieve it from other objects to ensure synchronization with the same device.)
2. Create a blit encoder. (This will be explained in more detail in future episodes.)
3. Initiate the copy operation from the buffer to the texture.
4. The source buffer object.
5. The offset from the start of the buffer (in case the buffer contains more than just the necessary image).
6. Bytes per row of the image (typically, this is bytes per pixel multiplied by the width, though alignment may introduce nuances).
7. The total number of bytes for the entire image (how much data needs to be copied).
8. The size of the original image (includes depth for 3D images).
9. The destination texture.
10. The destination slice if it’s a texture array.
11. The destination mipmap level, if the texture has allocated mipmaps and you’re uploading a specific level.
12. The destination origin in the texture, allowing you to upload a portion of it.
13. Don’t forget to end encoding!

This approach is the only way to upload data to a private texture from the CPU. Alternatively, you can copy from a shared texture, depending on what best suits your task.

From Raw Buffer

If your texture uses shared storage mode and is accessible from the CPU, you can directly copy data to it using the replace method:

texture.replace(
    region: MTLRegion(
        origin: .init(x: 0, y: 0, z: 0),
        size: .init(
            width: Int(imageBuffer.width),
            height: Int(imageBuffer.height),
            depth: 1)),
    mipmapLevel: 0,
    withBytes: imageBuffer.data,
    bytesPerRow: imageBuffer.rowBytes)

As you can see, this method doesn’t require a device instance or command buffers — everything happens directly in the texture’s memory. This is possible because of the shared storage mode.

Accessing

Sometimes, such as when building an image or video editing app, you may need to download the result of your GPU processing back to the CPU. In a previous episode, we discussed how to do this with buffers. While the process for textures is quite similar, there are some important nuances to consider.

If you’re working with a private texture, you’ll need to blit it firstly to a shared buffer before downloading the buffer’s content to the CPU. You can find the details on how to do this in the episode about buffers.

In some cases, if your texture uses shared storage and grants access to its buffer, you can retrieve the content as you would with a normal buffer:

let bytes = texture.buffer?.contents()

Alternatively, you can use the getBytes method (similar to replace, but with the data flow reversed):

texture.getBytes(
    imageBytes,
    bytesPerRow: imageBytesPerRow,
    from: MTLRegionMake2D(0, 0, width, height),
    mipmapLevel: 0)

Binding to GPU functions

We’ve now created a texture, uploaded content, and even downloaded it back. But how do we actually use it in GPU functions like shaders or compute kernels? This depends on the usage type and the function type. To bind your texture object to a GPU function index, you can use one of the following options:

computeEncoder.setTexture(texture, index: 0)         // (1)
...
renderEncoder.setVertexTexture(texture, index: 0)    // (2)
renderEncoder.setFragmentTexture(texture, index: 0)  // (3)

1. Binding the texture to a compute kernel at index 0.
2. Binding the texture to a vertex shader at index 0 (e.g., for use as a height map).
3. Binding the texture to a fragment shader at index 0 (typically for surface rendering).

If you need to bind a target texture (for writing into it) in a compute kernel, you can bind it as shown above. However, for rendering, you need to set it as a render target by attaching it to a color attachment in the render encoder (this could also be stencil or depth depending on your task):

let encoderDescriptor = MTLRenderPassDescriptor()
encoderDescriptor.colorAttachments[0].texture = texture

Note that there is always a texture attached — for instance: when using an encoder descriptor from MTKView, the texture bound to the view’s IOSurface is attached by default. An important moment is that the pixel format of the texture and the render pipeline attachment must match.

Conclusion

In this episode, we explored how to create, upload, and use textures in Apple Metal. From binding textures to shaders to handling render targets, understanding these concepts will allow you to effectively integrate textures into your GPU workflows.