(MTL S01E01) What is Apple Metal

This brief introduction aims to give you a clear idea of what Metal is — and what it isn’t. By the end of this article, you’ll understand how it compares to other graphics programming tools, discover what’s built on Metal, and explore potential areas where you can apply it.

At its core, Metal is a low-level framework designed for rendering and computation on the GPU. Unlike OpenGL and Vulkan, which are cross-platform standards, Metal has a final implementation that’s exclusive to Apple platforms (iPhone, iPad, Mac, Apple TV, and Apple Vision Pro).

Interestingly, when you run OpenGL on an Apple platform, it’s actually powered by Metal under the hood — you can confirm this by checking in a debugger or profiler.

An OpenGL texture on Apple platforms is built on top of a Metal texture

In terms of API, Metal is closer to Vulkan but with far less boilerplate code, as many settings are managed by the system — though you still have the ability to access them when needed. I’ll dive deeper into the details in future articles.

Where it’s used

Simply, Metal is used wherever raster graphics are rendered. Unfortunately, GPUs don’t offer native support for vector graphics (and no, line and triangle primitives don’t count as vector graphics). For vector rendering, you’ll still need to use CoreGraphics with the CPU or implement your own solution.

On a higher level, Metal powers many technologies such as SwiftUI, SpriteKit, SceneKit, CoreImage, and even serves as the backend for machine learning and math frameworks like TensorFlow, PyTorch, and JAX. It’s also the backend for OpenGL on Apple platforms.

There’re also various Metal-based libraries like MetalKit, Metal Performance Shaders, MPSGraph, and MetalFX (I may have missed something — feel free to correct me in the comments).

Those are just the standard frameworks, but there are plenty of third-party libraries that use Metal for GPU computing under the hood (like Unity, for example).

Where to apply it

I’ve listed a few frameworks that use Metal, but I won’t do the same for apps that use it, whether implicitly or explicitly. However, if your app involves any graphics or image processing, chances are high that you’ll need Metal.

Where can you apply Metal?

• Fast image processing (though not every algorithm is suitable). You can also take advantage of Metal Performance Shaders (MPS) for optimized routines.

Seamless stitching for texturing is achieved using a collection of Metal Performance Shaders (MPS)

• 2D or 3D graphics applications. This includes most games, image editing apps (like Pixelmator, Procreate, Affinity), and some video editing software.

Layer composition, color adjustments, effects, painting, etc in Pixelmator are implemented with Metal.

• Computation. When dealing with large datasets and applying the same processing kernel. However, keep in mind that transferring data to and from the GPU can sometimes take longer than processing it on the CPU.

You can compute and render spectrogram using Metal (Image for example)

• Machine Learning. You can use your own kernels, MPS, or MPSGraph for accelerated ML tasks.
• UI effects. Starting with iOS 17, SwiftUI allows for GPU-accelerated effects, and you can use CoreImage filters in UIKit for similar effects.

The black hole is rendered as SwiftUI element effect using Metal shader.

• Custom UI rendering. If you want to create custom interfaces with unique UX, as seen in apps like Procreate Dreams.

What is Metal not suitable for?

• Vector graphics. As I mentioned earlier, GPUs are not inherently designed to handle vector graphics, especially curves. However, there are some approaches to get around this limitation, though they usually involve custom implementations or hybrid solutions combining CPU and GPU processing.
• Small data computations. Typically, transferring small amounts of data between CPU and GPU memory can take longer than the computations themselves. Additionally, if you’re working with a small dataset, the GPU will likely be underutilized, making it less efficient than CPU processing for these cases.
• Multiplatform apps. Unfortunately (or fortunately, depending on your perspective), Metal is exclusive to Apple platforms. If you need true cross-platform support, you’ll have to use OpenGL or Vulkan instead.

Conclusion

• Metal is a low-level framework for GPU rendering and computing, providing direct access to the hardware for high-performance tasks.
• It’s used for a wide range of tasks, from machine learning to game graphics rendering.
• It does have some limitations, so you need to carefully consider whether it’s the right choice for your project.