# Core ML and Metal: how Mac apps run AI on video

> Discover how Core ML, Metal, and MPSGraph power real-time AI video on Apple Silicon Macs, from neural motion interpolation to intelligent HDR expansion.

Published 2026-10-31 · Libraries & tools · https://www.hazenstudio.com/maclc/blog/core-ml-for-video/

## In short

- Core ML and Metal allow Mac apps to execute deep learning models directly on Apple Silicon hardware.
- Apple Silicon's unified memory lets CPU and GPU access the same video frame without costly memory copies.
- MPSGraph executes complex neural networks on the Apple GPU with sub-millisecond per-frame inference.
- Running AI locally guarantees complete privacy with zero cloud latency and minimal energy draw.

Core ML and Metal are Apple's foundational frameworks that allow Mac software to execute trained machine learning models directly on Apple Silicon hardware. By leveraging your Mac's unified memory architecture, high-bandwidth GPU, and dedicated compute cores, these frameworks enable video players to run complex artificial intelligence tasks—such as real-time motion interpolation and neural HDR tone mapping—without sending a single frame to the cloud.

Where traditional video processing relied on rigid, hand-crafted mathematical algorithms, modern media tools increasingly turn to lightweight neural networks. Whether estimating motion vectors across complex scenes or intelligently expanding standard dynamic range colors into high-brightness highlights, Apple's developer tools make real-time video inference fast, quiet, and power-efficient.

## What are Core ML, Metal, and MPSGraph?

To understand how modern Mac apps process video through artificial intelligence, it helps to distinguish between the three distinct layers of Apple's machine learning software stack.

[Core ML](https://developer.apple.com/documentation/coreml) is Apple's high-level machine learning framework, first introduced in macOS 10.13 High Sierra. It takes deep learning models trained in external frameworks like PyTorch or TensorFlow and optimizes them for Apple hardware. Core ML acts as an intelligent runtime coordinator: when an application passes a frame of video to a Core ML model, the framework automatically decides which hardware execution unit will deliver the fastest, most energy-efficient result.

Metal is Apple's low-level, low-overhead graphics and compute API. While gamers know Metal for 3D graphics rendering, video engineers use Metal compute shaders for parallel numerical computation. Because video frames consist of millions of independent color pixels, the parallel architecture of the GPU is naturally suited for video processing.

Sitting between these two layers is [Metal Performance Shaders Graph (MPSGraph)](https://developer.apple.com/documentation/metalperformanceshadersgraph). Introduced in macOS 11 Big Sur, MPSGraph is a specialized linear algebra and tensor computation engine designed specifically for the Apple GPU. It allows developers to construct, compile, and execute multidimensional neural network computation graphs directly on the graphics processor. By compiling operations into fused GPU kernels, MPSGraph eliminates intermediate memory bottlenecks and enables fast processing speeds.

*Real-time AI video inference on Apple Silicon*

## How Apple Silicon divides AI tasks: CPU, GPU, and Neural Engine

Apple Silicon processors (M1 through M4) do not rely on a single execution core to run artificial intelligence models. Instead, Apple integrates three distinct compute architectures onto each system-on-chip:

1. **The Central Processing Unit (CPU):** Comprising high-performance and high-efficiency cores, the CPU excels at sequential logic, file demuxing, and memory control. Through the Accelerate framework and Apple Matrix Coprocessor (AMX) instructions, the CPU can execute vector mathematics, but it is not optimized for heavy per-pixel neural convolutions.
2. **The Graphics Processing Unit (GPU):** The Apple GPU features a broad array of parallel execution units backed by hundreds of gigabytes per second of memory bandwidth. For real-time video processing, the GPU is often the ideal processor: video frames already reside in graphics memory, allowing neural networks to modify pixel values without copying data across internal buses.
3. **The Apple Neural Engine (ANE):** The Neural Engine is a dedicated ASIC block engineered exclusively for deep learning tensor operations. It excels at low-power matrix multiplications using quantized INT8 or FP16 numbers. While exceptional for background image analysis or audio separation, streaming 60 or 120 uncompressed 4K video frames through the Neural Engine can introduce synchronization overhead if the decoded video frames must constantly transition between display textures and model tensors.

Developers can configure model execution programmatically through `MLModelConfiguration.computeUnits`. An app can instruct Core ML to use all available units, restrict processing to the CPU and Neural Engine, or target the GPU specifically to maintain tight synchronization with the display refresh cycle.

## Why unified memory makes Mac video AI exceptionally fast

On traditional PC architectures, running AI on a video stream involves severe hardware bottlenecks. A video frame decoded by the CPU or hardware ASIC must be copied across a PCIe bus into discrete graphics memory (VRAM) so the GPU can run neural shaders. Once processed, the resulting frame might need to be copied back across the bus for presentation. At 4K resolution (3840 × 2160 pixels), a single 10-bit color frame consumes approximately 33 megabytes. Copying 60 or 120 of these frames every second across external buses consumes substantial memory bandwidth and generates significant heat.

Apple Silicon eliminates this transfer penalty entirely through Unified Memory Architecture (UMA). The CPU, GPU, Media Engine, and Neural Engine all share a single pool of high-speed, wide-bus memory. 

When a video player decodes an HEVC frame using the hardware Media Engine, the uncompressed pixel buffer is written directly to unified memory. The GPU can immediately interpret that exact same memory address as an input tensor for an MPSGraph neural network, process the pixels in place, and display the resulting frame on your screen. Because zero bytes are copied across physical buses, memory latency drops to near-zero and total energy consumption remains minimal.

## How does Mac video software convert and run AI models?

Most machine learning research occurs in open-source Python environments like PyTorch. Because running a heavy Python runtime inside a lightweight macOS video player is inefficient, developers follow a standardized conversion pipeline using Apple's open-source tool, [coremltools](https://apple.github.io/coremltools/).

The conversion process inspects the PyTorch model graph, strips away training-specific code (such as backpropagation gradients and dropout layers), and translates standard neural operations into native Core ML primitives. During conversion, weights are typically quantized from 32-bit floating-point (FP32) down to 16-bit half-precision (FP16). On Apple Silicon, FP16 operations run at double the throughput of FP32 while preserving the visual precision needed for video.

Once converted into an `.mlpackage` bundle, macOS compiles the model into optimized machine code during application build time or on first launch. The compiled graph is mapped directly to Apple Silicon's hardware execution pipelines, ensuring maximum throughput.

## Two real-world examples of video AI in MacLC

To see how these technologies translate into everyday playback, look at how the open-source media player MacLC implements machine learning for video enhancement. Rather than using heavy generic models, MacLC uses two focused pipelines:

### 1. Neural frame interpolation with RIFE v4.25

Cinematic films are traditionally shot at 24 frames per second. While 24 fps provides a familiar filmic motion blur, fast camera pans can appear juddery on modern 120 Hz ProMotion Mac displays. 

MacLC bundles an implementation of [RIFE (Real-Time Intermediate Flow Estimation)](/maclc/blog/what-is-rife/). Using Core ML, MacLC runs RIFE v4.25 to analyze pairs of consecutive video frames, estimate intermediate bidirectional optical flow, and synthesize intermediate frames. This allows 24 fps films to be interpolated up to 60 fps or 120 fps in real time. Learn more about the mechanics in our guide to [motion interpolation](/maclc/blog/motion-interpolation/).

### 2. SDR-to-HDR highlight expansion via MPSGraph

Most legacy video content is mastered in Standard Dynamic Range (SDR), designed for older screens that max out at 100 nits. Modern MacBook Pro screens and Apple Pro Display XDR monitors can sustain 1,000 nits and peak up to 1,600 nits. Linear brightness boosting can clip highlights and wash out colors.

To solve this, MacLC provides an [SDR to HDR](/maclc/blog/sdr-to-hdr/) expansion engine. For its highest quality tiers (High and Maximum), MacLC does not rely on static formulas. Instead, it executes a small neural network directly on the Apple GPU using MPSGraph. 

Trained by Hazen Studio on high-dynamic-range open-source imagery from Poly Haven, this neural net analyzes incoming video frames and predicts highlight expansion curves. On an Apple M3 Max chip, this GPU graph completes inference in 0.45 milliseconds per 1080p frame on High and 0.53 milliseconds on Maximum. Because inference takes roughly half a millisecond, the GPU comfortably fits within the 16.6 ms frame budget of a 60 fps stream with headroom to spare.

MacLC runs its SDR to HDR model directly on the GPU with Metal Performance Shaders Graph. On an M3 Max, High quality finishes in 0.45 ms per 1080p frame and Maximum in 0.53 ms, lifting highlights while faces, shadows, and colors stay.

## Why on-device AI matters for video privacy and battery life

In recent years, many commercial video upscalers and enhancement tools have moved to cloud-based APIs. Users must upload video clips to remote server farms, wait minutes or hours for processing, and download the resulting files.

For daily video playback, cloud processing is impractical. A two-hour movie contains hundreds of gigabytes of uncompressed video data. Transmitting that data over consumer internet connections introduces unacceptable latency, costs money, and exposes personal home videos to third-party privacy risks.

Running artificial intelligence directly on your Mac through Core ML and Metal offers three decisive advantages:

* **Complete Privacy:** Your video files never leave your Mac's internal storage or memory. No telemetry, user habits, or video frames are transmitted to remote servers.
* **Instant Responsiveness:** Because models execute in real time, there is no pre-rendering or exporting phase. You open a video file, click play, and enjoy enhancements immediately.
* **Low Energy Consumption:** When neural models are engineered specifically for Apple Silicon GPU shaders, the processor remains cool. A well-designed MPSGraph pipeline draws little power, preserving battery during playback.

## What to expect from the future of AI video playback

As Apple Silicon continues to evolve across newer M-series generations, hardware acceleration for machine learning will become even more integrated into media pipelines. Future architectures promise wider memory bandwidth, specialized matrix instructions inside GPU cores, and deeper integration with video presentation layers.

For everyday viewers, this means media players can perform tasks that once required dedicated post-production render farms. Real-time [AI video upscaling on Mac](/maclc/blog/ai-video-upscaling-mac/), dynamic artifact reduction, intelligent dialogue enhancement, and perceptive [tone mapping](/maclc/blog/what-is-tone-mapping/) will continue to transform standard video files into rich, display-tailored experiences.

By building on native Apple frameworks like Core ML, Metal, and MPSGraph, Mac developers can deliver cutting-edge visual experiences that run smoothly, quietly, and entirely on device.

## Questions people ask

### What is Core ML used for in video apps?

Video applications use Core ML to run pre-trained neural networks for tasks like frame rate interpolation, super-resolution upscaling, noise reduction, and intelligent scene detection.

### What is the difference between Core ML and Metal Performance Shaders Graph?

Core ML is a high-level framework that compiles machine learning models to run across the CPU, GPU, and Neural Engine. Metal Performance Shaders Graph (MPSGraph) is a specialized compute engine that builds and executes custom tensor graphs directly on the Apple GPU.

### Does running AI on video drain a MacBook battery quickly?

When models are properly optimized for Apple Silicon's GPU shaders and low-precision FP16 mathematics, per-frame inference completes in fractions of a millisecond, preserving battery life compared to unoptimized CPU processing.

### Does on-device video AI require an internet connection?

No. Core ML and Metal execute entirely on local hardware inside your Mac. Your video frames never leave your computer, ensuring absolute privacy and zero network buffering.

---
MacLC is a free, open-source video player for Apple Silicon Macs: https://www.hazenstudio.com/maclc/ · download: https://www.hazenstudio.com/maclc/donate/