Introduction to the WebCodecs API
The WebCodecs API is a modern browser API that provides low-level access to built-in media codecs for encoding and decoding audio and video. Unlike traditional media APIs such as <video>, Media Source Extensions (MSE), or WebRTC, WebCodecs exposes the individual codec components directly to JavaScript. This gives developers fine-grained control over how media frames are processed, transformed, and rendered.
Before WebCodecs, web developers who needed to work with raw media frames had to rely on workarounds like decoding video into canvas elements or using WebAssembly-based codecs. These approaches were often slow, battery-intensive, and lacked hardware acceleration. WebCodecs solves this by exposing the browser's native, hardware-accelerated codecs through a clean JavaScript interface.
Why WebCodecs Matters
WebCodecs unlocks a wide range of use cases that were previously difficult or impossible on the web:
- Real-time video processing: Apply filters, effects, or computer vision algorithms to individual video frames with minimal latency.
- Custom streaming protocols: Build custom video conferencing, cloud gaming, or live streaming applications with precise control over buffering and frame timing.
- Video editing in the browser: Decode, transform, and re-encode video without server round trips.
- Transcoding pipelines: Convert media between formats efficiently using hardware acceleration.
- AI and machine learning: Feed decoded video frames directly into ML models for inference.
Browser Support and Feature Detection
WebCodecs is available in Chromium-based browsers (Chrome, Edge, Opera) and is being developed for other browsers. Always check for support before using it in production.
if ('VideoDecoder' in window && 'VideoEncoder' in window) {
console.log('WebCodecs is supported');
} else {
console.log('WebCodecs is not supported in this browser');
}
Core Concepts
VideoDecoder
The VideoDecoder takes encoded video chunks (such as H.264, VP9, or AV1 NAL units) and decodes them into VideoFrame objects. Each VideoFrame contains raw pixel data that can be drawn to a canvas, processed, or passed to other APIs.
VideoEncoder
The VideoEncoder takes VideoFrame objects and encodes them into EncodedVideoChunk objects. These chunks can then be multiplexed into a container format, transmitted over a network, or stored.
AudioDecoder and AudioEncoder
Similar to their video counterparts, AudioDecoder and AudioEncoder handle raw audio data and encoded audio chunks. They work with formats like AAC, Opus, and FLAC.
ImageDecoder
The ImageDecoder decodes images (including animated formats like GIF and AVIF) into VideoFrame objects, providing more control than the traditional Image element.
Decoding Video with VideoDecoder
Let's walk through a complete example of decoding an H.264 video stream. The process involves creating a decoder, configuring it, feeding it encoded chunks, and handling the output frames.
// Create a VideoDecoder
const decoder = new VideoDecoder({
output: (frame) => {
console.log('Decoded frame:', frame.width, 'x', frame.height);
// Draw the frame to a canvas
const canvas = document.getElementById('canvas');
const ctx = canvas.getContext('2d');
canvas.width = frame.width;
canvas.height = frame.height;
ctx.drawImage(frame, 0, 0);
// Always close frames when done to free memory
frame.close();
},
error: (e) => {
console.error('Decoder error:', e);
}
});
// Configure the decoder for H.264
decoder.configure({
codec: 'avc1.42E01E', // H.264 Baseline Level 3.0
codedWidth: 640,
codedHeight: 360,
hardwareAcceleration: 'prefer-hardware'
});
// Feed encoded chunks to the decoder
function feedChunk(data, timestamp, isKeyFrame) {
const chunk = new EncodedVideoChunk({
type: isKeyFrame ? 'key' : 'delta',
timestamp: timestamp, // in microseconds
data: data
});
decoder.decode(chunk);
}
// Example: decode a keyframe
const encodedData = new Uint8Array([/* H.264 NAL unit bytes */]);
feedChunk(encodedData, 0, true);
Understanding Codec Strings
Codec strings identify the specific codec and profile. Common examples include:
'avc1.42E01E'— H.264 Baseline Profile, Level 3.0'vp09.00.10.08'— VP9, Profile 0, Level 1.0'av01.0.05M.08'— AV1, Main Profile, Level 5.0, 8-bit'mp4a.40.2'— AAC-LC audio'opus'— Opus audio
You can check if a codec is supported using VideoDecoder.isConfigSupported():
const config = {
codec: 'avc1.42E01E',
codedWidth: 1920,
codedHeight: 1080
};
const support = await VideoDecoder.isConfigSupported(config);
if (support.supported) {
console.log('Configuration is supported');
decoder.configure(support.config);
} else {
console.log('Configuration is not supported');
}
Encoding Video with VideoEncoder
Encoding video involves creating VideoFrame objects from a source (such as a canvas, camera stream, or generated content) and passing them to a VideoEncoder.
let frameCount = 0;
const encodedChunks = [];
const encoder = new VideoEncoder({
output: (chunk, metadata) => {
encodedChunks.push(chunk);
console.log(`Encoded chunk: type=${chunk.type}, ` +
`timestamp=${chunk.timestamp}, ` +
`duration=${chunk.duration}, ` +
`bytes=${chunk.byteLength}`);
if (metadata.decoderConfig) {
console.log('Decoder config:', metadata.decoderConfig);
}
},
error: (e) => {
console.error('Encoder error:', e);
}
});
// Configure the encoder for H.264
encoder.configure({
codec: 'avc1.42E01E',
width: 640,
height: 360,
bitrate: 2_000_000, // 2 Mbps
framerate: 30,
keyFrameEveryNFrames: 60,
latencyMode: 'realtime'
});
// Create frames from a canvas and encode them
const canvas = document.getElementById('canvas');
const ctx = canvas.getContext('2d');
canvas.width = 640;
canvas.height = 360;
function encodeFrame(timestamp) {
// Draw something on the canvas
ctx.fillStyle = `hsl(${frameCount * 5 % 360}, 70%, 50%)`;
ctx.fillRect(0, 0, canvas.width, canvas.height);
ctx.fillStyle = 'white';
ctx.font = '48px sans-serif';
ctx.fillText(`Frame ${frameCount}`, 50, 180);
// Create a VideoFrame from the canvas
const frame = new VideoFrame(canvas, {
timestamp: timestamp, // microseconds
duration: 33_333 // ~30fps in microseconds
});
// Encode the frame
encoder.encode(frame, { keyFrame: frameCount % 60 === 0 });
// Close the frame to free memory
frame.close();
frameCount++;
}
// Encode 300 frames (10 seconds at 30fps)
for (let i = 0; i < 300; i++) {
encodeFrame(i * 33_333);
}
// Wait for all encoded chunks
await encoder.flush();
console.log(`Encoding complete. ${encodedChunks.length} chunks produced.`);
encoder.close();
Working with Audio
The audio codecs follow the same pattern as video. Here is an example of decoding Opus audio:
const audioDecoder = new AudioDecoder({
output: (audioData) => {
console.log('Decoded audio:', audioData.numberOfFrames, 'frames,',
audioData.sampleRate, 'Hz,',
audioData.numberOfChannels, 'channels');
// Process or play the audio data
// AudioData can be copied to an AudioBuffer for Web Audio API
audioData.close();
},
error: (e) => {
console.error('Audio decoder error:', e);
}
});
audioDecoder.configure({
codec: 'opus',
sampleRate: 48000,
numberOfChannels: 2
});
// Feed encoded audio chunks
const audioChunk = new EncodedAudioChunk({
type: 'key',
timestamp: 0,
data: new Uint8Array([/* Opus packet bytes */])
});
audioDecoder.decode(audioChunk);
Building a Complete Decode-to-Canvas Pipeline
Let's build a more complete example that reads an MP4 file, extracts H.264 chunks, decodes them, and renders to a canvas. This example uses the mp4box.js library for demuxing.
<!-- Include mp4box.js for demuxing -->
<script src="https://cdn.jsdelivr.net/npm/mp4box@0.5.2/dist/mp4box.min.js"></script>
<canvas id="output" width="640" height="360"></canvas>
<script>
const canvas = document.getElementById('output');
const ctx = canvas.getContext('2d');
let decoder = null;
let track = null;
async function decodeMp4(file) {
const arrayBuffer = await file.arrayBuffer();
arrayBuffer.fileStart = 0;
const mp4box = MP4Box.createFile();
mp4box.onReady = (info) => {
console.log('MP4 info:', info);
// Find the first video track
track = info.videoTracks[0];
// Configure the decoder
decoder = new VideoDecoder({
output: (frame) => {
canvas.width = frame.codedWidth;
canvas.height = frame.codedHeight;
ctx.drawImage(frame, 0, 0);
frame.close();
},
error: (e) => console.error('Decoder error:', e)
});
decoder.configure({
codec: track.codec,
codedWidth: track.track_width,
codedHeight: track.track_height,
description: getDescription(mp4box, track.id)
});
// Start extracting samples
mp4box.setExtractionOptions(track.id, null, {
nbSamples: 100
});
mp4box.start();
};
mp4box.onSamples = (trackId, ref, samples) => {
for (const sample of samples) {
const chunk = new EncodedVideoChunk({
type: sample.is_sync ? 'key' : 'delta',
timestamp: sample.cts * 1_000_000 / track.timescale,
duration: sample.duration * 1_000_000 / track.timescale,
data: sample.data
});
decoder.decode(chunk);
}
mp4box.start();
};
mp4box.appendBuffer(arrayBuffer);
mp4box.flush();
}
// Extract codec description (AVCC) from the track
function getDescription(mp4box, trackId) {
const track = mp4box.getTrackById(trackId);
if (track && track.mdia && track.mdia.minf &&
track.mdia.minf.stbl && track.mdia.minf.stbl.stsd &&
track.mdia.minf.stbl.stsd.entries[0].avcC) {
const avcC = track.mdia.minf.stbl.stsd.entries[0].avcC;
const description = new Uint8Array(avcC.size);
const view = new DataView(description.buffer);
avcC.write(view);
return description;
}
return undefined;
}
// Usage: select a file and decode it
const fileInput = document.createElement('input');
fileInput.type = 'file';
fileInput.accept = 'video/mp4';
fileInput.onchange = (e) => decodeMp4(e.target.files[0]);
document.body.appendChild(fileInput);
</script>
Processing Video Frames
One of the most powerful use cases for WebCodecs is real-time frame processing. You can decode frames, manipulate them, and re-encode them. Here is an example that applies a grayscale filter:
const decoder = new VideoDecoder({
output: async (frame) => {
// Draw to an offscreen canvas
const offscreen = new OffscreenCanvas(frame.width, frame.height);
const offCtx = offscreen.getContext('2d');
offCtx.drawImage(frame, 0, 0);
// Apply grayscale filter
const imageData = offCtx.getImageData(0, 0, frame.width, frame.height);
const data = imageData.data;
for (let i = 0; i < data.length; i += 4) {
const gray = data[i] * 0.299 + data[i+1] * 0.587 + data[i+2] * 0.114;
data[i] = data[i+1] = data[i+2] = gray;
}
offCtx.putImageData(imageData, 0, 0);
// Create a new frame from the processed canvas
const processedFrame = new VideoFrame(offscreen, {
timestamp: frame.timestamp,
duration: frame.duration
});
// Encode the processed frame
encoder.encode(processedFrame, { keyFrame: frame.type === 'key' });
// Clean up
frame.close();
processedFrame.close();
},
error: (e) => console.error('Decoder error:', e)
});
const encoder = new VideoEncoder({
output: (chunk) => {
// Store or transmit the encoded chunk
console.log('Encoded chunk:', chunk.type, chunk.byteLength, 'bytes');
},
error: (e) => console.error('Encoder error:', e)
});
decoder.configure({
codec: 'avc1.42E01E',
codedWidth: 640,
codedHeight: 360
});
encoder.configure({
codec: 'avc1.42E01E',
width: 640,
height: 360,
bitrate: 1_500_000,
framerate: 30
});
Capturing Camera Input with WebCodecs
You can use the MediaStreamTrackProcessor to get VideoFrame objects from a camera stream, then encode them with WebCodecs:
async function captureAndEncode() {
// Get camera stream
const stream = await navigator.mediaDevices.getUserMedia({
video: { width: 1280, height: 720, frameRate: 30 }
});
const track = stream.getVideoTracks()[0];
// Create a track processor to get VideoFrames
const processor = new MediaStreamTrackProcessor({ track });
const reader = processor.readable.getReader();
// Set up the encoder
const encoder = new VideoEncoder({
output: (chunk, metadata) => {
// Send chunk over network or store it
console.log('Encoded:', chunk.type, chunk.byteLength, 'bytes');
},
error: (e) => console.error('Encoder error:', e)
});
encoder.configure({
codec: 'vp09.00.10.08',
width: 1280,
height: 720,
bitrate: 2_500_000,
framerate: 30,
latencyMode: 'realtime'
});
let frameCount = 0;
// Read frames from the camera and encode them
while (true) {
const { done, value: frame } = await reader.read();
if (done) break;
encoder.encode(frame, { keyFrame: frameCount % 120 === 0 });
frame.close();
frameCount++;
// Check encoder health
if (encoder.encodeQueueSize > 10) {
console.warn('Encoder is falling behind');
}
}
encoder.close();
}
captureAndEncode();
Best Practices
Always Close Frames and Data Objects
VideoFrame and AudioData objects hold significant memory resources. Always call close() when you are done with them to prevent memory leaks.
const frame = new VideoFrame(canvas, { timestamp: 0 });
// ... use the frame ...
frame.close(); // Free the memory
// Check if a frame is closed
if (frame.closed) {
console.log('Frame already closed');
}
Handle Backpressure
Decoders and encoders have internal queues. If you feed data faster than it can be processed, the queue grows and latency increases. Monitor the queue size and apply backpressure:
function feedDecoder(chunk) {
if (decoder.decodeQueueSize > 5) {
// Wait before feeding more data
setTimeout(() => feedDecoder(chunk), 10);
return;
}
decoder.decode(chunk);
}
// For encoders, check encodeQueueSize
if (encoder.encodeQueueSize > 10) {
// Drop frames or wait
}
Use Hardware Acceleration When Available
Specify hardwareAcceleration: 'prefer-hardware' in your codec configuration to leverage GPU-based encoding and decoding, which is significantly faster and more power-efficient.
Check Codec Support Before Configuring
Always use isConfigSupported() before configuring a codec. Different platforms support different codecs and configurations:
async function safeConfigure(decoder, config) {
const { supported, config: supportedConfig } =
await VideoDecoder.isConfigSupported(config);
if (!supported) {
throw new Error(`Codec ${config.codec} is not supported`);
}
decoder.configure(supportedConfig);
}
Use flush() Before close()
Always call flush() before close() to ensure all pending operations complete:
// After feeding all chunks
await decoder.flush();
decoder.close();
await encoder.flush();
encoder.close();
Handle Errors Gracefully
Codec errors can occur due to corrupted data, unsupported configurations, or hardware issues. Always implement proper error handling and consider implementing a fallback:
const decoder = new VideoDecoder({
output: handleFrame,
error: (e) => {
console.error('Decoder error:', e.message);
if (e.message.includes('hardware')) {
// Fallback to software decoding
decoder.configure({
...currentConfig,
hardwareAcceleration: 'prefer-software'
});
}
}
});
Performance Considerations
WebCodecs is designed for high performance, but how you use it matters. Avoid creating unnecessary intermediate copies of frame data. When processing frames, use OffscreenCanvas in a Web Worker to keep the main thread responsive. The VideoFrame constructor can accept a BufferSource with format options, which is faster than going through a canvas when you only need to wrap existing pixel data.
// Wrapping raw pixel data directly (fast)
const frame = new VideoFrame(rawPixelBuffer, {
format: 'RGBA',
codedWidth: 640,
codedHeight: 360,
timestamp: 0
});
// Supported formats include: RGBA, RGBX, BGRA, BGRX, YUV420P, NV12
Using WebCodecs in a Web Worker
For best performance, run your codec operations in a Web Worker to avoid blocking the main thread:
// main.js
const worker = new Worker('codec-worker.js');
// Send encoded data to the worker
worker.postMessage({ type: 'decode', data: encodedChunkData });
// Receive decoded frames (as ImageBitmap) from the worker
worker.onmessage = (e) => {
if (e.data.type === 'frame') {
const bitmap = e.data.bitmap;
ctx.drawImage(bitmap, 0, 0);
bitmap.close();
}
};
// codec-worker.js
let decoder = null;
self.onmessage = (e) => {
if (e.data.type === 'init') {
decoder = new VideoDecoder({
output: (frame) => {
// Convert frame to ImageBitmap for transfer
frame.createImageBitmap().then((bitmap) => {
self.postMessage({ type: 'frame', bitmap }, [bitmap]);
frame.close();
});
},
error: (err) => console.error(err)
});
decoder.configure(e.data.config);
} else if (e.data.type === 'decode') {
const chunk = new EncodedVideoChunk(e.data.data);
decoder.decode(chunk);
}
};
Conclusion
The WebCodecs API represents a significant step forward for media processing on the web. By providing direct, low-level access to hardware-accelerated codecs, it enables developers to build sophisticated media applications — from real-time video processing and custom streaming solutions to in-browser video editing and AI-powered analytics — all without the overhead of WebAssembly-based codecs or the limitations of high-level APIs. While the API requires careful attention to memory management, backpressure handling, and cross-browser compatibility, the performance and flexibility it offers make it an essential tool for any developer working with media on the modern web. As browser support continues to expand, WebCodecs is poised to become the foundation for the next generation of web-based media experiences.