WebRTC

WebCodecs API Explained: Raw Codec Access in the Browser

Q: What are VideoEncoder and VideoDecoder used for?

VideoEncoder turns raw VideoFrame objects into compressed EncodedVideoChunk objects; VideoDecoder does the reverse, turning encoded chunks back into displayable frames.

by Nina Benkotic

9 min read

August 13, 2026

WebCodecs is a browser JavaScript API that gives developers direct access to the browser's built-in video and audio codecs, through raw VideoEncoder, VideoDecoder, AudioEncoder, and AudioDecoder objects, without going through the full WebRTC stack or the <video> element. It exists because a growing number of developers need frame-level control that WebRTC deliberately abstracts away: custom transport, per-frame processing, machine learning on raw pixels, none of which fit neatly inside an API designed to make peer-to-peer calls simple. This article looks at what the WebCodecs API actually is, what it gives you, how it relates to WebRTC, and when it's realistically worth reaching for.

Table of contents

The problem WebCodecs solves
The core building blocks
WebCodecs vs WebRTC
What you'd actually build with it
Browser support and the real-world caveats
Do you need WebCodecs or a platform that handles this for you?
Conclusion
Frequently asked questions

The problem WebCodecs solves

WebRTC and the <video> element are, by design, all-in-one. WebRTC bundles capture, encoding, transport, jitter buffering, and rendering into a single pipeline, and deliberately keeps the codec layer out of reach: you configure preferences, but you never touch a raw encoded frame. That's exactly right for a standard video call: nobody building a meetings product wants to hand-roll congestion control. But it becomes a limitation the moment you need something WebRTC wasn't built to expose, such as feeding frames into a machine-learning model before they're sent, muxing encoded output into your own file format, or shipping media over a transport WebRTC doesn't use.

WebCodecs solves this by exposing the encode and decode steps on their own, detached from capture and transport. You get raw codec access, and you're responsible for building the pipeline around it, which is precisely the trade-off developers reaching for this API are looking to make.

The core building blocks

The API surface is smaller than it might sound. Two families of objects do the work:

Raw media units. VideoFrame and AudioData hold uncompressed media. A VideoFrame carries pixel data plus metadata such as timestamp and duration, tied to whatever backing memory the browser is using at that moment.
Encoders and decoders. VideoEncoder and VideoDecoder do the actual codec work. A VideoEncoder takes VideoFrame objects and turns them into EncodedVideoChunk instances (the compressed bytes plus a little metadata, notably whether a chunk is a key frame or a delta frame). A VideoDecoder does the reverse, turning EncodedVideoChunk objects back into VideoFrame objects. AudioEncoder and AudioDecoder mirror this for audio, working with AudioData and EncodedAudioChunk.

A minimal pipeline looks like this:

Capture or generate a VideoFrame.
Hand it to a configured VideoEncoder.
Receive an EncodedVideoChunk back in a callback.
Send that chunk wherever you need it, for example over WebTransport, a WebSocket, or a data channel, or route it into a muxing library for a file.
On the receiving end, feed each chunk into a VideoDecoder, which hands you back VideoFrame objects.

What you do with those frames from there is up to you: paint them to a canvas, upload them to WebGPU or WebGL as a texture for GPU-side effects, read out their raw pixels for an ML model, or feed them into a MediaStreamTrackGenerator to turn them back into an ordinary video stream. Everything runs asynchronously and is best run off the main thread, in a web worker, since encode and decode callbacks can fire many times a second and would otherwise make the page sluggish.

None of this is free-form, though. Both encoders and decoders need to be configured with a specific codec string before they'll accept anything, and that string has to name not just the codec family but the profile and level too (for codecs other than VP8), since a browser's hardware decoder often only supports a narrow band of configurations. Getting this wrong doesn't crash the page; the encoder or decoder simply reports that it can't support the requested configuration, which is why most real implementations check support before committing to a codec rather than discovering the failure mid-stream.

WebCodecs vs WebRTC: how they relate (not compete)

The most common confusion is treating this as a straight replacement for WebRTC. It isn't. WebRTC still owns real-time peer connections, NAT traversal via ICE, congestion control, and the whole SDP offer/answer negotiation, and WebCodecs doesn't attempt to replicate any of it. WebCodecs is just the codec layer, now reachable directly rather than locked inside WebRTC's pipeline. If you need peer-to-peer calling with sensible defaults, WebRTC remains the tool; WebCodecs is what you use when you want to build something WebRTC's black box doesn't offer, and you're willing to supply the transport and error-recovery logic yourself.

That's exactly the gap the emerging WebCodecs-plus-WebTransport pattern is filling. WebTransport gives the browser a QUIC-based, HTTP/3 transport primitive with none of WebRTC's negotiation overhead, though it's a client-server transport (browser-to-server rather than peer-to-peer), so it complements WebRTC rather than replacing it outright. Pairing it with WebCodecs produces a more unbundled alternative to WebRTC for certain low-latency streaming scenarios: it's the browser-side stack on top of which newer protocols like Media over QUIC get built. We've covered WebRTC and Media over QUIC in detail elsewhere, so we won't retread that ground here, beyond noting how the pieces fit together.

What you'd actually build with it

In practice, WebCodecs tends to show up in a fairly specific set of projects:

Low-latency live streaming with a custom transport. This is the headline case: pairing WebCodecs with WebTransport lets you build a delivery pipeline tuned to your own latency and quality trade-offs rather than accepting WebRTC's defaults.
Frame-level processing, such as background removal or computer-vision analysis that needs to inspect or modify raw pixels before they're encoded. If that processing is happening inside a live WebRTC call, WebRTC's own Insertable Streams API can often do this without needing WebCodecs at all (more on that in the decision section below). WebCodecs earns its place for frame-level work that sits outside a live call altogether, where WebRTC's abstraction was never in the picture to begin with.
Cloud gaming and remote rendering. These workloads care intensely about encode latency and don't want a general-purpose conferencing stack in the way, so hardware-accelerated encode is appealing. In practice, though, hardwareAcceleration in a codec config is only a hint the browser is free to ignore, and actual hardware support varies by device and operating system. A production pipeline still needs to check isConfigSupported() and keep a software encode fallback ready, rather than assuming hardware acceleration will simply be there.
In-browser recording or transcoding. Taking a stream and re-encoding it into a different codec or container entirely client-side is practical now in a way it simply wasn't before WebCodecs existed, when the alternative was shipping media to a server or bundling a WebAssembly encoder.

Browser support and the real-world caveats

Support is strong and mature across Chromium: Chrome, Edge, and Opera have shipped full codec access since Chrome 94, several years ago now. Firefox caught up on desktop more recently, with full support landing from Firefox 130, though Firefox for Android still doesn't implement it. Safari's story took the longest: for several years Safari shipped only partial, video-only support, with full parity, including audio, arriving only in Safari 26. The practical upshot is that WebCodecs is a safe bet for a Chromium-first or desktop-first product today, but check current compatibility tables for your specific target browsers and versions before committing, particularly if mobile Firefox or older Safari releases are in your user base.

WebTransport, the transport half of the WebCodecs-plus-WebTransport pairing discussed above, took longer still to become universal. It only reached Baseline, meaning support across Chrome, Edge, Firefox, and Safari, in March 2026, when Safari 26.4 finally shipped it. Chromium has had it since 2022 and Firefox since 2023. So that combined stack has only been broadly deployable for a matter of months, not years, which is worth factoring in before betting a production system on it.

The bigger caveat isn't a browser gap. It's what you take on yourself once you drop below WebRTC's abstraction. WebRTC gives you loss recovery, synchronisation, jitter buffering, and congestion control for free, tuned over more than a decade of production use. Reach for WebCodecs and WebTransport instead, and none of that comes built in: you own bitrate adaptation, packet loss handling, and keeping audio and video in sync, because the browser is deliberately not making those decisions for you any more. WebTransport itself doesn't yet have equivalents to WebRTC's bandwidth-estimation and adaptive-bitrate algorithms, so a serious custom pipeline typically means implementing or borrowing that logic rather than assuming the transport handles it. That's the honest trade-off: lower-level control in exchange for owning the hard parts a bundled stack used to handle silently.

Do you need WebCodecs or a platform that handles this for you?

This is worth framing as a genuine build-versus-embed decision rather than assuming raw access is always the better answer. WebCodecs earns its complexity for teams building a genuinely custom media pipeline, a proprietary streaming protocol, a processing step that has to happen on raw frames, or a product shape WebRTC simply doesn't fit.

There's a middle path worth knowing about before you reach for WebCodecs, though. If all you need is frame-level access inside an otherwise ordinary video call, such as background blur or a computer-vision effect, WebRTC already has a way to do it without giving up its own transport. Insertable Streams (MediaStreamTrackProcessor and MediaStreamTrackGenerator) and WebRTC Encoded Transform let you read and modify frames mid-call while WebRTC keeps handling congestion control, NAT traversal, and sync. WebCodecs is the right call when you need more than that: a pipeline that lives outside a live WebRTC call entirely, a custom transport, or a container format WebRTC was never built to produce.

Most products that just need standard video calling don't need any of this, because WebRTC (or an embedded SDK built on it) already handles encode, transport, and loss recovery without any of that code needing to be written twice.

Digital Samba's real-time stack sits in that last category: a WebRTC-based pipeline with an SFU architecture and VP8 as the baseline codec, plus built-in recording and restreaming, covers the standard conferencing and broadcast cases without a team needing to hand-roll codec handling themselves. That's not a claim that Digital Samba uses or needs WebCodecs internally, but simply the other side of the decision this article has been building towards: know which category your product falls into before choosing your tools.

Conclusion

WebCodecs unlocks raw, frame-level encode and decode in the browser: real power for teams building custom low-latency streaming or per-frame processing pipelines, unnecessary weight for anyone just running standard video calls. The API surface is small and well defined, and browser support is now solid outside a few mobile and legacy gaps, but the responsibility that comes with dropping below WebRTC's abstraction is real. For more on where this fits against the transport layer, see our guide to Media over QUIC and our video codec guide for the codec context underneath it all.

FAQ

What is the WebCodecs API?

It's a browser JavaScript API that exposes the browser's built-in video and audio encoders and decoders directly, giving developers frame-level control without going through WebRTC or the <video> element.

How is WebCodecs different from WebRTC?

WebRTC is a complete, bundled real-time communication stack, with capture, encode, transport, and render all handled together. WebCodecs is just the codec layer, encode and decode on their own, leaving transport and everything else to the developer.

What are VideoEncoder and VideoDecoder used for?

VideoEncoder turns raw VideoFrame objects into compressed EncodedVideoChunk objects; VideoDecoder does the reverse, turning encoded chunks back into displayable frames.

Which browsers support the WebCodecs API?

Chrome, Edge, and Opera have supported it fully since Chrome 94. Firefox added full desktop support from Firefox 130, though not yet on Android. Safari's support was video-only for several years and only reached full parity, including audio, in Safari 26.

Does WebTransport work in all browsers yet?

Yes, as of March 2026, when Safari 26.4 shipped it and WebTransport reached Baseline across Chrome, Edge, Firefox, and Safari. Chromium has supported it since 2022 and Firefox since 2023, including on Android, but as a complete cross-browser stack it's a recent arrival.

When should you use WebCodecs instead of an embedded video SDK?

Reach for WebCodecs when you're building a genuinely custom media pipeline, a proprietary transport, frame-level processing, or a product shape that standard video calling doesn't fit. Most products needing ordinary video calling are better served by an embedded SDK that already handles encoding and transport.

Can WebCodecs be used with WebTransport for live streaming?

Yes, pairing WebCodecs with WebTransport is the emerging pattern behind newer low-latency streaming approaches such as Media over QUIC, giving developers a QUIC-based transport paired with direct codec access instead of WebRTC's bundled pipeline.