How is AI used in video coding?

Video has become a core part of modern digital life — from entertainment and communication to safety-critical and machine-driven applications. Making all this work depends on video coding, which compresses video so it can be transmitted and stored efficiently. Today, AI is taking video coding into a new phase: helping systems encode more efficiently, improve quality more adaptively, and support entirely new kinds of video applications. In this blog, we look at how AI is changing video coding — and what that could mean for the future of media and intelligent systems.

What Is Video Coding?

Video is everywhere. We use it for streaming films and sports, scrolling social media, joining video calls, online learning, gaming, security systems, and increasingly also in applications such as autonomous driving, robotics, drones, and industrial automation. In all these cases, video needs to be captured, transmitted, stored, and played back efficiently. That is where video coding comes in.

In simple terms, video coding includes compressing video so that it uses much less data while still delivering the quality, responsiveness, and reliability needed for its intended application. Without compression, video would be far too heavy for practical services. For example, a raw 4K video stream can require around 2 Gbps, while compression can bring that down to roughly 15 Mbps. Those three orders of magnitude reduction are what makes modern video services possible at scale — from streaming and video conferencing to connected cameras and machine vision.

This is hugely important because video makes up as much as 70% of internet and mobile traffic, and demand keeps growing as video quality, usage, and the number of connected devices all continue to increase. The volume of video traffic is expected to double in five years. In practice, video coding is the hidden technology that enables the entire video economy.

Traditionally, video coding has relied on highly optimized signal-processing methods and carefully engineered standards such as H.265/High Efficiency Video Coding (HEVC) and H.266/Versatile Video Coding (VVC). These technologies have improved dramatically over time, making video delivery more efficient generation after generation. But video applications are also becoming more demanding. We now want not only lower bitrates, but also better visual quality, lower delay for smooth interaction, more adaptability for network conditions, and support for new emerging use cases where video is consumed not just by people, but also by machines.

This is exactly why AI is gradually entering video coding. AI is opening up new ways to make compression smarter:

• speeding up video encoding,
• improving decoded video quality
• enabling new neural compression methods,
• and helping video systems adapt better to content, network, devices, and applications.

In other words, AI is not changing why we compress video — but it is changing how we do it, and how far video coding can go in the future.

How Video Coding Works

All modern video codecs, including the ones used in streaming today, follow what we call a block-based coding pipeline. The idea is simple: instead of sending full images, we try to remove redundancy. First, we split the image into blocks. Then we predict each block, either from neighboring pixels, or from previous frames. And instead of sending the full signal, we only encode the difference, the residual. That residual is then transformed, quantized, and entropy coded into a bitstream.

This pipeline has been extremely successful as it has improved compression efficiency roughly by a factor of two every standard generation. In this way, the latest video coding standard, H.266/VVC, delivers the same perceptual video quality as its predecessor, H.265/HEVC, but with half the bitrate. As most of the internet and mobile traffic is video, a modern codec like H.266/VVC can significantly improve the performance and efficiency of video transmission globally.

But the downside is that this block-based coding pipeline in modern video codecs is complex, especially on the encoder side where video is compressed for transmission or storage. Encoding a single video can require high computation and therefore encoders are typically implemented with the aid of efficient hardware chips on battery-powered devices like smartphones.

Now, AI does not replace this pipeline overnight. Instead, it is expected to enter gradually, one component at a time. Neural networks can enhance prediction, improve filtering, or even reconstruct higher-quality images. And importantly, these capabilities are now entering standards, e.g., through mechanisms like the Versatile Supplemental Enhancement Information (H.274/VSEI) standard, which allows signaling neural processing in the video bitstream. Within this framework video streamers among others can introduce and deploy their own AI models, in an interoperable way, to enhance the visual quality after decoding, e.g., by increasing video resolution or by removing coding artefacts.

So, the key idea is AI today is not replacing video coding, it is augmenting and evolving it, first as a hybrid coding pipeline by combining the strengths from conventional block-based and neural network-based video coding.

Where AI Meets Video Coding

Modern video compression is undergoing a transformation due to artificial intelligence (AI) and machine learning (ML). For decades, block-based video coding, e.g., standards like H.265/HEVC and H.266/VVC, used hand-crafted algorithms for tasks like motion prediction, transform coding, entropy coding, and rate–distortion optimization (RDO). Starting around the mid-2010s, AI/ML approaches began providing new tools and concepts to improve video coding. These can be grouped into four major categories, each representing a new research direction that augments or disrupts traditional video codec design.

AI for encoder optimization: AI can help make video encoding faster by guiding decisions like block partitioning and prediction mode selection without breaking compatibility with existing decoders integrated into billions of devices already on the market. As these AI encoder optimizations operate entirely on the encoder side, the output is still a fully standard-compliant bitstream and does not require any special decoder support. This means they are backward-compatible with existing decoders, making them attractive for integration into products and standards.

This is critical, especially for large-scale platforms like video-on-demand (VoD) services which process massive amounts of video to be consumed on numerous end-user devices. Standardization bodies such as the Joint Video Experts Team (JVET), a joint expert team of the International Telecommunication Union (ITU) and the International Organization for Standardization / International Electrotechnical Commission (ISO/IEC), actively encourage these techniques in next-generation video codec development, in reference codec software and test models, even if the bitstream syntax remains classical. The driving goals are reduced encoding cost (critical for real-time and large-scale streaming) and potential to improve coding efficiency.

AI-based enhancement tools in the codec pipeline (hybrid codecs): Neural filters can significantly boost video quality by removing compression artifacts or enhancing resolution after decoding or within the decoder. These can operate as post-filters or as part of the decoder’s in-loop filtering chain, improving visual results without changing the core bitstream. The advantage of post-filters is that they reside outside the coding loop, and so can be deployed with any codec. The H.274/VSEI standard enables encoders to signal neural post-filters in the bitstream, including model description, optional compressed weights, and when to apply them.

This allows content-adaptive enhancement, such as sequence-specific super-resolution or denoising. Importantly, it remains backward compatible – devices without support simply decode the base video. Nokia’s experiments show that real-time neural post-filters can deliver significant bitrate savings on top of H.266/VVC. On the other hand, some work explores in-loop learned filters, where enhanced frames are reused for prediction. While this could further improve compression, integrating complex neural networks into the decoding loop is challenging due to computational constraints. As a result, existing standards rely on lighter methods, keeping advanced neural filters optional via VSEI, ensuring compatibility and allowing decoders to opt in.

End-to-end neural compression (learned codecs): Here we move beyond the classical pipeline and use end-to-end neural networks to compress and reconstruct images and videos. Learned codecs have the advantage of being highly tunable (one can optimize them for perceptual quality metrics or for machine task accuracy) and can discover new coding tools automatically (e.g., specialized transforms and context models). They can also adapt on the fly by fine-tuning on specific content to boost performance. But they face practical barriers: each learned codec uses its own custom model and bitstream, requiring specialized decoders and powerful hardware, such as Graphics Processing Units (GPUs), Neural Processing Units (NPUs), or Tensor Processing Units (TPUs).

Unlike classical standards, there’s no universal decoder widely deployed for these new formats, which hinders adoption. Interoperability and complexity are key concerns, alongside hardware-decode availability, cross-device determinism, mobile power and latency constraints, and the lack of standardized end-to-end deployment. For these reasons, fully learned video codecs remain a research frontier and in pilot use for niche applications, while standardization bodies are investigating how to incorporate neural coding within future standards.

A concrete example is JPEG AI from 2025, the first international standard based on end-to-end neural image compression for still-image coding, illustrating how these approaches are already entering standards.

Neural representations & model compression: This is a more radical idea — where we don’t compress the signal directly but instead learn a model that represents the content itself. In this paradigm, the image or video content is “overfit” by a neural network as an implicit neural representation (INR) whose weights encode the content. The representation learning and compression problems are decoupled: first, a network is trained to represent the content, then the network’s weights are compressed – for example by using the Neural Network Coding (NNC) standard. NNC already provides a standards-based way to compress neural network parameters, and its future evolution toward tensor compression could make it relevant also for feature maps and latent representations.

Neural representations shift emphasis from coding raw signals to coding learned abstractions of signals. This approach can excel if content is highly structured or can be represented with fewer parameters than raw data. The complexity of training a network for each new piece of content and the overhead of transmitting model weights mean current INR-based codecs are not yet practical for general use. Nevertheless, the coupling of representation learning and compression is a promising research direction, potentially foreshadowing a future where AI systems themselves act as a new kind of codec. This approach has also been explored for volumetric video compression as such volumes can be represented as learned neural fields or Gaussian splats.

So, the key message here is that AI is influencing video coding at multiple levels — from incremental improvements to completely new approaches. What starts today as smarter encoding, enhancement, and optimization could, over time, lead to much broader changes in how video is represented, delivered, and even understood.

AI / ML Disruption in Video Coding

Now, beyond improving individual components, AI may lead to a much bigger shift. A forward-looking trend is the emergence of foundation models – very large neural networks pre-trained on diverse multimodal data, including images, text, video, audio, 3D content, etc. Traditionally, we have separate formats for these modalities like MP4 for video and AAC for audio. However, multimodal models learn shared representations across all these modalities, inspiring a unified representation paradigm. “One representation for all multimedia content” hints at a future where AI-native formats replace separate media-specific formats. In this vision, a single unified data representation (e.g., a shared latent space or model-driven code) could encode different modalities (image, video, text, audio, 3D), bridging traditional divides between media types. This means that in the future, content might not be stored or transmitted as separate formats – but as unified representations.

The rise of large multimodal models – which combine powerful vision, audio, and language understanding – is transforming video semantics and analysis. These models can interpret complex scenes, integrate visual and textual information, and even generate content across modalities. Their capabilities suggest new directions for compression and streaming: e.g., the Feature Coding for Machines (FCM) standard where compression optimizes for machine analysis tasks rather than human viewing, and semantic or scene-aware encoding where a video codec might adapt bit allocation using AI-detected scene content to improve efficiency or quality where it matters most. While a single format for all modalities remains a long-term vision, it’s guiding collaborations between industry and academia to drive research on multimodal foundation models and their integration with video technology.

The Future of Video Coding — and Nokia’s Role in It

AI is not replacing video coding. It is making it smarter, more adaptable, and better suited to a growing range of applications. In the near term, the biggest impact will likely come from hybrid approaches, where AI strengthens today’s proven coding pipelines through smarter encoding decisions, better post-processing, and more adaptive quality optimization. Over time, more radical approaches such as end-to-end neural codecs and learned representations may also become part of the picture, especially as standards and device ecosystems continue to evolve.

This matters because video must work reliably across an enormous range of environments: phones, TVs, tablets, browsers, cameras, vehicles, drones, industrial systems, and cloud platforms. That is why standardization remains so important. New AI-based coding tools need to be not only effective, but also interoperable, scalable, and practical to deploy. In other words, the future of video coding will not be defined by AI alone, but by how AI is integrated into standards, devices, networks, and real-world services.

This is where Nokia contributes. Nokia’s work in multimedia goes beyond individual research projects. The company contributes to video coding standards, compression technologies, and patented inventions that help modern video services work reliably at scale. Its contributions in multimedia have also been recognized with six Technology & Engineering Emmy Awards together with partners, and Nokia has created more than 5,000 inventions enabling multimedia products and services since 2000. As AI becomes part of video coding, Nokia’s work also looks at how video can be compressed, enhanced, and analyzed more intelligently for both viewers and AI-driven systems.

The key takeaway is simple: video coding is essential and it is entering a new phase. AI is:

• helping codecs make better decisions based on representative data,
• improve quality in new ways,
• and support emerging media experiences and machine-driven applications.

The future of video coding is therefore not about replacing what already works, but about evolving it — combining the strengths of established standards with the new possibilities opened by AI.