WHAT ARE CODECS AND WHY ARE THEY IMPORTANT?
If you work with digital video content or the software and systems that support it, you’ll be familiar with codecs. But for the rest us, our only encounter or experience with these vital digital components could be a common media player error message, like this one from Windows:
So, what are codecs and why are they so important for a company creating advanced light field displays?
We’ll start with the first question. The term “codec” is a portmanteau of “coder-decoder'' and is used to describe an algorithm and/or the mechanisms that implement it (hardware and/or software) that encode and decode digital data. This is important as it allows devices and programs to efficiently compress, transmit, and process large volumes of data.
In 2011, Jan Ozer, Contributing Editor at StreamingMedia.com described codecs as “the oxygen of the streaming media market; no codecs, no streaming media.” And this description holds true for 3D content and visual models for light field displays.
Within the light field display category, there is a need for increased data transmission capabilities, improved data codecs, and methods for the real-time delivery of multi-dimensional content. Existing video codecs simply weren’t designed to handle the bandwidth and compression requirements of interactive 3D content, so our team of developers and engineers got to work on building (and patenting) our own. We set out to address two key challenges: the streaming challenge and the real-time rendering challenge.
When it comes to streaming, a traditional 4K display at 30 frames per second requires ~6 Gbps (Gigabits per second) of raw pixel bandwidth, which is typically compressed to 10-30 Mbps for streaming services. A high-density light field display may have billions of rays, requiring many terabits per second of pixel bandwidth. As a result, for streaming, light field codecs must encode potentially 1000x more bandwidth than 2D codecs. It is impractical to target the same 10-30 Mbps streaming bandwidth requirement in this case, but it reinforces the magnitude of the problem and the importance of codecs for streaming optimization. Next-generation wired and wireless networks are targeting up to 10 Gbps links to the home, but an individual service is expected to consume only a portion of that. Therefore, even a high-end service such as light field streaming would ideally target data rates in the low Gbps range (or lower).
Our codecs also resolve the real-time rendering challenge. Light field or 3D scene data is deconstructed into layers before being sampled and rendered (encoded) to compress the data for transmission. The data is then decoded in real-time to construct a 3D scene on our light field displays. This enables interactive applications at reasonable frame rates of 30 or more frames per second. Avalon’s codec patent (US 10,432,944) and multiple pending applications address 3D image data encoding and decoding to allow us to drive a light field display in real-time.
Our approach delivers a codec method with optimized system transmission bandwidth and latency to provide for the production of a light field, in real time, with reliable resolution for applications like live video streaming and real-time interactive gaming. We’re proud to contribute to this active focus area alongside initiatives like MPEG Group, the Immersive Digital Experiences Alliance (IDEA), and JPEG-Pleno, which aim to provide a standard framework for representing new imaging modalities; the Optical Society (OSA) who seek consensus on a high bandwidth and display agnostic 3D display drive architecture; and the U.S. Air Force Research Laboratory's efforts to establish a Streaming Media Field of Light Display (SMFoLD) standard.