Mobile video 101: protocols, containers, formats & codecs

Mobile video as a technology and market segment can at times be a little complicated.

Here is simple syllabus, in no particular order of what you need to know to be conversant in mobile video. It is not intended to be exhaustive or very detailed, but rather to provide a knowledge base for those interested in understanding more the market dynamics I address in other posts.


Protocols:

There are many protocols used in wireless networks to deliver and control video. You have to differentiate between routing protocols (IP), transmission protocols (TCP & UDP), session control (RTP), application control (RTSP) and content control protocols (RTCP). I will focus here on application and content control.

These protocols are used to setup, transmit and control video over mobile networks

Here are the main ones:

• RTSP (Real Time Streaming Protocol) is an industry protocol that has been created specifically for the purposes of media streaming. It is used to establish and control (play, stop, resume) a streaming session. It is used in many unicast on-deck mobile TV and VOD services.
• RTCP (Real Time transport Control Protocol) is the content control protocol associated with RTP. It provides the statistics (packet loss, bit transmission, jitter...) necessary to allow a server to perform real-time media quality control on an RTSP stream.
• HTTP download and progressive download (PD). HTTP is a generic protocol, used for the transport of many content formats, including video. Download and progressive download differentiate from each other in that the former needs the whole content to be delivered and saved to the device to be played asynchronously, while the later provides at the beginning of the session a set of metadata associated with the content which allow it to be played before its complete download.
• Microsoft silverlight, Adobe RTMP and Apple progressive streaming. These three variants of progressive download are proprietary. They offer additional capabilities beyond the vanilla HTTP PD (pre-encoding and multiple streams delivery, client side stream selection, chunk delivery...) and are the subject of an intense war between the three companies to occupy the mindset of content developers and owners. This is the reason why you cannot browse a flash site or view a flash video in your iPhone.

Containers:

A container in video is a file that is composed of the payload (video, audio, subtitles, programming guide...) and the metadata (codecs, encoding rate, key frames, bit-rate...). The metadata is a set of descriptive files that indicate the nature of the media, its duration in the payload. The most popular are:

• 3GPP (.3GP) 3GP is the format used in most mobile devices, as the recommended container for video by 3GPP standards.
• MPEG-4 part 14 (.MP4) one of the most popular container for internet video.
• Flash video (FLV, F4V). Adobe-created container, very popular as the preferred format for BBC, Google Video, Hulu, metacafe, Reuters, Yahoo video, YouTube... It requires a flash player.
• MPEG-2 TS: MPEG Transport Stream is used for broadcast of audio and video. It is used in on-deck broadcast TV services in mobile and cable/ satellite video delivery.

Formats
Formats are a set of standards that describe how a video file should be played.

• H.263 old codec used in legacy devices and applications. It is mandated by ETSI and 3GPP for IMS and MMS but is being replaced by H.264
• H.264, MPEG4 part 10, AVC is a family of standards composed of several profiles for different use, device types, screen sizes... It is the most popular format in mobile video.
• MPEG2 is a standard for lossy audio and video compression used in DVD, broadcast (digital TV, over the air, cable, satellite). MPEG2 describes two container types: MPEG2-TS for broadcast, MPEG-2 PS for files.
• MPEG4 is an evolution of MPEG2, adding new functionalities such as DRM, 3D and error resilience for transmission over lossy channels (wireless for instance).  There are many features in MPEG 4, that are left to the developer to decide whether to implement or not. The features are grouped by profiles and levels. There are 28 profiles or part in MPEG 4. A codec usually describe which MPEG-4 parts are supported. It is the most popular format on the internet.

Codecs

Codec stands for encoding and decoding a media stream. It is a program that has the ability to decode a video stream and re encode it. Codecs are used for compression (lossless), optimization (lossy) and encryption of videos. A "raw" video file is usually stored in YCbCr (YUV) format which provides the full description of every pixel in a video. This format is descriptive, which requires a lot of space for storage and a lot of processing power for decoding / encoding. This is why a video is usually encoded in a different codec, to allow for a better size or variable transmission quality. It is important to understand that while a container obeys strict rules and semantics, codecs are not regulated and each vendor decides how to decode and encode a media format.

• DivX Proprietary MPEg-4 implementation by DivX
• WMV (Windows Media Video) - Microsoft proprietary
• x264 a licenseable H.264 encoding spoftware
• VP6, VP7, VP8... proprietary codecs developed by On2 technologies, acquired by Google and released as open source