Since computers have turned multimedia, video has been one of the most popular media consumed on such devices. As the computing world is getting more and more mobile, there is a patent need to enhance the experience of high quality video consumption in mobility situation. PDAs and smartphones have failed so far delivering such experience in terms of format support and in terms of quality. Who hasn't tried to play such videos and found that the video plays jerky or not at all? Good video playback experience with the main existing formats is one of the promises Mobile Internet Devices claim to bring. Let's see how this can be achieved on an Intel Atom processor-based MID.
The MID Promise
As mobile phones are gaining in capabilities, they started to support some video playback. This experience remains poor due to the screen size as well as the processing power of the platform that is somewhat limited. There is thus no way for example to have a wide variety of content supported. Usually the content is tailored to the device in size and in format. Basically there is virtually no chance that the video content you play on your PC, plays at all on your mobile phone. MIDs come with a new promise to the mobile device users: the capability to play most of the video content that a PC plays including full movies.
Owing to its 4,5-6 inch screen which has at least a resolution of 800x480, the visual experience of video playback on MID is very pleasant. The Intel Atom processor, being an x86 architecture, the MID can support most of the codecs available out of the box including the most common video formats like MPEG-2, MPEG-4, H.264 and VC-1. It allows the user to view the same video content he is downloading over the web and playing on its PC.
Nevertheless, it remained two drawbacks that prevent to deliver the full promise: The processing power necessary to decode some video files available on the internet may overcome the MID processing power. Decoding such heavily compressed content which are usually encoded with relatively high definition requires a lot of performance. The second is that decoding a video on the processor drains significantly the battery. In the case of a MID, it is essential that the user can view at least a full movie relying on its battery while he is in situation of mobility. That means autonomy of about several hours of video playback.
To solve the challenges mentioned above, Intel has integrated a video hardware decoder in the Intel GMA 500 MID chipset. On one hand, with this solution, the user can view up to two movies with one charged battery. The electrical power needed to decode a video stream is significantly decreased. On the other hand, the decoder can handle HD streams decode in the most common formats available: MPEG-2, MPEG-4 layer 2, H.264 and VC-1. To use the HW video decoder, the applications must send the video stream to the hardware decoder. With Windows, the hardware video decoder of the Intel GMA 500 is accessible through the DXVA API. With Linux, this is done in using a new public API called the VA API. The Intel GMA 500 chipset enables the Intel processor-based MID to offer the best in class video experience a user can expect on such a small device.
MID Hardware Video Decode
Windows and Linux drivers for the Intel GMA 500 are available to allow applications to access the hardware video decode feature. In this paper we will focus on how to enable this capability on Linux using the VA API.
Video decoding can be represented as a pipeline. It has some pure decoding stages like Variable Length Decode, Inverse Quantization, Inverse Discrete Cosine Transform, Motion Compensation, De-blocking (if applicable) and some post processing stages like color space conversion or scaling. The compressed data stream needs to go through all the stages to be properly decoded. The pipeline can be fed at any stage with the appropriate data. This is what we call an entry point.
The VA API is a public software API specification which provides access to graphics hardware acceleration for video processing. It is meant to enable hardware accelerated video decode at various entry points (VLD, IDCT, Motion Compensation, etc…) for the prevailing coding standards today (MPEG-2, MPEG-4/ASP/H.263, MPEG-4 AVC/H.264, and VC-1/WMV9), as well as hardware accelerated video post-processing such as de-interlacing, color adjustments, color space conversion and output scaling. The VA API provides much more functionality than the existing XvMC API. XvMC was designed to support MPEG-2 motion compensation only.
As this is a public API, it could be used in many graphical chipsets. Some may implement it partly, only giving access to a subset of entry points. In the case of the Intel GMA500 it supports VLD level offloads for all formats which includes VLD, iDCT, Motion Compensation and in-loop de-blocking (if applicable).
MID Video Acceleration Driver Architecture
The VA API provides an interface between a video decode application (client) and the hardware decode accelerator (server), to off-load video decode operations from the CPU. The basic operation paradigm is, for the client, to send parameters and compressed data buffers to the server using the API, while the server decompresses and post processes the bit stream it receives from the client.
From the client perspective, the server acts as a virtual decode pipeline dedicated to the application. Several applications can access the driver concurrently resulting in multiple video decode offloaded to the chipset. In this situation, each application has only access to its own virtual decode pipeline. Moreover, an application can create multiple virtual decode pipelines allowing the application to decode several streams concurrently.
The core API itself is windowing system agnostic, and could be implemented with a variety of windowing systems potentially. The current Linux video acceleration driver for the Intel GMA 500 is implemented with the X window system. The output rendering function (vaPutSurface) is to be used for X11 Drawables. It is particularly well suited for standard 2D video players which need to display video in a screen-aligned pixel rectangle.
With the first driver architecture based on DRI, it is not possible to map efficiently the video decoded in hardware, in a 3D environment. With the future evolution of the driver to DRI2, if the target drawable is an X pixmap, then the video output could also be used as a texture in the OpenGL pipeline with the ext_texture_from_pixmap extension in order to achieve more sophisticated visual effects.