An Overview for Developing End-to-End

http://www.informationdisplay.org/issues/2009/07/art6/art6.pdf

 

Enthusiastic consumer response to recent 3-D theatrical releases is demonstrating users’ appetite for 3-D content, and 3-D seems destined to become the next killer application for thehome-entertainment industry. Until then, the standardization of 3-D formats for mastering and distribution is critically important for a successful introduction to the consumer market.

 

by William Zou

 

TODAY, the most noticeable development in the display industry has been the focus on 3-D. Of course, the use of 3-D for printing, the movies, and even TV is not new. It has been envisioned, promoted, and implemented for more than 100 years. There is no question why 3-D is a natural extension to the 2-D presentations we already enjoy today, and it provides a more interesting and attentiongetting medium for viewers. But interest has faded every time 3-D has been promoted, and 3-D movies or TV have never quite moved beyond the gimmick stage – that is until recently. In 2005, particularly after Disney’s release of 3-D versions of Chicken Little in movie theaters, the industry rediscovered the huge business potential of 3-D.

 

3-D theatrical releases are now generating more revenue in movie theaters than 2-D releases. For the recent theatrical release of DreamWorks Animation’s Monsters vs. Aliens, nearly 60% of the revenue in the first week was generated from 3-D cinemas – which comprised only 28% of the total theaters screening the film. Because tickets to 3-D versions of films cost more than their 2-D counterparts, people are clearly willing to pay more for 3-D. The latest generation of digitalcinema projectors and 3-D glasses has enabled the success of 3-D at the box office. With the adoption of the Blu-ray HDTV format for next-generation home entertainment and large-display TV sets in the home, movie studios and other content providers see the potential for huge revenues from 3-D content consumption in the home as well as in theaters.

 

Meanwhile, consumer-electronics manufacturers see 3-D as a significant differentiator that could encourage consumers to continue investing beyond 1080p (full high definition). The advancement of 3-D display technology, including the availability of large screen sizes, high resolutions, and high frame rates (120 Hz and higher) will help enable high-quality 3-D content presentation in the home. 3-D is now being seen as the killer application that will not only bring people back to movie theaters, but also enable a whole range of new services and business to be offered to the home consumer.

 

3-D End-to-End Systems

This article provides an overview of 3-D endto end systems, as well as the issues and challenges involved in developing solutions. It will focus on industry efforts toward 3-D standards development and offer a glimpse into the substantial work remaining in those standardization efforts. Figure 1 shows the process flow for delivering 3-D to end users, be it to movie theaters (top of diagram from left to right) or to mobile phones and home TVs (lower part of diagram from left to right). The issues involved in getting 3-D into the home, including content creation, mastering, distribution, and the displays themselves, will be discussed in the following.

 

 

Content Creation. There are three main approaches to creating 3-D content:

1.     Live camera capture: capturing stereo paired images simultaneously. Live events require the use of stereo cameras. In addition to the conventional stereo camera rigs, a new class of 3-D cameras has been developed for capturing 2-D and depth mapping in real time using rangefinders. Rangefinders are usually An Overview for Developing End-to-End Standards for 3-D TV in the Home laser or infrared camera devices that are used to provide depth maps for a given scene. Depth maps can also be calculated on a pixel-by-pixel basis in real time or offline.

 

2.     Computer-generated images (CGIs): creating a stereo pair of views by rendering two views in parallel with a small angular separation in the two views. Computer-generated content is typically considered the easiest method of stereo generation. The rendering system can render one or more related views depending on the application.

 

3.     2-D–to–3-D conversion: processing of existing 2-D imagery and extraction of depth information from non-stereo depth cues to create a polygon-based image. This process deconstructs the 2-D image into a series of objects (also known as segmentation), assigning relative depth to each object, then filling in occluded areas. The conversion can be either real time or non-real time. 2-D converted material may prove to be essential for a successful transition from 2-D to 3-D.

 

Mastering.

Mastering involves formatting and packaging original 3-D content in a standardized format for post-production and down-stream distribution. It is ideal that a single master is used for all distribution channels. Generally, the master package will undergo additional processing (compression, storage, and physical transport) before being delivered into the distribution system. The master format is ideally in an uncompressed domain so that the highest quality can be maintained for downstream process and distribution.

 

 

Fig 1: A conceptual model for delivery of 3-D content illustrates the role of the 3-D Home Master (lower left in red-outlined box) with regard to

standardizing content and delivery for end-user devices (at right). Source: “Report of SMPTE Task Force on 3-D to the Home,”

 

 

Distribution/broadcast/emission:

For 3-D distribution/broadcast/emission, the master (either from live camera or play-out systems)  is taken into the distribution systems for delivery to the home. Because of bandwidth constraints, the signal is typically compressed to reduce the data rate to meet specific channel requirements. A stereoscopic image pair consists of a left-eye image and the corresponding right-eye image. Therefore, the support of stereoscopic 3-D content for distribution to homes requires two times more bandwidth than 2-D content. In order to reduce the bandwidth requirement while preserving visual quality as much as possible, and also supporting 2-D device compatibility, various 3-D formats have been developed:

 

1.     Spatial compression. In order for stereoscopic content to fit into existing transmission/ storage infrastructures, one simple approach is to use spatial compression. This can be done by sub-sampling left- and right-eye images and then compressing them into a single 2-D image frame. The sub-sampled left and right images can be packed in a top/bottom, side-by-side, line/column interleaved or checkerboard fashion. All these formats achieve the goal of using no more than 2-D bandwidth. However, the downside is loss of spatial resolution and incompatibility with conventional 2-D displays. Embedding and transmitting additional side information with the formats and using a post-process such as pixel interpolation/ smoothing could mitigate the problem of loss of spatial resolution.

2.     Time multiplexing. By multiplexing  left- and right-eye images in the time domain, 3-D content becomes 2-D, but the frame rate is increased 100%. This approach could be implemented using some existing standards and interfaces, and possibly be supported by some consumer devices already in the home. The advantage of time multiplexing is that full spatial resolution for each eye can be provided. For some distribution platforms such as packaged media, the high bandwidth/bit rate of using time multiplexing might not be an issue, while for others the 100% increase of bandwidth/ bit rate is prohibitive.

3.     2-D + depth map/delta/metadata. The key benefit of this approach is the support of 2-D device compatibility – display, set-top box (STB), etc. The format consists of 2-D images plus associated depth maps for each 2-D image. A conceptually similar approach using a single 2-D video stream with added metadata can be used for stereo image pairs. Instead of encoding two separate 2-D image streams, conventional video data can be used to represent, for example, the left half of the stereo pair, while metadata in the form of a difference map or delta map is used to encode the right half as a function of the left. The 2-D + depth map/delta/metadata can be encoded with standards-based compression techniques (e.g., MPEG-2, AVC) for distribution. At the receiver, the 2-D portion can be decoded by conventional 2-D STB/TV while the data of depth map/delta or 3-D metadata is ignored. New 3-D devices (e.g., STB, 3-D TV set) can decode the complete 3-D stream and render it on a 3-D display. • Color coding. Using color-coded images for left-/right-eye presentation (such as with the anaglyph method) for 3-D TV has been with us for decades. It is the only format that can be used for 3-D presentation with a pair of color-coded glasses on any 2-D display. Certainly, color coding produces poor quality and does not provide a compelling 3-D viewing experience.

 

3-D Displays in the Home

There are many different types of 3-D display technology and each has a different set of advantages and disadvantages. Currently, there are three types of 3-D displays commercially available on the market.

 

• DLP-based rear-projection TV using the checkerboard format.

• Plasma TV using checkerboard or line/column interleaving.

• LCD TV using page-flip (120-Hz frame rate with time multiplexing) or a linealternative micro-polarized screen. The companion article to this piece, “3-D Displays in the Home” by Andrew Woods, describes these devices in more detail, but they all require wearing glasses (either passive or active) for 3-D viewing. Autostereoscopic 3-D displays do not require viewers to wear glasses and can also provide multiple viewing spots with good spatial resolution. Although glassless 3-D viewing is considered the future, large screen multi-view autostereoscopic displays will not be commercially available in the short term. All of the 3-D display technologies mentioned above will be co-existent and consumers will have to deal with the choices of quality, price point, and the question of whether they will accept the wearing of glasses for long-term 3-D viewing. The adoption of well-researched standards can help minimize consumer confusion and the need to educate the end-user by facilitating the most viable options.

 

Issues and Challenges for Delivering 3-D Content to the Home

The key issues and challenges for delivering  3-D content to the home include a lack of industry standards for content mastering and distribution, high cost of 3-D content production and distribution infrastructure, and multiple incompatible display technologies. Selection of the right 3-D formats for specific distribution channels heavily depends on:

1.     3-D quality and extensibility (future quality improvement).

2.     Efficiency of implementation (storage, transport, computation, rendering).

3.     2-D compatibility (distribution requirement and 2-D display).

4.     Compatibility with the existing consumer devices.

5.     Production and implementation cost.

6.     Technology maturity and time to market. The following is a list of possible distribution channels, including their pluses and minuses:

 

Terrestrial broadcast. This is the most bandwidth-constrained distribution platform. Each TV station has only a 6 MHz (U.S. and Japan) or 8 MHz (European countries) channel bandwidth. It is also the most regulated and standards-based distribution. With limited RF bandwidth per station, constraint by legacy-transmission standards, and the current state of consumer TV receivers in the home, it is highly desirable to transmit 2-D backwardcompatible formats that could be received by both 3-D receivers and legacy 2-D receivers. This approach would require compressed 3-D bit streams for emission in the format of 2-D + metadata (e.g., depth-map, delta, etc.) so that a 2-D legacy receiver could decode the 2-D portion and ignore the 3-D side information, while a 3-D receiver could display the 3-D content. An alternative would be to simulcast 2-D and 3-D streams (not necessarily with identical content), so that each uses a

 

3-D TV standards fraction of the channel. It will be important to assess the likely transmission bit-rate requirements for terrestrial broadcast and to understand what factors affect the quality of the 3-D experience as perceived by viewers. The relevant international standards bodies addressing terrestrial broadcast standards are ATSC (Advanced Television Systems Committee), DVB (Digital Video Broadcasting), and ITU-R (International Telecommunication Union Radiocommunications Sector).

 

Cable. Cable is relatively less constrained with regard to bandwidth than terrestrial broadcast. 3-D services could be treated just like today’s Video on Demand (VoD) model or with dedicated 3-D channels for those with a special STB that supports 3-D decoding and presentation. This business model provides a flexible requirement for cable to go to either a 3-D-only or 2-D-compatible format solution. Cable could be one of the distribution platforms that starts offering 3-D early when sufficient 3-D-ready sets are available in the home. Like the broadcast industry, the cable industry follows standards-based implementation. Two standards development organizations playing key roles in defining standards are the SCTE (Society of Telecommunications Engineers) and DVB.

 

DTH/Satellite to the home. Unlike cable or terrestrial broadcast, DTH (direct to home) is basically a closed system – individual operators such as DirecTV, Dish Network, and BSkyB can do whatever they want. As is the case with cable, bandwidth constraint is less of a problem, although the rollout of HD content, especially the addition of large numbers of local HD channels in the US, consumes large chunks of transponder bandwidth. DTH could offer 3-D as a dedicated channel, either using 3-D-only or 2-D-compatible formats.

 

Packaged media. It is expected that packaged media such as Blu-ray discs will be the main platform for 3-D home entertainment, as driven by Hollywood studios and their desire to capture the home-entertainment segment by leveraging their growing 3-D theatrical content library and success in the 3-D box office. The Blu-ray disc format has large disc storage space (25/50 GB) and supports advanced video and audio CODECs. The format also supports dual-stream decoding (picture-inpicture and secondary audio features). The video decoding can support up to a 40-Mbps data rate and the 1080p format.

 

IPTV/Internet/Mobile/Cellphone. IPTV (Internet protocol TV) is much like cable service, although some varieties are based on point-to-point switching networks, while others are RF-overlay on fiber or FTTH (fiber to the home). These networks can adopt VoD-like 3-D services or transmit dedicated streams for 3-D content to the home. Mobile TV and cell phones can also provide 3-D content using single-view autostereoscopic screens. There have been announced market trials for 3-D-capable cell phones and services. Internet content download and streaming to PC and TV are now popular and will play a key role in future content distribution and consumption. SMPTE and some industry forums have already started developing standards for broadband distribution, and their 3-D support is expected.

 

What Standards Are Needed? Standards are playing important roles in developing an end-to-end value chain for 3-D to the home. The key questions to be answered are what standards are needed and when to standardize 3-D TV. The benefits of standardization include interoperability, low-cost consumer devices, and consumer confidence. Ideally, standards should be implemented before any proprietary system becomes the de facto standard on the market. The right timing to standardize a technology depends on factors such as technology maturity as well as future-proofing for extensibility. Another issue is single/worldwide standard vs. multiple/regional standards to meet specific requirements. A single worldwide standard is best, and this is certainly possible, especially for 3-D content mastering. The requirements and business models for the distribution platforms are very different, and therefore a onesize- fits-all distribution format might not be feasible. The fragmentation of distribution standards/formats might split and confuse the market and the consumer, and a worldwide standard would go far toward alleviating this problem. The entire value chain can be segmented into four categories as shown in Fig. 2. With the exception of content creation, standards are needed in each of the following value chain areas:

 

Content creation. There is no need to standardize content creation. The Society of Motion Picture and Television Engineers (SMPTE) uses the term 3-D source master to refer to the image format or file package where content originates.

 

3-D home mastering. It is important to standardize a single worldwide 3-D home master format for all distribution channels. (The SMPTE defines “3-D Home Master” as an uncompressed, unencrypted image format or file package derived from a 3-D source master.) The 3-D Home Master is intended to be used in the creation of 3-D distribution data. However, any future standards creation should adequately study whether it is feasible to create a single 3-D home master vs. multiple masters (each for a different distribution channel or set of channels).

 

Distribution. In this case, distribution is considered the last mile to the home. Distribution includes physical transmission channels/ platforms such as terrestrial broadcast using RF frequency, cable HFC networks, satellite to home (DTH), packaged media, and the Internet. Consumer devices. There are many incompatible 3-D display technologies on the market and there is no clear trend indicating that a single display format is emerging in the near future. Furthermore, it is assumed that there might be multiple distribution formats to meet specific uses for distribution. Therefore, it is critical to make sure that any consumer playback device (e.g., Blu-ray player, STB, game console, etc.) can communicate with any 3-D display device in the home and that 3-D content from any playback device is displayed properly. This requires developing industry-wide interface standards and signaling specifications.

 

 

 Fig. 2: The value chain is a simplified version of the flow diagram appearing in Fig. 1.

 

 Content Creation 3D Home Mastering Distribution Consumer Package Media Broadcast Device Worldwide 3-D Standards Development

 

A number of entities are actively engaged in the development of standards for different portions of the 3-D pipeline. These organizations include the CEA (Consumer Electronics Association), ATSC, DVB, ITU, SCTE, BDA (Blu-ray Disc Association), DVD Forum, ISO/IEC/MPEG, and SMPTE. As the chair of the SMPTE Task Force on 3-D to the Home, the author will focus on SMPTE’s effort, while providing a brief survey of some of the activities of other organizations.

 

Society of Motion Picture and Television Engineers (SMPTE). In August 2008, the SMPTE established a task force to define the parameters of a stereoscopic 3-D mastering standard for content viewed in the home. The project, called 3-D to the Home, was the first step in propelling the 3-D home-entertainment industry forward by setting the stage for a standard that will enable 3-D feature films and other programming to be played on fixed devices in the home, regardless of the delivery channel. With cross-industry participation and contribution, the task force defined the concept of a 3-D home master, identified use cases from the perspective of various entities in the supply chain for 3-D content to the home, and developed requirements for the 3-D home master. In March 2009, the task force completed its assignments and published its report, which will form the basis for developing actual format standards within SMPTE. This report has been distributed to other SDOs (standards development organizations) in support of their development of related distribution and presentation standards. The task force defined a 3-D home master as follows: “Uncompressed and unencrypted image format or file package derived from a 3-D Source Master. The 3-D Home Master is intended to be used in the creation of 3-D Distribution Data.” Based on the use cases developed by the task force, the minimum requirements for the specification for a single 3-D home master were defined. The key parameters include the following nine categories: image content, audio synchronization, graphical overlays, subtitles, closed captions, backward compatibility with 2-D and 3-D metadata, ancillary metadata, and evaluation criteria. In addition to the use cases and requirements described in the report, the task force recognized that there are various unknowns, which may (in the future) impact the format of the master package. These include evolving display technologies, future distribution channels, production and authoring techniques and needs, and unexplored psychophysical characteristics of the human visual system. A group within the SMPTE standards committees started working on defining specifications based on the requirements developed by the task force in June 2009, with core standards expected to conclude within a year. As in the development of the 3-D home master requirements, the SMPTE committee will work closely with other SDOs that will develop companion standards for complete end-to-end interoperability. A brief listing of other standards groups follows. For more information, please see the SMPTE Web site at www.smpte.org.

 

Consumer Electronics Association (CEA). CEA’s Video Systems Committee has launched a new standards activity aimed at establishing standards for 3-D video. It observed that many broadcasters, DVD distributors, and CE manufacturers are moving forward with 3-D, and concluded that establishing a standard for transporting 3-D video over an uncompressed high-speed digital interface was very important.

 

Advanced Television Systems Committee (ATSC). ATSC is an international organization that is developing standards for digital television and is responsible for developing DTV standards for digital terrestrial broadcast.

 

Digital Video Broadcasting (DVB) project. DVB is at this stage looking for information on what technologies and capabilities are available. For example, the project committee is examining backward compatibility and looking for commercial requirements for work items that sit within the scope of the DVB expertise.

 

International Telecommunication Union (ITU). The subject of 3-D TV is currently being considered by Working Party 6C (program production and quality control), as part of the work of ITU-R Study Group 6 (Broadcasting Service). In October 2008, the ITU-R approved a new study question on digital 3-D television broadcasting. This is essentially a call for proposals for 3-D TV. The question calls for contributions on systems that include, but also go beyond, stereoscopy.

 

Society of Cable Telecommunications Engineers (SCTE). In early 2009, SCTE established a standards project called 3-D Over Cable. This effort will investigate the distribution of 3-D video content over cable networks. One result will be the identification of any changes that are necessary or desirable in existing SCTE standards to facilitate the provision of 3-D content by cable operators.

 

Blu-ray Disc Association (BDA). BDA is a voluntary membership group creating, upholding, and promoting the BD formats and developing Blu-ray Disc specifications. Due to the large disc storage capacity, 1080p picture quality, and 7.1 audio, Blu-ray Disc is the ideal platform for bringing 3-D technology to mainstream home entertainment. The BD format is backed by the Hollywood studios and major CE manufacturers and will be a distribution platform delivering the highestquality 3-D content to the home in a relatively short time frame. In May 2009, the BDA formed a 3-D task force to add advanced 3-D technology into the Blu-ray format.

 

DVD Forum. The DVD Forum is an international association of hardware manufacturers, software firms, content providers, and other users of Digital Versatile Discs. It issued an RFI (request for information) on 3-D technology in 2008 and received several proposals with various encoding formats. The DVD Forum then conducted backwards compatibility tests on the legacy DVD players using these proposed formats. In early 2009, SENSIO’s 3-D spatial compression format was accepted as an optional DVD standard.

 

ISO/IEC/MPEG. In the past, MPEG developed standards related to 3-D video. ISO/IEC 23002-3 (also referred to as MPEGC Part 3) specifies the representation of auxiliary video and supplemental information. This specification supports the well-known 2-D + depth format, including metadata to adjust the rendering of a 3-D scene based on viewing characteristics such as display size and viewing distance. The 5th Edition of the ITU-T Rec. H.264 | ISO/IEC 14996-10 Advanced Video Coding (AVC) standard, which will be published soon, includes extensions for multiview video coding (MVC). The Multiview High Profile is specified in this edition of the standard, which utilizes existing high-profile tools (excluding interlaced) for inter-view prediction to achieve improved compression of stereo and multiview video. Signaling of scene acquisition information (i.e., camera parameters) and multiview scene information (i.e., maximum disparity) is also specified. Such metadata may be used for 3-D display

 

3-D TV standards processing. MPEG is also in the process of developing an amendment of AVC that standardizes additional capabilities in the context of 3-D. Conclusion Recent 3-D theatrical releases have demonstrated significant consumer desire for and acceptance of 3-D content. 3-D is poised to become the next killer application for the home-entertainment industry. Standardization of 3-D formats for mastering and distribution is critically important for a successful 3-D introduction to the consumer. The SMPTE 3-D Home Master is being developed as the cornerstone of the entire content chain. It will provide high-level image-formatting requirements for the source materials authored and delivered by content developers. It will additionally provide requirements for the delivery of those materials to all distribution channels; from physical media to terrestrial, satellite, cable, and other streaming service providers. Beginning in 2009 and continuing down the road, the industry needs to work hard to build a complete value chain for delivering 3-D content to the home. With the completion of the above specifications, 3-D home entertainment will become a reality

 

William Zou is the chairman of the SMPTETask Force on 3-D to the Home. He isemployed by DTS, Inc. in Agoura Hills, CA.The full report on 3-D to the Home fromThe Society of Motion Picture and Television

Engineers (SMPTE) comprises 76 pages,including a 3-D glossary, use cases, minimumrequirements, recommendations, appendices,and references. It is recommended for anyone seeking a comprehensive introduction to manyaspects of 3-D systems and the issues that need to be considered in planning for 3-D to the home and is available for download from the online SMPTE Store (https://store.smpte.

org) for $20.