Stereo3D broadcasting: where has it got to and where is it going?

http://www.quantel.com/repository/files/whitepaper_Stereo3DbroadcastingJan09.pdf

 

January 2009

 

Two years ago, if someone had said to you ‘Stereo3D TV will be the next big thing’ would you have taken them seriously?  

 

In the next few years, assuming the world economy doesn’t turn up more unpleasant surprises, we may just

be at the very beginning of the introduction of Stereo3D Broadcasting.      

 

Hollywood is already embracing Stereo3D film. Now, around the world, Broadcasters too are starting to

seriously investigate – even plan for – Stereo3D.

 

Totally new technology introduction always generates lots of questions.  

 

This paper or blog tries to help provide some accurate answers to six common questions about what

services could we expect, how could we get them and when might they happen.  

 

However in the end this is a personal view, based on two years of experience of tests, discussions,

presentations and actual product development. We hope it helps shed some light on the current discussion

and feel free to drop us a line if you agree or disagree or just want to talk.

 

Mark.horton@quantel.com

Trevor.francis@quantel.com

 

 

What is the probability that Stereo3D will happen?

 

Very high. Here are some pertinent ‘lessons from history’ from the

introduction of sound and colour in film:

 

‘Talking pictures will never replace the silent drama’

Joseph Shenck, United Artists, C.1928

 

‘I consider the so-callled “all-talkie”….nothing but rotten trash’

Sergei Eisenstein, 1929  

 

‘The reaction to talking pictures is somewhat problematical. Talking throughout the entire picture has a tendency to retard the action’

Pat Powers, Cinephone, 1929

 

“The process of Color motion picture photography [has] never been perfected…it would tire and distract the eye, take attention from faces and acting and facial expression, blur and confuse the action….”  

Douglas Fairbanks, 1930

 

And then a few years later:

 

‘Whether color can make black and white pictures as obsolete as sound made silent pictures, is, as suggested, quite another question. The silent picture was slain overnight by the jawbone of Al Jolson, whose Jazz Singer threw a hitherto sceptical industry bodily into speaking likenesses. But color is not so pronounced a revolution as sound’

Fortune Magazine, 1934

 

By early 1929 all the important studios in Hollywood had become thoroughly sound conscious. This was a great help to us in introducing color. Prior to that studio executives were loth to permit any change

whatsoever in their established method of photography and production. But with the adoption of sound, many radical changes became necessary. Technicolor was always confronted with objections photographing in color required more light, different costumes, a knowledge of color composition, additional time, and one or the other of these points, plus the added forceful argument that it cost more money, made it difficult for us to get started.  

 

In my opinion, the turning point came when we ourselves produced the series of short subjects. By entering the field as a producer, by keeping very careful records of our time and money schedules, and by openly discussing with studio executives everything that we were doing as we went along, we dissipated most of the prevailing misinformation.

Technicolor Founder H. Kalmus, 1938

 

A personal favourite book is Alvin Toffler’s influential 1970’s ‘Future Shock’ which predicted a near-future where the speed of change is faster than we can absorb. He predicted this would disorient people and generates many negative reactions. Here we are in that future and he was right. Film and TV are technology driven businesses, yet as in many other industries, many people dislike change.  

 

A very recent case is Digital Intermediate – the process of digital colour correction of movies. Digital

Intermediate took about two and a half more years than actually needed to gain full acceptance, mainly

because of some highly emotional resistance, although the benefits were quite literally there for anyone to see. Now, almost every movie goes through the DI process.  

 

Of course, many much-hyped film innovations prove nothing but fads and there’s nothing wrong with some healthy scepticism. However, as individuals and as an industry, we have a responsibility to rationally think about new technology. The world isn’t going to wait around for us.  

  

Objections fade if and when it’s clear a new technology delivers real benefits – either to the content maker or the audience. The most compelling argument is of course money. If a new technology can either save money or make more money that tends to get attention. Show business is a business.    

 

Digital Stereo3D film making is already being enthusiastically embraced by Hollywood because of current evidence that the numbers work.  

 

Assuming you have a film that already appeals to an audience, Stereo3D presentations get more cash through the door. Last year, Hannah Montana smashed box office records and this year dozens of titles are in production.   

 

There are some who think Stereo3D is just a passing fad. Some say that Stereo3D already failed once in the 1950’s and that what counts is making good content. However, similar arguments were used against sound and colour in film. When film makers of the 30’s made good films that used sound and colour that audiences would have gone to see anyway then sound and colour thrived. What sound and colour did was enhance the audience experience.  

 

Now good movies - that happen to use Stereo3D - are getting good box office. As long as we make good

films there is no reason for Stereo3D films to be a short lived fad. In the analogue 1950’s we simply did not

have the technology to make consistently good stereo films in a reasonable time and at a reasonable cost

and there were many issues with projection. Now with digital capture, digital post and digital projection, the

situation is wholly different.  

 

It follows that the key argument in favour of Stereo3D Broadcasting is the overwhelmingly positive reaction

the general public has to it. As an industry we do well to always consider the general public. They are

everyone’s customers. We aren’t here just to sell to each other.  

 

What exactly do we mean by Stereo3D broadcasting?

 

One definition of Stereo3D broadcasting might be:

 

‘The transmission of a high quality left eye and right eye signal to the viewer of a Stereoscopic ready

domestic Television with horizontal parallax information extracted by viewer glasses that helps convey depth’.  

 

This intentionally narrow definition excludes:

 

•        Anaglyph (which is a single signal with colour used to convey depth. Although simple to do, poor

•        colour quality makes this an unattractive option)  

•        Narrowcasting to D-Cinema theatres (although this is an interesting new area)

 

Just now we would exclude multi-view stereo that does not need glasses. This is a bit complex to broadcast

today, is lacking in resolution and currently has limited ‘sweet spots’. So, right now this is not quite ready for

prime time, especially as some proposed camera systems are being studied that use five rather than two

sensors. However, things are changing very fast and we should keep an open mind on systems that do not

require glasses, as some very smart people are working on this now and this may become very significant in

the future.  

Let’s break the production process down into sections – shooting, galleries and trucks, post production suites

and transmission.

 

Shooting  

 

A pair of matched cameras, typically spaced at roughly adult eye distance, are used to capture the image.  

 

The horizontal offset produces a binocular disparity. That binocular disparity, together with other information

in a scene, including the relative size and height of objects, occlusion, sharpness and detail, linear

perspective, object density, differential brightness, shadows, haze, and relative motion is used by the mind to

create depth perception.

 

No special cameras are needed, however since the only difference between the eyes should be horizontal

disparity (and not for example color, geometry or focus differences) care is needed to match and

geometrically align the cameras.  

 

Special rigs are used which range from inexpensive and simple (which can produce good stereo but only if

carefully set up and skilfully used) to highly sophisticated rigs, like the 3ality Digital systems shown here, with

advanced mechanics and electronics which need minimal set up times and are easy to use.  

 3ality Digital rigs have been in action recently in many Broadcast tests including NFL and Sky

 

Shooting for Stereo3D is a different aesthetic than conventional TV. The audience is far more involved in the

image and wobbly camera work or whip pans are usually avoided, in favour of a more immersive and

smoothly paced style that draws the audience in.

 

One important topic to understand is ‘where to place the action’. Up to now much stereo has been shot for

large screens, not for domestic television sets. One artistic judgement is whether to place the objects of

interest in a shot on, behind or in front of the TV screen by ‘converging’ – which means ‘toeing’ the cameras

inwards or outwards in a similar way to how we move our eyes when we look at objects close up or further

away. 

 Positive, Negative and Zero Parallax

 

Placing objects behind the screen (positive parallax) gives a ‘window’ effect i.e. the viewer is looking through

a frame to a scene behind1 Placing objects in front of the screen (negative parallax) gives the appearance of

action happening in the viewer’s room.  

 

An ‘in your face’ style is great for certain kinds of action but care is needed not to push objects so far that the

viewer can’t ‘fuse’ them and they get an uncomfortable double image.  

 

Also care is needed framing shots where the action comes out of screen, if the objects are cut off by the

screen edges. This gives a so called ‘conflict of cues’ – the image is in front of the screen yet is also

occluded by the screen frame so must therefore be behind the screen.  

 

This is much more of an issue in Television than for example in IMAX where the edges of the screen are

much further out from the point of view of the audience. Converging on an object (i.e. there is no horizontal

difference between the left and right images) places an object on the screen (zero parallax).

 

So, there are new considerations, as there were when color was introduced, but unlike the changeover to

colour, the cameras are the same, you just need more.  

 

Production galleries and trucks

 

In some recent tests, existing production switchers (vision mixers if you are from the UK) and existing DVEs

were used. Production switchers can tie crosspoints allowing two cameras to be cut as if they were one and

DVEs can be used to horizontally slide images, simulating convergence control. Graphics inserts can be

made with a twin key and fill, introducing a horizontal offset between the signals. We don’t quite have all the production manufacturers committed to Stereo3D yet but this is changing fast and

expect plenty of news at NAB 09.

 

One point to mention in passing is that some stereo tests use a mux/demux workflow which combines left

and right eye into a single stream, allowing it to be passed down a single cable (or transmitted as a single

datastream). While this reduces the resolution of the image, one curious property of stereo is that the image

still appears sharp. The way we perceive stereo is still not fully agreed upon but we do know that the brain

creates stereo rather than passively capturing it and, if supplied with two lower resolution signals, many

viewers ‘see’ a high quality result.      

 

There are a number of tests in progress, some using different methods to the ones mentioned but the

common conclusion so far is positive – it can be done.

 

Post production suites  

 

The ‘fourth dimension’ of stereo is time. As well as removing any stereo or non stereo image errors a critical

role for post production is to create Stereo3D that is comfortable to watch over extended periods of time.

That means handling the z-space information on a shot so that it is technically and artistically correct and

also handling z-space over a sequence of shots, so that the eyes can comfortably adjust.  

 

Shooting fast moving Stereo3D content is quite different from shooting stills (and can be different from

feature film work where there may be more opportunity to set up takes). However, with a little experience any

competent artist can learn how to post produce stereoscopically and there are new measurement and

correcting tools coming on the market to make life easier.  

 

Audio post is an interesting area as 5.1 can be used very effectively to enhance depth.

 

Today, given the right tools, post production of Stereo3D content is becoming simpler, quicker and cheaper

than ever before.

 

Stereo images can now be accurately measured and corrected for example using the new 3ality Digital

SIP2100. 

Transmission Schemes

 

Transmitting two full quality Stereo3D HD signals as independent streams is impractical as it uses up

significant bandwidth and risks the two signals picking up unwanted differential artefacts or getting out of

sync. There are several schemes that aim to remove this including Side By Side, Checker Board and

Stereo3D specific compression.  

In Side by Side, a single signal is created that ‘squashes’ the left and right eye into a single picture. This is

then expanded out by the viewing device.  

‘Side by side’ horizontal squeezing of the original images to one picture. Pictures are then unsqueezed in

final playout.

 

Because we makes stereo in our brains by some very complex and not fully understood methods, despite

the reduction in resolution, pictures still look very good. If done properly, the actual quality is higher than

many of the excessively compressed HD services being broadcast today.

 

A single cable can carry the signal around a Broadcast facility prior to transmission and because the signal

travels as a single stream, there is no possibility of two signals getting out of sync. Sensio and Hyundai

currently use this method.  

 

In ‘checkerboard’, each eye is placed in a single frame rather like a chessboard with white spaces as one

eye and black as the other.  

Texas Instruments Checker boarding scheme Again there is no possibility of two signals getting out of sync. A number of TV vendors already use this.

 

Specific Stereo3D compression can use a variety of schemes that take advantage of the similarity between

the eyes to intelligently send only needed data.  

 

Monocular picture streams have much redundant image content between a group of frames which can be

reduced through use of long GOP compression. Stereo3D can take advantage of this too but also remove

redundant data between left and right views.  

 

Compression can be used to reduce payload by removing redundancy between eyes over time. This

example is ‘Worldline’      

  

Using two channels of non Stereo3D compression is very risky to use for Stereo3D because of the possibility

of concatenation or compression artefacts introducing differences between the eyes.  

 

However good Stereo3D compression schemes are likely to find favour and companies like TDV already

have working systems on the market.

Will Stereo3D Broadcast really work?

 

Yes. However, like sound and colour, not everyone is a believer. Here are some current objections:  

 

The EBU's head of emerging media David Wood said at IBC: "There is a degree of scepticism that we have

enough technology breakthroughs to make 3DTV work at the moment………. the technology is seriously

flawed in two main areas. The furthest point on the screen needs to be fixed at 6.5cm - the same as the

distance between human eyes. This can be achieved by projection in a cinema but there's no way of

knowing what size a viewer's display will be. Secondly there is a conflict between the focus and the

convergence of our eyes when viewing 3D. These psycho-physical limitations can never be solved with a

stereoscopic system….The real long-term future of 3DTV - and we are talking 50 years - is Object Wave

recording, a subset of which is the hologram"

TVB Europe, October 2008

 

The first point being raised here is that stereoscopic images are created in the mind by presenting two

images of a scene with a horizontal parallax difference that provides binocular disparity. That disparity makes

the viewer perceive depth. The amount of horizontal parallax is important for camera operators (or CGI

artists). Too little or too much and the viewer will not get the view that the Director intended (and in the worst

case too much can lead to headaches and fatigue). It is correct that we need to understand and manage that

depth relationship.  

 

Just as we need to manage the amount of colour or sound in a picture (and we don’t know how loud or what

colour settings viewers are using at home) we need to manage the amount of horizontal parallax. There are

two aspects to this – training and technology. If people know that a project will be shown on TV it is possible

to take that into account during shooting and post to set up an average that will work on any screen. There is

now also technology on the market to help people get it right, or if necessary automatically correct the image

(just as in colour or sound we have metering but also legalisers). The technology can be used during

shooting, post production or delivery.    

 

The second point being raised is that we normally focus and converge on the same point when we look

around but when we look at a Stereo3D TV screen, we focus on the screen plane but converge at a different

point (the implication being that this is unnatural and uncomfortable). However, beyond a certain distance

from a screen the subjective effect is negligible.    

 

Stereoscopic expert Lenny Lipton deals with the topic of accommodation and convergence on his blog

http://lennylipton.wordpress.com/2008/06/11/reality-check/. Lenny wrote ‘Foundations of the Stereoscopic

Cinema’ one of the definitive reference texts on Stereo3D.  

 

However, here is a different kind of objection.

 

James McQuivey, analyst at Forrester Research, said: "It's a very restricted viewing experience. You need to

sit in a certain part of the room, and wear glasses. You can't turn your head and chat to your friend and more

importantly, even if there were 100 3D films - and there are not - is that really enough to justify $1000 extra

on a television. That is a big premium to pay for just a film every week or so."

The Telegraph January 2009

 

While it’s true that there will be areas in peoples living rooms which give a better view that others, those

areas will typically correspond to where they sit anyway. It’s true that you need to wear glasses but as we’ll

discuss later there is no definitive evidence yet about consumer acceptance.  

 

The lack of content argument didn’t stop sound and colour.  

 

It is true that we don’t have much content yet. However, one immediately available source of content will

come from Hollywood as the number of new Stereo3D titles grows every month. In our tests, good Stereo3D

film content made for the big screen usually works well on Stereo3D TVs with just a little adjustment. This is

a ‘sunk cost’ as the movies are being made anyway, so with normal rights payments and some remastering

there is an existing pool of film content. There are also many Stereo3D titles from the last century2

 – we’ve

been looking at how restoration can work and it seems it makes sense for some of the better ones to be

remastered. This would take a bit of time and money but is perfectly feasible.    

 

If a Broadcaster wants to put a film channel on air, there could be enough new or remastered content for a

film channel to start sometime in 2010 and each year after that the available content will rise.  

 

Of course, as anyone following the tests and real content creation taking place at the moment can tell you,

there’s going to be much more than film to watch – during 2007/2008, music concerts, game shows, wildlife

documentaries, tennis, ice hockey, soccer, air races, winter sports, basketball, martial arts and American

football and much more were all shot in Stereo3D.

 

Finally the cost of TV sets argument doesn’t hold up in the long term. Stereoscopic TV sets will come down

in price to a small percentage above a monoscopic set. If the demand and the will are there, any short term

issues will go away.  

 

Hollywood is very interested in Stereo3D Blu-Ray right now and that (along with Stereo3D games) will help

drive TV purchases irrespective of Broadcasting.  

Here are a few of the remastered titles available. http://www.sensio.tv/en/home_theater/3d_dvd/available/default.3d

How are Broadcasters thinking about Stereo3D now?

As a new choice. Childhood is called ‘the formative years’ for a reason. We grew up, like many, watching just

a few channels of analogue TV. TV was something you watched passively and there wasn’t much choice.

There was plenty of bad TV but some was excellent. Some of the advertising was really good too.  

 

But the world moved on and we need to move on too.

 

For kids growing up now, things are vastly different:

 

•        Channel proliferation means massive audience dilution for the advertiser.

•        Time shifting set top boxes allow viewers to fast-forward through commercials or just miss them altogether. 

•        Phones are delivering content.  

•        PC and Mac based home entertainment provided by games, video download sites and Internet  social networking sites are yet more alternatives to sitting down at home to watch conventional TV.    

 

Games consoles also appeal to the young and the not-so-young. Games console manufacturers are starting

to look seriously at Stereo3D themselves. That creates new levels of competition for conventional

broadcasters but also presents an opportunity for Stereo3D Broadcasters. There will be an existing

population of Stereo3D TV sets which have been bought for gaming.  

 

Kids are growing up with lots of screens delivering entertainment and information. Conventional broadcast

TV has no inalienable right to be the delivery system of choice in the future. All that said, after a hard day at

work, or at school, it’s still great as many people still do, to sit down and relax in front of a TV – especially if

you established that habit in your formative years. News and sports have also been shown to hold a good

share of the audience, if done well.  

 

All this is well known – the question is ‘what do Broadcasters want to do about it?’ One answer is to simply

concentrate on reducing costs and maximising efficiency. That is a perfectly valid argument for some

broadcasters to take. However the efficiency argument alone can’t bring in new revenue or win new

audiences. 

 

There are many new potential sources of revenue - that have nothing to do with Stereo3D - that

Broadcasters can adopt to offset the decline in the old 30 second commercial. These are all perfectly valid

choices, however amongst hundreds of channels Stereo3D will be a strong differentiator. Stereo3D will be

one of many choices – but a spectacularly different choice.

 

In fact some Broadcasters considering offering Stereo3D TV are counting on limited competition, at least in

the early days.  

 

What’s the business case for Stereo 3D broadcasting?  

 

Digital Stereo3D Broadcasting is now being seriously studied because it is a potentially lucrative business.

The technology to begin stereo broadcasting exists now and the first pilot channel is already transmitting in

Japan. Show the general public and you get a positive reaction.  

 

A key debate will be around start up costs. There were initial objections to colour TV on the grounds of start

up cost and colour took many years to roll out. Interestingly the technology costs today of moving from

monoscopic broadcast to stereoscopic broadcast are much lower than the inflation adjusted costs in the 50’s,

60’s and 70’s of moving from black and white broadcast to colour.  

 

Compare at a headline level what technology a Broadcaster needs to change:

 

·         Monochrome to Colour  Monoscopic to

·         Stereoscopic

·         Comments

·         Cameras  New cameras  Same cameras    Existing HD cameras used in matched pairs are suitable for stereo  

·         Storage    New VTRS  Same storage  Existing NAS or SANs can be used. Pairs of

·         VTRs can be used.

·         Some new VTRS can record Stereo signals (e.g. Sony).

·         Production galleries 

·         New production switchers

·         Existing production switchers.

·         Existing DVEs

·         Most current production switchers and twin channel

·         DVEs can be repurposed for Stereo

·         Post production  New equipment  Some new, some current  

·         Transmission  New  Current  

  

So, while there are costs (and not all of them are covered here), on a technology level they are nowhere near

what some people may imagine. Can we get people to wear glasses to watch TV?  

 

Yes. This is the objection that is the most difficult to refute technically because it is based on the tricky field of

human behaviour. The evidence so far is ‘yes – most but not all people will be prepared to wear glasses’.

About three quarters of the world’s adult population have some level of eyesight issue that glasses or contact

lenses would improve. In those countries currently most interested in Stereo3D broadcasting, the majority of

the adult population own glasses. Of course there is a potential vanity issue, however no one much minds

wearing sunglasses outdoors if it’s too bright and the need to wear earphones didn’t seem to slow down

sales of the Sony Walkman in the eighties.  

 

Why then do we hear assertions like ‘you’ll never get people to wear glasses to watch TV’. Polarised

stereo3D glasses are close cousins of Polarised sunglasses3

. No one much seems to violently object to

sunglasses – it’s a multi-billion dollar business - and if you already wear glasses, it will be possible to have

prescription Polarised versions made of them for Stereo3D TV viewing.

 

This objection can stem from a number of different factors:

 

•        Negative experience of badly made or badly displayed Stereo3D films.

•        Negative experience of viewing content while wearing anaglyph glasses.

•        Poor stereo acuity.

•        An honest commitment to auto stereoscopic (glasses free systems).

 

Anyone who has watched poor Stereo3D (including all Anaglyph) is likely to be sceptical that audiences can

be persuaded to watch it over extended periods of time. We wouldn’t want to either.

 

A percentage of the population have impaired stereo acuity (figures of 6% to 8% are sometimes quoted).

They may see some stereo effect but often wonder what the fuss is all about. It may be worthwhile having

you eyes re-checked if you are planning to work in stereo, as a change in prescription can help someone

with an astigmatisim.

The good news is that watching stereo at home will look much more like the first picture than the second :-)

 

Glasses free Stereo3D is a great long term objective but for today glasses will be the way forward.

 

Earphones and headphones used to be seen as ‘geeky’ too. Sony and later Apple made them fashionable.

Maybe we need someone to pick up the challenge of producing fashionable glasses.  

 

iPod is doing quite well with earphones isn’t it?

 

 There are also TV systems using active glasses and these, along with colour frequency based glasses are all possible ways of showing high quality TV pictures. Conclusion

 We know how to shoot, produce, post produce and broadcast Stereo3D today. The content will be there in

the next few years if there is a will to create it. TV sets are already on the market, more will be coming.

Stereo3D games are already on the market and DVD will be available.

 

Who knows what percentage of the population will accept wearing glasses? We don’t and neither does

anyone making definitive sounding negative pronouncements. Maybe they saw bad content. What we’re

reasonably certain of, based on personal experiences of working with many thousands of people here at

Quantel, with partners, or on the road at events, is that if you show good Stereo3D correctly and use good

glasses, many a sceptic is quickly won over. Seeing is believing.

 

There are lots of smart and dedicated folks in all kinds of companies working hard on making Stereo3D

Broadcasting happen right now. If we want Stereo3D Broadcasting, we can have it.

 

Since the general public looks like they will want it too, the questions now are increasingly ‘how’ and ‘when’

not ‘if’.

 

This isn’t the whole story and there is much more to be said. Please drop us a line if you want to talk through

any of these topics in detail.

 

Mark.horton@quantel.com

Trevor.francis@quantel.com