Mark Schubin on Mobile 3D: Muscles Matter

http://www.mobilizedtv.com/mark-schubin-on-mobile-3d-muscles-matter

Mark Schubin is a multiple Emmy-Award-winning SMPTE Fellow who has worked professionally in television since 1967.

Some people get seasick, and the result of a non-hiker’s first walk up a hill is likely to be sore muscles. What does that have to do with Mobilized TV?  Fans of 3-D better hope it’s a lot.

Clearly, technology issues are associated with 3D viewing on a mobile device.  Instead of just one image, two are needed–one for each eye.  The

display needs to prevent the wrong eye from seeing its partner’s image.  Are glasses needed?  That’s a bummer.

Engineers have been tackling those issues and have already come so far that glasses-free 3-D was demonstrated on a tiny mobile-phone screen at the April 2009 National Association of Broadcasters convention.  But that’s technology.  Then there is psychophysics.

Unrelated to psychics or psychobabble, psychophysics is the science of psychological responses to physical stimuli.  An example of a physical stimulus is the picture on a mobile-phone screen.  An example of a psychological response is liking the picture enough to want to pay for it.

Psychophysics has already played a role in mobile TV.  Handheld at a typical viewing distance, a mobile-phone screen creates a smaller retinal image than do other video displays.  So, when Fox delivered a mobile-TV version of the popular series 24, each “mobisode” had a very short duration (initially one minute, later increased to three), with louder sound effects, more close-ups, bigger bullet holes, and more blood.

Mobile 3-D will likely face the same issues of a small retinal image and a tired device-holding arm.  But there are two other major considerations. One is called the “vergence-accommodation disparity” or sometimes the convergence-accommodation disparity.  Convergence is the aiming of the eyes at a particular point.  In 3-D, that point can be on the plane of the screen, behind it, or in front of it. Accommodation is the focusing of the eyes’ lenses on a particular point.  There are some 3-D images, involving holography, moving mirrors, or volumetric displays, in which accommodation can be tied to convergence.  For traditional stereoscopic 3-D, however, accommodation is always at the plane of the screen.  The single accommodation distance and the varying convergence distances of stereoscopic 3-D create a perceptual disparity.  The muscles moving the eyes report one depth to the brain; those focusing the lenses report another.

At the 2009 SMPTE Digital Cinema Summit, University of California Professor Martin S. Banks described experiments he had performed concerning that perceptual disagreement.  “This is really the first evidence that a vergence-accommodation conflict can cause fatigue and discomfort.”

It’s not an entirely new discovery.  Writing in The American Journal of Physiological Optics, Leonard Troland said, “The basis for eye strain is undoubtedly to be found not in any direct effect of the stimulus upon the eye but in a demand for overexertion of the ocular musculature.”  “Studies… indicate that one of the most common causes of eye strain consists in an unconscious attempt on the part of the observer to modify the normal coordination of the ocular reflexes of accommodation and convergence.”  That was in July 1926.

DreamWorks head (and 3-D fan) Jeffrey Katzenberg, speaking at the International Broadcasting Convention in September 2008, said the last thing 3-D should do is “make your audience hurl.”  But, as seasickness shows, perceptual conflict can, indeed, lead to vomiting.

Banks suggested to the Digital Cinema Summit that the conflict might be reduced by increasing a viewer’s “zone of comfort.”  For any given viewing distance, there is a range of convergence distances that might be acceptable.  A cinema viewer 50 feet from a screen, for example, might not mind convergence distances that vary from, say, 40 feet to 60 feet. A mobile-TV viewer’s eyes, however, might be just 18 inches from a screen.  If the scene shows an image requiring convergence 30 feet away, that’s unlikely to fall into anyone’s zone of comfort.  Or is it?

In 1895, the Lumiere brothers presented the first cinema audience with motion pictures of a train arriving at a station.  They were silent, black-&-white, jittery, and showed the train moving at an angle, but that was enough, according to a contemporary report, to cause an audience member to jump up in fear until the last car had passed through the frame.

In 1919, Thomas Edison staged a “tone test” at a concert hall, defying members of the audience to tell the difference between a live opera singer and a phonograph recording of her voice.  A reporter for the Pittsburgh Post wrote that he couldn’t.

Today, we might laugh at the idea that the Lumiere and Edison audiences couldn’t tell playback from reality, but that’s only because we have been taught the difference.  Perception is learned.

Consider seasickness.  First-time sailors tend to suffer from it, but those who’ve spent long periods on boats get over it.  Muscles, too, can be trained.  First-time hill walkers get sore muscles; long-time hikers don’t.

So it’s possible that viewer training will get around the convergence-accommodation 3-D problem even on mobile screens.  A short 3-D piece, followed by a recovery period, and then another stereoscopic sequence might train eye muscles in a manner similar to that by which a marathon starts with short sprints.

Unfortunately, eye-fatigue and nausea are only one of the perceptual issues associated with 3-D mobile TV.  There’s also, for example, the infinity-interpupillary problem.  It’s best not to overexert what lies between the ears.  So that other problem will be discussed in Part II.  Which other problem?  The one that experienced 3-D sailor might call “i-i.”

Mobile 3-D, Part II: Music to One’s Eyes?

This is part 2 of a two-part series by Mark Schubin, a multiple Emmy Award-winning SMPTE Fellow who has worked professionally in TV since 1967. Read Part 1 here.

In 1908, Camille Saint-Saëns composed what was probably the first film score. What does that have to do with mobile 3-D? It could be a solution (of sorts) to the infinity-interpupillary problem.

Any form of stereoscopic 3-D entertainment must deal with three categories of issues. Technical issues include the delivery of two motion-picture streams instead of one, alignment of dual imagery in both time and space, and mechanisms for getting the correct view–and only the correct view–to each eye.

Then there are program-production issues, including where to make different objects appear (in front of, at, or behind the screen) and how else to make use of the depth axis. When Alfred Hitchcock shot Dial M for Murder in 3-D, he was shocked to discover how barren the sets looked when he viewed the first rushes stereoscopically. Many more props had to be added before the experienced director was satisfied that the depth was properly filled.

The Society of Motion-Picture and Television Engineers (SMPTE) devoted an entire webcast of its Professional Development Academy (PDA) in June of 2009 to the subject, “Producing Stereoscopic Content: What Makes Great 3-D Great and What Can Go Wrong.” How, for example, can distant scenes be made to look like they’re in 3-D (long distances dilute the effect) without artificially separating the camera pickups of the two eye views, which would make people seem tiny, as though viewed by a giant?

As the existence of the SMPTE PDA webcast shows, effort is being devoted to production issues. And, as the appearance of a glasses-free 3-D mobile-phone display at the exhibit of Korea’s Electronics and Telecommunications Research Institute (ETRI) at the 2009 National Association of Broadcasters (NAB) convention shows, effort is also being applied to the technical issues.

The third category, however, is psychophysical issues–how viewers respond psychologically to the physical stimuli of 3-D imagery. Part I of this series covered one psychophysical issue, the convergence-accommodation disparity (viewers’ eyes point to one distance while their eyes’ lenses might focus on another). That disparity can introduce eye strain and fatigue–perhaps even nausea–but, in the same way that experienced sailors can get over the perceptual disparity that causes seasickness, 3-D viewers who become sufficiently accustomed to stereoscopic imagery might be able to watch without discomfort.

The infinity-interpupillary problem is different. When looking at something infinitely distant, eyes point straight ahead, creating parallel views separated by the same distance as the centers of their pupils — nominally a little more than two-and-a-half inches in a human adult. If a screen were to be placed in front of those eyes–at any distance away–the stereoscopic left-eye and right-eye images on that screen would have to be that same interpupillary distance apart.

In 3-D, “infinity” can actually be pretty close. When looking at something at a distance of 120 feet, our eyes each “toe-in” from looking straight ahead by less than five hundredths of one degree each. At 30 feet, it’s just two tenths of a degree. At even just six feet, there’s less than one degree of angular change per eye.

That’s not too much of an issue for 3-D in a movie theater. The left- and right-eye images of a distant object can be placed 2.5 inches apart. Unfortunately, if that’s done for a 30-foot-wide screen, then they’ll be five inches apart when the same movie is shown on a 60-foot-wide screen, requiring unnatural eye divergence.

On a 15-foot screen, everything would appear to be too close. On a home TV, the same object would seem really too close. That’s why 3-D TV might not be able to “repurpose” 3-D movies directly. As for yet smaller displays, is the screen on a mobile phone even 2.5 inches wide?

Perhaps that’s one reason why 3-D TV has yet to take off. The first 3-D TV broadcast was in 1928, and by 1953 Business Week ran the headline “3-D Invades TV.” Stereoscopic TV is hardly new. The idea that 3-D needs to recreate the depth cues of the real world, however, is arguable. Is 3-D, like sound, color, and increased detail, the next step towards reproducing reality? Or might it be more like film music?

No one would argue that music adds to the reality of dramatic or comedic programming. We don’t go through life with a band of personal musicians providing a background to our emotional states. But there’s also little doubt that music can enhance that programming. The same Alfred Hitchcock who tried 3-D in Dial M for Murder also performed an experiment in Psycho. To save money, the director shot the movie with the crew that worked on his TV series. But, according to Steve Vertlieb, in Midnight Marquee magazine in 2002, “Hitchcock himself had begun to have serious misgivings about the picture. It seemed somehow flat and lifeless, and he gave serious thought for a time to cutting the film down to an hour and releasing it as a part of his long running television series.”

What changed his mind was Bernard Hermann’s music score. It converted what might have been a mere TV episode into a movie classic, the American Film Institute’s number-one thriller of all time.

Could 3-D be similar to movie music? The stereoscopic scene shown on the mobile phone at the ETRI exhibit at NAB 2009 was hardly realistic. The trees in a forest glade seemed just inches apart. At the same time, the view was captivating. Imagine! There was a 3-D picture on a mobile phone!
Stereoscopic 3-D will do nothing to make the images on a mobile screen seem bigger, more detailed, or more realistic. But, if it entices users to pay for what they otherwise wouldn’t (or pay more for what they otherwise would), who cares?