Several recent advances point the
way toward real-time holographic television for telepresence, entertainment,
and teleoperation. This article introduces the concepts and requirements for
such systems, then presents the MIT Media Lab's work to make them practical
by V. Michael Bove, Jr., and Daniel Smalley
THE NOTION of holographic television appeals to the popular imagination, figuring prominently
in science-fiction movies and carrying enough cachet that the word "holographic"
is often applied to systems that are not really holographic and sometimes not even
3-D, such as the "Pepper's ghost" illusions that have been used to re-create deceased
celebrities onstage. Holographic TV also potentially provides some important technical
advantages in that unlike two-view 3-D TV, it supplies in a consistent fashion all
the visual cues to object shape and position, including focus ("accommodation")
and motion parallax, increasing both viewer comfort for extended viewing and perceptual
accuracy for precise tasks.
Research since the early 1960s has attempted to build true holographic television,
but until very recently the prospect has seemed distant. The authors' group has
for several years concentrated on developing holographic displays suitable for consumer
applications, adding constraints of mass manufacturability, low cost, and compatibility
with mass-market computational resources such as might be found in PCs or game consoles.
A resurgence of consumer interest in 3-D displays, combined with several relevant
technological developments, makes this an opportune time to explore re-imagining
holographic displays as part of a home in the near future rather than in fictional
spacecraft in the far-off future.
Before we consider technical requirements for building such a device, it is important
to define precisely what a holographic display is, namely, a system that uses diffraction
of light to reconstruct light wavefronts (or lightfields) associated with a desired
visual scene. It is sometimes added that the diffraction pattern should be generated
by interference between a coherent reference beam and coherent light reflected by
a scene (or at least by a computational simulation of the interference), but for
a display designer the physical characteristics of the necessary diffraction patterns
are what matters.
Like all 3-D displays, holographic displays are bound by the behavior of light
and – despite cinematic special effects to the contrary – cannot create
images in free space or project them across a room. As shown in Fig. 1, from the
point of view of the viewer, all parts of a reconstructed object must have the display
Fig. 1: From the eye's viewpoint, an object
reconstructed by a hologram cannot extend past the edges of the display device.
The realism of holographic displays and also the difficulty
of building them can be traced to the physics of diffraction. A device is required
– usually called a spatial light modulator (SLM) – that can change the
amplitude (by varying its transmittance) and/or phase (by varying its index of refraction)
of light waves passing through it with a fine enough pixel pattern that diffraction
over a useful range of angles (which will be the viewing angle of the display) occurs.
For typical display applications this means a pixel pitch of about the same size
as the wavelength of visible light (or around half a micron).
Half-micron pixels may be smaller than the pixels in typical current microdisplays,
but an even bigger challenge lies in the fact that physics constrains the pixel
size to stay the same no matter the size of the display. Thus, such a display will
need about 2 million pixels per scan line per meter of image width. It's not too difficult to make a tiny direct-view
holographic image by illuminating a micro-display with a laser, but scaling this
up to useful dimensions by tiling such displays together is a complicated proposition.
Tiling of small devices nevertheless has been employed by several researchers to
create larger images; if the device is fast enough, the tiling can also be done
optically, where one device images in several positions sequentially.
A strategy that relaxes the need for such small pixels is to make
a hologram with a smaller view angle (and thus larger pixel pitch in the SLM) and
then use a steerable light source to direct the hologram where an eye tracker sees
the viewer's pupils.1 The pixel count and computational requirements
reduce significantly if the hologram is made to have parallax only in the horizontal
direction, as the vertical resolution then reduces to that of an ordinary television
image, and the computation of each scan line can be carried out independently.
For clarity, the following diffraction discussion will
be done in one dimension, with the extension to two dimensions straightforward (for
a horizontal-parallax-only display, the process will happen in only one dimension).
If a beam of monochromatic light enters a sinusoidal diffraction pattern, some
will pass straight through (the undiffracted or zero-order beam) and beams will
also come out at an angle to either side of the zero-order beam (the first-order
diffracted beams). The resulting angle is a function of the ratio of the wavelength
of the light to the spatial frequency of the pattern, while the amount of light
that is diffracted is a function of the contrast of the pattern. Note here that
the R/G/B illuminators will need to be monochromatic, as broadband illumination
will diffract across a range of angles, leading to a blurry image.
Such grating patterns are created in optical holograms through
interference, but the mathematics for synthesizing them from a three-dimensional
model of a scene are tractable. This process can be carried out similarly for a
computer-graphics model or for real imagery if sufficient information about the
real scene can be captured.
If the spatial frequency of the sinusoid varies (a "chirped" grating),
as in Fig. 2, and the pattern is illuminated with collimated light, beams at the
higher-frequency end will be diffracted at more of an angle than those on the other
end, giving the appearance that the light is coming from a point emitter.
Fig. 2: Because
a "chirped" diffraction pattern bends light by varying angles, when it is illuminated
by collimated light, the result appears to be a point emitter at a particular
(x,y,z) position. A scene can be built up by superposing about as many of these
as there would be pixels in an ordinary image of similar apparent resolution.
Note that for this diffraction method to work, there must be a
mechanism – which could be as simple as a barrier – for keeping the
undiffracted and opposite-order light from reaching the viewer's eyes. It should
be apparent that a 3-D scene could be built up by summing up chirped gratings corresponding
to the points making up the scene. Such calculations are within the capabilities
of modern PC or game-console graphics processors.2
Capture and Transmission
It is commonly assumed that the massive pixel count
of display holograms makes transmitting real-time holographic television nearly
impossible, and even if data compression could somehow reduce the data rate to something
manageable, the requirement for coherent illumination of the scene and extremely
short exposure times for moving imagery (because the scene has to be stationary
to within a small fraction of a wavelength of light during the exposure) would still
render the process impractical. Nevertheless, since the early days of holography,
analysis and experiments have been carried out for coherent capture and real-time
Recent advances in two areas have opened up an alternative approach:
non-holographic capture and calculation of the holographic interference pattern
at the receiving end. Ordinary cameras have become small and inexpensive enough
to permit building dense arrays of them, while small lightfield and rangefinding
cameras have also become available. The outputs of each of these image acquisition
strategies are more compact and easier to transmit than the holograms that would
result from capturing the same scenes at the equivalent image resolutions, and current
graphics processors and digital signal processors provide enough processing to do
the necessary diffraction-pattern computation.
Thus, researchers have recently been able to demonstrate scene
capture for holographic displays using a camera array,6 a lightfield
camera,7 or a rangefinding camera.8 The authors' group at
the MIT Media Lab has shown that the Microsoft Kinect can be used as the camera
for a holographic television system, with Internet transmission of the resulting
data and conversion to horizontal-parallax-only holograms at video rates on a standard
PC with three graphics cards. Because the Kinect produces perspective views and
the rendering algorithm needs an orthographic camera in the parallax direction,
it is necessary to perform calibrated geometric correction as part of the process.
Because a hologram captured with a single camera will have missing occluded regions
visible from viewer positions far from that of the camera, for complex scenes or
displays with large viewing angles it may be necessary to merge data from more than
one rangefinding camera, requiring yet more calibration and correction. The resulting
holograms have been demonstrated on both our display and a refreshable polymer display
developed by the University of Arizona, College of Optical Sciences (Fig. 3).
Fig. 3: This hologram was generated
from a Kinect camera and displayed on a refreshable photorefractive polymer
display. Courtesy University of Arizona College of Optical Sciences.
Several generations of holographic video displays have
been built at the MIT Media Lab since the groundbreaking Mark I display premiered
by Stephen Benton and his students in 1989.9 Our current display project
continues with the Scophony geometry used in its predecessors, where instead of
a common light-modulator technology, the diffraction patterns are created by acoustic
waves in a transparent material; as the pattern moves with the speed of sound, such
systems need a mechanical scanner to provide a stationary hologram. Recent systems
in our laboratory have employed our own lithium niobate guided-wave light modulator
in place of the earlier bulk-wave acousto-optic modulators. These devices –
similar to surface-acoustic-wave filters – can be fabricated with a modest
two-mask process. In a guided-wave modulator, a waveguide is created just under
the surface of the material, light is coupled into the waveguide, and diffraction
is created by surface acoustic waves. One of our prototype modulators is shown
in Fig. 4.
Fig. 4: In
this photograph of a guided-wave light-modulator chip, laser light enters at
left and diffracted light exits at right.
Our ultimate target is to fabricate devices with 480 or more independent
waveguides, providing sufficient bandwidth to support large displays, but our immediate
goal is to demonstrate a full-color horizontal-parallax-only 100-mm-wide proof-of-concept
desktop display of SDTV resolution with a bill of materials in the hundreds of dollars.10
Figure 5 shows the basic architecture: the light output from the
modulator passes through a lens, horizontal and vertical scanners (where the vertical
scanner will eventually not be needed when the modulator has as many channels as
the display has scan lines), a parabolic mirror, and a vertical diffuser. Because
the diffracted light has a rotated polarization from the zero-order beam, removing
the latter can be done with a polarizer. We first verified the operation of the
display optics with a bulk-wave modulator (Fig. 6 shows a small full-color image)
and are now using our guided-wave modulator. Full details of our experiments will
be presented in an upcoming publication, but the optical design has proven to work,
and Fig. 6 shows a full-color (though not 3-D) image from the display system.
Fig. 5: This prototype display
consists of a power supply (a), folded aluminum chassis (b), mirrors i and j
(c), transform lens (d), light modulator (e), laser source (f), modulator driver
cards with DVI-A inputs (g), phased-lock-loop control to drive the polygon (k),
vertical scanner (l), parabolic reflector (m), and anisotropic diffuser (n).
The vertical scanner will not be needed when the number of channels in the modulator
increases to match the number of scan lines in the hologram.
Fig. 6: This single view of
a small holographic stereogram test image (only 26 scan lines, not full screen)
was displayed on the system shown in Fig. 5.
Holographic Television: A Work in Progress
A brief article such as this can just touch upon the
basic principles; readers interested in exploring in depth the current state of
display holography may want to look at a comprehensive recent technical overview
by the author.11 Much research and development remains to be done before
holographic television is an everyday consumer product, but the practicality of
real-time holographic viewing is being enabled by progress in a variety of technologies.
For a Q&A with V. Michael Bove about the processing demands of holographic
see the October 2012 issue of Information Display.
Research described in this article has been supported
by consortium funds at the MIT Media Lab and by a gift from Intel Corp. We also
thank NVIDIA for providing graphics hardware used in this work.
1R. Haussler, S. Reichelt, N. Leister, E.
Zschau, R. Missbach, and S. Schwerdtner, "Large real-time holographic displays:
From prototypes to a consumer product," Proc. SPIE Stereoscopic Displays and
Applications XX 7237, 72370S (2009).
2Q. Y. J. Smithwick, J. Barabas, D. E. Smalley,
and V. M. Bove, Jr., "Interactive holographic stereograms with accommodation cues,"
Proc. SPIE Practical Holography XXIV 7619, 761903 (2010).
3E. N. Leith, J. Upatnieks, B. P. Hildebrand,
and K. Haines, "Requirements for a wavefront reconstruction facsimile system," J.
Soc. Motion Picture Television Eng. 74, No. 10, 893–896 (Oct. 1965).
4L. H. Enloe, J. A. Murphy, and C. B. Rubinstein,
"Hologram transmission via television," Bell Syst. Tech. J. 45, No.
2, 335–339 (Feb. 1966).
5R. J. Doyle and W. E. Glenn, "Remote real-time
reconstruction of holograms using the Lumatron," Appl. Opt. 11, No.
5, 1261–1264 (1972).
6P-A. Blanche, A. Bablumian, R. Voorakaranam,
C. Christenson, W. Lin, T. Gu, D. Flores, P. Wang, W.-Y. Hsieh, M. Kathaperumal,
B. Rachwal, O. Siddiqui, J. Thomas, R. A. Norwood, M. Yamamoto, and N. Peyghambarian,
"Holographic three-dimensional telepresence using large-area photorefractive polymer,"
Nature 468, No. 7320, 80–83 (2010).
7R. Oi, T. Mishina, K.Yamamoto, and M. Okai,
"Real-time IP-hologram conversion hardware based on floating point DSPs," Proc.
SPIE Practical Holography XXIII 7233, 723305 (2009).
8 J. Barabas, S. Jolly, D. E. Smalley, and
V. M. Bove, Jr., "Diffraction specific coherent panoramagrams of real scenes," Proc.
SPIE Practical Holography XXV 7957, 795702 (2011).
9P. St.-Hilaire, S. A. Benton, M. Lucente,
M. L. Jepsen, J. Kollin, and H. Yoshikawa, "Electronic display system for computational
holography," Proc. SPIE Practical Holography IV1212, 174–182
10D. Smalley, Q. Smithwick, J. Barabas, V.
M. Bove, Jr., S. Jolly, and C DellaSilva, "Holovideo for everyone: a low-cost holovideo
monitor," Proc. 9th International Symposium on Display Holography (ISDH 2012)
11V. M. Bove, Jr., "Display holography's
digital second act," Proc. of the IEEE 100, No. 4, 918–928 (2012).