The Progress of Light-Field 3-D Displays
Light-field displays represent an exciting and promising technology for the future. We introduce the principles of light-field displays, describe different types of multi-user light-field systems, and discuss their relative merits.
by Xu Liu and Haifeng Li
THREE-DIMENSIONAL display technologies have been a topic of research for over a century.1,2 Many techniques have been developed to create ideal displays with a 3-D effect, from classical stereoscopy, autostereoscopy,3,4 and integral displays5 to volumetric or holographic displays.6–9
Traditionally, researchers have used the theory of geometric optics in the construction of 3-D displays; examples include stereoscopic displays, autostereoscopic displays,3,4 and classical integral displays.2,5 In these systems, the presentation of 3-D scene content to the observers’ left and right eyes is considered according to the image perception principles of human vision. But in the case of holographic displays,10,38 wave-optics theory is used, which means both the light amplitude and phase distribution describe the radiation of the light from a real 3-D scene. A display that can represent both the intensity and the phases of a 3-D scene would be considered the “ideal” 3-D display.
Light-field displays derive from the concept of computational imaging. They are based on the distribution of light rays in a 3-D scene that are used to generate a 3-D display. They convert the phase distribution of a wavefront into angle distributions of light rays, and thus can enable occlusion and the correct perception of the 3-D scene.
The Principles of Light-Field Displays
As mentioned previously, the concept of light-field displays comes from “light field” imaging in the area of computation imaging. The phrase “light field” was coined by Gershun11 in a paper on the radiometric properties of light in 1936. The “light field” was redefined by Adelson and Bergenin12 in 1991 as part of a description of the plenoptic function of a natural scene that was used to present an imaging effect in computer graphics.
The plenoptic function can be expressed in the following way: P(x, y, z, θ, ϕ, λ, t), where x, y, z are the 3-D coordinates that describe the location from which light is being viewed or analyzed; θ, ϕ, describe the direction of the light; and λ and t are the wavelength of the light and the time of the observation, respectively.
For simplicity, the light field of a 3-D scene can be described with five-dimensional spatial parameters (x, y, z, θ, ϕ), as shown in Fig. 1(a). These form a 5-D spatial parameter space.
Considering the flat boundary around the 3-D scene, Gottler13 and Levoy14 in 1996 put forward a 4-D parameter space for the light-field presentation P (s, t, u, v) instead of 5-D space. The 4-D parameter space can perfectly describe the light-field distribution from the geometrical point of view, as shown in Fig. 1(b).
Fig. 1: The spatial parametrization of plenoptic and light-field models are shown at left and right, respectively. (a) The plenoptic model. (b) The light-field model.
It is clear that the dimensions of the parameter space are different among volumetric, stereoscopic, horizontal parallax only, and holographic displays. For holographic displays, we generally use the wave-optics theory, in which the light wave is expressed as
where r is expressed as (x, y, z), and the direction of k can be expressed as (θ, ϕ). Thus, holographic displays have the same parameter space as the plenoptic-rays model.
For volumetric 3-D displays,7,8 all the spatial voxels have the same luminance regardless of observation angle so that there is no angular parameter. These displays have a three-dimensional spatial parameter space P(x, y, z).
For stereoscopic displays, right eye and left eye images are needed; therefore, the parameter space is just two dimensional. In the case of a high-density viewing-angle autostereoscopic display, assuming the number of views is N, the parameter space is N times the two-dimensional parameter space N × P(x, y).
It is obvious that the more dimensions the parameter space of the light field has, the more “real” the 3-D scene it presents can be. Light-field parameter space analysis helps determine whether a 3-D display technique is or is not a “real” 3-D display.
The four parameters for describing the light field (s, t, u, v) or (x, y, θ, ϕ) indicate that in order to display a true 3-D image, two additional dimensional parameters are needed than with a 2-D display. If we take 1000 picture elements in one dimension, the light-field 3-D display will have a data rate at least 106 times higher than the 2-D display because of the two extra dimensions. This data rate cannot be accomplished by a normal-video-speed spatial light modulator (SLM). It needs an SLM with a much higher data-rate display ability, either high speed or high resolution, or both. Using a single high-speed SLM, we could optically scan the image with mirrors or other means while displaying the video data in a time-sequential manner to create the light-field display. Alternately, we could employ many SLMs in parallel and optically align them together to reduce the data rate required on each one.
There are two possible ways to generate the light field: (1) The first we will call the “rays angular multiplex” (or angular light integral) method, and, in this case, each SLM presents its own array of rays with unique distribution x, y, θ, ϕ to form all the distribution of light rays from a single x, y image location. These SLMs then together form the array of different spatial images for the 3-D display [Fig. 2(a)]. (2) The second case we will call the “image multiplex” (or image integral) method, and in this case each SLM presents all the x, y values at a unique θ, ϕ to form a unique viewpoint. In this case, the observer can see one SLM image in one direction and the light field of viewpoints is again made up from the total array of SLMs [Fig. 2(b)].
Fig. 2: Above are two possible configurations of SLMs for the generation of light fields. (a) Rays angular multiplex. (b) Image multiplex.
In practice, because of the low data rate of current display devices, we typically reduce the whole space parallax to the horizontal parallax only, to bring the spatial parameter space to three parameters (i>x, y, θ). This type of display system just ensures the perfect horizontal light field, but omits the light-field difference in the vertical direction. Almost all the techniques reviewed here are horizontal-parallax-only light-field-display systems.
There are two possible ways to meet the high data rate needed for a good 3-D light-field display: (1) time multiplexing or (2) spatial multiplexing. Time multiplexing approaches use a scanning type of light-field display based on a high-speed lighting source or high-speed modulator. These types of systems will be discussed in the next section. Spatial-multiplexing approaches use an integral type of light-field display based on multi-projector arrays. These are discussed in the section after the next one. In either method, one can use either rays-angular-multiplexing or image-multiplexing techniques. The various combinations are described in the next two sections.
Scanning-Type Light-Field-Display Systems (with Rotating Structure)
Scanning LED Arrays: LEDs have a very fast lighting speed (about 50 nsec), so they can do high-speed signal modulation and are a good candidate for 3-D light-field displays. Conventional LEDs are a Lambertian light emitter, but we can use a moving light slit in front of an LED to create directional light rays. Using a rotating cylindrical distributed LED array together with a higher-speed rotating light slit scanner, we can create a 3-D light-field display as shown in Fig. 3.
Fig. 3: The basic principle behind scanning LED-type light-field displays is shown above.17
Endo16 had proposed the basic theory behind this method in 2000. And a display system with a few colors was shown in 2005 by Yendo.17 Sony developed an LED-based small “RayModeler,” a 360° autostereoscopic 3-D display prototype with a display size of 27 cm in height and 13 cm in diameter, in 2009. In 2010, Zhejiang University (ZJU)18 developed the biggest color-scanning LED light-field-display system to date, with a size of 65 cm in height by 80 cm in diameter. The system presented dynamic video and 3-D color imagery and also demonstrated an interactive effect. These three scanning LED systems are shown in Fig. 4.
Fig. 4: Scanning LED-type light-field-display systems are shown from (a) Yendo, (b) Sony, and (c) ZJU.
Another way to use LEDs as a light-field-display medium was proposed by Yan.18 He used a high-density color LED display panel combined with a light-ray controller screen to form a high-speed display panel. This method is based on image multiplex synthesis. The panel rotates around its center axis and is addressed sequentially while the image light field is synchronized with the rotation.
Principally, if we increase the number of LEDs in each array and increase the LED array number in the circle, we can achieve better performance. But, in practice, due to the size limit of current color LEDs, this technique is more suitable for large-sized 3-D displays.
Scanning 45°-Tilted Special Diffusion Screen Technique: Cossairt19 proposed a method to address the lack of occlusion in volumetric displays by changing the scanning diffusion screen into a direction diffusion screen. His method simulates the generation of a light-field 3-D display. In 2007, Jones proposed the theory of light-field displays, and it was the first time that people began using the term “light-field display” and that a real light-field single-color dynamic display had been presented.20
Jones used the concept of light-field imaging in 3-D displays by inverting the light-ray propagation direction. He proposed a rendering method for light-field displays in which the observer sees the image composed by different projector images (see Fig. 5).
Fig. 5: Above are two different views of the 45°-tilted special-diffusion screen technique as implemented by the black-and-white 3-D light-field display from Jones’s system.
This system used one high-speed DMD SLM to form a high-frame-rate black-and-white projector. It employed a standard programmable graphics card to render over 5,000 images/sec of interactive 3-D graphics, projecting 360° views with 1.25° separation with up to 20 updates/sec and a 45°-tilted diffusion selective reflection screen (DSRS) rotated at 30 revolutions/sec.
Later on, researchers at Zhejiang University developed a special LED color-sequential high-speed projector.21 It can project 8,000 single-bit images/sec with a resolution of 1024 × 768. Combined with a 45°-tilt DSRS, it was the first time that vivid colors were shown with dynamic 3-D light-field imagery (see Fig. 6).
Fig. 6: Above are shown the 45°-tilted special-diffusion screen technique implemented by Zhejiang University (left) and views from different angles of a 3-D image displayed by the system (right).
This technique used a DMD as a high-speed SLM and could achieve a data rate approaching 3 Gbit/sec, which is very good for a low-resolution horizontal-parallax-only 3-D light-field display. Through the scanning of the tilted DSRS, one can get a good light-field display, but the display region is combined with the rotating DSRS region, and the influence of the movement of mechanical parts in the ambient light causes problems. Moreover, as the display size increases, the tilted DSRS will increase greatly, further increasing the problems caused by the mechanical movement.
Scanning Flat Special Diffusion Screen Technique: The scanning flat DSRS technique was proposed in 2010.22,23 In this technique, in place of a 45°-tilted DSRS, we used a special DSRS screen that can reflect diffuse light tilted 30° (see Fig. 7) in a vertical direction and reflect only in the horizontal direction. This flat screen was used as the light-ray scanner. It rotated at 1800 rpm, and the high-speed color projector projected 21,000-frames/sec images on the screen. Through the scanning, one can create a vivid 3-D-scene light field floating above the flat screen.
Fig. 7: The flat special diffusion screen technique is shown in two working modes.
There are two types of screens that can be used as the scanner. One is the reflective DSRS screen. The display works in the reflective mode. The projector is put on top of the scanning screen. For the reflective mode, the scanning screen is a highly reflective diffusion selective screen. Therefore, the ambience illumination light has serious influence. The other one is the transitive mode. The projector is working in the transmitted form, and a transmitted diffusion selective screen (TDSS) is used. In this case, the TDSS scanning screen has low reflectance of ambient light, resulting in better contrast and, hence, better 3-D performance in high-ambient-light situations.
In order to get that better performance, we developed an RGB color projector system using three high-speed DMDs to generate the R, G, B high-frame-rate images, respectively. Each RGB channel has a capability of 21,000 frame/sec.23 This provides a near-perfect color effect in the display.
The flat-scanning-screen technique can enable an interactive floating 3-D display. It produces a very “real” floating 3-D image display that can be made interactive by monitoring the observers’ gestures and eye movements with a camera and responding to them. In Fig. 8, the 3-D display Olympic mascot Jingjing appears next to the real toy. The floating image has obvious volume.
Fig. 8: Shown above are examples of 3-D display performance from a system based on the flat special diffusion screen technique.
As mentioned above, this system currently can only deliver a horizontally correct light-field display. Su24,25 proposed a method that uses interactive effects to achieve vertical light-field-display information. He used a specially designed 360° lens imaging system to track the surrounding observers’ eyes and displayed the corresponding correct vertical light-field imaging to the corresponding observer.
Integral-Type Light-field-Display System (with Multi-Projector Array)
An integral-type light-field display is different from the scanning variety. Instead of using a high-speed SLM or a light modulation source, we used a large number of image generators working in parallel to project images on a special directional transmission diffusion screen (DTDS). Through the special diffusion effect and with a large number of display image generators, we can achieve high-data-rate processing and generate the entire light-field distribution of the 3-D scene. In principle, to achieve good light-field-display performance, the multi-projector array works in an angular multiplex mode. It means that each projector must have its outlet pupil seamlessly adjacent with those of its neighboring projectors, as shown in Fig. 9
Fig. 9: The principles of an angular multiplex light-field-display system30 are shown above.
Obviously, the key issues here are how to work a large number of SLMs in parallel and what is the DTDS that can direct the SLM light to the desired direction? Different DTDSs require different image-generation methods and different arrangements of the image generators.
In general, there are flat screens with multi-projector systems, curved screens with multi LCD systems, and surround-type light-field-display systems.
Flat Screen with Multi-Projector System: There are many papers that present the 3-D light-field displays with flat special diffusion screens and multi-projector systems. The first near-commercial product is the work by the team from Holografika,26,27 in which multiple projectors are arranged in such a way that the output pupil of each projector seamlessly abuts the ones from its nearest neighbors. Shang28,29 used a special holographic screen to meet the needs of the DTDS’s properties and set up a large flat-screen multi-projector light-field-display system in 2009. Samsung presented its large flat-screen system at Display Week in 201330 with a 300-Mpixel multi-projection 3-D display that had a 100-in. screen and a 40° viewing angle (Fig. 10).
Fig. 10: Shown above are three different embodiments of multi-projector systems with flat special diffusion screens. (a) the HoloVizio system.27 (b) Shang’s system. (c) Samsung’s system.
Because micro-projectors have become cheaper, many people have tried to use micro-projector arrays to generate the flat-screen 3-D display systems.31
It must be mentioned that for the flat DTDS, the field of view is limited by the field angle of each projector lens. The curved screen has an advantage in increasing the field of view. Because large numbers of projectors have been used in the display, the degradation due to mismatching fringes in the display is a problem that needs to be solved.30
Curved Screen with Multi-LCD System: This configuration employs three LCD units together with an arc DTDS, forming the light field of a 3-D scene in the central region. All three units and the diffuser are set in different concentric arcs.33,34 As shown in Fig. 11, the LCD panels are divided into numerous sub-display regions (or mosaic images) for different views. The number of sub-displays determines the number of views (angular resolution), and the number of pixels in each sub-display region determines the spatial resolution of the 3-D image. There is a direct tradeoff between the spatial and angular resolution since the product of the number of sub-displays and the number of pixels per sub-display is fixed and equal to the total pixel count of the LCD panel. Each lens and corresponding LCD region make up a so-called projector. All the light beams projected by these “projectors” converge at the arc center that is defined as the center of the reconstruction area.
Fig. 11: The basic configuration for a three-panel curved-screen light-field system uses LCDs and a vertical diffusion screen.
The 3-D display unit is scalable so that multiple display units are utilized to provide a large viewing range horizontally with the most feasible modularization. Each lens projects the pixels of an LCD sub-image in the form of a series of directional rays, which then construct the light field with other rays projected by the lens array. This 3-D display is limited by the pixel count of the LCD.
Surround-Type Light-Field Displays: The surrounding-type light-field display can increase the observers’ angular range to 360°. It is a system that can display the 3-D light-field image in the center of a certain volume. The observers can move around the volume to obtain a different point of view from different positions.36,38 The system consists of N projectors aligned in the same horizontal plane and arranged such that the lens pupils of successive projectors form a continuous region from P1 to PN (Fig. 12). Because of a limited number of projectors, rays are projected discontinuously and horizontally from these discretely positioned projectors. That is to say, without a cylindrical directional diffuser, screen observers would only obtain a series of discontinuous emitting exit pupils of the projectors. To smooth the discontinuity of rays, a cylindrical directional diffuser screen is set in the front.
Fig. 12: The projection principle of surrounding-type light-field 3-D displays appears above. (a) The light rays formed on DTDS. (b) Different views: v1 and v2.
A display system with 360 projectors has been set up in ZJU.37 The projectors used here are DMD-based LED-light-source color projectors with 800 × 600 pixels. The cylindrical diffusion selective screen is 4 m in diameter and 1.8 m high. The screen diffuses the light vertically about 60° and diffuses about 5° horizontally (Fig. 13).
Fig. 13: The above images show schematics (top) and display performance of a surrounding light-field display with 360 projectors.37
The other type of surrounding light-field display involves a projector array surrounding the outside of a cylindrical direction diffusion screen. The observer is located at the central region of the cylindrical screen. In this case, a strong immersion effect will be perceived with a very wide viewing angle.37
Ongoing Challenges of Big Data and More
As discussed above, light-field 3-D displays can show a very good-looking 3-D image “floating” in the air, and the observers can watch a real 3-D scene from different points of view around the display with the naked eye. But to display a high-quality 3-D image, one needs a huge amount of display data. The minimum amount of 3-D display data needed depends on the 3-D scene volume displayed. The angular resolution of the human eye is about 1 arc minute. For a given spatial volume of a V-sized 3-D display, if N is the number of display voxels, then N/V is the density of display voxels. If the observer watches the 3-D scene from a distance L, the neighboring voxels will provide an angular interval of about (V/N)1/3/L to the observer. If we take 1′ as the limiting resolution of the eyes, we will have the relation:
(V/N)1/3 ≤ PI*L/60/180/2
If we want to have a 3-D display with a volume of about 10 cm3, and the observer is 50 cm from the display, we can estimate that the minimum number of voxels required is 109. This number is achievable with the current state-of-the-art SLM devices through the use of parallelism and multiplexing.
As we have explained, for techniques showing light-field 3-D, no matter what scanning or multiplex SLM integral approach is used, the total available data rate is limited. In this case, we have to decide how much of the data should be used to present the angular information and how much for the resolution information, in the 3-D scene. We must balance these two parameters by considering the light-field-display method used and observation position in order to obtain the best possible 3-D display quality.
To compare the performance of the different techniques, we can look at the data rate of the display system. The higher data rate a display system uses to present an image, the better the performance should be. Currently, for a 360° surrounding light-field display system 10 cm3 in size, a data rate of at least 10 Gbit/sec is needed for displaying. This means that the development of high-data-rate spatial light modulators and data-processing methods are the key factors in the further development of high-performance 3-D displays.39 The other way to achieve high performance is to use high-density low-data-rate SLMs to operate in parallel so the system as a whole can get to the required high data rate. From the current state of the art and the advances that are being made, we can expect that 3-D light-field displays will be the first feasible “real” 3-D display technology of the near future.
Acknowledgment
These works were supported by several research funds. We thank the National Basic Research Program of China (973 Program) (2013CB328802), the National High Technology Research and Development Program of China (2012AA011902), and the National Nature Science Foundation of China (61177015).
References
1E. Lueder, 3D Displays (Wiley, 2010).
2J. Hong et al., “Three-dimensional display technologies of recent interest: principles, status, and issues, Applied Optics 50, No. 34 , H87-H115 (December 2011).
3N. A. Dodgson, “Autostereoscopic 3D Displays,” Computer 38, 31–36 (2005).
4Y. Takaki and T. Dairiki, “72-directional display having VGA resolution for high-appearance image generation,” Proc. SPIE 6055, 60550X (2006).
5J-H. Park, K. Hong, and B. Lee, “Recent progress in three-dimensional information processing based on integral imaging,” Applied Optics 48, No. 34, H77-h96 (December 2009).
6J.-H. Park et al., “Three-dimensional display scheme based on integral imaging with three-dimensional information processing,” Opt. Express 12, 6020–6032 (2004).
7G. E. Favalora, “Volumetric 3D displays and application infrastructure,” Computer 38, 37–44 (2005).
8G. E. Favalora, “100 Million-voxel volumetric display,” Proc. SPIE 4712, 300–312 (2002).
9X. Xie, “The investigation of data voxelization for a three dimensional volumetric display system,” J. Opt. A Pure Appl. Opt. 11, 045707 (2009).
10T. Kreis, P. Aswendt, and R. Hofling, “Hologram reconstruction using a digital micromirror device,” Opt. Eng. 40, 926–933 (2001).
11A. Gershun, “The Light Field” (Moscow, 1936, trans. by P. Moon and G. Timoshenko), J. Math. and Physics 18, 51–151 (1939).
12E. H. Adelson and J. R. Bergen, ‘‘The Plenoptic Function and the Elements of Early Vision,’’ Computation Models of Visual Processing, M. Landy and J.A. Movshon, eds. (MIT Press, 1991), pp. 3–20.
13S. J. Gortler et al., ‘‘The Lumigraph,’’ Proc. ACM Siggraph (ACM Press, 1996), pp. 43–54.
14M. Levoy and P. Hanrahan, ‘‘Light Field Rendering,’’ Proc. ACM Siggraph (ACM Press, 1996), pp. 31–42.
15R. Yang, X. Huang, S. Li, and C. Jaynes, “Toward the Light Field Display: Autostereoscopic Rendering via a Cluster of Projectors,” IEEE Transactions on Visualization and Computer Graphics 14, No. 1, 84–96 (Jan.–Feb. 2008).
16T. Endo et al., “Cylindrical 3-D video display observable from all directions,” Proceedings of Pacific Graphics 300–306 (2000).
17T. Yendo et al., “The Cylindrical Lightfield Display,” ACM SIGGRAPH 2005, Emerging Technologies 16.
18C. Yan et al., “Omnidirectional multiview three-dimensional display based on direction-selective light-emitting diode array,” Optical Engineering 50(3), 034003 (2011).
19O. S. Cossairt et al., “Occlusion-capable multiview volumetric three-dimensional display,” Applied Optics 46, 8 (Mar), 1244–1250 (2007).
20A. Jones et al., “ Rendering for an Interactive 360° Light Field Display,” ACM SIGGRAPH (Association for Computing Machinery, 2007).
21C. Yan et al., “Color three-dimensional display with omnidirectional view based on a light-emitting diode projector,” Applied Optics 48, 4490–4495 (2009).
22X. Xia et al., “Omnidirectional-view three-dimensional display system based on cylindrical selective-diffusing screen,” Applied Optics 49, 4915–4920 (2010).
23X. Xia et al., “A 360-degree floating 3D display based on light field regeneration,” Optics Express 21, 11237–11247 (2013).
24C. Su et al., “A penetrable interactive 3D display based on motion recognition,” Chinese Optics Letters 12(6), 060007 (2014).
25C. Su et al., “An Interactive 360° Floating 3D Display Based on Gesture Recognition,” SID Symposium Digest of Technical Papers 45 (Book I), 1278–1281 (2014).
26T. Balogh et al., “Large Scale Interactive Holographic Display,” IEEE VR2006 Conference, March 25–29, Alexandria, Virginia, USA
27http://www.holografika.com/
28X. Shang et al., “Demonstration of a large size real time full color three dimensional display,” Opt. Lett. 34, 3803–3805 (2009).
29F. C. Fan, S. Choi, and C. C. Jiang, “Use of spatial spectrum of light to recover three- dimensional holographic nature,” Applied Optics 49, No. 14, 2676–2685 (2010).
30J-H. Lee et al., “Optimal Projector Configuration Design for 300-Mpixel Light-Field 3D Display,” SID Symposium Digest of Technical Papers 44, Book I, 400–403 (June 2013).
31R. Horstmeyer, S. B. Oh, and R. Raskar, “View-Dependent Displays and the Space of Light Fields,” arXiv preprint arXiv:1008.0034 (2010).
32R. Otsuka, T. Hoshino, and Y. Horry, “Transport: all-around three-dimensional display system,” Proc. SPIE 5599, 56–65 (2004).
33Y. Peng et al., “Self-calibration three-dimensional light field display based on scalable multi-LCDs,” Journal of the Society for Information Display 20, 653–660 (2012).
34Y. Peng, H. Li, Q. Zhong, and X. Liu, “Liquid-crystal-display-based touchable light field three-dimensional display using display-capture mapping calibration,” Applied Optics 51, 6014–6019 (2012).
35Y.-f. Peng et al., “Large-sized light field three-dimensional display using multi-projectors and directional diffuser,” Optical Engineering 52, 017402-017402 (2013).
36Q. Zhong et al., “Surround-type Light Field Display with Immersive Experience,” SID Symposium Digest of Technical Papers 45, 517–519 (2014).
37Q. Zhong et al., “Multiview and light-field reconstruction algorithms for 360° multiple-projector-type 3D display,” Applied Optics 52, 4419–4425 (2013).
38S. Yoshida et al., “Prototyping of Glasses-Free Table-Style 3D Display for Tabletop Tasks,” SID Symposium Digest of Technical Papers 211–214 (2010).
39E. Smalley et al., “Anisotropic leaky-mode modulator for holographic video displays,” Nature 498, 313–317 (2013). •