Color and Displays in the Space–Time Continuum

Novel methods of synthesizing display color by hybrid color processing in both the spatial and temporal domains can enhance the image quality and improve the performance of color displays.

by Louis D. Silverstein

EINSTEIN did not address color or dis-plays in relation to the space–time continuum. But now that I have captured your attention with this intriguing title, I would like to briefly explore how the human visual system (HVS) enables the mixing or synthesis of colors in space and time and to describe some novel methods for synthesizing display color. We will see how a hybrid spatial–temporal approach to display color synthesis can enhance the image quality and improve the performance of color displays.

A common problem of all color displays, regardless of whether they are of the self-luminous or non-self-luminous type, is the synthesis of a full-color image from a limited set of primary colors. In order to fully understand the principles of color generation in electronic-display systems, the basic mechanisms that allow color synthesis to occur must be briefly explored.

The theoretical foundation underlying color synthesis is the trichromatic theory of color vision. This basic theory postulates that all colors are analyzed by the HVS through a set of responses corresponding to the transformed spectral sensitivities of three different populations of photosensitive receptors in the human retina. Each receptor population is selectively sensitive to a finite spectral bandwidth that approximates separate long (red), middle (green), and short (blue) wavelength response functions. These receptor signals are in turn neurally processed and combined in a complex manner to produce what is ultimately experienced as color.

Because the outputs of only three distinct populations of wavelength-sensitive receptors are combined to produce the sensation of the entire spectrum of color, the appearance of any color can be matched by the mixture of three appropriately selected primary-color stimuli (Fig. 1).1 Note that the existence of separable opponent-process channels enables different spatial and temporal characteristics for each channel. It can also be seen that the photoreceptors sensitive to short wavelengths (blue) contribute little or nothing to luminance because they provide no input to the intensity or luminance channel.

 

fig_1_tif

Fig. 1: This simplified schematic representation outlines the structure and functional organization of human color vision from photoreceptors through neural processing.

 

The concept of additive mixtures of chromatic luminous sources may be the most basic operating principle enabling the development of full-color electronic displays. In principle, the simplest form of additivity is obtained by direct superposition of three differently colored beams of light or colored images. However, two other characteristics of the HVS offer enormous flexibility in devising techniques for synthesizing color.

Since the HVS is quite limited in both the spatial and temporal resolution of visual input, spatial or temporal patterns composed of three appropriately selected primary colors are sufficient for producing the full range of colors when the spatial or temporal frequencies of the patterns exceed the respective resolution limits. Spatial resolution of the HVS is basically limited by the optics of the eye and the fineness of the retinal mosaic of cone-receptor elements, while temporal resolution is limited by the finite temporal bandwidth of the retinal photoreceptors and other neuronal elements. These limits in spatial and temporal resolution, or, more precisely, the fact that integration occurs beyond these limits, permit the phenomena of spatial- and temporal-additive color synthesis to occur. Figure 2 shows the limitations of HVS resolution in the spatial domain.2 Note that the luminance channel supports the highest spatial resolution while the two color channels are much more limited in their ability to resolve spatial detail. In particular, the resolution of the blue–yellow channel is severely limited and reveals that only relatively coarse spatial patterns can be resolved by the short-wavelength visual mechanism. Similar differences in visual-channel resolution have been found in the temporal domain.3

Conventional Approaches

Several approaches to color synthesis have traditionally been employed for electronic displays. The most successful of these conform to the principles of additive color mixture and include optical superposition, spatial synthesis, and temporal synthesis.1 Direct optical superposition of three primary-color images is commonly used in projection-display systems. However, while optical superposition is an effective method of color synthesis for projection systems, it is not readily amenable to most direct-view color-display technologies.

Spatial color synthesis has by far been the most successful method of color synthesis and remains the foundation of modern direct-view color-display technology. The two most successful electronic-color-display devices available, the shadow-mask color cathode-ray tube (CRT) and the color liquid-crystal display (LCD), conform to this principle.

 

fig_2_tif

Fig. 2: Psychophysical investigations have provided clear indications of the spatial resolution of the luminance (intensity) and opponent color channels (red–green and blue–yellow) of the human visual system.

 

fig_3_tif

Fig. 3: The spatial-additive approach to synthesizing color in displays has been very effective, but it sacrifices image resolution. The commonly used vertical-stripe color pixel mosaic (configured for square full-color pixels) is shown, along with the spatial Nyquist limits of the mosaic and the associated fixed-pattern noise.

 

Although spatial-additive color synthesis has been extremely successful, the method has two significant limitations that may inhibit the continued evolution of high-resolution color-display technology. The first limitation is the obvious sacrifice of potential display resolution because the use of available spatial area for color synthesis reduces the spatial imaging potential of the display. The other limiting factor is the fixed-pattern noise produced by the mosaic of primary-color subpixels, and particularly the blue subpixel elements (Fig. 3).

The spatial Nyquist limits shown in the figure define the two-dimensional (2-D) baseband of spatial frequencies that can be rendered without aliasing, while the fixed-pattern noise modulation spectrum indicates the residual luminance modulation (0 < m < 1) present in the mosaic when all of the color subpixels are activated to produce a flat white field. Both of these analytical evaluations of the imaging potential of the sampling mosaic were obtained via a 2-D Fourier transform of the mosaic pattern. Figure 3 clearly reveals the sacrifice in spatial resolution resulting from spatial color synthesis, as well as the high levels of fixed-pattern noise modulation along the horizontal dimension. It can easily be established that the fixed-pattern noise is dominated by the blue (B) subpixels since they account for only about 8% of the luminance in a D65 white field when the set of sRGB primary chromaticities1 is assumed.

Different pixel mosaics yield different spatial-frequency basebands and levels of fixed-pattern noise. For example, the RGB delta triad mosaic which uses an alternating half-subpixel phase shift in the rows provides a substantially larger baseband and significantly lower levels of fixed-pattern noise than that for the stripe mosaic. These attributes make the delta triad mosaic desirable for television monitors and electronic camera viewfinders. Nevertheless, even for improved geometries such as the RGB delta triad mosaic, the reliance on spatial color synthesis and the allocation of display area for all three primary-color subpixels is wasteful of potential resolution and a paradigm for fixed-pattern noise. In particular, the display area allocated to the B subpixels is especially wasteful because the B subpixels contribute little to luminance and short wavelengths are processed only at a very low spatial resolution by the HVS.2

Temporal color synthesis, or field-sequential color, avoids the loss of spatial resolution that is inherent to spatial synthesis and does not produce fixed-pattern noise. However, two important limitations of temporal color synthesis constrain the efficacy of field-sequential-color displays. First, although the field-sequential approach produces effective additive-color mixtures, residual luminance differences between the time-varying com-ponents can produce observable luminance flicker for temporal frequencies at or above those at which effective chromatic integration has taken place.1,3 This is a consequence of the higher temporal resolution in the luminance channel than in either of the two opponent-process color channels.

A more difficult limitation results from relative movement between the displayed image and the viewer's retina, whether the motion arises from the image or from the viewer's head and eye movements. In either case, the time-varying color components are no longer imaged on the same retinal region, and the observer experiences what has come to be known as "color breakup" or "the rainbow effect."

 

fig_4_tif

Fig. 4: One possible configuration of STColor for a direct-view LCD utilizes an illumination source switchable between yellow (Y) and blue (B) along with an LCD panel and pixel mosaic consisting of a checkerboard pattern of magenta (M) and cyan (C) color filters. When the Y illuminant is activated during one temporal field, the display output consists of a checkerboard pattern of red (R) and green (G) subpixels. Activation of the B illuminant during an adjacent temporal field yields a display output of homogeneous B subpixels. The temporal combination of the two fields yields a full-color display.

 

Avoiding color breakup for RGB field-sequential-color displays in the presence of large high-velocity saccadic eye movements requires color field rates well in excess of those needed to eliminate flicker and can easily exceed 1000 Hz when the display luminance and contrast are high.4 The current de facto standard for sequential-color field rates is in the range of 360–480 fields per second. These high field rates impose severe bandwidth requirements on field-sequential-color displays and their drive electronics and make the temporal isolation of primary-color image fields very difficult.

Image quality has been an important driving force behind the rapid evolution of display technology. In all major market segments, the momentum toward higher display resolution and enhanced color quality is inescapable. This, in turn, has exposed the limitations of both spatial and temporal color synthesis and raises the question as to whether either method for synthesizing color can, by itself, fully satisfy the ever-increasing demands on display image quality. Clearly, new approaches to color synthesis are required to sustain the evolution of display technology.

Synthesizing Colors in Both Space and Time

Hybrid spatial and temporal color synthesis, herein designated as STColor, is a novel approach to the problem of display color synthesis that distributes the color synthesis function across both the spatial and temporal domains.5 The approach is fundamental and can be adapted to most color-display technologies, including direct-view matrix displays of both the non-emissive and emissive types, projection displays with matrix image sources, and near-to-eye virtual displays. The general approach reduces the number of primary-color subpixel elements from three to two, arranges the two primary subpixel elements in a 2-D checkerboard mosaic, and synthesizes the remaining third primary-color element by temporal synthesis.

One possible configuration of STColor for a direct-view LCD illustrates the general principles of the method, which must be adapted to specific applications (Fig. 4). In this configuration, an illumination source switchable between yellow (Y) and blue (B) spectral power distributions is combined with an LCD panel and a pixel mosaic consisting of a checkerboard pattern of magenta (M) and cyan (C) color filters. The sources of illumination may be combinations of light-emitting diodes (LEDs), cold-cathode fluorescent lamps (CCFLs), laser diodes, or any other potential source with suitable spectral, temporal, and intensity characteristics. When the Y illuminant is activated during one temporal field, the display output consists of a checkerboard pattern of red (R) and green (G) sub-pixels. Activation of the B illuminant during an adjacent temporal field yields a display output of homogeneous B subpixels. The temporal combination of the two fields yields a full-color display. Since the homogeneous B field provides twice the space-average amount of B luminance required to achieve white balance, light throughput efficiency of the system may be enhanced by doubling the relative duration of the R/G fields. Although there are many possible ways to partition the spatial and temporal components, this configuration isolates the short-wavelength contribution to the temporal domain and thus capitalizes on the unique structure and attributes of the HVS color channels.5

 

fig_5_tif

Fig. 5: There are two different modes of addressing for STColor displays. The left side of the figure shows the addressable pixel locations, the pixel window, and the subpixel weights for the R/G and B fields according to a logical full-color pixel-addressing mode. The right side of the figure shows the addressable pixel locations, the pixel window, and the subpixel weights for the R/G and B fields according to a subpixel addressing mode using a 3 x 3 spatial-filter kernel.

 

The regular, symmetric 2-D checkerboard spatial mosaic provided by STColor enables either logical full-color pixel addressing (Fig. 5, left column) or a simple, efficient implementation of subpixel addressing using a minimal 3 x 3 spatial-filter kernel (Fig. 5, right column). The latter offers single subpixel addressability in both dimensions and preserves local color balance.

Many alternative combinations of illuminants and color filters are possible, but restricting the short-wavelength component to the temporal domain offers unique benefits for image quality because the visual processing of short wavelengths is extremely limited in both spatial and temporal resolution. It can be seen in Fig. 4 that the R/G field determines the spatial properties of the display and carries most of the luminance information while the B field principally serves as a color-mixture function.

As far as the perceptual impact of the temporal component of STColor is concerned, prior research has found that perceived color breakup is strongly determined by the luminance and contrast of the alternating color fields.4,6 Because the R/G field contains both of the high-luminance and high-contrast components, and alternates in time with the temporally diffuse and low-luminance B field, perceived color breakup is alleviated and the required field rates will be determined primarily by flicker.

The temporal frequencies required to preclude observable flicker reach an asymptote at approximately 80 Hz even for fully modulated temporal waveforms at high levels of retinal illuminance.7 It is estimated that STColor field rates in the range of 120–160 fields per second will be sufficient to provide a flicker-free display essentially devoid of color break-up, although this will depend on overall display luminance and contrast. This constitutes a dramatic reduction in temporal bandwidth relative to the 360–480 fields per second needed today.

Another Configuration

An alternative configuration of STColor utilizes a combination of C and Y illuminants with a color pixel mosaic composed of a checkerboard of M and G color filters (Fig. 6). When the Y illuminant is activated during one temporal field, the display output consists of a checkerboard pattern of red (R) and green (G) subpixels. Activation of the C illuminant during an adjacent temporal field yields a display output consisting of a checkerboard pattern of G and B subpixels. As in the configuration described previously, the temporal combination of the two fields yields a full-color display. Since both fields generate G subpixels, the space/time average G luminance must be suitably adjusted to achieve white balance. This combination of illuminant fields and color pixel mosaic preserves most of the image-quality benefits of the configuration shown in Fig. 4, while providing lower inter-field luminance modulation and thus enabling lower flicker-free field rates.

Other combinations of temporal illuminant fields and spatial color-filter arrays can be constructed to achieve different display-performance objectives, such as extending the display color gamut or enhancing the luminous efficiency of the display. Researchers from Philips Laboratories have recently prototyped and demonstrated displays based on a hybrid approach to color synthesis that achieve an impressive expansion of the display color gamut and significant improvements in luminous efficiency.8,9

 

fig_6_tif

Fig. 6: This alternative STColor configuration uses a combination of C and Y illuminants with a color pixel mosaic composed of a checkerboard of M and G color filters. This combination preserves most of the image-quality benefits of the configuration of Fig. 4 while providing lower interfield luminance modulation, thus enabling lower flicker-free field rates.

 

The M/C spatial mosaic for the STColor configuration shown in Fig. 4, along with the spatial resolution (Nyquist limits) of the mosaic and the associated fixed-pattern noise, makes two results readily apparent (Fig. 7). The spatial-frequency baseband is dramatically expanded relative to the RGB stripe mosaic and the fixed-pattern noise has been suppressed to very low levels of modulation and shifted to higher spatial frequencies along the diagonal direction. For an equivalent horizontal density of pixels – and number of column drivers – and assuming a requirement for square pixels, STColor offers an increase in effective resolution (as indexed by increases in the spatial-frequency baseband in horizontal and vertical dimensions) of approximately 200% over the RGB stripe mosaic and 50% over the RGB delta triad mosaic. Fixed-pattern noise is substantially and significantly reduced relative to either of the RGB display configurations.

The increased resolution and reduced fixed-pattern noise offered by STColor can enable a reduction in pixel density relative to displays relying only on spatial color synthesis using RGB mosaics. This, in turn, enables a reduction in the number of column drivers and increases the aperture ratio of the display.

Conclusions

A novel method of synthesizing display color by hybrid color processing in both the spatial and temporal domains has been described. This new method enhances the effective display resolution and addressability, as well as providing very-low fixed-pattern noise relative to displays using spatial color synthesis alone. Marked reductions in color field rate, minimization of color breakup, and improved light efficiency relative to displays using temporal color synthesis can be realized with STColor.

In addition, the method provides many degrees of freedom for selecting combinations of temporally switched illuminants and color-filter mosaics to achieve different display performance objectives, such as extending the display color gamut or enhancing the luminous efficiency of the display.

The STColor approach is adaptable to a wide variety of direct-view and projection-display systems and offers great potential for enhancing the performance and reducing the cost of future color displays.

References

1L. D. Silverstein, "Color in Electronic Displays," SID Seminar Lecture Notes, Vol. 1, M-13/1-63 (2003).

2B. A. Wandell and L. D. Silverstein, "Digital Color Reproduction," in The Science of Color, S. Shevell (ed.), 2nd edition (The Optical Society of America, Washington, DC, 2003), pp. 281–316.

3D. Varner, "Temporal Sensitivities Related to Color Theory," J. Opt. Soc. Am. A, 474–481 (1984).

4D. L. Post et al., "Predicting Color Breakup on Field-Sequential Displays: Part 2," SID Symposium Digest Tech. Papers 29, 1037–1040 (1998).

5L. D. Silverstein, "STColor: Hybrid Spatial-Temporal Color Synthesis for Enhanced Display Image Quality," SID Symposium Digest Tech. Papers 36, 1112–1115 (2005).

6L. Arend, J. Lubin, J. Gille, and J. Larimer, "Color Breakup in Sequentially Scanned LCDs," SID Symposium Digest Tech. Papers 25, 201–204 (1994).

7D. H. Kelly, "Visual Responses to Time-Dependent Stimuli: Amplitude Sensitivity Measurements," J. Opt. Soc. Am. 51, 422–429 (1961).

8S. J. Roosendall et al., "A Wide-Gamut High-Aperture Mobile Spectrum Sequential LCD," SID Symposium Digest Tech. Papers 36, 1116–1119 (2005).

9M. J. J. Jak et al., "Spectrum Sequential LCD," SID Symposium Digest Tech. Papers 36, 1120–1123 (2005). •

 

fig_7_tif

Fig. 7: The M/C spatial mosaic for the STColor configuration of Fig. 4, along with the spatial resolution (Nyquist limits) of the mosaic, and the associated fixed-pattern noise are shown.

 


Louis D. Silverstein is the Founder and Chief Scientist of VCD Sciences, Inc., 9695 E. Yuca St., Scottsdale, AZ 85260-6201; telephone 480/391-1326, fax 480/391-0186, e-mail: lou-s@vcdsci.com. He is a Fellow of the SID.