Resolving Resolution

Accurate metrics for 3-D displays will be more difficult to determine than those for 2-D displays. The author suggests it may be necessary to look beyond 2-D and one-eye paradigms.

by Edward F. Kelley

THERE are presently two main competing commercial liquid-crystal-display (LCD) stereoscopic technologies for home television that requires the use of glasses: temporally multiplexed (TM) displays that use active glasses and spatially multiplexed displays that are often called patterned-retarder (PR) displays that use passive glasses. There has been quite a bit of discussion about the resolution of these displays on the Internet, where the claim is often made that PR displays have only half the resolution of TM displays because for PR displays each eye only uses half of the input image for stereoscopic imagery. This article examines this claim and attempts to understand what our eyes see in terms of resolution for the PR display compared to that of the TM display. (By "our eyes," the author means the human visual system, including the brain.¹) This paper does not discuss other features of these different technologies – only resolution, and only resolution for the presentation of stereoscopic imagery. The use of these displays for two-dimensional imagery is not discussed.

TM Displays

Some temporally multiplexed LCDs might present information at a refresh rate of 120 Hz, but with black frames temporally interleaved with the imagery: left eye, black frame, right eye, black frame. The active glasses are synchronized with the left-eye/right-eye sequence as each eye's information is displayed. Thus, each eye would receive 30 Hz and both eyes combined would receive imagery at 60 Hz. Each eye sees the entire 1920 x 1080 full resolution of the input stereo imagery. Newer TM LCDs employ 240 Hz (left, black, black, black, and black, black, right, black), resulting in 60 Hz per eye.

PR Displays

In the PR displays, each horizontal line has the opposite circular polarization, and the passive glasses have a left-circular polarized filter for the left eye and a right-circular polarized filter for the right eye. In its simplest manifestation, here is an example of how the PR display generates an image: The left-eye image can be composed of the odd lines from the left-eye input image, providing 540 lines with information content interleaved with black lines. Then the right-eye image would be composed of the even lines from the right-eye input image, resulting in 540 lines also interleaved with black lines. The information lines of the left-eye image fall along the interstitial black lines of the right-eye image and vice versa. This results in a two-eye image of 540 lines from the left input signal interleaved with 540 lines from the right input signal – a spatially multiplexed display. Each eye receives a 60-Hz image at the same time; i.e., each eye sees half the pixels available in the input, but the combined eyes see full resolution (1920 x 1080), and each pixel is addressable in the combined image. Note that if any averaging is employed that combines the horizontal lines of each left- or right-eye image in some way, then there may be a resulting loss of resolution. This will depend upon the manufacturer. If the default mode of operation of the PR display is producing some kind of strong averaging of the horizontal lines, then an update of the software that runs the display may be required to provide an improved resolution without significant processing (this was true in the author's case).

For two-dimensional viewing not requiring glasses, both display technologies are full-high-definition (HD) resolution at 1920 x 1080. However, various experts have claimed that because of this spatial multiplexing in the 3-D mode, the vertical resolution (horizontal lines) of PR displays is one-half the vertical number of pixels – half the HD resolution. On the surface, this seems to be a reasonable argument. Under what conditions is the claim true and are there any conditions under which it is not?

Evaluating Displays with Blinders On

Some individuals have attempted to evaluate the resolution of PR displays by viewing them with one eye and judging them accordingly. In such a case, a person would see an interstitial black line interleaved with the 540 lines of visible image (one line of image then one line of black), whereas the other eye would reverse the interleaved configuration; hence, the claim of half-resolution. Is using two-dimensional resolution patterns and one eye a fair evaluation? The equivalent for TM displays would be to evaluate the 120-Hz temporal performance with only one eye and judge them on that basis. This would result in pronounced 30-Hz flicker with the measurement being made with only one eye. Would that be fair?

The answer to both questions is no. We have two eyes and it would seem that the performance of these displays should be evaluated on the basis of a two-eye-based metrology. However, that kind of metrology has not yet been fully developed, for example, within standards groups such as the International Committee for Display Metrology (ICDM) in its production of the Information Display Measurement Standard (IDMS).²

Criteria for Fair Evaluations

In attempting a comparison of different technologies, it is important to remember that display settings can dramatically affect measured and perceived resolution. Contrast, sharpness, luminance, gamma, etc., can affect the perception of resolution comparisons between displays. Because of this, it is critical that the displays compared must present static patterns in their 3-D mode and that they present them in the same way, as much as possible – they must look the same to the extent possible. Consider the faces pattern in Fig. 1 for setting up displays supplied with the printed version of the IDMS.

Fig. 1: These patterns of faces were modified for (2 x 960) x 1080 use with stereoscopic displays.

The faces can look fine in one display and noticeably incorrect in another, yet the colored ramps can be deceivingly similar. Thus, before we even attempt to compare stereoscopic displays and their resolutions, we must ensure that the displays render such patterns as similarly as possible: We should set them up so that the colors and the faces look the same on both displays. That is sometimes hard to do. The perception of resolution can be especially affected by sharpness settings. The appearance of dark text on gray or colored backgrounds can show sharpening artifacts that contribute to the perception of increased resolution. There are also special patterns that emphasize sharpening artifacts, but they must be tailored for stereoscopic displays and evaluated using a 3-D display mode.

Image Examples

Figure 2 shows a single frame from the Disney movie Tangled.³ If we focus attention on an isolated highlight in the right eye of the main character, Rapunzel, Fig, 2(a) shows the stereoscopic (3-D) image in the PR display, Fig. 2(b) shows the 2-D rendering also in the PR display, and Figs. 2(c) and 2(d) show magnified views of the highlight.

Fig. 2: This raw photograph of a movie frame shows the 3-D PR image without glasses (a) and the 2-D image (b). The focus is on the specular highlight (c) and (d) in her right eye.

Figure 3 displays the comparison details. Figure 3(a) shows the manually converged image in the PR display, which is how it is seen with two eyes from an optimal distance from the display. (The manual convergence is achieved with common image-processing software by cutting and pasting the rows together to obtain a properly converged highlight.) Figure 3(b) shows the same 2-D image of the highlight shown in Fig. 2. A comparison between the two shows that half the resolution has not been lost. Figure 3(c) shows what the highlight would look like if the PR display were exhibiting half the resolution. Clearly, this is not what we observe in Fig. 3(a). Figure 3(d) shows the right-eye image of the TM display, which is effectively equivalent in resolution to the 2-D image in Fig. 3(b).

Fig. 3: A comparison of PR and TM displays shows (a) the manually converged image of the PR display (what we see at the optimal viewing distance), (b) the 2-D image of the highlight from Fig. 2, (c) a manually converged image that exhibits half-resolution that is artificially made from the image in (a), and (d) the right-eye image from the TM display.

Differences in the appearance of images rendered by the PR and TM displays at this magnification, such as the presence of larger black gaps between the horizontal lines of the PR display, may be attributed to differences in the pixel layouts between the displays that result in different fill factors. The fact that Figs. 3(a) and 3(b) are not identical would indicate that this manufacturer is using some kind of processing, which may reduce the effective resolution slightly, but certainly not by half. However, each line in Fig. 3(a) has a different number of pixels contributing to the highlight, whereas a half-resolution rendering would be like Fig. 3(c) with line duplication.

In some cases, a likely problem with people evaluating the resolution of the PR display may arise from being too close to the display. Depending upon the acuity of our vision, if we are closer than three screen heights (see textbox⁴) from the display surface, then our eyes do something we do not realize: They will converge the interleaved horizontal lines together, perhaps by one eye moving up or down a small angle on the order of a minute of arc, a phenomenon similar to the nulling of small vertical displacements due to vertical phoria resulting from extra-ocular muscle imbalance, or this line convergence may be a cortical result – an area for further research. The author favors optical convergence as the answer because when there is any head tilt, the eyes must both horizontally and vertically converge the stereoscopic images, and with a slight head tilt, our eyes readily make the vertical adjustment without discomfort. Additionally, the converged lines are slightly brighter than the individual lines, but this is difficult to see because of the non-linear response of the eye (a factor-of-two luminance increase appears approximately 26% brighter to the eye based upon a lightness calculation). This convergence of the horizontal lines leaves black lines interleaved between bright lines (interstitial black lines), which does reduce the effective resolution of the display by half. The resulting image appears similar to what we see when using only one eye with the black interleaved lines. This line convergence with an accompanying loss of resolution occurs when we are closer than the optimum viewing distance for media imagery.

When we back far enough away from the PR display surface to reach the acuity limits of our vision, then our eyes do not converge the horizontal lines and the observed image appears without the interleaved black lines.

These effects are difficult to observe without practice. You can move in close to the PR display and see those black lines clearly, but when you try to take a picture of the screen at a similar distance without any polarization in front of the camera, you do not see the black lines! This fact proves that the interstitial black lines that we observe when close are an artifact of our eyes vertically converging the bright lines together either optically or cortically. In the case of the optical vertical merging of the lines, the eyes do not have much of an angle to adjust. Horizontal angular adjustments to render objects properly are much larger. The TM displays can be viewed closer than this ideal distance without such problems and maintain their full resolution.

But aren't we losing a lot of information by eliminating half of the pixels? Perhaps not. Of necessity, there may be quite a bit of redundancy in the 1920 x 1080 rendering for both eyes. Given any single object that we look at in our artificial 3-D space, the rendering of that object is usually almost the same in both eyes. It is the retinal disparity between our eyes that provides the impression of three dimensions. Once we fixate upon an object in our 3-D view, if each eye observed a substantially different rendering of that object, then we would experience binocular rivalry and it would look strange or even create discomfort if the area over which the difference occurs is large. The 3-D effect is produced by the horizontal retinal disparity and different vergence of our eyes on each object in the scene. It is not produced by a single object having different renderings and textures for each eye. Thus, the combining of the left and right eyes in an interleaved fashion does not make most of the scene half-resolution provided that we navigate that scene from an ideal distance or farther so that the horizontal lines do not merge together. For the cases of small surfaces and objects where each eye can see different renderings, such as some glossy surfaces, more vision research will be needed to see how the stereoscopic rendering is affected by the PR display interleaving. Usually such areas are small and any interleaving disparity may not be objectionable or noticeable.

The author and a few friends compared several 3-D Blu-ray movies simultaneously on the PR and TM displays (by means of a splitter) while sitting at the correct distance from each display so that the pixels were at the limit of resolution (3.2+ screen heights). If the displays were adjusted to look the same as explained above, then the quality of the images as far as resolution is concerned appeared to be the same, full-HD resolution (1920 x 1080) in both displays. These simple observations of 3-D movies, of course, do not constitute carefully controlled vision studies.

Resolving the resolution issues in stereoscopic displays is not going to be as easy as it was for two-dimensional displays and will require further considerations, measurements, and vision studies than what have been presented here. However, what we have seen through this discussion is that two-dimensional and one-eye arguments may not be the correct way to proceed. The metrics for stereoscopic displays may need more refining through careful metrology, and the evaluation of resolution may require further vision studies.

References

¹Many thanks to Dr. Louis D. Silverstein for his help in getting the vision terminology correct and for a number of other comments that were incorporated into this article.

²The Society for Information Display's Definitions and Standards Committee oversees the International Committee for Display Metrology – see www.icdm-sid.org.

³Tangled, by Walt Disney Animation Studios, distributed by Walt Disney Studios Motion Pictures, © 2010 Disney Enterprises, Inc. This is a Blu-ray 3D movie. We used a frame from chapter 8 of 13 at the approximate time of 1:00:05; this is where Rapunzel is saying "easy." The exact frame is important because the highlight changes from frame to frame. As her face approaches a static position, you will note her eyes jump to her right a little (the viewer's left); we use the sixth frame after the jump. If you go to the seventh frame you will see artifacts of the scene transition in the image.

⁴The Encyclopaedia of Medical Imaging, H. Pettersson, ed. (Taylor & Francis, UK, 1998), p.199. •