Unlocking the Interactive Capabilities of Large Outdoor Displays

If digital signage is to achieve maximum effect, it must first grab the attention of the viewer. It can be fun too, but fun alone is not enough, since advertisers are in the business for one purpose only – to sell more product. Here, we describe an experiment in attention grabbing.It works. It's fun. But how will it translate into a business advantage?

by Daniel Michelis and Florian Resatsch

TECHNOLOGICAL DEVELOPMENT is changing the ways viewers interact with displays and their content. In particular, the increasing availability of wireless and sensor technology makes a broader range of human–computer interaction possible. The development of improved video- and audio-tracking not only supports object tracing, but also enables the reception of wireless, digitally augmented information about objects. By using these new sensor technologies, we can develop new forms of interaction in "invisible" environments, using "smart" inhabitable spaces to interact with context-sensitive applications.

Screen size is changing simultaneously in both directions – the smallest screens are getting smaller and the biggest are getting bigger. Small displays of ever-increasing quality are being integrated into various objects encountered in our normal daily lives, whereas large displays are increasingly being used in outdoor areas, enabling so-called "public viewing." The ubiquitous availability of mobile devices and their networked communication technology has completely freed up the ways in which we can interact with displays. For example, interaction is no longer linked to a specific locality where a keyboard or touch pad might be located, but now can happen wherever a display can interact with a mobile phone. Furthermore, the physical form of the interface is changing and now can adapt itself to the user's requirements. Apart from conventional desktop interaction with screens, computers have become a ubiquitous part of our daily lives. The interaction between man and machine takes place during our leisure time, in art, and in culture. The central function of most interfaces is no longer limited to supporting the user in performing task, but rather has shifted toward interfaces that are pleasurable, aesthetically pleasing, expressive, creative, culturally relevant, and even provocative.

---

Daniel Michelis is a Research Associate at the University of the Arts in Berlin, Germany, IEB, Hardenbergstraße 9A, 10623, Berlin, Germany; telephone +49-30-726-29-83-24, fax -83-9, e-mail: michelis@ieb.net. Florian Resatsch is a Researcher at the University of the Arts Berlin, Berlin, Germany.

Based on the technological developments mentioned above, we will now present and discuss an interactive media facade in Berlin conceptually developed and evaluated in 2006. The interaction design was based on an attempt to introduce known and familiar interactions to the world of digital media in public spaces. As a result, the concept of using distorting mirrors, which are used in many fairs and amusement parks, was chosen as the underlying interaction component. These distorting mirrors are pleasing to many people because their functional principle can be discovered intuitively. We will show why "magical" mirrors should be considered to be potentially interesting interaction components for passers-by.

Human Love Affair with Reflection

Long before the development of the electronic display, the mirror was used as a medium for visual simulation. Virtual worlds have been simulated for hundreds of years. The term "virtuality" was originally defined as the opposite of reality – the virtual distinguishes itself from the real and denotes a fictional world. For centuries, the mirror was the central instrument for the creation of a virtual world, and the creation of illusion was its inherent function. The virtual images that arise through reflections reflect back the real image fictionally.

Today, this ability is emulated by digital media technologies. Through the development of photography, film, radio, television, and computers, today's world is inundated with images that imitate the virtuality of the mirror's image.¹ With the introduction of new media, all forms of representation developed by man in the past 5000 years can be translated into digital form. Consequently, a variety of digital techniques for visual stimulation has taken root that operates within the tradition of past media. They consistently fulfill the same goal by satisfying the needs of the viewer for visual stimulation. The content and purpose of the presentation have not changed; rather, the technique and form have.

Through the use of digital technologies, new opportunities arise to satisfy man's age-old desire for experiencing fictional worlds. Independent of content and fictional histories, the desire for immersion is at the fore. In striving to experience fictional worlds, humans search for an experience similar to that of jumping into a swimming pool or the ocean – the experience of being completely submerged in another reality. We enjoy leaving our familiar world behind and exploring the characteristics of a new environment. We want to swim around and see new possibilities arise. The feeling of experiencing a fictional virtual place is, according to Murray, "pleasurable in itself."²

Previous Attempts

An early example of a working approach of using new media to create the experience of immersion was the ALIVE project of the MIT Media Lab. The core component of the project was a "magic mirror" in which viewers found themselves next to a comic character who directly interacted with them. The figure followed any movement of the user and appeared to be alive on its own within the mirror.

The digital mirror of the ALIVE project was an interface that inserted a picture of the user into a virtual world. The function of the mirror was implemented in hardware; in this case, a camera that recorded the picture and the screen that visualized the user next to the comic figure. In the development of new media, the camera is the central interface for users and the screen is the visualization component. A more recent approach was the Augmented Mirror project developed by the Ambient Intelligence Group at MIT. The aim of the project was to develop intuitive interfaces using the mirror as the primary interface. The research goals were described as "investigating several applications of the technology, including communication with mobile phones to collect behavior data, face recognition to identify members of a family, as well as clothing-store and instant-messaging applications."

The production of illusion in the 20th century was primarily overtaken by mass-media photography, film, and video, which were then displaced by the computer with its display that Manovich also designates as an "illusion generator." In the digital age, analog glass mirrors have been displaced as a medium of visual production. In their place stands the digital screen as the dominant interface between man and computer.³

The Investigation

Digital mirrors used in the interaction with passers-by are also central to the media facade described in this article. The aims of the installation were to encourage a debate on digital media and to research the possibilities and limitations of technological interaction. The major elements of the installation were four life-sized displays in the building's casino (Fig. 1), three projectors in the foyer, and background projectors in the tower, foyer, and casino (Fig. 2).

Interactive Outdoor Screens. The four life-sized displays on the ground floor are the central point of the installation, which ranges along an entire side of the building (Fig. 3).

The displays could show full-color pictures, but the cameras are only black and white. Therefore, the color blue is the basic color of the whole facade. Once people start interacting with the installation, the projection in the tower, the foyer, and the casino change to a warmer color, i.e., from blue (cold) to red (warm). The main displays do not change and stay blue (Figs. 2 and 4).

As the main interactive component, the screens invite the observer to play in front of and interact with four different mirror effects that we called aura, progression, luminary, and flexibility (see, for example, Fig. 5).

Aura: The mirror is loaded with virtual energy. An aura of this energy develops around the viewer in front of the mirror. It reacts to movements with flame-like clouds that surround the viewer like polar light.

Progression: Fast-growing flowers follow the movement of the viewer and grow over the entire mirror. Once the viewer gets out of sight, the plants shrink and disappear.

Luminary: The building is covered by a pulsating star of bright zeros and ones. Once the viewer steps before the mirror a current of zeros and ones develops around the viewer's body. The numbers react to the movements of the viewer, so the viewer can direct the cloud number in the mirror.

Flexibility: A magic ribbon moves over the mirror. The viewer can take it in hand and write artful figures on the mirror and the viewer's mirror image (Fig. 6).

Projections in the Foyer

The projection screens above the main entrance of the building act as a picture gallery of still photos taken in front of the mirrors. The aim of this picture gallery is to attract the attention of people from a distance (Fig. 4). The pictures build a colorful contrast against the color gradient of the building. This additional part of the installation follows the theory of media richness and reach. The large projections show passers-by that something is going on at or around the building.

Technical Solution

The actual installation is based on an information-technology infrastructure installed throughout the entire building. The four mirrors, which are the main interaction component, all face the Rosenthaler Strasse, a street in the main tourist area around the "Hackesche Hoefe" in central Berlin. Every mirror set consists of two single high-resolution digital-light-processing (DLP) displays showing a single life-sized picture. A camera is installed between the DLP displays and the glass facade of the building to create an image of the person in front of the screen.

The fastest moving spot in the camera picture is tracked and integrated into the interaction. The captured picture is then transformed with the respective illusion and displayed on the screens. The software to generate the visual illusions was based on a system developed by Ars Electronica Futurelab and was programmed by the Berlin-based media company DEON.

Contrary to classic systems engineering in which requirements, specifications, and architectural and operational implementations are key components, we created an art installation based on a technical prototype to experiment with user reactions. Based on the usage numbers, we have demonstrated that the approach was very successful at attracting the attention of passers-by compared to static posters or standard urban-screen-content provisions.

Usage Patterns

Based on our experience with previous implementations of services advertised in public spaces, most of them fail to attract the attention of people in the street. Many available digital services, such as mobile city guides, are only used by those who already knew about their existence. The visual illusions we used exploited the basic interest of humans in the mirror. Once people see their distorted and enhanced reflection in the mirror, they start acting.

This has positive implications on the cost of providing content. Most screens used in urban areas need intensive content management with updates throughout the day to sustain the level of interest of the viewers. In this case, the attractor is the person himself or herself, and the interaction leads to an increased number of users looking at and using the displayed content.

In November and December 2006, during a series of 13 experiments, a total of almost 5000 passers-by were observed. Within the scope of the experiments, for example, the reaction times of the particular effects were altered. By slowing down the reaction time of the effects by 1–2 sec, the behavior of the useralso changed since they no longer obtained any immediate feedback about their movements.

In this series of experiments, the number of displays operating was varied from one to four. Additionally, the number of different effects in parallel use was also varied. The goal of this variation was to examine what role the choice of factors play in usage behavior. Here, the complexity of the effects, the continuity of interaction – i.e., the regularity of the effects and the playback of the recorded image – were all varied. In a further part of the investigation, the mirroring effects were modified. Instead of playing back the recorded "mirror image," all that was shown was the representation of the effect that reacts to the movement of the viewer. Finally, the actual mirror image and the interaction effect were played back on all four displays.

Initial Results

The results of the investigations are currently being assessed and they are being further expanded by means of a survey. However, the initial results are as follows. In interacting, groups exhibit a very special dynamic. If a group of two or more people pass by the public displays, most often, only one person becomes the active one who starts the interaction. This person pauses, then actively participates in the interaction, drawing the attention of the other group members toward himself or herself. Typically, the others initially stand by and hesitantly observe the interaction. In the parts of the investigation in which more than one display was available, the other members of the group began to use the free displays and explore the interaction possibilities. Little by little, the entire group participated in the interaction. In the cases in which there was only one display, the group became impatient, and the most active person was encouraged to rejoin the group and move on.

We also measured the effect when passers-by were no longer shown the mirror images. In clear weather, roughly 30% of passers-by used the Magical Mirrors installation when it was functioning normally. After turning off the mirror images, fewer than 5% of passers-by used the displays. Thus, it appears that dynamic interaction with the display increases usage by 500% or a factor of 6.

If the visual illusion (in other words, the reflection) is enough to make people aware of the screens and generate an interest that is high enough to stay and play, the content may change slightly into information. This means that in future applications, once the viewer interacts with the display, the image on the display could change to content such as advertising or information

Quoting Donald Norman,⁴ an application today should follow three design axioms: simplicity, versatility, and pleasurability. Simplicity means that the complexity of the added service is that of the task, not of the tool. For the mirror approach, people can interact with the service very easily, providing location-based information. Versatility requires that a property of the application is to encourage novel and creative interaction. The system allows people to interact very creatively with the information in a manner of their own choosing. This means that passers-by start interacting with the dynamic "information" that the display would normally provide. Finally, using such products should be enjoyable. While Norman uses these axioms for information appliances, they are applicable to any kind of interaction device.

We have shown that a system designed in accordance with Norman's axioms increases the number of eye contacts with the screens per day. When used for advertising, the interaction should be very unobtrusive and should not disturb the usage of the services. The advertising can be placed within the interaction, letting people play with logos or images of products instead of the artistic "magical mirror" interactions we tested in the prototype. Selling more products directly can be achieved through the integration of mobile-phone interaction or direct Internet access to the screens.

The company LocaModa^™ from Boston, Massachusetts, U.S.A.,⁵ has developed an interaction system allowing users to call phone numbers displayed on the screen and interact with the content on the screen. For example, a user can call a number to cause more detailed information on a particular product to be displayed. This form of interaction allows further improvements in the future, such as the use of SMS text messaging to order a product or to receive personalized information.

The advantage is clear, but if digital display screens are to be used effectively as advertising media, they must be configured to be used in ways that grab the attention of their target audience. In today's fast times, attention grabbing is one of the most difficult challenges, and providing a means to interact with the screen is a solution.

Conclusion

Although many interactive technologies have been introduced to markets all over the world, many innovations fail.⁶ Many failures occur because people are more likely to accept an innovation if they do not need to drastically change their previous behavior. An innovation tends to be successful if the product itself changes, but the way in which it is used follows familiar habits and patterns. The question we must therefore address is "How do we proceed in developing human-centric applications in ways that do not force people to drastically change their behavior?"

Our installation showed that people are very likely to use new ways of interacting with displays in public spaces if the method of interaction is based on the familiar. By using advanced camera and software technologies, we have created the illusion of a mirror, but with additional and more versatile functionality by which viewer will see more than his or her own reflection and can enter a virtual world and directly interact with it. The media facade is an example of a "Mediated Space" that attracts passers-by and motivates them to interact with the mirrors. The interaction was designed as much as possible to be intuitive and natural. A digital illusion generator can meet the demands of fascination, curiosity, and manipulation in ways that static mirrors and conventional advertisements could never have. The possibilities of this technology are something new; the desire of the human mind to travel into fictitious worlds through visual illusions is the same. As we have shown by the experiments presented above the use of mirror images as a familiar interaction style can lead to a significantly increased awareness of big displays in public spaces.

Acknowledgment

All photographs courtesy of Verena Eidel.

References

¹D. Michelis, H. Send, and T. Schildhauer, Locative Data, Prototype Research with RFID i-com, Vol. 5, No. 2, S. 38-42 (2006).

²D. Michelis, T. Nicolai, F. Resatsch, and T. Schildhauer, "The Disappearance of the Screen: Research on Audible Interfaces in the Ubiquitous Computing Environment," ICMB '05, 262–266 (2005).

³S. Benford, H. Schnädelbach, B. Koleva, R. Anastasi, C. Grennhalgh, T. Rodden, J. Green, A. Ghali, and T. Pridmore, "Expected, Sensed, and Desired: A Framework for Designing Sensing-Based Interaction," ACM Transactions on Computer-Human Interaction 12, No. 1, S. 3–30 (2005).

⁴M-L.Ryan, Narrative as Virtual Reality: Immersion and Interactivity in Literature and Electronic Media (John Hopkins University, Baltimore, London, 2001).

⁵J. Murray, Hamlet on the Holodeck, (MIT Press, Cambridge, Massachusetts, 1998).

⁶L. Manovich, The Language of New Media, (MIT Press, Cambridge, Massachusetts, 2001). •