Developing the First Commercial Product that Uses Multi-Touch Technology
The first commercial product to use multi-touch technology – JazzMutant's Lemur music controller – came to life in 2004. Through the story of the development of the technology, this article draws the outlines of the current state of the art in the emerging realm of multi-touch technologies. In particular, this article focuses on passive-matrix multi-touch technology.
by Guillaume Largillier
START-UP COMPANIES come in three types: those that open new market opportunities for emerging technologies, those that create or apply new technologies to improve existing products, and those that innovate by introducing a new product concept meant to satisfy the needs of an emerging (and usually fast-growing) market. Before it started offering multi-touch technologies to OEMs through component or IP licenses, JazzMutant (now renamed Stantum) fell into the last category. The company's initial motivation was to develop a new generation of input devices for real-time music and media applications that would overcome the numerous limitations of existing control devices. From this starting point, JazzMutant developed the world's first commercial product to utilize multi-touch technology.
New Approach Needed
The democratization of personal computers has deeply changed the way we work and live in the past two-and-a-half decades. This phenomenon has been even more visible in the domain of music production. The music-software industry is growing amazingly fast, dynamic, and highly competitive. For the past 15 years, every music trade show has yielded batches of new software tools and powerful virtual instruments. As a consequence, today's serious hobbyist making music at home on a laptop is much better equipped than many hardware-based professional studios of the mid-1990s. Even so, the key issue in 2002 when JazzMutant launched this project was that general-purpose input devices (e.g., the mouse and keyboard) were totally ill-adapted and unsuitable for music production. A mouse and keyboard are designed for iterative sequences of actions (typewriting, pointing, scrolling, etc.), whereas music production requires manipulating a plurality of parameters at once. A musician or sound engineer must simultaneously control numerous parameters in real time, regardless of the stage of production.
For this reason, music-hardware manufacturers have developed different types of dedicated input devices, called music controllers, each of which presumably fills a function that was painful to manage with a mouse or a QWERTY keyboard. Most music controllers available on the marketplace prior to 2004 were – and to a large extent, still are – simple extensions or literal imitations of the devices used in the analog world. Indeed, the most popular controllers were either master keyboards (a device that looks like a piano) or control surfaces that imitated an analog mixing console [a large table-sized panel featuring a plethora of sliding controls (faders) and rotary knobs].
Such devices do not require any "conceptual" adaptation effort from the user, since they reproduce what was used in the past. However, these dedicated controllers suffer from their lack of versatility, lack of modularity, and lack of visual feedback. These controllers are so inflexible that rather than using them, most musicians kept working directly on computer screens with mice and QWERTY keyboards, regardless of how unintuitive and ill-adapted they were. Our ambition was to overcome this issue in order to bring music-controller hardware that embodies the same endless innovation that characterizes music software to the fingertips of musicians.
The original product concept was a touch-screen-based controller supplied with a user interface (UI) software editor. This would give each user the ability to design his/her own set of interface layouts that exactly fit his/her needs, his/her favorite software, and the way he/she believes a controller should work. For instance, at one instant, the user could be mixing several sources with virtual faders. The next instant, he/she could be controlling an equalizer or a software synthesizer using other kinds of UI objects. For music applications, an interface that cannot manage several functions at once is ultimately useless. This means that a tactile interface for music applications must be multi-touch (Fig. 1).
The main bottleneck at the time was that the only multi-touch technologies available (such as those from Tactex and FingerWorks) were not transparent. In other words, they did not provide any visual feedback, which was a major drawback in our project. At the same time, none of the numerous touch-panel technologies available [analog resistive, surface capacitive, infrared, surface acoustic wave (SAW), etc.] was capable of detecting and tracking more than one finger at once, even though the first developments in this field date from the early 1980s. After carefully reviewing the state of the art, we quickly concluded that to make a real product out of the original concept, we would first need to develop a transparent multi-touch sensing system that enabled the user to manipulate several UI objects at once.
For our purposes, multi-touch technology means the capacity to detect and track the activity of at least two contact points at the same time (i.e., during a single data-acquisition frame); preferably, this number should be unlimited. This differs from some improper (yet common) definitions of multi-touch that refer to the capacity to sense alternately two input/contact points.
Early Development: Making a Choice
In order to better understand our development process, it is important to specify our product requirements.
We needed our multi-touch display to combine both touch input and visual feedback. So this ruled out multi-touch capacitive touch pads (i.e., those from Tactex, FingerWorks, etc.) that could not be placed on top of a display to allow direct interaction with the user interface. This also disqualified optical systems (such as FTIR or video tracking) because their form factors prevent them from being integrated into portable devices such as laptops, PDAs, smart phones, game pads, or music controllers. Since then, such technologies have been integrated into various multi-touch surfaces such as MS surface, Mitsubishi Electric Research Laboratories' diamond touch,etc. Scalability was very important in order to drop the modules into a complete product line (portable controllers, desk-size controllers, etc.).
In addition to being capable of detecting as many contact points as the user has fingers, the product required the touch screen to have a very low latency because rapid response is a very critical aspect of audio applications. The tech-nology also had to be rugged and reliable for any conceivable use such as outdoor venues and smoky and/or wet locations such as clubs, etc.
Similarly, musicians need a touch technology that behaves in the most accurate and reliable fashion even if their fingers are wet or sweaty – this makes many other touch technologies (SAW, for example) unsuitable for such applications. Finally, we did not want to wait until the middle of the 21st century to see our products on the marketplace; in other words, we wanted the technology to rely as much as possible on an existing, mature industrial process. These preliminary specifications greatly dictated our R&D strategy and, moreover, it informed the choice we made and our orientation.
In theory, there were two types of touch-sensing techniques that potentially could deliver multi-touch: active matrix and passive matrix. An active-matrix touch screen is made up of an array of "taxels" (tactile elements), i.e., independent touch-sensitive dots. Such dots can be active components of any sort, such as transistors or photodiodes. We quickly realized this approach would require an unnecessary, over-complicated, and expensive manufacturing process for an uncertain result. As far as we know, there is no product of this kind available in the marketplace as of the date of this article, although several TFT-LCD manufacturers have recently displayed small-sized active-matrix touch-display prototypes.
The alternative is passive matrix. A passive-matrix touch screen comprises a grid arrangement of two transparent conductive layers. The bottom layer features an array of rows, whereas the top layer features a perpendicular array of columns, both layers being spaced from each other with an insulating material. The conductive layers can be made with a traditional process (patterned ITO or thin metal wires). Matrix touch screens such as digital resistive or projected capacitive have been known for years – for a comparison of these two technologies, see Table 1. However, such touch screens are not multi-touch capable by themselves. Indeed, it is a prioriimpossible to detect several contact points located on the same line or column, unless a particular method is used to control and drive them. Herein lies the secret of passive-matrix multi-touch technology, and this is where we focused our research from 2002 to 2003.
Multi-Touch "Core" Development
The development of our multi-touch tech-nology began with a conventional 5-in. digital-matrix touch panel featuring a grid of 20 rows and 25 columns, which was fairly easy to obtain. Although not as popular as analog-resistive sensors, digital touch screens had been commonly used for years in application areas such as industrial controls. The problem with passive-matrix touch screens is that the nodes are not independent cells; a node is basically the intersection of two perpendicular conductive tracks. Using conventional controllers, it was impossible to activate two intersections at the same time (Fig. 2).
The main purpose of this first prototype was to develop a new scanning method that would allow us to read the individual state of each node of the matrix, thus making each intersection of the grid work as an independent sensor. Usually, to read the data of a passive-matrix touch screen (digital-resistive or projected-capacitive), the conventional process consists of energizing the columns one by one and then scanning each of the rows to see if there is any change in the row's electric characteristic (contact or change in capacitance). However, this method does not reliably allow the detection of several contact points on the same row or column due to the interaction between the points.
We designed a new controller during the summer of 2003 capable of delivering an image of the entire matrix 20 times per second, giving the state of each intersection. Our controller set all the columns – except the one being energized – to high impedance. In this way, current could not propagate through the inactive columns and affect the measurement. Then, while a selected row is being measured, the other rows are connected to ground. This method allowed us to measure the state of each node of the matrix independently.
Although this first prototype could track several touches at the same time, it was far from satisfactory. For instance, from time to time, "ghost" touch points appeared in some areas of the touch screen where there was no finger at all. In some cases, contact points hid each other because the resistance of the transparent material (ITO) composing the columns and the rows increases proportionally to the length of the tracks. Thus, the potential measured at the lower left corner of the sensor was greater than the potential measured at the upper right corner. As a consequence, the initial touch point tended to absorb a large part of the electrical potential of the driven column. The potential measured at a second point was therefore too low to be detected.
The solution lay in using an adjustable voltage comparator at the output of a row to determine whether the voltage observed was sufficient to be recognized as resulting from the action of a finger on the touch screen at the location being scanned. The reference value of the comparator (comparison threshold) is decremented at each row measurement. Thus, the comparison values of the last rows are lower than those of the first rows, which allow a contact point located at the lower left or the upper right to be detected in the same manner. After several further improvements to the controller, we had a reliable multi-touch system by the end of 2003.
This method of detecting and following the activity of several touch points (fingers, styli, etc.) was detailed in our first patent application in France in early 2004 (see U.S. Patent Application 2007/0198926 for the latest English-language version.) At this stage of development, we had proof that it was possible to get reliable multi-touch data from a passive-matrix sensor, but were still far from having a suitable system for our product.
Integrating Hardware and Software
The next step entailed improving the controller even further. We wanted a multi-touch controller scan rate of at least 60 Hz because real-time music applications require an instantaneous response with imperceptible delay. We also needed a matrix touch screen featuring a much higher resolution as well as a low activation force. For our application, we targeted a minimum resolution of seven rows per centimeter, or about 50 nodes per square centimeter. This ensured that a light finger touch would cover a minimum of about eight cells. Note that the actual touch-point resolution depends on the size of the touch area, which is affected both by the size of the finger and the touch pressure. On the final product, the effective resolution is typically two or three times greater than the matrix resolution.
Anyone involved in touch-screen design knows that developing a suitable sensor for a particular application is a matter of balancing numerous parameters: the spacing and location of the silicone dots, the thickness of the PET, the finish coating, etc. Since we decided from the beginning to use an industrial-class high-brightness TFT display featuring high contrast and a wide viewing angle, the optical aspects were much less important to us than the mechanical characteristics. This decision significantly reduced the optical constraints. After trying several sensor designs (and manufacturers), we finally developed a well-balanced touch screen that enabled us to release the first fully functional prototype of our first product, the Lemur music controller (Fig. 3), in July 2004. In July 2005, long before multi-touch technologies became a popular topic of conversation, we officially released the Lemur.
Since then, apart from further improving the multi-touch sensing technology and introducing a second-generation product – the Dexter, a digital audio workstation controller (Fig. 4) – an important part of our research and development has been dedicated to software development and UI design. Multi-touch technologies promise to bring new ways to interact with computers and electronic devices, making the user experience much more playful and productive at the same time. However, making effective use of multi-touch technology definitely requires a new approach to interaction strategy and user-interface design. Multi-touch interaction is not only a matter of hardware, but software as well.
Beyond improving our own products, we have developed some general-purpose multi-touch interaction techniques. As a result of this R&D, we introduced the first prototype of a multi-touch-enabled tablet PC at the SIGGRAPH 2007 conference. This proto-type featured a vector-graphics application that enabled the user to quickly sketch drawings in an extremely intuitive and user-friendly fashion. The UI of this application combines simultaneous finger touches and stylus input.
Conclusion
Multi-touch display technology holds great promise for future product development. By focusing on simplicity in the manufacturing process, cost efficiencies, and how we could use existing technologies, we were able to bring the Lemur music controller – believed to be the world's first commercial multi-touch display product – to market in just 3 years. •
Fig. 4: JazzMutant's most recent product introduction, the Dexter digital audio workstation controller.