Technologies and Requirements for Digital Pens
N-trig, a maker of digital pens and an innovator of touch technology, describes what to look for and what to expect when selecting a digital pen for mobile devices such as smartphones, tablets, and subnotebooks.
by On Haran
MANY digital-pen-equipped devices have been introduced recently.1–4 This is, in part, because users are demanding a more capable tablet than one solely designed for media consumption. While the technologies behind these pens vary significantly, they all are subject to varying levels of jitter, linearity, and accuracy, and they all have requirements such as pressure sensitivity and palm rejection that must be addressed in order to provide a feel to the user that is similar to that of pen and paper. In addition, the ideal digital pen should be cost effective and integrate easily into all kinds of mobile devices. Besides enabling content creation and manipulation, the digital pen should complement the man– machine interface by introducing mouse-like capabilities such as accurate on-screen pointing, left and right button clicks, and “hover over” detection not possible through touch alone. Figure 1 shows various examples of content creation with digital pens.
Fig. 1: Digital pens enable content creation such as note taking, document annotating, drawing, and signing.
Digital-Pen Technologies
Digital-pen technologies can be classified as shown in Fig. 2. Device-integrated pens enable interaction with and writing on device screens, whereas offline pens capture inking on paper or dedicated substrates and subsequently transfer the collected data to the devices. (Inking is the industry keyword for the act of transferring user pen gestures into digital representations.) Offline pens require increased cognitive activity on the part of the user and tend to reduce the overall experience to some degree. Touch-sensor-integrated pens interact with the grid of capacitive-touch electrodes that exist on most laptops, tablets, and smartphones, providing simultaneous pen and touch as well as palm rejection and therefore providing a better user interface with the device. Other pen technologies – including optical, ultrasonic, and electromagnetic – require the addition of dedicated sensing components within the device, increasing the device cost, thickness, and power, and thereby compromising the touch integration.
Fig. 2: Digital pens can be roughly classified by the above attributes.
Touch-sensor-integrated digital pens can be further classified as a passive stylus or active pen, with physical operation principles shown in Fig. 3. The passive stylus operates by projected-capacitance (p-cap). In p-cap, electrical signals are driven on a set of conductive transparent electrodes deposited on one axis of the touch-screen sensor, while synchronized sensing of electrical signals takes place on a perpendicular electrodes set. The signal transfer between the axes depends on their “proximity,” which is formally indicated by CJ, their junction capacitor. When an object, such as a human finger, is placed on the device screen in the vicinity of the junction area, CJ> drops by 10–30%, depending on the touch-sensor design, leading to weaker sensed signals and, consequently, touch detection. While it works well for human-touch detection, p-cap poses challenges when it is utilized for passive styli because those employ fine tips (1–2 mm in diameter) to enable visibility of the underlying pen inking. As a result of the fine tip, the relative reduction in CJ is only around 1–3%. This weak effect challenges passive-stylus operation in the presence of user touch, electrically noisy environments, high and low temperatures, and other non-optimal working conditions. Furthermore, it is not easy to distinguish a passive stylus from fingernails and other fine user touches. The usage of passive styli is therefore restricted to less demanding applications. In some cases, powered electronics are used to enhance the weak passive-stylus signal by injecting a synchronized interfering signal to the touch screen to improve detection, but performance and feature shortcomings still exist compared to that of active-pen technology.
Powered active pens13 drive unique modulated signals from the pen’s tip to the horizontal and vertical grid of electrodes. In this operation mode, sensing takes place on both axes simultaneously, with the received signal magnitudes being proportional to the tip capacitance (= proximity) to the electrodes. Signal processing is employed to digest all received signals and extract accurate tip positioning. These modulated content-rich signals are orthogonal to the touch-sensing signals, and therefore active-pen sensing is not confused with touch sensing. Through their modulation, these signals provide additional information about the pen and its state (tip pressure, buttons pressed) and can be captured even when the pen is hovering above the device.
Figure 3 also shows the components of a digital-pen solution for both active pens and passive styli – the pen, the sensor, and the controller. The sensor, which is embedded within the device, senses the presence of the pen in its vicinity, and transfers analog signals to the controller. The controller applies analog and digital processing to the signals; extracts the pen information amidst the surrounding noises generated by the device display, power adaptor, and other peripherals; and forwards “pen reports” to the host device operating system, which acts upon the user instructions and operations embedded in these reports. This process enables pen reports that are more frequent in occurrence compared to touch events, improving the user experience.
Fig. 3: Active pens and passive styli both interact with the touch sensor, but operate differently.
Requirements
Despite the variety of technologies and options available for digital pens, a unified set of requirements can be defined to ensure that users will be satisfied with pen performance. Initial steps were taken by Microsoft, focusing on several mandatory performance metrics.5 A more comprehensive set is discussed below, including automated (jitter, accuracy, linearity, edge behavior, tilt-error, and pen lag) and manual (missing/extra strokes, inking in hover) inking testing, as well as a description of pen requirements beyond inking.
It should be noted that pen performance does not only depend upon the pen but also on the sensor size, position, and design, and on the controller configuration, digital signal processing, and firmware; thus, pen verification should be conducted for the entire device, including pen, sensor, and configured controller.
Achieving high-quality inking is the cornerstone of a good digital pen. Various types of inking tests are discussed below. Automated testing highlights specific inking gaps. The testing is performed using a robot having X-Y-Z axes (see Fig. 4). Prior to performing the tests, the device should be properly aligned on the robot platform. Automated testing can also be applied to other digital pen features, as is shown in Fig. 4 on the right, where a pressure testing apparatus verifies active-pen pressure reports vs. the actual mechanic pressure applied by the tip to the device.
Fig. 4: A robotic apparatus enables accurate verification of inking (left) and pressure (right).
Jitter, linearity, and accuracy are major positioning metrics that are shown graphically in Fig. 5. Jitter refers to the maximal distance between pen reports obtained for a static pen. The robot thus positions a pen statically on the digitizer, and a related application measures the maximal distance between the resulting pen reports. Most pen controllers employ processing techniques to keep static pen reports uniform, hence this test should result with “0” jitter. Industry commonly acceptable non-processed pen jitter should not surpass 0.2 mm (being 2% in magnitude of 1-cm-sized characters).
Fig. 5: Jitter, accuracy, and linearity are compared in terms of physical pen position vs. digital reports.
Linearity is tested by slowly (<5 mm/sec) drawing diagonal lines across the screen in different orientations. The deviations of the reports from the best-fitted line are measured to yield the “linearity.” A pen linearity up to 0.4 mm is commonly acceptable in the industry today. Typical linearity measurements for an active pen vs. a passive stylus are shown in Fig. 6.
Fig. 6: Linearity testing demonstrates a performance gap between active pens and passive stylus.
For positioning accuracy, the test robot places the pen at different points across the device and ensures that the delta gap between the actual placement and the reported averaged position matches the specifications (typically <0.4 mm). In order to be accurate, the controller processing should account for the sensor design and positioning within the device.
The robot then tilts the pen to different angles, while maintaining the same tip-device physical touch position. Ideally, the device should report a steady position. In reality, though, the interaction between the pen and the device is not limited to their physical touch position, and therefore the reported position is biased to the direction of the tilt. Figure 7 shows this tilt-induced error, with the cursor reported at a shifted position due to the pen tilt. Industry acceptable tilt error is 0.75 mm at 30° tilt.
Fig. 7: The tilt accuracy of a digital pen is an important consideration in terms of accuracy and ergonomics.
Edge inking is tested by drawing lines from within the interior of the sensor towards the boundary and in the inverse direction. Figure 8 shows an example of incorrect edge inking, where lines are not well terminated when reaching the device boundary.
Fig. 8: Edge-inking imperfection is shown for lines drawn from the interior of the sensor toward the edges.
Pen “lag” represents the gap between the actual and reported inking in terms of space and time. A cursor lagging behind when the pen is moved quickly illustrates this deficiency. Testing lag includes an optical beam or microphone to monitor the actual inking time, which can be compared to the reported one. Lag exists due to several mechanisms, among which are the limited refresh rate of the controller (usually around 100 Hz), processing applied within the controller, and non-immediate interaction with the device OS and drivers.
Manual content-oriented inking testing addresses inking gaps that do not manifest themselves in robot testing. Figure 9 shows results from a manual inking test conducted by placing a paper and a sheet of carbon copy on the device. The digital pen writes both to the device and to the paper via the carbon-copy sheet. Overlaying the two sets of written elements highlights the inking gaps.
Major inking gaps relate to missing or extra strokes. Missing strokes are those that were conducted by the user, yet not reported at all by the pen controller. Missing a first stroke may occur when controllers do not react quickly enough to the pen presence, as may be the case when they have switched to idle modes to save power or to other non-optimized modes. Extra strokes may occur when the pen is not touching the device, yet is close enough that the sensor acquires significant signals. This problem is referred to as “ink in hover” and results in text elements being erroneously connected. High-quality pens and sensors contain dedicated mechanisms that enable clear identification of whether the pen is touching the device or not, avoiding these kinds of extra strokes. Other non-accurate inking effects may result from inaccurate sensor readings or from non-optimized controller processing.
Manual inking tests will lead to different results based on the user and writing type. The faster the writing, the larger the pen-inking gaps; handwriting inking problems will be manifested differently when a user is working in English, Chinese, or Arabic, and between small vs. large letters. It is thus advisable to conduct this test with different types of characters, written at different speeds, and in different sizes.
Figure 9, right, shows typical inking results obtained using an active pen and a passive (stylus) pen, manifesting the gaps between their operations.
Special inking testing focuses on advanced features – including tip pressure, pen tilt, and tail eraser – available in some digital pens. Tip pressure measurement11 enables users to draw thicker lines when the digital pen applies more pressure to the device. Pen-tilt measurement enables a user to draw thicker lines with a tilted pen, mimicking the action of a tilted pencil. Yet another inking feature is a tail eraser that mimics the pencil’s counterpart. Every provided feature should be calibrated and verified to ensure its operation.
Inking testing in different real-life environments should be conducted. Some digital pens are challenged under moisture, some when the temperature changes significantly, some when noisy power adaptors are connected to the devices and others when a user touches the screen with his finger or palm during the pen operation. Inking-quality degradation under these disturbances is expected to be negligible for high-quality pens.
Fig. 9: At left, an inking test uses carbon copy and paper placed over a display as a reference for digital writing. The center example shows good inking in pink, missing strokes in white, and extra strokes in red. At right, active and passive stylus inking is compared.
Touch-Sensing Integration
In a device supporting touch sensing and digital-pen sensing, integration between these user interfaces is a desirable attribute. The digital-pen solution should be aware of touch events, whereas the touch solution should be aware of digital-pen events. This awareness enables adaptive operation leading to a more robust and rich user experience. Specifically, this capability enables users to interface with a device using both touch and pen simultaneously. An expanded input vocabulary is afforded by supporting unimodal pen, unimodal touch, and multimodal pen + touch inputs. A recent study demonstrated6 the richness of interactions available when compound gestures based on pen and touch primitive input operations (tap, drag, hold, and others) are supported.
Another integration challenge, described below, is known as palm rejection, relating to a natural human behavior that poses a challenge to the touch sensing system6–8 Especially for large devices (>10 in.), the user places his palm on the device as part of the natural writing process. Since this is a non-intentional touch, it is expected that touch systems will correctly identify the palm and avoid reporting it to the device OS. Yet, in reality, under certain conditions, touch controllers fail to classify the palm correctly, as can be tested using large conductive tokens (Fig. 10). High-quality digital pens and sensors, and, specifically, touch-sensor-integrated ones, are able to complement the touch-sensor analysis, ensuring that no false palm reports will be issued. For this integration, awareness of the pen presence should be available when the pen is hovering above the sensor while it is being held by the user.
Fig. 10: Palm-like large tokens are placed and moved on the tested device, and if the device “draws” (as is the case in the figure), it indicates poor palm rejection.
General Pen Requirements
In terms of pen performance, durability, ergonomics, noise, surface friction, and cost are all important considerations.
Durability: The quality of the pen operation should remain consistent over years of operation. To avoid loss of the pen, it is ideal that it be able to be connected to the device, possibly using a dedicated storage cavity or “garage.” Most pens require internal power sources. To avoid user frustration, the pen or controller should alert the user when the power source is depleted. Pens should be robust enough to avoid breakage following a drop from 1 m (representing the fall from an office desk).
Pen ergonomics: The pen should be lightweight and the weight should be well-balanced. Preferably, a device should work with pens of different diameters to optimize the grip for both children and adults. Hiroshi9 found that thick (12 mm) pens reduce muscle load and mitigate fatigue as opposed to thin (8 mm) pens. This finding contradicts the drive to garage the pens in thin devices, which favors very thin (5.5 mm) pens. Ravindra10 found that different pen diameters and shapes lead to different accuracies and speeds of writing; hence, the optimal pen should also depend on the planned usage.
Friction and noise: Some digital pens utilize metallic tips that generate loud noise when used to write on the cover glass of devices. Lieberman7 commented that cultural differences, such as responding to the sound of a stylus used on a screen, will have a dramatic effect on utilization and acceptance of digital pens.
Furthermore, the metallic–glass interface is a low-friction one, with a kinetic coefficient of friction mK<0.2 (dependent upon the glass type and coating), and, as a result, the tip often glides over the device surface in a way the user had not intended. High-quality pens utilize alternative materials and tip designs to ensure that writing noises are kept to a minimum and increase mK above 0.2, providing better control over tip movement. Interchangeable tips are convenient for enabling users to optimize their personal writing experience. mK measurement is detailed in ASTM D1894.12
Writing the Future
As discussed earlier in this article, the functions of digital pens extend beyond writing because the pen represents an alternative man–machine interface (MMI) versus touch. Thus, functions such as “mouse over” can be accomplished by a “pen hover” feature and “mouse right click” may be enabled by introducing mechanical or capacitive buttons on the pen and ensuring that the pen controller will recognize and properly report when these buttons are pressed.
Since the digital-pen sensor and controller are embedded within the device, when a user purchases a device today, the user is restricted to a certain digital-pen technology. However, within this technology, some pen suppliers provide customers with different options for pens, ranging from basic models enabling touch, which is simply more accurate than a finger, up to high-end pens that enable reporting accurate 3-D positioning and orientation, tip pressure, support interaction with the device via additional buttons, and other added value.
This article has discussed some of the drivers for the introduction of device-integrated digital pens, classifying pen solutions and highlighting the benefits of on-device touch sensors integrated with active pens. Requirements related to touch integration and pen inking, enabling intuitive annotation, note taking, drawing and painting, collaboration, and other considerations, were also detailed.
Digital-pen technology has matured over the last decade to provide rich user interaction beyond pen-on-paper. Yet, as more and more people use digital pens for different applications and real-life environments, new needs and challenges occur, providing a basis for continuous research and development.
In addition to convenience and usability, digital pens offer environmental benefits. Worldwide consumption of paper has risen by 400% in the past 40 years, leading to an increase in deforestation, with 35% of harvested trees being used for paper manufacture.14 Focusing on the users’ needs and ensuring matching requirements will grow the desire for digital pens, driving towards a pen-on-paper-style user experience that will lead to a paper-less greener world.
References
1http://www.n-trig.com/
2http://www.microsoft.com/surface/en-us/products/surface-pro-3
3http://store.sony.com/vaio-active-pen-zid27-VGPSTD2/cat-27-catid-All-Computers-Tablet-Accessories
4http://www.samsung.com/global/microsite/galaxynote/note
5http://msdn.microsoft.com/en-us/library/windows/hardware/hh998618.aspx
6K. Hinckley, “Manual Deskterity: An Exploration of Simultaneous Pen + Touch Direct Input,” CHI 2010 (April 10–15, 2010), http://research.microsoft.com/en-us/um/people/benko/publications/2010/final-revised-alt-chi-pen-plus-touch.pdf, https://www.youtube.com/watch?v=9sTgLYH8qWs
7B. A. Lieberman, “Pointing the Way: Designing a Stylus-driven Device in a Mobile World” (2013), http://software.intel.com/sites/default/files/article/380819/pointing-the-way.pdf
8S. C. Paine, ”Digitizers and Ultrabooks. What People Want, Design Recommendations and Developer Tips. (Video Series),” http://ultrabooknews.com/2013/01/28/digitizers-and-ultrabooks-what-people-want-design-recommendations-and-developer-tips-video-series/
9H. Udo, “An Electromyographic Study of Two Different Types of Ballpoint Pens,” Industrial Health 38, 47–56 (2000).
10R. S. Goonetilleke, ”Pen Design for Improved Drawing Performance,” Applied Ergonomics 40, Issue 2, 292–301 (March 2009).
11R. Rodriguez, “Sony N-trig devices now pen pressure sensitive in 32-bit Photoshop,” SURFACE PRO ARTIST (January 10, 2014).
12ASTM D1894 – 14, “Standard Test Method for Static and Kinetic Coefficients of Friction of Plastic Film and Sheeting”
13http://en.wikipedia.org/wiki/Active_pen
14S. Martin, “Paper Chase,” Ecology Communications, Inc., http://www.ecology.com/2011/09/10/paper-chase/ •