MOTIVATION & AIMS:

Computers are getting faster and smaller every day. Many products, notebook and laptop personal computers, pen-based computers and personal organizers, are designed to be as small (`portable') as 
possible.  While up to now their size was limited by hard disk, memory chips, battery and power supplies, the lower bound is now increasingly dependent on the size of the input/output devices.  The resolution of 
the human eye limits the size of the screen, and the dimensions of the fingers fix the minimum size of keyboards and mice.  The desire to lower these bounds motivates the search for alternative ways for 
humans to communicate with computers and the development of new input/output devices such as audio and visual interfaces. 

Audio and visual interfaces present two big advantages. First they can be implemented as very small devices with the current VLSI technology. Second, in some circumstances, they will allow the design 
of more natural interfaces than keyboards and mice. From this point of view, one of the natural ways of inputting data into the computers seems to be the use of handwriting.  So far, there are some devices 
that interface between handwriting and computers, such as electronic tablets or digitizers for on-line capturing and optical scanners for off-line conversion. However, all of them are bulky and complicated to 
use, increasing the complexity of the whole system. The goal of this project is to develop a small and portable visual interface that can be built using video technology and computer vision techniques. 

The new input system consists of a camera, a piece of paper and a pen. The camera, pointed at the sheet of paper, images the handwriting and the trajectory of the pen is tracked so that the 
handwriting is recognized. 
 

DESCRIPTION OF THE SYSTEM

We use computer vision techniques and estimation theory to track the position of  the pen tip in the image plane [1,2,3]. 
 

USING:  A template of the pen tip An uncalibrated video camera A model of handwriting generation dynamics A recursive estimator WE CAN: Track the pen tip throughout a sequence of images with 1-2 pixels accuracy  Detect the positions in which the pen is up and not writing.

REAL-TIME IMPLEMENTATION 

The system was implemented on a Pentium platform equipped with a Texas Instruments C80 DSP board. We achieved real-time, processing 30 frames per second. The following figure shows a user writing on a piece of whiter paper. The user wrote the word "mario", that is displayed on the monitor of the computer. 
 

RESULTS 

In the following figures we show several handwriting sequences acquired in real-time with our system. 
 
 

ACHIEVEMENTS

We have developed a new way of input data for computers. The system works in real-time (30 Hz) and  is able to track the position of the pen tip while a user is writing.  The system is able to detect when the pen is up and not writing. The recovered handwriting path provides enough spatio-temporal for performing handwriting recognition. 

PUBLICATIONS

[1] M. E. Munich and P. Perona, "Visual Input for Pen-Based Computers", CNS Technical Report CNS-TR-95-01, California Institute of Technology, 1995. 

[2] M. E. Munich and P. Perona, "Visual Input for Pen-Based Computers", in Proceedings of the 13th International Conference on Pattern Recognition, Vienna, August 1996. 

[3] M. E. Munich and P. Perona, "Visual Input for Pen-Based Computers", in Proceedings of the 3rd International Conference on Image Processing, Lausanne, September 1996.