Specifying Gestures by Example

Specifying Gestures by Example
Dean Rubine
Citiation:- http://dl.acm.org/citation.cfm?id=122753

At the time when the Gesture-based Interfaces were pretty naive and not extensively researched, Dean Rubine at CMU, Pittsburgh, published a paper that described how gesture recognizers could automatically be created from example gestures, thereby removing the need of hand-coding them. It was a great leap in the sense that any Gesture-based application, till that date, required the use of a hand-coded Gesture Recognizer, a module to distinguish between the gestures. In his paper, he shared the idea of using GRANDMA (Gesture Recognizers Automated in a Novel Direct Manipulation Architecture), a toolkit for rapidly adding such gestures to Direct Manipulation Interfaces (Click and Drag Applications). He further talked about GDP (Gesture-based Drawing Program), a gesture-based Application built using GRANDMA.

GRANDMA comes across a toolkit for incorporating recognizers that could identify Single-stroke and Multi-finger 2-D gestures in application with Click-and-Drag Interfaces (e.g. GDP). The gestures used in GDP are also Single-stroke. The start of a gesture begins with the positioning of the mouse cursor and pressing the mouse button. The user can draw the gesture by moving the mouse. The end of the gesture is signalled either by the release of the mouse button or if user stops the mouse for 2 seconds with the mouse button still pressed. Figure 1 below shows how the different gestures can be drawn.

The gestures can be manipulated at runtime. i.e. the gesture designer can, on the fly, can modify the way the input for a pre-defined gesture is taken leaving the output untouched. Figure 3 below indicates how inputs for any gesture belonging to different classes can be edited at the run-time.

Once the new input method for a gesture is defined, it needs to be trained by entering examples of that gesture. As per the author, 15 examples and 13 features per gesture class are enough for the training and recognizing purposes. Figure 4 below shows the examples of a gesture input for DELETE operation.

The low-level recognition of single-stroke gestures consists of three parts:

1. Training: After the user enters the example set for any gesture class, the weights are calculated corresponding to all the feature sets. To develop the model for each gesture class, the author didn't use the Iterative techniques. He, however, used a method termed as Linear Discriminator to compute the weights that was dependent on calculating mean, covariances and Inverted Common Co-variance matrix.
2. Classification: Once the training is complete, the user can immediately start using that gesture. Classification of any defined gesture class is done by using a Linear Classifier. The values for pre-defined feature set are evaluated on the drawn gesture and evaluated with the weights of the classes calculated in the Training Set. The classification of a gesture is simply the class that maximizes the value.
3. Rejection: This step comes into picture only if there is an ambiguity in the deciding the class of the gesture. To resolve the tie, probability that the gesture belonged to the classified class is computed. For values of Probability less than 0.95, the classes are rejected.

Results
Figure 8 below shows the comparison between the recognition rate and the training set size.

The results clearly indicate that this method works pretty well with the classes size restricted to 15. Also, it indicates that about 98% gestures can be classified correctly with 15 or more example sets per class. The accuracy begins to drop as the number of the classes increase and size of the example set per class decreases.

Critique

The idea of creating gestures inputs at the run-time is really good, given that this space was pretty untouched. Moreover, the recognizer need not be hand-coded. It also supports the idea of Eager Recognition, which, is really user-friendly. Once the system recognizes the gesture, it automatically completes it. It also supports Multi-Finger recognition. Use of Single-stroke gestures also help in avoiding the segmentation problems that are associated with Multi-stroke gestures. Moreover, its really hard for user to indicate the end of a Multi-stroke gesture.

However, it also suffers from the problems associated with Single-stroke gestures. e.g. An Equal-to (=) gesture input would actually end up looking like a gesture for 'z'. Not only, such a gesture is non-intuitive, this can give user hard-time in case he needs to use the similar gestures for both. Moreover, it also places the limitation that the interface incorporating such a mechanism should also be implemented using GRANDMA.

Future Work

This idea can be extended to Three-dimensional space and probably, incorporate the use of Multi-Strokes.  Moreover, it can be made to integrate into the existing application that hasn't been built using GRANDMA.