Which features are better? - Geometric or Gesture

What?! No Rubine Features?: Using Geometric-based Features to Produce Normalized Confidence Values for Sketch Recognition

Paulson, B., Rajan, P., Davalos, P., Gutierrez-Osuna, R., and Hammond, T.

Citiation:

http://srl.tamu.edu/srlng/research/paper/23?from=/srlng/research/

With the advent of Tablets PCs and touch screens, Sketch-based applications are assuming more significance day-by-day. They are one of the best ways to help convey ideas and designs. However, at the same time it is challenging as it becomes really important to understand the intention of the sketcher.

Previous Work:

The previous work done to identify the sketches mainly focused on two approaches : Gesture-based and Geometric-based. These two approaches were tried in isolation by different researchers.

Gesture-based:

This approach was tried by Rubine and Long to identify gestures. The approach has the advantage of using statistical classifiers that are really fast and sound. It also produces Normalized Confidence Values that can be used for alternative interpretations (e.g. differentiate between circle and ellipse). 

This approach mainly focuses on how the sketch was drawn and not how the final sketch looks like. Since every user has his way of drawing a given sketch, this approach is highly user-and-style dependent. As a result, this requires training on example sets of each gesture class respective to every user. Moreover, this approach is tightly coupled with the scale and rotation of the gestures and imposes user to start drawing from specific points and in certain directions only. So, this approach suffers heavily when it comes to draw "free" or "natural" sketch recognition.

 

Geometric-based:

This approach was tried as a part of low level recognizer (PaleoSketch) for identifying primitive shapes. It relies upon finding errors between the sketched shape and the ideal version of the sketch. It makes use of a series of mathematical tests and formulas to perform the recognition. This mainly focuses on how the final sketch looks like rather than how it was drawn. So this approach is independent of the user and his style and hence, performs well for different users.

However, this approach is riddled with couple of disadvantages. It uses numerical thresholds and heuristic hierarchies that are difficult to analyze and optimize. Inferences can't be generalized easily as the classification is not statistical. Alternative interpretations for a given sketch are also difficult if multiple error measures per shape are used.

So, the author, in this paper, went a step ahead and tried the hybrid approach in such a way as to take advantage of using both the approaches. To allow fast and alternative interpretations, a statistical (Quadratic) classifier was used. The geometric features ensured that the model remained user independent.

The data for performing the tests was taken from the PaleoSketch. Following on the lines of Paleosketch, a total of 1800 sketch examples (belonging to 9 different shape classes) by 20 users were taken and split into two equal halves. Each half contained 900 examples sketched by 10 distinct users. The idea behind this was to find out if the system is really agnostic to user's style. This also helped them compare results with PaleoSketch.

The full feature set didn't provide the accuracy as reported by PaleoSketch. So, the paper also used Multi-dimensional Scaling method (Sequential Forward Selection) for Optimal Feature Subset Selection and was able to obtain comparable results.

Using this, the system determined that 14 out of 31 Geometric were significant to obtain good results. Out of the 14 features, the main features were related to Complex shapes and Polylines as they were hard to recognize. 

The most significant observation was that only 1 gesture-based feature (Total Rotation) was found to be optimal. This mainly resulted from the Data split approach based on user. Since most of the gesture-based features are user-dependent, the Data split approach rendered all of them insignificant.

The results, thus, prove that the geometric based feature set is more significant than geometric-based-recognition which was the core of many sketch recognition applications.

Critique:

This approach culled out the best parts of both the gesture-based as well geometric-based techniques. Using the features from the both the techniques augured well in achieving good accuracy for recognizing sketches.The idea of Feature Subset Selection to use the subset of Optimal Features is also good as this helped in leaving out the unnecessary features. Moreover, using such approach made the system user-independent which is always at the core of any Sketch Recognition.

However, the paper only discusses about One Example (Pendulum) when it comes to recognizing Complex shapes. This doesn't throw light on what all sorts of Complex sketches it would be able to handle.