The current education systems in elementary schools are usually using traditional teaching methods such as paper and pencil or drawing on the board. The benefit of paper and pencil is their ease of use. Researchers have tried to bring this ease of use to computer-based educational systems through the use of sketch-recognition. Sketch-recognition allows students to draw naturally while at the same time receiving automated assistance and feedback from the computer. There are many sketch-based educational systems for children. However, current sketch-based educational systems use the same sketch recognizer for both adults and children. The problem of this approach is that the recognizers are trained by using sample data drawn by adults, even though the drawing patterns of children and adults are markedly different. We propose that if we make a separate recognizer for children, we can increase the recognition accuracy of shapes drawn by children. By creating a separate recognizer for children, we improved the recognition accuracy of children’s drawings from 81.25% (using the adults’ threshold) to 83.75% (using adjusted threshold for children). Additionally, we were able to automatically distinguish children’s drawings from adults’ drawings. We correctly identified the drawer’s age (age 3, 4, 7, or adult) with 78.3%. When distinguishing toddlers (age 3 and 4) from matures (age 7 and adult), we got a precision of 95.2% using 10-fold cross validation. When we removed adults and distinguished between toddlers and 7 year olds, we got a precision of 90.2%. Distinguishing between 3, 4, and 7 year olds, we got a precision of 86.8%. Furthermore, we revealed that there is a potential gender difference since our recognizer was more accurately able to recognize the drawings of female children (91.4%) than the male children (85.4%). Finally, this paper introduces a sketch-based teaching assistant tool for children, EasySketch, which teaches children how to draw digits and characters. Children can learn how to draw digits and characters by instructions and feedback.

Artifacts made by humans, such as items of furniture and houses, exhibit an enormous amount of variability in shape. In this paper, we concentrate on models of the shapes of objects that are made up of fixed collections of sub-parts whose dimensions and spatial arrangement exhibit variation. Our goals are: to learn these models from data and to use them for recognition. Our emphasis is on learning and recognition from three-dimensional data, to test the basic shape-modeling methodology. In this paper we also demonstrate how to use models learned in three dimensions for recognition of two-dimensional sketches of objects. / Singapore-MIT Alliance (SMA)

Intelligent tutoring systems (ITS) have proven to be effective tools for aiding in the instruction of new skills for young kids; however, interaction methods that employ traditional input devices such as the keyboard and mouse may present barriers to children who have yet learned how to write. Existing applications which utilize pen-input devices better mimic the physical act of writing, but few provide useful feedback to the users. This thesis presents a system specifically designed to serve as a useful tool in teaching children how to draw basic shapes, and helping them develop basic drawing and writing skills. The system uses a combination of sketch recognition techniques to interpret the handwritten strokes from sketches of the children, and then provides intelligent feedback based on what they draw. Our approach provides a virtual coach to assist teachers teaching the critical skills of drawing and handwriting. We do so by guiding children through a set of exercises of increasing complexity according to their progress, and at the same time keeping track of students' performance and engagement, giving them differentiated instruction and feedback. Our system would be like a virtual Teaching Assistant for Young Kids, hence we call it TAYouKi. We collected over five hundred hand-drawn shapes from grownups that had a clear understanding of what a particular geometric shape should look like. We used this data to test the recognition of our system. Following, we conducted a series of case studies with children in age group three to six to test the interactivity efficacy of the system. The studies served to gain important insights regarding the research challenges in different domains. Results suggest that our approach is appealable and engaging to children and can help in more effectively teach them how to draw and write.

With the proliferation of touch screens, sketching input has become popular among many software products. This phenomenon has stimulated a new round of boom in free-hand sketch research, covering topics like sketch recognition, sketch-based image retrieval, sketch synthesis and sketch segmentation. Comparing to previous sketch works, the newly proposed works are generally employing more complicated sketches and sketches in much larger quantity, thanks to the advancements in hardware. This thesis thus demonstrates some new works on free-hand sketches, presenting novel thoughts on aforementioned topics. On sketch recognition, Eitz et al. [32] are the first explorers, who proposed the large-scale TU-Berlin sketch dataset [32] that made sketch recognition possible. Following their work, we continue to analyze the dataset and find that the visual cue sparsity and internal structural complexity are the two biggest challenges for sketch recognition. Accordingly, we propose multiple kernel learning [45] to fuse multiple visual cues and star graph representation [12] to encode the structures of the sketches. With the new schemes, we have achieved significant improvement on recognition accuracy (from 56% to 65.81%). Experimental study on sketch attributes is performed to further boost sketch recognition performance and enable novel retrieval-by-attribute applications. For sketch-based image retrieval, we start by carefully examining the existing works. After looking at the big picture of sketch-based image retrieval, we highlight that studying the sketch's ability to distinguish intra-category object variations should be the most promising direction to proceed on, and we define it as the fine-grained sketch-based image retrieval problem. Deformable part-based model which addresses object part details and object deformations is raised to tackle this new problem, and graph matching is employed to compute the similarity between deformable part-based models by matching the parts of different models. To evaluate this new problem, we combine the TU-Berlin sketch dataset and the PASCAL VOC photo dataset [36] to form a new challenging cross-domain dataset with pairwise sketch-photo similarity ratings, and our proposed method has shown promising results on this new dataset. Regarding sketch synthesis, we focus on the generating of real free-hand style sketches for general categories, as the closest previous work [8] only managed to show efficacy on a single category: human faces. The difficulties that impede sketch synthesis to reach other categories include the cluttered edges and diverse object variations due to deformation. To address those difficulties, we propose a deformable stroke model to form the sketch synthesis into a detection process, which is directly aiming at the cluttered background and the object variations. To alleviate the training of such a model, a perceptual grouping algorithm is further proposed that utilizes stroke length's relationship to stroke semantics, stroke temporal order and Gestalt principles [58] to perform part-level sketch segmentation. The perceptual grouping provides semantic part-level supervision automatically for the deformable stroke model training, and an iterative learning scheme is introduced to gradually refine the supervision and the model training. With the learned deformable stroke models, sketches with distinct free-hand style can be generated for many categories.

Pen-based interfaces utilize sketch recognition so users can create and interact with complex, graphical systems via drawn input. In order for people to freely draw within these systems, users' drawing styles should not be constrained. The low-level techniques involved with sketch recognition must then be perfected, because poor low-level accuracy can impair a user's interaction experience. Corner finding, also known as stroke segmentation, is one of the first steps to free-form sketch recognition. Corner finding breaks a drawn stroke into a set of primitive symbols such as lines, arcs, and circles, so that the original stoke data can be transformed into a more machine-friendly format. By working with sketched primitives, drawn objects can then be described in a visual language, noting what primitive shapes have been drawn and the shapes? geometric relationships to each other. We present three new corner finding techniques that improve segmentation accuracy. Our first technique, MergeCF, is a multi-primitive segmenter that splits drawn strokes into primitive lines and arcs. MergeCF eliminates extraneous primitives by merging them with their neighboring segments. Our second technique, ShortStraw, works with polyline-only data. Polyline segments are important since many domains use simple polyline symbols formed with squares, triangles, and arrows. Our ShortStraw algorithm is simple to implement, yet more powerful than previous polyline work in the corner finding literature. Lastly, we demonstrate how a combination technique can be used to pull the best corner finding results from multiple segmentation algorithms. This combination segmenter utilizes the best corners found from other segmentation techniques, eliminating many false negatives (missed primitive segmentations) from the final, low-level results. We will present the implementation and results from our new segmentation techniques, showing how they perform better than related work in the corner finding field. We will also discuss limitations of each technique, how we have sought to overcome those limitations, and where we believe the sketch recognition subfield of corner finding is headed.

Sketch recognition allows computers to understand and model hand drawn sketches and diagrams. Traditionally sketch recognition systems required a pen based PC interface, but powerful mobile devices such as tablets and smartphones can provide a new platform for sketch recognition systems. We describe a new sketch recognition library, Strontium (SrL) that combines several existing sketch recognition libraries modified to run on both personal computers and on the Android platform. We analyzed the recognition speed and accuracy implications of performing low-level shape recognition on smartphones with touch screens. We found that there is a large gap in recognition speed on mobile devices between recognizing simple shapes and more complex ones, suggesting that mobile sketch interface designers limit the complexity of their sketch domains. We also found that a low sampling rate on mobile devices can affect recognition accuracy of complex and curved shapes. Despite this, we found no evidence to suggest that using a finger as an input implement leads to a decrease in simple shape recognition accuracy. These results show that the same geometric shape recognizers developed for pen applications can be used in mobile applications, provided that developers keep shape domains simple and ensure that input sampling rate is kept as high as possible.

When asked to draw, most people are hesitant because they believe themselves unable to draw well. A human instructor can teach students how to draw by encouraging them to practice established drawing techniques and by providing personal and directed feedback to foster their artistic intuition and perception. This thesis describes the first methodology for a computer application to mimic a human instructor by providing direction and feedback to assist a student in drawing a human face from a photograph. Nine design principles were discovered and developed for providing such instruction, presenting reference media, giving corrective feedback, and receiving actions from the student. Face recognition is used to model the human face in a photograph so that sketch recognition can map a drawing to the model and evaluate it. New sketch recognition techniques and algorithms were created in order to perform sketch understanding on such subjective content. After two iterations of development and user studies for this methodology, the result is a computer application that can guide a person toward producing his/her own sketch of a human model in a reference photograph with step-bystep instruction and computer generated feedback.

One of the challenges students face in studying an East Asian (EA) language (e.g., Chinese, Japanese, and Korean) as a second language is mastering their selected language’s written component. This is especially true for students with native fluency of English and deficient written fluency of another EA language. In order to alleviate the steep learning curve inherent in the properties of EA languages’ complicated writing scripts, language instructors conventionally introduce various written techniques such as stroke order and direction to allow students to study writing scripts in a systematic fashion. Yet, despite the advantages gained from written technique instruction, the physical presence of the language instructor in conventional instruction is still highly desirable during the learning process; not only does it allow instructors to offer valuable real-time critique and feedback interaction on students’ writings, but it also allows instructors to correct students’ bad writing habits that would impede mastery of the written language if not caught early in the learning process. The current generation of computer-assisted language learning (CALL) applications specific to written EA languages have therefore strived to incorporate writing-capable modalities in order to allow students to emulate their studies outside the classroom setting. Several factors such as constrained writing styles, and weak feedback and assessment capabilities limit these existing applications and their employed techniques from closely mimicking the benefits that language instructors continue to offer. In this thesis, I describe my geometric-based sketch recognition approach to several writing scripts in the EA languages while addressing the issues that plague existing CALL applications and the handwriting recognition techniques that they utilize. The approach takes advantage of A Language to Describe, Display, and Editing in Sketch Recognition (LADDER) framework to provide users with valuable feedback and assessment that not only recognizes the visual correctness of students’ written EA Language writings, but also critiques the technical correctness of their stroke order and direction. Furthermore, my approach provides recognition independent of writing style that allows students to learn with natural writing through size- and amount-independence, thus bridging the gap between beginner applications that only recognize single-square input and expert tools that lack written technique critique.

Sketch recognition is an active research field, with the goal to automatically recognize hand-drawn diagrams by a computer. The technology enables people to freely interact with digital devices like tablet PCs, Wacoms, and multi-touch screens. These devices are easy to use and have become very popular in market. However, they are still quite costly and need more time to be integrated into existing systems. For example, handwriting recognition systems, while gaining in accuracy and capability, still must rely on users using tablet-PCs to sketch on. As computers get smaller, and smart-phones become more common, our vision is to allow people to sketch using normal pencil and paper and to provide a simple microphone, such as one from their smart-phone, to interpret their writings. Since the only device we need is a single simple microphone, the scope of our work is not limited to common mobile devices, but also can be integrated into many other small devices, such as a ring. In this thesis, we thoroughly investigate this new area, which we call acoustic based sketch recognition, and evaluate the possibilities of using it as a new interaction technique. We focus specifically on building a recognition engine for acoustic sketch recognition. We first propose a dynamic time wrapping algorithm for recognizing isolated sketch sounds using MFCC(Mel-Frequency Cesptral Coefficients). After analyzing its performance limitations, we propose improved dynamic time wrapping algorithms which work on a hybrid basis, using both MFCC and four global features including skewness, kurtosis, curviness and peak location. The proposed approaches provide both robustness and decreased computational cost. Finally, we evaluate our algorithms using acoustic data collected by the participants using a device's built-in microphone. Using our improved algorithm we were able to achieve an accuracy of 90% for a 10 digit gesture set, 87% accuracy for the 26 English characters and over 95% accuracy for a set of seven commonly used gestures.

Freehand sketching is a natural and crucial part of everyday humaninteraction, yet is almost totally unsupported by current user interfaces. With the increasing availability of tablet notebooks and pen based PDAs, sketchbased interaction has gained attention as a natural interaction modality.We are working to combine the flexibility and ease of use of paper and pencilwith the processing power of a computer, to produce a user interface fordesign that feels as natural as paper, yet is considerably smarter. One of themost basic tasks in accomplishing this is converting the original digitizedpen strokes in a sketch into the intended geometric objects. In this paper wedescribe an implemented system that combines multiple sources of knowledge toprovide robust early processing for freehand sketching. We also show how thisearly processing system can be used as part of a fast sketch recognition system with polynomial time segmentation and recognition algorithms.