**Project Summary**

My PhD thesis topic focused on the integration of handwriting-based interfaces with intelligent tutoring systems for mathematics. Traditional lessons in Cognitive Tutors on how to solve algebra equations require students to interact with an interface very different than that of solving equations on paper. Keyboard-and-mouse-based interfaces impose extraneous cognitive load on students during problem-solving and distract them from the math. A natural interaction experience using the handwriting modality can directly support the two-dimensional math notations such as fractions and exponents that more constrained, traditional interfaces do not. Yet, state-of-the-art handwriting recognition accuracy may not be good enough for students to use to solve problems, distracting them with the need to fix recognition errors. With my co-advisors, Ken Koedinger at CMU and Jie Yang now at NSF, I designed, developed and evaluated a proof-of-concept handwriting-based interface for an algebra tutoring system. On the user interaction and learning side, we found that students are able to solve more problems in the same amount of time, while achieving similar learning gains on the post-test, when using handwriting vs typing. On the technology and algorithms side, we found that the use of problem-solving context significantly improved handwriting recognition for the algebra domain.

**Project Background**

Standard lessons in Cognitive Tutors (built by Carnegie Mellon‘s PACT Center) on how to solve algebra equations require students to interact with an interface very different than that of solving equations on paper. Although some alternative interfaces have been explored, typically students use menus to perform manipulations such as adding a quantity to both sides of an equation and then type in the result of such manipulations using the keyboard. There are several weaknesses to this style of online tutoring, especially for mathematics.

- Keyboard-and-mouse-based interfaces may impose extraneous cognitive load on students during problem-solving and distract them from the math concepts of import, by requiring them to learn and account for idiosyncracies of the type-in interface that are irrelevant to the learning event.
- Typing mathematics on the computer generally involves some degree of linearization of the input (for instance, “2^x” instead of placing the
`x`as a superscript. A more natural interaction experience using the handwriting modality can more directly support the two-dimensional math notations such as fractions and exponents that more constrained, traditional interfaces do not, allowing students to construct their solutions and express their understanding in an unconstrained way. - More generally, current standard interfaces for entering mathematical equations on computers are arguably limited and cumbersome. Mathematical notations have evolved to aid visual thinking and yet text-based interfaces relying on keyboard-and-mouse input do not take advantage of the natural two-dimensional aspects of mathematical equations.
- Choosing solving steps from a menu rather than requiring the students to generate the next step independently can encourage non-learning-oriented behaviors such as “gaming the system”.

Since Cognitive Tutors are not intended to *replace* teachers or traditional classroom instruction and tests, these lessons could be a weakness in the curriculum with respect to encouraging transfer from the Cognitive Tutor experience to the paper experience. Due to its similarities to paper-based mathematics, pen-based handwriting input may be faster, more efficient, and more preferable for entering mathematics on computers.

Our results have shown that, in addition to more general usability gains for speed and user satisfaction [Anthony et al, 2005], students are able to solve more problems in the same amount of time, while achieving similar learning gains on the post-test [Anthony et al, 2007a]. We were able to enhance the accuracy of the handwriting recognition used in the interface through use of the tutoring system’s knowledge of the student learning state as a source of context [Anthony, 2008]. A more natural interaction experience could lead to decreased cognitive load, better performance, and improved usability in the general case, and learning gains and improved transfer to paper tests in the intelligent tutoring case [Anthony et al, 2007a].

**Project Status and Findings**

This project ran from 2004-2008. We completed several formative user studies and created a prototype system. Major findings include:

- General users entering calculus-level mathematics equations and expressions on the computer were faster in handwriting than typing (using Microsoft Equation Editor) by a factor of 3! This advantage increased as equation length and complexity increased. They also rated the handwriting modality more highly on a post-sessions Likert scale [Anthony et al, 2005].
- In the same study, users also input equations in a multimodal handwriting-plus-speech method, which ended up being faster and better liked than the keyboard-and-mouse method and was not much worse than handwriting alone. Also, users’ speech while writing differed from when speaking alone. Finally, user errors in handwriting and speech were non-overlapping, a fact which a multimodal recognition system could use to improve overall performance [Anthony et al, 2006a].
- Students learning or reviewing algebra equation solving who used a handwriting interface experienced similar learning gains as measured by improvement from pre-test to post-test than those who used a keyboard interface. However, the handwriting students finished the lesson in about half the time of the keyboard students [Anthony et al, 2007a].
- In the same study, students experienced a greater degree of what we call “transfer to paper.” In conditions in which there was a modality switch (e.g., writing on the pre-test to typing in the interface to writing on the post-test), there was a lower correlation in performance during training vs on the tests, indicating that performance during tutoring is more predictive of performance during testing when students use the same modality to solve the problems [Anthony et al, 2007a].
- In general, students and users indicate that handwriting is a more suitable, natural and useful input modality for mathematics on the computer than typing [Anthony et al, 2005; Anthony et al, 2008a].
- Use of worked examples as part of the tutoring system helped provide a feed-forward instructional intervention to mitigate against the lack of step-targeted feedback which was unfavorable due to inadequate raw handwriting recognition accuracies [Anthony, 2008].
- Tutor-provided context such as knowledge about student skills, common student errors and the correct answer to the current problem combined with the recognition hypotheses improved raw recognition accuracy by 10% points, decreasing the proportion of time the system would erroneously interrupt the student as s/he worked [Anthony, 2008].

*Current project status:* This project is no longer active.

**Documents**

*Thesis Proposal*

- Coming soon!

*Publications*

- Anthony, Lisa. (2008) “Developing Handwriting-based Intelligent Tutors to Enhance Mathematics Learning.” Ph.D. thesis, Human-Computer Interaction Institute, School of Computer Science, Carnegie Mellon University. [pdf]
- Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2008b) “Toward Next-Generation, Intelligent Tutors: Adding Natural Handwriting Input.”
*IEEE Multimedia*15(3), pp. 64-68. [camera copy] [DOI link] - Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2008a) “How Handwritten Input Helps Students Learning Algebra Equation Solving.”
*Technical Report CMU-HCII-08-100*, 1 Mar 2008. [pdf] - Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007b) “Adapting Handwriting Recognition for Applications in Algebra Learning.”
*ACM Workshop on Educational Multimedia and Multimedia Education (EMME’2007)*, Augsburg, Germany, Sep 2007, pp. 47-56. [pdf] - Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007a) “Benefits of Handwritten Input for Students Learning Algebra Equation Solving.”
*International Conference on Artificial Intelligence and Education (AIEd’2007)*, Los Angeles, CA, Jul 2007, pp. 521-523. [pdf] - Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2006b) “Towards the Application of a Handwriting Interface for Mathematics Learning.”
*IEEE Conference on Multimedia and Expo (ICME’2006)*, Toronto, Canada, Jul 2006, pp. 2077-2080. [pdf] - Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2006a) “Entering Mathematical Equations Multimodally: Results on Usability and Interaction Design.”
*Technical Report CMU-HCII-06-101*, 15 Mar 2006. [pdf] - Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) “Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer.”
*ACM Conference on Human Factors in Computing Systems (CHI 2005)*, Portland, OR, 4 Apr 2005, pp. 1184-1187. [pdf]

*Presentations*

- 04/2005 — Short Paper presentation at CHI’2005. [pdf]
- 07/2006 — Special Session on Distance Learning paper presentation at ICME’2006. [pdf coming soon]

*Posters*

- 02/2005 — PSLC Advisory Board Meeting: Poster Session [pdf]
- 05/2005 — PSLC Annual NSF Site Visit: Poster Session [pdf]
- 04/2006 — HCII 12th Anniversary Celebration: Poster Session [pdf coming soon]
- 05/2006 — PSLC Annual NSF Site Visit: Poster Session [pdf coming soon]
- 10/2006 — Science of Learning Centers Symposium: Poster Session [pdf coming soon]

**People**

*Former:*

- Dr. Lisa Anthony (as a graduate student, contact person)
- Dr. Ken Koedinger (Co-PI)
- Dr. Jie Yang (Co-PI)
- Thomas Bolster (graduate research assistant)
- Keisha How (undergraduate research assistant)
- Andrea Knight (graduate research assistant)

**Selected Bibliography**

*Handwriting, Speech, and Mathematics*

- Blostein, D. and Grbavec, A.: Recognition of Mathematical Notation. In
*Handbook on Optical Character Recognition and Document Analysis*, Wang, P.S.P. and Bunke, H. (eds) (1996) 557-582. - Brown, C.M.L.: Comparison of Typing and Handwriting in “Two-Finger” Typists.
*Proceedings of the Human Factors Society*(1988) 381-385. - Fateman, R.: How Can We Speak Math? http://www.cs.berkeley.edu/~fateman/papers/speakmath.pdf.
- Kanahori, T., Tabata, K., Cong, W., Tamari, F., and Suzuki, M.: On-Line Recognition of Mathematical Expressions Using Automatic Rewriting Method.
*Proceedings of the IEEE International Conference on Multimodal Interfaces (ICMI’00)*(2000) 394-401. - Matsakis, N.E.: Recognition of Handwritten Mathematical Expressions. Master’s theses,
*Massachusetts Institute of Technology*(1999) Cambridge, MA. - Miller, E.G., Matsakis, N.E., and Viola, P.A.: Learning from One Example Through Shared Densities on Transforms.
*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’00)*(2000) 464-471. - Microsoft.:
*Microsoft Word User’s Guide Version 6.0*(1993), Microsoft Press. - Smithies, S., Novins, K., and Arvo, J.: Equation Entry and Editing via Handwriting and Gesture Recognition.
*Behaviour and Information Technology*20 (2001) 53-67. - xThink.:
*Math Journal*(2003) http://www.xthink.com/MathJournal.html.

*Multimodal and Recognition Technologies*

- Blum, A. and Mitchell, T.: Combining Labeled and Unlabeled Data with Co-Training.
*Proceedings of the Workshop on Computational Learning Theory (COLT’98)*(1998) 92-100. - Clark, H. and Brennan, S.: Grounding in Communication. In
*Perspective on Socially Shared Cognition*(eds. Resnic, L.B., Levine, J., and Sag, S.). APA, Washington, D.C. (1991) 127-149. - Oviatt, S.: Mutual Disambiguation of Recognition Errors in a Multimodal Architecture.
*Proceedings of the CHI Conference*(1999) 576-583. - Oviatt, S., Coulston, R., and Lundsford, R.: When Do We Interact Multimodally? Cognitive Load and Multimodal Communication Patterns.
*Proceedings of IEEE International Conference on Multimodal Interfaces (ICMI’04)*(2004). - Oviatt, S., DeAngeli, A., and Kuhn, K.: Integration and Synchronization of Input Modes During Multimodal Human-Computer Interaction.
*Proceedings of the ACM Conference on Human Factors in Computing (CHI’97)*(1997) 415-422.

*Education, Cognitive Tutors, and Psychology*

- Aleven, V.A.W.M.M. and Koedinger, K.R.: An Effective Metacognitive Strategy: Learning by Doing and Explaining with a Computer-Based Cognitive Tutor.
*Cognitive Science*26 (2002) 147-149. - Anderson, J.R., Corbett, A.T., Koedinger, K.R., and Pelletier, R.: Cognitive Tutors: Lessons Learned.
*The Journal of the Learning Sciences*4 (1995) 167-207. - Hausmann, R.G.M. and Chi, M.T.H.: Can a Computer Interface Support Self-explaining?
*Cognitive Technology*7 (2002) 4-14. - Locke, J.L. and Fehr, F.S.: Subvocalization of Heard or Seen Words Prior to Spoken or Written Recall.
*American Journal of Psychology*85 (1972) 63-68. - Sweller, J.: Cognitive Load During Problem Solving: Effects on Learning.
*Cognitive Science*12 (1988) 257-285.

last revised 04/25/2012