About Me

I am the Research Director of the Natural Language Processing Group in Research & Development at Educational Testing Service in Princeton, New Jersey. My research interests span natural language processing for educational technology, automated essay scoring and evaluation, discourse and sentiment analysis, argumentation mining, education policy, English language learning, and writing research. The intersection of my interests has led to these two inventions: E-rater®, an automated essay evaluation application, and Language Muse, an instructional authoring tool for teachers of English learners.  I co-organize this yearly workshop (usually at NAACL or ACL): Innovative Use of NLP for Building Educational Applications (with Joel Tetreault and Claudia Leacock).  

Application-wise, my research focus targets designing NLP-centered systems for the service of learning. I have significant experience managing system design from the early idea stages through product implementation . I believe that working with users from the very early stages is critical, and this is also one of my favorite aspects of system design. In this context, I have spent a lot of time collaborating with K-12 instructors and university faculty.  I spend a lot of time reading about the Common Core State Standards Initiative and working with university faculty and administration to inform my thinking about building new systems that support writing instruction and assessment.  I think that Understanding Language is a great initiative with a great group of people that illustrates the critical role of language in the Common Core Standards, especially with regard to English learners, and addresses important policy issues about equity in education. 

I have become increasingly more engaged in  automated writing evaluation and writing research, advocating for building capabilities that cover a greater breadth of genres and relevant writing constructs to improve support for writing instruction and assessment.

Here is my full CV. 

Grant Awards

• Department of Education, Institute of Education Sciences, Education Technology Program, Grant Award # R305A140472 , Goal #2: Technology-Assisted Generation of Linguistically-Relevant Instructional Activities to Support ELLs in Content and Language Learning in the Content Areas, $1,499,000, Principal Investigator (with Co-PI, John Sabatini) (July 2014  - July 2017).

• Department of Education, Institute of Education Sciences, Education Technology Program, Grant Award #R305A100105, Goal #2: A Technology-Rich Teacher Professional Development Intervention that Supports Content-Based Curriculum Development for English Language Learners, $1,499,000, Principal Investigator. (March 2010  - March 2014).

• Burstein, J., Elliott, N., and Molloy, H. (accepted). Working on the Ground to Inform Automated Writing Evaluation: Method and Theory. To appear in Special Issue: CALICO Journal 33.1, 2016 (Guest Editors: Volker Hegelheimer, Ahmet Dursun, Zhi Li).

• Shermis, M., Burstein, J., Elliot, N., Miel, S., and Foltz, P. (in press). Automated Writing Evaluation: A Growing Body of Knowledge. In the Handbook of Writing Research (Eds. C. MacArthur, S. Graham, and J. Fitzgerald): Guilford Press: NY.

• Burstein, J., Shore, J., Sabatini, J., Moulder, B.,Lentini, J., Biggers, K., and Holtzman, S. (2014). From Teacher Professional Development to the Classroom: How NLP Technology Can Enhance Teachers’ Linguistic Awareness to Support Curriculum Development for English Language Learners. Journal of Educational Computing Research, 51(1): 119-144.
  
• Somasundaran, S., Burstein, J., and Chodorow, M. (2014). Lexical Chaining for Measuring Discourse Coherence Quality in Test-taker Essays. The 25th International Conference on Computational Linguistics (COLING), Dublin, Ireland, August 23-29, 2014.
• Burstein, J., Somasundaran, S., Chodorow, M. (2014). Finding your inner-annotator: An experiment in annotator independence for rating coherence quality in essays. In Proceedings of the Eight Linguistic Annotation Workshop in conjunction with International Conference on Computational Linguistics (COLING), Dublin, Ireland, August 23-24, 2014.
• Beigman Klebanov, B., Madnani, N., Burstein, J., and Somasundaran, S. (2014). Content Importance Models for Scoring Writing From Sources. In Proceedings of the Annual Meeting of the Association of Computational Linguistics, Baltimore, MD. June 23-25, 2014.
• Burstein, J., Sabatini, J., & Shore, J. (2014). In Ruslan Mitkov (Ed.), Developing NLP Applications for Educational Problem Spaces, Oxford Handbook of Computational Linguistics. New York: Oxford University Press. Currently published online: www.oxfordhandbooks.com.
  
• Beigman Klebanov, B., Burstein, J., and Madnani, N. (2013). Sentiment Profiles of Multi-Word Expressions in Test-Taker Essays: The Case of Noun-Noun Compounds. Accepted to ACM Transactions for Speech and Language Processing.
• Beigman-Klebanov, B., Madnani, N, Burstein, J. (2013). Using pivot-based paraphrasing and sentiment profiles to improve a subjectivity lexicon for essay data. Transactions of the Association for Computational Linguistics.
• Burstein, J., Sabatini, J., Shore, J., Moulder, B., and Lentini, J. (2013). A User Study: Technology to Increase Teachers’ Linguistic Awareness to Improve Instructional Language Support for English Language Learners. In Proceedings of  the Workshop for Improving Textual Accessibility  in conjunction with the Annual Meeting of the North American Association for Computational Linguistics, Atlanta, Georgia,  June 14, 2013
• Burstein, J., Tetreault, J., and Chodorow, M. (2013). Holistic Annotation of Discourse Coherence Quality in Noisy Essay Writing. In the Special issue of Dialogue and Discourse on: Beyond semantics: the challenges of annotating pragmatic and discourse phenomena (Eds.  S. Dipper, H.  Zinsmeister, and B. Webber).
  
• Burstein, J., Tetreault, J., & Madnani, N. (2013). The E-rater® Automated Essay Scoring System. In Shermis, M.D., & Burstein, J. (Eds.), Handbook of Automated Essay Scoring: Current Applications and Future Directions. New York: Routledge.
• Burstein, J., Tetreault, J., Chodorow, M.,Blanchard, D., & Andreyev, S. (2013). Automated Evaluation of Discourse Coherence Quality in Essay Writing. In Shermis, M.D., & Burstein, J. (Eds.),  Handbook of Automated Essay Scoring: Current Applications and Future Directions. New York: Routledge.
• Burstein, J., Beigman-Klebanov, B., Madnani, N., & Faulkner, A.  Sentiment Analysis Detection for Essay Evaluation. (2013). In Shermis, M.D., & Burstein, J. (Eds.),  Handbook of Automated Essay Scoring: Current Applications and Future Directions. New York: Routledge.
• Madnani, N., Burstein, J., Sabatini, J., and O’Reilly, T. (2013). Automated Scoring of a Summary-Writing Task Designed to Measure Reading Comprehension. In Proceedings of the North American Association for Computational Linguistics Eighth Workshop Using Innovative NLP for Building Educational Applications,  Atlanta, Georgia, June 13, 2013.
• Burstein, J., Shore, J., Sabatini, J., Moulder, B., Holtzman, S., & Pedersen, T. (2012). The Language Muse system: Linguistically focused instructional authoring ETS RR-12-21. Princeton, NJ: ETS. 
• Burstein, J., Flor, M., Tetreault, J., Madnani, N., & Holtzman, S. (2012). Examining linguistic characteristics of paraphrase in a test-taker summaries. ETS RR-12-18. Princeton, NJ: ETS.
• Burstein, J. (2012).  Fostering Best Practices in Writing Instruction and Assessment with E-rater®. In “Writing Assessment in the 21st Century—Essays in Honor of Edward M. White, Norbert Elliott and Les Perelman (Eds.). 
• Beigman-Klebanov, B., Burstein, J., Madnani, N., Faulkner, A., and Tetreault, J. (2012). Building Subjectivity Lexicon(s) From Scratch For Essay Data. In Alexander Gelbulkh (Ed.), Springer Lecture Notes in Computer Science. Berlin: Springer-Verlag.
• Burstein, J. (2012). Automated Essay Scoring and Evaluation.  In Carol Chapelle (Ed.)., The Encyclopedia of Applied Linguistics. Wiley Blackwell. Marlden, MA.
• Burstein, J., Tetreault, J. and Andreyev, S. (2010). Using Entity-Based Features to Model Coherence in Student Essays. Proceedings of the HLT/NAACL Annual Meeting, Los Angeles, June 2010.
• Shermis, M. D., Burstein, J., Higgins, D., & Zechner, K. (2010). Automated essay scoring: Writing assessment and instruction. In E. Baker, B. McGaw & N. S. Petersen (Eds.), International Encyclopedia of Education (Vol. 4, pp. 20-26). Oxford, UK: Elsevier.
• Burstein, J., Shore, J., Sabatini, J., Lee, Y., & Ventura, M. (2007). The automated text adaptation tool. In Demo Proceedings of the the annual conference of the North American chapter of the Association for Computational Linguistics (NAACL-HLT 2007), Rochester, NY.
• Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater v.2.0. Journal of Technology, Learning, and Assessment, 4(3).
• Higgins, D., Burstein, J., and Attali, Y. (2006).  Identifying Off-Topic Student Essays without Topic-Specific Training Data.  In J. Burstein and C. Leacock (eds). Special Issue of Natural Language Engineering on Educational Applications Using NLP.
• Shermis, M. D., Burstein, J., & Leacock, C. (2006). Applications of computers in assessment and analysis of writing. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of writing research. New York: Guilford Publications.
• Burstein, J. and Higgins, D. (2005). Advanced Capabilities for Evaluating Student Writing: Detecting Off-Topic Essays Without Topic-Specific Training. Proceedings of the International Conference on Artificial Intelligence in Education, July 2005, Amsterdam, The Netherlands.
• Burstein, J., Chodorow, M., and Leacock, C. (2004). Automated Essay Evaluation: The Criterion Online Service, AI Magazine, 25(3), 27-36.
• Higgins, D., Burstein, J., Marcu, D., and Gentile, C. (2004). Evaluating Multiple Aspects of Coherence in Student Essays. Proceedings of the HLT/NAACL Annual Meeting, Boston, May 2004.
• Burstein, J. and Wolska, M. (2003). Toward Evaluation of Writing Style: Overly Repetitious Word Use. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics. Budapest, Hungary.
• Burstein, J., Marcu, D., and Knight, K. (2003). Finding the WRITE Stuff: Automatic Identification of Discourse Structure in Student Essays. In S. Harabagiu and F. Ciravegna (Eds.) Special Issue on Advances in Natural Language Processing, IEEE Intelligent Systems, Vol.18, no. 1 pp. 32-39. 
• Burstein, J., Kukich, K., Wolff, S., Lu, C., Chodorow, M., Braden-Harder, L., and Harris, M. D.  (1998). Automated Scoring Using A Hybrid Feature Identification Technique.  In the Proceedings of the Annual Meeting of the Association of Computational Linguistics, August, 1998. Montreal, Canada. 

Recent Talks

• Invited Speaker, Automated evaluation of writing: Expanding the body of knowledge for writing in the disciplines, Presented at the Symposium on Linguistic Perspectives on Discipline-Specific Language: Transforming the Teaching & Assessment of STEM Writing, Iowa State University, May 1-2, 2014: http://stemwriting.engl.iastate.edu/activities/symposium/
• Co-Organizer (with Norbert Elliot)/Presenter, "Investigating Genre for Writing Measurement and Automated Writing Evaluation, Annual Meeting of the National Council of Educational Measurement, Philadelphia, PA, April 2014.Genre-AES_NCME2014.Final.pdf
  0.5 MB
  
• Invited Speaker, Workshop on Writing and Readability Tools, University of Utrecht, Utrecht, The Netherlands, January 31, 2014.
• Keynote Speaker, The Far Reach of Multi-Word Expressions in Educational Technology, Ninth Workshop on Multi-Word Expressions, NAACL 2013, Atlanta, Georgia.
• Keynote Speaker, Technology for Second Language Learning , Iowa State University, September 22, 2012, Ames, IA
• Technische Universität Darmstadt, Computer Science Department, Design Considerations for NLP Applications for Educational Purposes, July 6, 2012.

• Automatic Essay Scoring System Using Content-Based Technologies, U.S. Patent Number, 6,115,683, issued September 5, 2000  (with Susanne Wolff, Chi Lu, and Randy Kaplan)
• System and method for computer-based automatic essay scoring U.S. Patent Number, 6,181,909, issued,  January 30, 2001  (with Bruce Kaplan, Don Rock, Martin Chodorow, Chi Lu, Susanne Wolff, Lisa Braden-Harder, and Karen Kukich)
• System And Method For Computer-Based Automatic Essay Scoring, U.S. Patent Number,  6,366,759 issued,  April 2, 2002  (with Bruce Kaplan, Don Rock, Martin Chodorow, Chi Lu, Susanne Wolff, Lisa Braden-Harder, and Karen Kukich)
• Methods of Automated Essay Analysis, U.S. Patent Number, 6,796,800, issued, September 28, 2004. (with Daniel Marcu, Slava Andreyev, Martin Chodorow, and Claudia Leacock)
• Automated Essay Scoring, U.S. Patent Number 7,088,949, issued August 8, 2006. (with Slava Andreyev and Chi Lu)
• Automated Annotation, U.S. Patent Number 7,127,208, issued October 24, 2006. (with Daniel Marcu)
• Method And System For Detecting Off-Topic Essays Without Topic-Specific Training, U.S. Patent Number 7,711,312, issued May 4, 2010. (with Derrick Higgins).
• Method and system for determining text coherence, U.S. Patent Number 7,720,675, issued May 18, 2010. (with Derrick Higgins, Daniel Marcu, and Claudia Gentile).
• Methods for automated essay analysis, U.S. Patent Number, 7,729,655, issued June 1, 2010 (with Daniel Marcu, Slava Andreyev, Martin Chodorow, and Claudia Leacock)
• Method and System for Automated Item Development for Language Learners 7,762,816, issued July 27, 2010.
• Automated Essay Scoring, U.S. Patent Number 7,769,339, issued August 3, 2010 (with Slava Andreyev and Chi Lu)
• Automated Annotation, U.S. Patent Number 7,796,937, issued September 10,  2010 (with Daniel Marcu).
• Automated Essay Scoring System, U.S. Patent Number 7,831,196, issued November 9,  2010 (with Yigal Attali and Slava Andreyev).
• Method and System for Automatic Generation of Adapted Content to Facilitate Reading Skill Development for Language Learners, U.S. Patent Number 8,239,762, issued August 7, 2012 (with Jane Shore, John Sabatini, and Matthew Ventura).

• Method and System for Instructing Language Learners. U.S. Patent Application Number Pending, filed April 1, 2005.
• Building Subjectivity Lexicons for Sentiment Analysis Systems for Use in Essay Scoring U.S. Patent Application No. 11948-501-888, filed October 21, 2011 (with Beata Beigman-Klebanov, Nitin Madnani, Adam Faulkner, and Joel Tetreault).
• Building Subjectivity Lexicons for Sentiment Analysis (with Beata Beigman-Klebanov, Nitin Madnani, and Adam Faulkner), filed March 2012.
• Discourse Coherence Systems with Lexical Chaining, Application No. 61/949,499, filed March 7, 2014 (with Swapna Somasundaran and Martin Chodorow).
• Automated Scoring of a Summary-Writing Task Designed to Measure Reading Comprehension, No. 61/802,808, filed  March 18, 2013 (with Nitin Madnani).