CIC is now initiating a series of pilots in close collaboration with UTS academics across the faculties, and other units (HELPS; Jumbunna; IML) to test the potential of language technologies to provide rapid, formative feedback on draft writing.
This builds on the work at The Open University’s Knowledge Media Institute (KMi), initiated by Simon Buckingham Shum, in collaboration with Ágnes Sándor, a linguist in the Parsing & Semantics Group at Xerox Research Centre Europe, Grenoble.
KMi research demonstrated the exciting potential of combining Xerox’s natural language processing with KMi’s knowledge cartography approaches, which helped define the concept of Contested Collective Intelligence:
De Liddo, Anna; Sándor, Ágnes and Buckingham Shum, Simon (2012). Contested Collective Intelligence: rationale, technologies, and a human-machine annotation study. Computer Supported Cooperative Work (CSCW), 21(4-5) pp. 417–448. Open Access Eprint: http://oro.open.ac.uk/31052
Duygu Simsek‘s PhD at KMi began testing the applicability of rhetorical parsing to student writing, and won the LAK14 Best Demo movie award. Investigations into visualization using these approaches were reported at LAK13 (the source of the above snapshot):
Simsek, D., Buckingham Shum, S., Sándor, Á., De Liddo, A. and Ferguson, R. (2013). XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scientific Metadiscourse. 1st International Workshop on Discourse-Centric Learning Analytics. (3rd International Conference on Learning Analytics & Knowledge, 8 April 2013, Leuven, Belgium). Open Access Eprint: http://oro.open.ac.uk/37391
Taibi, D., Sándor, Á., Simsek, D., Buckingham Shum, S., De Liddo, A. and Ferguson, R. (2013) Visualizing the LAK/EDM Literature Using Combined Concept and Rhetorical Sentence Extraction, 1st Learning Analytics and Knowledge Data Challenge at Learning Analytics and Knowledge (LAK ’13), Leuven, Belgium. Open Access Eprint: http://ceur-ws.org/Vol-974/lakdatachallenge2013_07.pdf
The latest research findings from Duygu’s research will be presented at the 5th International Learning Analytics & Knowledge Conference (LAK15): Scaling Up: Big Data to Big Impact in a few weeks.
Duygu, S., Sandor, Á., Buckingham Shum, S., Ferguson, R., De Liddo, A. and Whitelock, D. (2015). Correlations between automated rhetorical analysis and tutors’ grades on student essays. In: 5th International Learning Analytics & Knowledge Conference (LAK15): Scaling Up: Big Data to Big Impact, 16-20 March 2015, Poughkeepsie, NY, USA (Forthcoming), ACM. Open Access Eprint: PDF
When assessing student essays, educators look for the students’ ability to present and pursue well-reasoned, strong, valid and logical arguments, and for their ability to use examples, evidence and personal interpretations for and against a particular position throughout a written piece. Such scholarly argumentation is often articulated by rhetorical metadiscourse. Educators will be necessarily examining metadiscourse in students’ writing as signals of the intellectual moves that make their reasoning visible. Therefore students and educators could benefit from available powerful automated textual analysis that is able to detect rhetorical metadiscourse. However, there is a need to validate such language technologies in higher education contexts, since they were originally developed in non-educational applications. This paper describes an evaluation study of a particular language analysis tool, the Xerox Incremental Parser (XIP), on undergraduate social science student essays, using the mark awarded as a measure of the quality of the writing. As part of this exploration, the study presented in this paper seeks to assess the quality of the XIP through correlational studies and multiple regression analysis. As a result, we found statistically significant correlations between the output of the XIP and grades of student essays.