A3R: Authentic Assessment Analytics for Reflective Writing

Authentic Assessment Analytics for Reflection (A3R) is an exciting new project that DVC(ES) (Deputy Vice-Chancellor Education and Students Prof. Shirley Alexander) is funding over the coming year, jointly led by the Connected Intelligence Centre (CIC) and Institute for Interactive Media and Learning (IML), as part of the UTS priority to transition towards increasingly authentic assessment practices.

Authentic assessment is concerned with student performance in practice-oriented tasks reflecting experiences beyond the educational environment. To be authentic the task needs to mirror the complexity and high level thinking that is required to solve problems in real-world settings.

The A3R project will target the development of students’ reflective writing skills as required by a range of authentic assessment tasks across the university. This will involve the development and piloting of formative feedback generation through a web app called Academic Writing Analytics (AWA). AWA has the potential to provide rapid analytical feedback on students’ reflective writing, at scale.

What is reflective writing?

Boud, Keogh and Walker (1985) provide a useful perspective:

When a person writes reflectively, they consider a specific event or situation and write about how it impacts them personally. This can include how they are emotionally affected by the situation, or the ways in which they are challenged to consider new ideas, different perspectives, or new ways of doing things.

In authentic assessment, this is usually within the context of the profession in which they intending to become a part of, and so it the writing also includes how the situation affects their understanding of themselves as an emerging professional. For example: “How does this affect me as a developing engineer?”, or “How does this change the way I think of becoming a teacher?”

When used effectively, reflective writing tasks can not only deepen learners’ understanding of key concepts, but can also help them critically appraise their developing professional identity, and build qualities for lifelong learning.

What is AWA?

AWA (Academic Writing Analytics) analyses writing and provides feedback designed to help the writer improve their writing. For the A3R project, AWA is being developed to provide feedback for the improvement of reflective writing. Students will upload their texts, AWA will analyse them and provide a combination of written and visual feedback for the writer. For example a student submits a journal entry to AWA, which then advises the student in which aspects of the text need improvement. The student can write a new draft of the text, or perhaps the student can keep in mind what needs to be improved in a future journal entry.

AWA does not focus so much on content, but looks for evidence that the writer has reflected on a learning experience, on how they are progressing in a subject and on how they analyse certain concepts relevant in the learning experience. AWA is not intended to be a replacement for human marking of summative assessment tasks. It is meant to be part of a suite of tools that students are free to use in drafting their writing.

The Authentic Assessment Analytics for Reflection (A3R) Project

This project has the potential to provide rapid analytical feedback on students’ reflective writing, at scale. Assessing writing, even formatively, is extremely time-consuming — whether a brief first assignment, a draft essay, a thesis chapter, or a research article in preparation for peer reviews. However, there is evidence that timely, personalised feedback is one of the key factors impacting learning, and students consistently request more responsive, quality feedback.

“If AWA fulfils its promise, we are moving to a scenario of being able to offer 24/7 formative feedback to learners, on their own drafts or any other text they choose to reflect on. This feedback could also form the basis for discussion with peers and/or tutors, a provocation for sharing their understandings of what deep reflective writing ‘looks like’ made visible in new ways.” (Buckingham Shum et al, 2016)

Timeline

April 2016

Currently gathering historic written data to help establish a baseline on which to compare AWA assisted student writing.
This will help ‘fine tune’ AWA so that they can improve the way it gives feedback.
AWA also needs to be sensitized to what good reflective writing is in the context of various disciplines.

May – July 2016

Gather feedback from subject tutors on their user experience of the tool. How confident they are and whether the interface is meaningful and useful.
They hope to learn more about using AWA to improve the quality of student reflective writing, and how students perceive its usefulness in the context of their subjects.

July – October 2016

The software will be tested with over 1000 students from a number of disciplines.
The team will collect feedback from the students on their experience with the software, and the extent to which the analysis and generated feedback is helpful for the improvement of their writing.

A3R: Authentic Assessment Analytics for Reflective Writing