PhD Stage 1 Seminar: Leonie Payne – Bringing Rigour to the Evaluation of Higher Education

This is a PhD Stage 1 seminar to present and discuss the proposed gap in the literature review, identified research questions, early results and plans for further investigations. All are welcome to attend, and to provide constructive feedback that will assist the student in preparing the future work for thesis submission.

Thesis Title: Bringing Rigour to the Evaluation of Higher Education

PhD Candidate: Leonie Payne

Abstract

Student evaluations of teaching (SETs) are increasingly used in high stakes decisions that affect stakeholders at all levels of an organisation. However, claims about bias are frequently used to discredit SET findings. Whilst much attention has been paid to the way in which this bias can affect academic opportunities, SETs are also increasingly used in institutional rankings and other comparative settings. Indeed, the Australian Federal Government announced in August 2019 that aspects of the Quality Indicators for Learning and Teaching (QILT) Student Experience and Graduate Outcomes surveys will form two of the four key metrics used to allocate performance based funding of Australian Universities from 2020. This thesis will seek to create a more robust approach to the analysis of bias in SETs. It will demonstrate that the methodology used in making claims of bias is frequently poorly theoretically motivated, and reliant on small samples and hypothesis testing, which leaves the field prone to the replication crisis.

The aim of my PhD research is to provide a sounder theoretical and statistical footing upon which to place the analysis of SETs. I will achieve this by establishing a mathematical and conceptual model of bias that will assist those analysing SETs to understand where and how bias may creep into the survey methodology used. I will also develop statistical techniques and tools that can be readily applied by university evaluation units to understand SET data, including the potential impact of biases which are identified through use of the conceptual model. Early contributions at this stage of my PhD project include a prototype mathematical and conceptual model, analysis of bias creeping into the 2018 QILT Student Experience Survey methodology, and application of Monte Carlo resampling techniques to address non-response bias on a publicly available University of New South Wales dataset. I demonstrate that Monte Carlo resampling techniques and parameter space estimation (mapping) show promise to assist comparisons to be made on a like-for-like basis. This research will allow for more equitable comparisons of SET data at individual, cross faculty, and cross institutional levels.

Short bio
Leonie Payne is a PhD candidate in the Connected Intelligence Centre (CIC) of the University of Technology Sydney. Her research is focused on developing a statistically robust system for analysing student evaluations of teaching surveys, and developing visualisation and analytical tools that can be implemented in higher education settings. These tools will assist non-statistically trained audiences to make more equitable comparisons at individual, cross faculty, and cross institutional levels.