Raters' Rating Quality in Assessing Students' Assignment: An Application of Multi-Facet Rasch Measurement

Abstract

Assessing students' assignments is essential as it reflects students' understanding and achievements. This study evaluates the marking quality among lecturers at the Teacher Education Institute (IPG) using the Multi-Facet Rasch Measurement (MFRM) model. Two hundred thirty-two students from the Postgraduate Diploma in Education (PDPP) program submitted written assignments, which were assessed by experienced lecturers. The analysis was conducted using FACETS 4.1.1 software, involving three main facets: candidates, raters, and assessment criteria. The findings indicate that the instrument used is valid in terms of construct and meets unidimensionality requirements. The reliability value of the raters was high (0.81), and the rater separation index exceeded the set threshold (2.09), indicating stability in marks given by lecturers. However, two raters were identified as showing misfit and overfit patterns respectively, suggesting inaccuracies in scoring. The Wright map and unexpected response analysis also revealed differences in the severity among raters and potential bias. These findings are valuable for the IPG in improving monitoring of inter-rater reliability and marking consistency. This study also shows that MFRM can provide comprehensive information and contribute to understanding the analysis of assessor consistency with quantitative evidence. MFRM is a suitable alternative model to overcome the limitations of the Classical Test Theory (CTT) statistical model, especially in analyses involving multiple raters.