User experience evaluation of electronic moderation systems : a case study at a private higher education institution in South Africa

Abstract

The transformation of a manual paper-based moderation process into an electronic moderation (eModeration) process poses unique challenges. These challenges concern academic processes, people and the user experience of interactive systems. eModeration can improve the user experience of assessment processes while lowering the risk of delaying the process or losing scripts. Despite the benefits associated with optimising assessment procedures, particularly examination procedures, the use of eModeration in South Africa is limited. There are several possible reasons for a lack of eModeration adoption ranging from infrastructure and technical issues through to organisational and human factors. The focus of this study is on the human factors involved in eModeration. Since no User Experience Evaluation Framework for eModeration existed at the time of this research, an in-depth study was conducted based on the experiences of eModeration users in the context of private higher education institutions. The study focused on identifying the most important user experience constructs for the evaluation of an eModerate system within the context of private higher education institutions in South Africa towards proposing a framework. The study was based in the fields of Information Systems and Human-Computer Interaction with eModeration being the application domain. The research used a Design Science Research methodology, which involved the development and testing of a User Experience Evaluation Framework for eModeration. The data generation methods included interviews with deans, eModerators and management, as well as a survey that included responses from both moderators and deans. The research was conducted at Midrand Graduate Institute and evaluated at Monash University. The study makes a validated contribution towards identifying the most important user experience constructs. The identified constructs were utilised in the design and development of the User Experience Evaluation Framework for eModeration, which can be used along with the evaluation criteria tool to evaluate eModerate systems.