TEACHER PERCEPTION OF GENAI AS AN ASSESSMENT AID<br/>Bruce Ma and Martin Ma<br/>Center for Language Education, Hong Kong University of Science and Technology, Hong Kong<br/>lcmbruce@ust.hk; lcmartinma@ust.hk<br/>ABSTRACT<br/>The emergence of Generative AI (GenAI) in November 2022 has initiated a proactive integration of this technology into pedagogical practices <br/>at the HKUST Center for Language Education (CLE). Apart from encouraging and teaching students to use GenAI, HKUST has made <br/>assessment-related GenAI tools accessible to faculty. One such tool is Pregrade - a grammar and spelling grader with the capability to <br/>assess and comment on writings based on a detailed rubric (Pregrade, 2024). HKUST’s second year Technical Communication course, <br/>designed for Engineering students, spearheads the university's integration by incorporating various GenAI learning strategies. This course’s <br/>ESP nature presents difficulties for language teachers who have little to no engineering background as one of its assessments revolves <br/>around analyzing the ethics of engineering incidents. The large amount of information that is needed to grade this assessment poses a <br/>challenge to the efficiency and consistency in grading. Pregrade has limited application in this case as the assessment requires specific <br/>content-based grading which it does not currently offer (Pregrade, 2024). Therefore, a tailored, content-based solution was developed: our <br/>course-specific GenAI bot designed to provide feedback for teachers.<br/>While tertiary students have a strong inclination to embrace GenAI (Arowosegbe, 2024; Chan &amp; Hu, 2023; Kohnke, 2024; Li et al., 2024), <br/>our literature review highlights the transformative potential of GenAI tools in language assessment and teacher feedback practices as well. <br/>Studies indicate that AI can enhance grading efficiency and provide valuable, accurate feedback to students (Mohamed, 2023), while also <br/>streamlining the assessment process, ultimately saving teachers time and effort (Koraishi, 2023). Yet, there is a gap in research examining <br/>the pedagogical effects of these tools, particularly regarding student and teacher attitudes toward AI integration in the classroom (Har &amp; Ma, <br/>2023). While considerable research has focused on student attitudes, teachers' perceptions and experiences have received relatively less <br/>attention. This study aims to explore teachers' perceptions and experiences with our course-specific GenAI bot that serves as a marking <br/>assistant, emphasizing its usability, reliability, efficiency, and overall impact on teaching practices. By addressing this gap, the study <br/>contributes to the ongoing dialogue about the role of GenAI in educational assessments and its broader implications for teaching.<br/>According to Ma (2017), the integration of technology provides an innovative alternative for delivering written feedback on student writing <br/>more effectively and efficiently. This shift is particularly evident in the development of Automated Writing Evaluation (AWE) tools, which <br/>utilize computer-based educational technology to offer students constructive feedback on their writing. The emergence of GenAI has the <br/>potential to revolutionize the use of AWE tools in assessing writing as it can enhance traditional AWE systems by providing more contextaware feedback. Unlike conventional AWE tools, which often rely on rule-based algorithms, GenAI can understand and generate natural <br/>language, allowing it to offer suggestions that are more aligned with the specific contexts and needs.<br/>Building on this content-aware feature, our course-specific GenAI bot, based on Claude-3.5-Sonnet-200k, is pre-loaded with a <br/>comprehensive knowledge base of engineering incidents. It is designed to assist teachers in evaluating students’ engineering analytical <br/>reports by comparing the evidence presented in student writings against this knowledge base. The bot generates accessible summaries of <br/>the analysis in both text and tabular formats, highlighting any missing components to enable teachers to quickly identify gaps in the reports.<br/>An explanatory mixed methods approach is adopted for this study, combining quantitative and qualitative data collection techniques to <br/>provide a comprehensive understanding of teachers' experiences with the GenAI bot. Specifically, these experiences relate to the extent to <br/>which the bot supports teachers in assessing whether students can identify factors contributing to engineering disasters, highlight the <br/>interplay between these factors, analyze engineers’ accountability, apply ethical codes to hold relevant engineers responsible, and evaluate <br/>the overall quality of students’ analyses in connecting these elements. We are currently in the data collection phase of the study, which is <br/>scheduled to span two semesters, ending in May 2025.<br/>REFERENCES<br/>Arowosegbe, A. (2024). Students’ perception of generative ai use for academic purpose in UK higher education. <br/>https://doi.org/10.20944/preprints202405.1158.v1<br/>Chan, C., &amp; Hu, W. (2023). Students’ voices on generative AI: Perceptions, benefits, and challenges in higher education. International <br/>Journal of Educational Technology in Higher Education, 20(1). https://doi.org/10.1186/s41239-023-00411-8<br/>Har, F., &amp; Ma, B. W. L. (2023). The future of education utilizing an artificial intelligence robot in the Centre for Independent Language <br/>Learning: Teacher perceptions of the robot as a service. In C. Hong &amp; W. W. K. Ma (Eds.), Applied degree education and the <br/>shape of things to come. Springer, Singapore. https://doi.org/10.1007/978-981-19-9315-2_3<br/>Kohnke, L. (2024). Exploring EAP students' perceptions of GenAI and traditional grammar checking tools for language learning. Computers <br/>and Education: Artificial Intelligence, 7, 100279. https://doi.org/10.1016/j.caeai.2024.100279<br/>Koraishi, O. (2023). Teaching English in the age of AI: Embracing ChatGPT to optimize EFL materials and assessment. Language Education <br/>and Technology, 3(1), 55–72. https://langedutech.com/letjournal/index.php/let/article/download/48/37… to <br/>Day 1<br/>Back to <br/>Day 2<br/>Back to <br/>Day 3<br/>Li, Y., Deng, Y., Peng, B., He, Y., Luo, Y., &amp; Liu, Q. (2024). Generative artificial intelligence in Chinese higher education: Chinese <br/>undergraduates’ use, perception, and attitudes. Frontiers in Educational Research, 7(4). <br/>https://doi.org/10.25236/fer.2024.070401<br/>Ma, B. (2017). The study of teacher written feedback: The effectiveness of electronic feedback on student writing revisions (Doctoral thesis, <br/>Durham University).<br/>Mohamed, A. (2023). Exploring the potential of an AI-based chatbot (ChatGPT) in enhancing English as a foreign language (EFL) teaching: <br/>Perceptions of EFL faculty members. Education and Information Technologies. https://doi.org/10.1007/s10639-023-11917-z<br/>Pregrade. (2024). Pregrade user guide, Pregrade. AI, https://app.pregrade.ai/guide/index.html#<br/>KEYWORDS<br/>assessment, written corrective feedback, GenAI, automated writing evaluation, ESP<br/>BIODATA<br/>Dr. Bruce Ma is an educator and researcher with a focus on writing feedback, learner autonomy, and second language writing. His research <br/>interests also include the integration of generative AI in teaching and learning, as well as computer-aided language learning.<br/>Martin Ma specializes in English for Academic Purposes (EAP) and English for Specific Purposes (ESP). His research interest includes <br/>applying Systemic Functional Grammar (SFG) notions, such as Thematic Progression and Grammatical Metaphor, to teaching, as well as <br/>applying SFG concepts through GenAI

<p>This book chapter provides an overview of Temi, an autonomous, video-oriented personal assistant robot which was deployed within the Centre for Independent Language Learning (CILL) at The Hong Kong Polytechnic University. The artificial intelligence robot was chosen principally because of its role as a Robot as a Service (RaaS). Such a service can deliver greater self-improvement and better learning strategies (e.g. Cohen, A. D. (2014). Strategies in learning and using a second language (2nd ed.). Routledge., Dörnyei et. al., 2015, Wenden, Learner strategies for learner autonomy, Prentice Hall, 1991, Yang, Frontiers in Psychology 12:600, 218–600, 218, 2021) as well as foster beneficial attitudes and skills towards the users’ long-term language learning success. Through its cloud-based system, Temi offers users access to dynamic interactions and enhanced CILL services, during the COVID-19 pandemic. As a whole, it appears that the introduction of Temi has proven to be an effective strategy to augment learners’ autonomy. It further allows administrators to rethink how CILL services are conducted during human resource shortages.</p>