ChatGPT and Assessment (Part 3): University Assessment

"ChatGPT-3 does not provide you with an answer to your prompt, it provides you with an output. That is a different thing altogether" (Kate Lyndsay, 2022)

Continuing the series on ChatGPT and Assessment (Pt 1 and Pt 2), let's look at how the emergence of tools such as ChatGPT has disrupted the nature of education (see ChatGPT and Education), especially University Assessment. 

Designing Assignments in the ChatGPT Era has this to say: 

"Some instructors seek to craft assignments that guide students in surpassing what AI can do. Others see that as a fool's errand - one that lends too much agency to the software...Either way, in creating assignments now, many seek to exploit ChatGPT's weaknesses."

 The article then goes into detail of how to do this. Either way, it recognises that assignments, and clearly (although not stated here) assessment in general has been disrupted and there is no going back. 

Educators have started initiatives to take advantage of this disruption, some of which I've already shared on this blog. A recent discover is Creating a Collection of 101 ideas to use AI in education, which is a call to action to crowd-source useful ways the new tools can be used effectively in education. 

When it comes to assessment, Kate Lindsay's post on ChatGPT and the Future of University Assessment is an interesting read. Beginning by stating how academia has been 'stunned' by the tool's essay writing and usability (Herne, The Guardian, 2022), and the concern that students can "now easily produce AI-assisted written work with minimal effort, dumbing down the value of their university degree", she moves on to predict: 

The sheer volume and scale of what’s coming will be meaningfully different and ultimately challenge the foundations upon which we measure that ability to think – university assessment.

 Kate then presents four possible scenarios for university assessment: 

1. Ban it: A "futile exercise" - "banning it is simply trying to implement an analogue solution to digital problem", and it is "questionable whether it is the technology or our approach to assessment that is problematic."
2. Return to pen and paper exams. Kate isn't totally against "encouraging more synchronous writing exercises"or giving "opportunities for students to write in class and learn to approach writing as a practice of learning as well as a demonstration of it."
3. Develop AI literacies as part of Student Assessment. ChatGPT can be incorporated into student assessment, with activities such as prompt competitions, essay improvement exercises, and fact checking. This is a good option as AI is not going away, and ChatGPT is "technology we are all going to be (and already are) engaging with", so helping students to use the technology responsibly and critically is "part of preparing them for the world of work."
4. Assess 'Humanness'. Bearing in mind that ChatGPT-3 "can not consult, critique and cite third party sources, it can not refer to recent real world events or published material, it can’t demonstrate higher-level thinking, argue or have original ideas", there is a need to approach assessment so students are asked to demonstrate sophisticated thinking.  Because the "more detail and facts you ask for, the more it falters" if educators adopt "assessment techniques that measure learners on critical thinking, problem-solving and reasoning skills rather than essay-writing abilities" then tools such as ChatGPT will not be an issue. Asking students to expain their thinking processes, using oral and video assessments would help with this. 
5. Using ChatGPT to support Assessment Processes. Training the tool to support grading could be an option. 

Kate has also a framework for approaching AI and university assessment on her blog, and has shared a collection of articles on Wakelet related to AI and University Assessment. 

Mentioned in Kate's first blog post, this guide by Ryan Watkins, Update your course syllabus for ChatGPT, suggests ten creative assessment ideas which incorporate ChatGPT or ask students to do something that ChatGPT can't:

1. Prompt competetions. Choose a question with no single right answer and ask students in groups to use ChatGPT to write prompts to answer the question, later sharing them and the answers in class and rating the best one. 
2. Reflect and improve. Students use ChatGPT to answer a question, then reflect on the answer, check and change the results, using track changes so the teacher can see how their answer differs from the original ChatGPT output. 
3. Re-vision. After using ChatGPT, students rewrite the output from a different perspective, expanding it and correcting any errors. 
4. Dual assignments. Give students the option of using ChatGPT or not. Those who use it should track changes to show what they change. 
5. Mind Maps. Ask students to create mind maps to illustrate an answer, as ChatGPT can't make visual representations of content in this way. 
6. Debates. Students debate an issue and are assessed on their performance. 
7. Videos or podcasts. Students record audio or video to answer a question. 
8. Explain your thinking. Ask students in addition to writing an answer / essay, to add comments that explain their thinking, or to do this seperately in an audio or video file. 
9. 2x2 matrix. Students compare two concepts using this or a Venn diagram. 
10. Next time. Students use ChatGPT to answer a question, then reflect on what they learned from this. 

What is clear to me from all of these reflections and suggestions, is that the approach to Assessment needs to change. Finally, I tend to agree with what JISC has stated in relation to AI writing tools, that: 

'We should really regard them as simply the next step up from spelling or grammar checkers: technology that can make everyone’s life easier’ (Weale, The Guardian, 2023)