ChatGPT and Assessment (Part 3): University Assessment

"ChatGPT-3 does not provide you with an answer to your prompt, it provides you with an output. That is a different thing altogether" (Kate Lyndsay, 2022)

Continuing the series on ChatGPT and Assessment (Pt 1 and Pt 2), let's look at how the emergence of tools such as ChatGPT has disrupted the nature of education (see ChatGPT and Education), especially University Assessment. 

Designing Assignments in the ChatGPT Era has this to say: 

"Some instructors seek to craft assignments that guide students in surpassing what AI can do. Others see that as a fool's errand - one that lends too much agency to the software...Either way, in creating assignments now, many seek to exploit ChatGPT's weaknesses."

 The article then goes into detail of how to do this. Either way, it recognises that assignments, and clearly (although not stated here) assessment in general has been disrupted and there is no going back. 

Educators have started initiatives to take advantage of this disruption, some of which I've already shared on this blog. A recent discover is Creating a Collection of 101 ideas to use AI in education, which is a call to action to crowd-source useful ways the new tools can be used effectively in education. 

When it comes to assessment, Kate Lindsay's post on ChatGPT and the Future of University Assessment is an interesting read. Beginning by stating how academia has been 'stunned' by the tool's essay writing and usability (Herne, The Guardian, 2022), and the concern that students can "now easily produce AI-assisted written work with minimal effort, dumbing down the value of their university degree", she moves on to predict: 

The sheer volume and scale of what’s coming will be meaningfully different and ultimately challenge the foundations upon which we measure that ability to think – university assessment.

 Kate then presents four possible scenarios for university assessment: 

1. Ban it: A "futile exercise" - "banning it is simply trying to implement an analogue solution to digital problem", and it is "questionable whether it is the technology or our approach to assessment that is problematic."
2. Return to pen and paper exams. Kate isn't totally against "encouraging more synchronous writing exercises"or giving "opportunities for students to write in class and learn to approach writing as a practice of learning as well as a demonstration of it."
3. Develop AI literacies as part of Student Assessment. ChatGPT can be incorporated into student assessment, with activities such as prompt competitions, essay improvement exercises, and fact checking. This is a good option as AI is not going away, and ChatGPT is "technology we are all going to be (and already are) engaging with", so helping students to use the technology responsibly and critically is "part of preparing them for the world of work."
4. Assess 'Humanness'. Bearing in mind that ChatGPT-3 "can not consult, critique and cite third party sources, it can not refer to recent real world events or published material, it can’t demonstrate higher-level thinking, argue or have original ideas", there is a need to approach assessment so students are asked to demonstrate sophisticated thinking.  Because the "more detail and facts you ask for, the more it falters" if educators adopt "assessment techniques that measure learners on critical thinking, problem-solving and reasoning skills rather than essay-writing abilities" then tools such as ChatGPT will not be an issue. Asking students to expain their thinking processes, using oral and video assessments would help with this. 
5. Using ChatGPT to support Assessment Processes. Training the tool to support grading could be an option. 

Kate has also a framework for approaching AI and university assessment on her blog, and has shared a collection of articles on Wakelet related to AI and University Assessment. 

Mentioned in Kate's first blog post, this guide by Ryan Watkins, Update your course syllabus for ChatGPT, suggests ten creative assessment ideas which incorporate ChatGPT or ask students to do something that ChatGPT can't:

1. Prompt competetions. Choose a question with no single right answer and ask students in groups to use ChatGPT to write prompts to answer the question, later sharing them and the answers in class and rating the best one. 
2. Reflect and improve. Students use ChatGPT to answer a question, then reflect on the answer, check and change the results, using track changes so the teacher can see how their answer differs from the original ChatGPT output. 
3. Re-vision. After using ChatGPT, students rewrite the output from a different perspective, expanding it and correcting any errors. 
4. Dual assignments. Give students the option of using ChatGPT or not. Those who use it should track changes to show what they change. 
5. Mind Maps. Ask students to create mind maps to illustrate an answer, as ChatGPT can't make visual representations of content in this way. 
6. Debates. Students debate an issue and are assessed on their performance. 
7. Videos or podcasts. Students record audio or video to answer a question. 
8. Explain your thinking. Ask students in addition to writing an answer / essay, to add comments that explain their thinking, or to do this seperately in an audio or video file. 
9. 2x2 matrix. Students compare two concepts using this or a Venn diagram. 
10. Next time. Students use ChatGPT to answer a question, then reflect on what they learned from this. 

What is clear to me from all of these reflections and suggestions, is that the approach to Assessment needs to change. Finally, I tend to agree with what JISC has stated in relation to AI writing tools, that: 

'We should really regard them as simply the next step up from spelling or grammar checkers: technology that can make everyone’s life easier’ (Weale, The Guardian, 2023)

 

Text adventures with ChatGPT

 

Here's another great idea for using ChatGPT with English language students that I first saw the educator Doris Molero talk about on Facebook. Some months ago I had the pleasure to talk to Doris about some of her work. You can listen to my chat with Doris on Teachers Talk Radio here. 

The original prompt / idea was: 

This is a wonderful game to play with your students. Here you have some variations on the same game: "You are a text-based video game where you give me options (A, B, C, and D) as my choices. The setting is Lord of the Rings. I start out with 100 health" "You are a text-based video game where you give me options (A, B, C, and D) as my choices. The setting is Winnie the Pooh. I start out with 100 health" Your imagination is the limit. have students play in groups and share their games and results.

I tried out some variations to see what the results would be. The first one was...

You are a text-based adventure game where you give me options (A, B, C, and D) as my choices. The setting is New York City in the Future where robots are common. I start out with 100 health.

This is what ChatGPT came up with and how the game developed as I started playing it.

You wake up in the future New York City and see that the world has changed. Robots are now a common sight. You have 100 health points and are ready to start your adventure.

You find yourself standing at a crossroads.

A. Walk down the busy street and see what the city has to offer B. Enter a dark alley and explore C. Visit a local store to purchase supplies D. Head to the nearest bar to ask for directions

Which option do you choose (A, B, C, or D)?

I selected 'A'

You walk down the busy street and take in your surroundings. You see all sorts of robots going about their business. Some are performing manual labor while others are interacting with humans in a friendly manner.

You come across a street performer who is playing music with a unique robotic instrument.

A. Stop to listen to the performance B. Offer to make a donation to the performer C. Keep walking and ignore the performer D. Ask the performer about the instrument

What would you like to do (A, B, C, or D)?

I next chose 'B'

You approach the street performer and offer to make a donation. The performer gratefully accepts and thanks you for your generosity.

As you continue down the street, you come across a robot that has broken down and is in need of repair.

A. Attempt to repair the robot yourself B. Find a nearby store to buy parts to repair the robot C. Leave the robot and continue on your way D. Report the broken robot to the authorities

What would you like to do (A, B, C, or D)?

Here, I chose 'B'

You head to a nearby store to purchase the necessary parts to repair the robot. After acquiring the parts, you return to the broken down robot and begin to repair it. Your technical skills come in handy and you successfully fix the robot.

The grateful robot thanks you and offers to assist you on your journey in any way possible.

A. Accept the robot's help B. Decline the robot's offer and continue on your own C. Ask the robot to guide you to the nearest bar D. Ask the robot to show you around the city

What would you like to do (A, B, C, or D)?

I selected 'A' because I was interested in seeing what this vague answer would lead to. I'm also noticing the game is trying very hard to try to make me go to a bar, which is interesting, but not ideal if you were doing this activity with children. I suppose adding 'suitable for children' to the prompt would be a good idea in that context. 

You accept the robot's offer and the two of you set out to explore the city together. As you walk, the robot shares some interesting facts about the city and its advancements in technology.

You come across a large building with a sign that reads "Robot Arena."

A. Enter the Robot Arena and participate in a battle B. Watch a battle at the Robot Arena C. Keep walking and avoid the Robot Arena D. Ask the robot about the Robot Arena

What would you like to do (A, B, C, or D)?

I next chose 'A ' because I was interested in seeing how the combat system worked. I think adding the 'I start out with 100 health' or similar to the prompt will lead to Chat GPT adding combat or opportunities to lose health in the game. On another occasion, I'll explore this further and omit this part of the prompt to see what happens.

You and the robot enter the Robot Arena and sign up for a battle. The arena is packed with spectators and the atmosphere is electric. Your battle is against a tough opponent, but with the help of the robot, you manage to emerge victorious.

The crowd cheers and you and the robot are awarded a prize.

A. Keep the prize and continue exploring the city B. Donate the prize to a local charity C. Use the prize to upgrade your equipment D. Share the prize with the robot

What would you like to do (A, B, C, or D)?

OK, so no damage. My health is still at 100 and there's no sense of any peril in the game, which is problematic. I want to play a game that has some conflict in it or where there are some stakes to it. I think working on the prompt more to ensure that is the case and also adding a part of the prompt so there is an ending after a number of turns would probably be a good idea. As it stands, I think the game feels like it could go on forever. I chose 'B' and decided to end the game here. 

You decide to donate the prize to a local charity. The charity thanks you for your generosity and you continue your journey through the city, proud of the good deed you have done.

As you walk, you come across a large park.

A. Take a rest and relax in the park B. Visit a nearby lake to fish C. Keep walking and avoid the park D. Play a sport with the local robots

What would you like to do (A, B, C, or D)?

So, apart from the running commentary above, how did I feel about using this type of game for language practice for students? I definitely think it has potential and as mentioned above, I think working on the prompt and further playtesting would help streamline the game ChatGPT comes up with. At the moment the language practice seems fine - there's a range of verbs and vocabulary that's interesting enough. It would make an interesting reading activity for students. 

However, as a game, it feels there's a lot to be desired. There's no sense of peril so far, and it feels like it's never-ending. Adding a prompt to limit the amount of turns and perhaps devising a point system to reward the 'best' choices would help - then students could play to see how many points they got rather than losing health. I'll definitely be experimenting more with this. Thanks for the idea, Doris!