Organic Materials AI assignment has students grading ChatGPT's homework

January 27, 2025 by Alyx Dellamonica

An assignment that gives students an opportunity to assess the benefits and limitations of artificial intelligence is having its second run this year in Organic Materials Chemistry (CHM456/CHM1304)

“Most students are using AI in their daily lives. That makes this a good time to build it into the class,” said Professor Dwight Seferos, who created the assignment along with graduate student Bryton Varju, and who is running it for the second time this semester. 

The assignment challenges students to use analysis and critical thinking as well as fact checking, by having them examine AI-generated essays on topics covered in class, explained Seferos. Critical analysis of the essays will reveal errors made in the process.  

“AI can generate a lot of things... and they might be correct or not.” 

Varju, a graduate student working in the Seferos group, was instrumental in designing the assignment. “I wrote a Chemistry Teaching Fellowship Program (CTFP) proposal outlining a writing assignment where students would edit short essay answers generated by ChatGPT, when prompted with exam-style questions.” 

Bryton Varju poses in PPE, holding up a small square object.
Bryton Varju (supplied image)

Originally from BC, Varju was introduced to Prof. Seferos and his research when he was invited to a departmental seminar at Simon Fraser University. “I was doing my MSc and though his work was totally different to the inorganic chemistry I was working on at the time,I found the talk very intriguing. After I finished my MSc, I packed up and drove across the country to start my PhD in Dwight's lab.” 

Varju’s research focuses on degradable conjugated polymers, plastics that can function as semiconductors. “The idea is that they are sturdy enough to use in an electronic device but will fully break down (no microplastics) when exposed to a specific trigger, like sunlight. Eventually, these materials can be used to make fully degradable electronic devices.” 

The inspiration for the classroom assignment was ChatGPT, a generative chatbot utilizing a large-language model (LLM), which was released publicly on Nov. 22, 2022. “The idea of asking students to edit essay responses is that it gives them a chance to interact with ChatGPT and evaluate how it responds to high-level questions, while simultaneously improving their grasp on the course material by simulating a peer review process.” 

The Chemistry Teaching Fellowship Program provides financial incentives and opportunities to develop specific teaching initiatives, under the supervision of a faculty member.  The objective is to provide a high-quality learning experience in developing pedagogical tools that would benefit both the graduate student in their future careers and directly enhance the quality of instruction currently in the department. Varju’s successful proposal enabled implementation of the new assignment into the course last year. 

“To be honest, the assignment ran more smoothly than I was expecting! Generating the list of exam-style short answer questions was the most time-consuming aspect of the project.” 

"I was very nervous ChatGPT would generate completely erratic responses, but I was relieved to find that it mostly behaved. The main problems that were to do with ChatGPT misinterpreting a question, and usually just adding a few clarifying words solved the problem.” 

“The benefit of working with an upper-year undergraduate/graduate student course is that everyone already has experience with written assignments, so the only novel aspect to them is the use of ChatGPT.” 

Varju noted public fears surrounding AI, not to mention the speed with which these technologies are being implemented into familiar services and online ecosystems. "My experience with LLM generative chatbots from running this assignment has taught me that they are able to generate writing that is passable to non-experts but usually does not hold up to the scrutiny of those actively engaging with the field.” 

“A lot of the concern I have heard comes from how rapidly ChatGPT and related generative chatbots can create convincing writing, essentially allowing platforms with minimal fact-checking before publication (YouTube, most social media sites) to be flooded with artificial content.” 

“The danger is not that the chatbots will become "smarter", but rather that they will be more and more difficult to identify. The input of verified experts is going to become very important to all of the fields affected by this.” 

“Since current students are positioned to be the next generation of experts and will have to grapple with the existence of this technology for all of their professional lives, it's important to get them thinking about this early.” 

Seferos agreed, and said he is pleased to be running the assignment again, this time with graduate student Ebad Noman.  

“The most important thing is the student feedback. I really value suggestions from our students. They put us at the front line of these new educational approaches.” 

Usefully integrating AI into advanced courses like Organic Materials Chemistry has potential to be a valuable educational tool, fostering critical thinking and analytical skills. As AI continues to evolve, such assignments will be crucial in preparing the next generation of scientists and professionals to navigate and leverage these advancements responsibly. Seferos and Varju's innovative approach exemplifies forward-thinking educational strategies that can equip students for the future.