This article is part of a series of case studies from McMaster instructors that explore perspectives and approaches for integrating or limiting the use of AI in learning environments. –
Dr. Kevin Browne is currently a Professor at Mohawk College teaching in the Computer Science & IT programs, as well as a Sessional Instructor at McMaster University teaching in the Software Engineering Technology Program. Dr. Seshasai Srinivasan is currently an Associate Professor at McMaster University and is the Chair of the Software Engineering Technology program. Together they have developed a scalable asynchronous interview format for oral exams.
The Software Engineering technology program at W Booth School has developed a scalable asynchronous interview format for oral examinations. This innovation addresses challenges posed by Generative AI, which has made traditional written assessments and online quizzes less reliable methods to assess student understanding.
To maintain academic integrity while preserving the flexibility of a fully online course, faculty members like Dr. Kevin Browne have implemented a process using MS Teams and the A2L platform. In this format, students record themselves verbally answering quiz questions while sharing their computer desktop screens. An automated program handles the administrative burden of creating individual MS Teams channels, allowing the process to scale from one student to hundreds. Instructors and TAs then grade these recordings and transcripts asynchronously using standard rubrics.
The system has proven successful in a class of 50+ students, with no critical implementation challenges. Future goals include scaling to even larger classes through group interviews and introducing frequent, low-stakes video assessments to build student comfort with the format. This method also ensures data privacy by storing recordings on McMaster servers.
W Booth School of Engineering Practice and Technology’s Software Engineering Technology (SFWR) is a fully virtual program in which all lectures and assessments happen online. Assessments include quizzes, assignments, challenge projects, as well as exams. Exams are usually held on camera but without the contentious proctoring software that requires students to surrender the control of their computer to a fully invasive monitoring protocol. Other assessments such as quizzes, projects, portfolios, and assignments are offline.
The current challenge is that with the emergence of generative AI (GenAI), students can simply use these tools as a response system to produce solutions to the various assessments. Specifically, the tools can prepare well-structured and sophisticated responses for a wide variety of questions. Further, traditional assessments are often modular, meaning they can be broken into independent tasks that current AI models excel at solving. As a result of this, most assessments can be completed by the students without mastering the course content, which makes them ineffective at evaluating student understanding. The assessments might still be effective as practice and/or portfolio-building opportunities, which provide different types of value that students generally praise and appreciate. But the most fundamental purpose of assessments is to measure and verify that actual learning has occurred, and if instructors and institutions cannot do this via assessments then they cannot certify the competency of students with confidence.
Ultimately, there is a risk of graduating large cohorts of students who are not competent in their disciplines, affecting their employability, leading to declining industrial productivity, and eventually hindering the socioeconomic progress of our society. Additionally, there is a significant fear that stakeholders, including employers and alumni, will devalue the degree if the university fails to ensure the integrity of its assessments. Thus, assessment dynamics must undergo a complete transformation in the age of AI.
Instructors within the school have considered a combination of solutions to address this issue. Among others, the foremost is the adoption of the scalable asynchronous interview process for oral examinations of students. First piloted in Dr. Kevin Browne’s course, this is now being used across the program in several courses.
The Interview Process: The highlights of the interview process are as follows:
The instructions (Appendix 1) are provided to students on how to launch the interview and the steps they need to follow. A crucial aspect of this process is that students are required to start their video recording and screen sharing before the A2L quiz containing the questions is started, as this captures the delivery of the questions themselves to the students on the recordings via screen sharing. This prevents students from simply copying and pasting questions into GenAI tools to get the answers before providing them in the video, as if they were to do so it would be captured on video.
All students who enrolled in the pilot course completed the mandatory video interviews as part of the course midterm. To date, the asynchronous individual interview process has successfully been implemented in a class of approximately 50 students by the faculty members of the Software Engineering Technology program. Instructors have been able to individually and flexibly mark each students’ video interview using standardized rubrics.
Scheduling individually conducted live oral exams with each student would consume onerous amounts of class time and require large diverse question banks to ensure students tested earlier were not able to effectively assist students tested later by sharing their test questions with them. This asynchronous format allows the testing to occur simultaneously, greatly reducing the time required to conduct the testing, and reducing the need for question randomization. At the same time, grading can occur afterwards, and be completed by a teaching assistant, preserving class time for instruction or other activities.
While the results are early and limited to a few classes, thus far the class performance with this assessment format more closely resembles that of a written proctored exam than a “take home test” or assignment, in terms of being able to differentiate and measure a range of understanding among students. Some students answer some questions incorrectly or fail the assessment by answering too many questions incorrectly, whereas other students excel and answer many or all questions correctly, as would be expected on a proctored in-person written test. For example, the first attempt pass rate on the asynchronous oral exam midterm in the pilot course was 80%. With this approach, even satisfactory student responses to questions contain the sort of mild or partial inaccuracies indicative of real student responses, again resembling proctored in-person written tests, or responses to questions in a job interview.
One challenge with implementation has been a failure on the part of some students to follow the procedure, for example by failing to share their screen or forgetting to turn on record. This challenge is being effectively addressed with adjustments such as ungraded practice opportunities and/or a final reminder on the first page of the quiz itself regarding the technical configuration required before students proceed to the quiz questions themselves. In terms of accommodations, a student with an accommodations letter was granted the accommodation to write out their answers on paper in front of them before saying them aloud, but beyond this no new accommodations have been required.
Students also report anxiety with the format through feedback surveys and course evaluations. However, test anxiety also occurs with proctored in-person tests, and this anxiety is virtually universally accepted in academia as a worthwhile trade-off for the high integrity provided by these tests. The oral exam format can help students to practice similar oral communication skills as will be required during job interviews. We surveyed students after the pilot course midterm and asked them, “Compared to written assessments in other courses, how well do you think the asynchronous oral exam format assessed your actual understanding of ideas in the course?”, and the majority of students actually thought the oral exam format assessed their understanding better.
Thus far, these challenges and new learnings encountered have not been detrimental to the assessment’s utility and are being effectively addressed. Notably, the most recent summative student evaluation of the course learning experience where this assessment was adopted was a mean value of 8.5 out of 10, which is in line with previous evaluations of the course which did not use this assessment format.
We would like to imbibe this as a standard assessment practice in every course in the Software Engineering Technology program, in a similar way that proctored final exams are the standard expectation for in-person courses. Further, we would also like to introduce video assessments on a regular basis throughout the length of the course with pass/fail grading to enable students to develop their comfort with oral video examinations prior to the final summative assessment.
Appendix 1 – Instructions for the Interview The interview will open at 9:00am on April 17th and you must begin by 9:10am.
The following steps should be followed (Please see the video as well for the instructions):
The interviews will be marked based on the video recording saved in each private channel.
A variation on this process is being tested in Summer 2026 where students simultaneously create meetings in their private Teams channels, sharing their microphone, camera and screen, and then a question to answer is broadcasted to all Teams private channels at a scheduled time as a message from the instructor. Students receive the question at the same moment in time and answer it in their individual meetings. This variation simplifies the process somewhat by eliminating the need to use the A2L quiz feature.
—
Browne, K., & Srinivasan, S. (2026), A scalable asynchronous interview format for oral exams. Licensed under Creative Commons BY-NC-SA 4.0.
Teaching in the Age of AI, Updates
Spotlight on SOTL, Updates