Mitigating Generative AI Exploitation Through Isolated Practical Assessment: An Empirical Investigation of Assessment Integrity and Individual Learning Outcomes in Systems Penetration Testing and Cyber Security Education

Project leader(s): Michael Bowkis Lee Campbell

Theme: Innovative assessment
Project faculties: STEM
Status: Current
Date: February 2026 to December 2027

This project uses a mixed methods approach to investigate whether curriculum innovation in assessment design can address the disruption caused by generative AI in practical computing education. TM359, Systems Penetration testing, provides a natural experimental context: text-based assessments (TMA01, EMA) where AI tools excel, and a practical assessment (TMA02) where AI cannot substitute for genuine competency.

The research will construct and test a practical assessment requiring students to complete challenges and provide evidence of their results. Quantitative analysis will compare grades across assessment formats, while qualitative data from surveys and interviews will capture student experiences and identify circumvention strategies.

The module structure enables direct comparison within the same student cohort. Where AI tools can assist with text-based assessments, we expect continued grade inflation. Where students work within an isolated computing environment that AI tools have no knowledge of, grades should reflect genuine competency.

An analogy: surgeons must demonstrate operative competency during actual procedures, making immediate decisions without external consultation. Cybersecurity professionals face the same demands. The 2025 cyber attacks on Marks & Spencer and Jaguar Land Rover illustrate the consequences when organisations lack genuine expertise.

The redesigned TMA02 will present students with scenarios and questions on one side and an isolated computing environment on the other, requiring extraction of specific information from systems configured uniquely for each scenario. AI tools will not be able to directly provide valid answers.

Three scenarios with scaffolded progression move from structured guidance through reduced support to independent work. Comparing performance across AI-susceptible and AI-resistant assessments will provide robust evidence of whether this approach restores alignment between grades and genuine competency.