Podiatry Virtual Assessment Project

In 2021, Cardiff Metropolitan University was looking to build on the successes the Podiatry department had experienced during the early stages of the pandemic in creating a virtual assessment process for their level 6 (third year) students. Previously, students were assessed under ‘clinical l’ conditions on campus using real patients.

The need for a safe, virtual method to measure each student’s performance had led to a temporary solution in 2020. Although the approach was 'temporary', it displayed significant benefits, which is why Cardiff Metropolitan university wished to transform it into a solution that could stand the test of time, through the pandemic and beyond.

Therefore, the intention for PDR was to understand what had already been done, assess in which ways it had been successful and identify what opportunities remained for improvement. This led to the creation of a suite of concepts for a new version of the Podiatry Diagnosis Exam, which were prototyped, tested and further developed in order to provide functional deliverables and an action plan for implementation in the new academic year.

A BESPOKE, RESEARCH-LED APPROACH

We began with a general review of the current assessment processes and touchpoints, including all current documentation. From this, we developed a basic student journey map based on the initial research we had collected at this stage.

We then did a cognitive walkthrough of the assessment process with PDR staff playing the roles of students and assessors to better understand the potential pain points and logistical elements involved in delivering this service to students. From this understanding, we were able to construct a survey that was distributed to the level 6 students the week after their Podiatry Diagnosis exam where they used the existing virtual assessment system. The responses allowed us to learn much more about the real student experience.

We also conducted in-depth interviews with 4 students and 2 staff members to delve deeper into the realities of both running and sitting the exam. We used Thematic analysis techniques to break down and analyse the findings from the survey and the interviews.

We were also fortunate enough to be able to do usability analysis on 5 of the student’s exam recordings using a further layer of thematic analysis that helped draw out actionable insights using STEP cards. All of these insights allowed us to develop a revised journey and service blueprint of the current examination process including both the students’ and staffs’ journeys.

Once we had concluded the findings from the research, we ran a co-creation session with 5 Podiatry staff as well as Jeff Lewis, Professor of Flexible Learning at Cardiff School of Sport and Health Sciences. Alongside this, we also ran an ideation session with PDR staff leading on from the staff co-creation.

CRITICAL FINDINGS FROM OUR RESEARCH

From our detailed analysis and assessments with the key stakeholders we discovered a number of important findings including:

Virtual assessments are more work for staff.

A lot of work goes into planning, organising, and creating the necessary material for the exam before the students are even made aware of the exam, and running these ‘virtually’ can be more labour-intensive than the traditional, in-person method. Additionally, from talking to staff who run the assessment the heavy workload would only increase with larger cohorts.

There are pros and cons to remote working and virtual examinations.

We discovered that simulating the exam allows for more control over exam content, as the conditions that students experience are chosen by staff ensuring a fair level of difficulty between students. The virtual set-up also helps to create an equal playing field and helps stop students from cheating or copying each other which is observed during in-person assessments.

However, there were clear distractions in the students’ and assessors’ homes which interrupted the student's examinations.

The majority of students surveyed and interviewed didn't feel prepared for what to expect on the exam day.

Although the students felt prepared to discuss the content they had revised, they were confused about what to expect from the structure as well as the materials provided as part of the exam despite being given information about the exam structure during lectures and further conversations with lecturers.

The layout of the Case Study material could be improved.

We discovered issues with the way people would read through the case-study material as well as issues around data not being realistically presented in the document, including photos and videos not being clear enough for students to see.

During the exam, clarity is key.

There appeared to be a lack of a standardised procedure for staff to follow during the exam which resulted in differences in how the examination was delivered to students. As a result, this impacted the student's understanding as well as the thoroughness of the procedure that stops students from attempting to cheat.

PROJECT OUTCOMES & FURTHER OPPORTUNITIES

We created three concepts for the client, each a natural step in the progression of the latter.

Concept 1 was a ‘ready to implement’ solution that required no investment by the university and could be tested by PDR and then delivered to Cardiff Met to use immediately.

Concept 2 was then a near-future concept and a development of the first concept, which offered similar solutions but in a platform that was tailor-made for the service and could be rolled out across multiple different courses within the school.

Concept 3 was a far-future concept that incorporated virtual reality apparatus to help replicate the in-person patient diagnosis system, ensuring students received a fair and appropriately difficult clinical experience while also being conducted remotely.

PDR produced all the necessary components for concept 1. This included an interactive digital Case Study, designed to be more practical for university staff to propagate and update as well as being laid out in a more user-friendly way for students to navigate. The usability of the case study was tested, and a template version was also produced so that staff could populate new case studies efficiently. Using the data from an existing case, an example past paper was also produced which the intention of being used to walk students through the exam process, reducing the confusion students experienced when engaging with materials they hadn't used before the examination.

This entire package was produced in software that staff had prior experience using to ensure uptake and assist the rollout of this service for the 2021/22 academic year.

PDR produced all the necessary components for concept 1. This included an interactive digital Case Study, designed to be more practical for university staff to propagate and update as well as being laid out in a more user-friendly way for students to navigate. The usability of the case study was tested, and a template version was also produced so that staff could populate new case studies efficiently. Using the data from an existing case, an example past paper was also produced which the intention of being used to walk students through the exam process, reducing the confusion students experienced when engaging with materials they hadn't used before the examination.

This entire package was produced in software that staff had prior experience using to ensure uptake and assist the rollout of this service for the 2021/22 academic year.

KEY TAKEAWAYS

Alongside the solutions we developed, we identified 3 key takeaways that should be considered before any similar projects of this kind in the future:

1 - By creating templates and scripts that are easy and simple for all staff to follow and populate, workload can be shared, and consistency can be managed.

2 - Outsourcing the production of video content would also reduce staff workload and increase the pool of case study content.

3 - Providing students with a dedicated past paper and interactive/visual walkthrough of what to expect on exam day would alleviate a lot of student anxiety without assisting them with technical content.