How to analyse PE exam papers: my step-by-step process

In any exam, there are three fundamental variables. They are:

1. Examined content
2. Examined skills
3. Examined assessment objectives

Beyond this, all exams can be analysed for a further set of secondary variables or trends. They are:

• Scale of mark allocation per question
• Typical range of applications
• Typical sub-sections of a paper
• Typical paper structure

When we break a paper down this way, it makes the analysis of it far easier.

In recent years, there have been many attempts by teachers to analyse exam papers. I have personally published numerous models of how this can be done. However, all of the versions I have seen—including my own previous attempts—have all focussed on a narrow range of the variables above and typically the examined content only. For these reasons, Mark Mills (Business Manager - Service) and I have established a new model to analyse papers and it is this I will present below.

 Step 1: Extract all the content 

Unfortunately, this is laborious and, because of the copyrighted nature of the exam-board content, is not sharable. I screenshot every question and every related mark scheme and store them in paper order on one single sheet which I title “raw”.

Step 1: Extract all the content

Our uses of this tab are twofold:

1. When questions or mark schemes are required, they are stored here. They are easily findable through filters.
2. The source of the base analysis of the questions which leads into step 2.

 Step 2: Accurately annotate all raw content 

Step 2: Accurately annotate all the raw content

Once again this is manual. I go through the document line by line and annotate for all of the following:

• Marks total
• AO1, AO2, AO3 mark contribution. Add further AOs if required.
• Main skill (for questions with multiple skill, I select the more challenging skill)
• Specification point
• Secondary specification point (for questions where two specification areas are tested)

The accuracy of this step is extremely important, as errors will throw out the results from the sheet. I do not rush this process and I take this analysis with a clear mind and no distractions.

 Step 3: Begin to analyse skills 

Step 3: Begin to analyse skills

The next step is to identify the allocation of marks per skill. I do this by drawing out the data from the raw sheet that relates to question commands.

I have rarely—in fact never—seen this done by anyone else. When papers are analysed, it is almost universally for content. For me, the skills are my primary objective and, thus, are on the first analysis tab.

 Step 4: Begin to analyse mark allocation 

Step 4: Begin to analyse mark allocation

Once again, this is rarely analysed by teachers. This sheet alone can inform exactly how your students should be practising and testing in preparation for their formal assessments. It should also inform how we create core assessments, end of unit tests and mock exams/PPEs.

 Step 5: Begin to analyse assessment objectives 

Step 5: Begin to analyse assessment objectives

Assessment objectives are very misunderstood. When I analyse papers, I always find that their contribution varies from paper to paper. If you look closely at the video, no two papers have the same allocation. Furthermore, analysing assessment objectives forces me, the teacher, into ensuring that my classroom builds these specific behaviours or, put it another way: if the assessment requires K&U, application and evaluation and analysis and our classrooms do not cause these behaviours, students WILL underperform in the assessment.

 Step 6: Begin to analyse specification content 

Step 6: Being to analyse specification content

This is the step that most teachers will start with. In my opinion, it should be the last/penultimate step. There are two primary aims for this sheet:

1. To establish which topics are most heavily tested and need to be “over-learned”.
2. To establish which topics are both substantial and have been lightly tested in recent papers and need to be “over-learned”.

Please note that any teacher or business that attempts to make predictions is foolish. Only the exam writer can predict an exam. But the specification analysis gives me an excellent context in which to work.

Teachers should be very cautious when providing this information to students. Students, like all human beings, will try to find the path of least resistance during challenging moments like exam series. Therefore, I use terms like “over-learn” very specifically. I NEVER make exam predictions to students.

 Step 7: Make yearly paper summaries 

Step 7: Make yearly paper summaries

Once all the data has been inputted and analysed it can be manipulated in numerous ways. One of the most useful is to create a sheet for paper summaries which shows how each paper has been structured. This process is extremely useful when writing assessment materials for students to use. In my context, I use this sheet to help me plan National Diagnostic Exams and to ensure they are as close to “the real thing” as possible.

Now to the big question: I already know that I am going to receive requests to share these documents with colleagues. I get asked this on a regular basis. However, I am going to disappoint you and state categorically that I will not be sharing our raw analysis documents. There are two reasons for this:

1. I am not legally entitled to share copyrighted material such as exam board questions and mark schemes.
2. The documents are live and they are sensitive. The formulae are complex so adding a row or deleting a column by mistake or transferring the document between different iterations of software can mean it doesn’t work anymore. This would mean that, once shared, the tendency for support requests from teachers to my core team would be very high. 

Now to the good news: Instead of sharing the documents for teachers to populate themselves, every March I am going to publish the most important data from the documents as a free download for teachers. I will structure these documents so that you can provide them to students and so that they can act as learning resources directly in your classroom. I hope that this will enable you to get the most of the analysis without having to do the legwork and also in a well designed and presented format.