Multiple choice is a favourite method for many organisations and the people employed to write their assessments, however there are a few things to consider when choosing multiple choice as your choice for collecting evidence which contributes to an assessment of a unit of competency. This article explains why using multiple choice can see you in breach of the rules of evidence and principles of assessment including validity, sufficiency, reliability, flexibility and fairness.
Are you using multiple choice to assess skills?
It is important to assess the skill of a student by having them demonstrate it. If you are asking knowledge questions for a component of the unit that clearly requires a skill to be demonstrated, you are not using a valid assessment method by asking them a multiple-choice question. Many people tell me they also ask the student to demonstrate the skill in a practical task, however if that practical task is sufficient for a student to deemed satisfactory in that component, then asking them multiple choice questions in addition can lead to over assessing a student. Knowledge questions should only be used to assess knowledge.
Are you asking students to recite knowledge provided to them in the content?
There are a couple of issues here you need to consider, firstly, is the content you are asking them to recite declarative or is it your interpretation of information? Declarative knowledge is that which could be defined as a standard, legislation, principle, or regulation, in other words it can be described as an industry ‘truth’. If you have interpreted the ‘truth’ into something more subjective, your ‘correct’ answer may not be as definitive as you may think.
Let me provide you with an example here.
When you ask a student ‘what is the minimum requirement for the distance between a stove top and an installed range hood?’, they can give you a clear factual response, as this is declarative knowledge. There are Australian standards that specify this as a ‘truth’.
On the other hand, if you ask a student a multiple-choice question like ‘how do you bake a cake?’ and give them a list of options, you may be expecting the correct response to be the recipe you provided to them in your content. This is called procedural knowledge, and we have all heard the saying ‘there is more than one way to skin a cat’. This recipe is subjective, as there are many methods, yet you are insinuating that this recipe is the only way to bake a cake. Here you are inadvertently misleading the student into thinking there is only one correct answer when in fact there are many. They may even select the ‘incorrect’ answer but have a decent understanding of how to bake a cake and have a good justification as to why they chose one of the other answers.
If this is the case, the student should be provided with an opportunity to justify why they believe their answer is correct. Unfortunately, this particular assessment method does not allow for this to occur unless the student is asked after the assessment task has been marked.
This means that for every subjective multiple choice question you ask a student, you need to have a provision for asking the student why they chose the answer you were not expecting and this takes time if you are going to provide this opportunity to each and every student. In this case, you are better off asking the student a short answer question that simply asks, ‘explain how you would bake a cake.’ Of course the answers may vary but the principles should still apply.
If you decide you do want them to prove they can recite the content, consider this first. Is memorising subjective content appropriate for the AQF level you are writing for or is it below the cognitive expectations for the qualification? If you are assessing students for a certificate III or higher, the student needs to show they understand concepts and issues, employers want them to be able to think and work things out.
Secondly, if you think it is appropriate to ask a student to memorise subjective content as part of their assessment for competency, have you explained to the student that they need to refer to the content as a required resource for the assessment task? For many reasons, a student may not have read the content, and if the knowledge is not a ‘truth’, especially if they are experienced in the field and have a different but valid understanding, they are not necessarily required to read the content unless specified as a requirement of undergoing the assessment eg. a scenario designed to assess contextual knowledge. I often see multiple choice questions presented to students without adequate conditions for assessment and listed resources, and this situation falls directly into this category.
Is using multiple choice for factual knowledge any better?
So, lets assume you have decided to make life easier for yourself and you are only going to assess factual knowledge in your multiple choice. There are a number of other considerations you need to address here. Firstly, how are you going to ensure the student knows the correct answer and is not eliminating the incorrect answers to reach the right answer? We have all watched Hot seat or other game shows where we see the contestants use logic to reach the most viable answer. Yes, problem solving is a great skill to have, and a wonderful bi product for assessing this employability skill, but is assessing this employability skill the intention of this question and if so, how do you know they didn’t know the answer? To put this more bluntly, if they get the answer correct, there is one of 3 reasons, they either guessed and got lucky, they knew the answer or they have great problem solving skills, but you cannot provide evidence as to which of these it is and therefore this is not a valid assessment method. You only have evidence that they got the question right, not that they knew it.
Let’s now be less optimistic and think about what happens if they get the answer wrong. In VET you are required to provide the student with the opportunity to reach competency, therefore you are obligated to allow the student to resubmit the assessment. If you provide the student with the same questions to answer, they now have less options to choose from and a greater chance of guessing the right answer. Your once invalid assessment method has now become even more invalid.
Using question banks?
Now if you are clever, you have already considered this, and you have developed a bank of questions (possibly using an LMS) and these are presented to a student at random. This is great to support authenticity as it provides different questions to different students so they cannot tell each other the numbers to select. Of course, to do this properly takes a lot of effort and it is not enough to just create a question bank for the knowledge requirements for your unit of competency. This is because when you randomise your questions, you cannot guarantee an adequate question for each component you are intending to assess will be presented to the student, they may get three questions for one component and none for another. To mitigate this, you need to set up a question bank for each component of the unit you are intending to assess, and yes, this takes far more time than writing short answer questions and having a well written marking guide to assess the student responses. Are you resourced to develop multiple questions as a contingency for this?
Is a multiple choice question something they will experience in the workplace?
When we read the assessment conditions of a unit of competency, they often require assessment to be conducted in the workplace or in a simulated environment. I am personally yet to be asked multiple choice questions by my manager or clients to ensure I know what I am doing. I am however often asked WHEN I am doing something, WHY I am doing it that way. Funnily enough, if I was a student learning how to do my current job, it would be more practical to ask me to write an essay, as writing blog posts like this is something I do regularly. Not everyone will of course, but it’s about choosing the right assessment method for the qualification, and the tasks a student will do after they have gained their qualification. This is how we should be assessing in the VET environment.
This is not to say that you should NEVER use multiple choice, but with the considerations above, think about whether the use of multiple choice questions as a summative assessment method is really the right thing to do. This method has its place, but it is not in formal assessment of competency. It is best used as a tool for the student to gage how they are progressing in a formative manner, and should be reserved as an informal activity to consolidate learning, not assess it.
