Blog 021: 2024 Summer Assessment Technology

August 21, 2024

I find that I'm able to ingest new information more quickly and deeply during the summer when my mind is not cluttered with all of the day-to-day responsibilities that mount in autumn. In turn, Mountain Measurement would like to encourage you to capitalize on the extra headspace that you may have during the summer months to explore the features of your assessment technology.

You may want to consider learning how to create test questions using some of the alternative item types that you have never used before. Mountain Measurement encourages you to start with the easier item types and progress to the more innovative ones. Have you ever created a matrix item type? Give it a try. If your assessment technology is capable of it, how about a bowtie? Give that one a try too. In order to stretch yourself, you might want to try converting some of your multiple-choice items into some of these alternative ones. I suggest that you convert at least 3 different items to each of the item types that you are learning. Edit them and polish them, so that they can legitimately replace the more traditional items that you have used in the past. When you are finished, compare the traditional and its corresponding alternative item. If you're happy with the new one, then pat yourself on the back. If not, ponder the item some more and see what you can do to improve it. Remember that just like good writing, item development takes a lot of revision and editing.

One other assessment technology feature that you may want to consider learning about is the examination or item statistics section. Although the capabilities of these features vary greatly from system to system, almost all of them will provide you with the number or percentage of students that provided each response to each question. This information is useful in determining if the item was too easy or too challenging for your students. From there, you may want to find out if your system will give you the item score/total score correlation for each item which is also known as the point-biserial or the discrimination of the test item. Although the item score/total score correlation can run from -1 to 1, in reality, you'll typically see values between -0.2 and 0.5. The greater the value, the better the item is at differentiating between high performers and low performers. If the item has a value less than zero, then the item is likely to have some issues. If you have any items with an item score/total score correlation less than 0, you may want to check if the item is keyed correctly or consider if one of the distractors (incorrect answers) may be more plausible than you had originally considered.

Another important area to learn about is the content coding capacities of your assessment technology. The first step may be entering a new nursing framework into your system and all of the topics associated with that framework. Each system is likely to have its own lingo for what they call frameworks and topics. Some may call them domains and sections and others may call them outcomes and objectives. Regardless of what they are called, be sure to enter all of the topics associated with each framework. For example, when entering the "Categories of Client Needs" framework, be sure to enter each of the eight subcategories. Keep in mind that other faculty members are likely to have already created frameworks and you should use the pre-existing frameworks whenever possible. This will allow you track the performance of your students within each topic across courses. From there, be sure to code each of your test items to one and only one topic within each framework. See Figure 1 for a diagram of how a single item was mapped to six different nursing frameworks. Mountain Measurement recommends that you consider coding your items to the following common nursing frameworks:

The objectives of your course(s)
Your program's end-of-program student learning outcomes (EPSLO)
Categories of Client Needs (i.e., NCLEX Test Plan)
Steps of the NCSBN Clinical Judgment Measurement Model
Categories of Human Functioning
Categories of Health Alteration

Figure 1. Content Coding of Item 3527 to one topic with each of six different nursing frameworks.

If you take the time to explore some of this terrain during the summer, I am confident that you will be more likely to use these features when creating, editing, and evaluating your assessments. Ultimately, this will result in more useful and valid assessments for you, your students, and your program.