Blog 011: Understanding the Impact of the Modified NCLEX® Exam on the NCLEX® Program Reports

• reducing the maximum number of operational items,
• reducing the maximum allotted time,
• not administering any pretest items,
• and removing the special NGN sections

in a webinar produced in April 2020.[1] In October 2020, pre-test items and the NGN special section were reintroduced. See Table 1 for the details of these modifications.[2]

Table 1: Details of the Modifications of the NCLEX-RN® and NCLEX-PN® Examination

To ensure that these changes did not negatively impact nurse educators using the NCLEX® Program Reports, Mountain Measurement, Inc. conducted research on the impact of the changes to the NCLEX® Examination on the information provided in the NCLEX® Program Reports.

NCLEX® Program Reports

Data Integrity

Because the testing centers were either closed or not fully operational between March 16 and March 25, 2020, NCSBN decided to modify the NCLEX® Program Reports reporting periods which cover this timeframe.  Typically, the NCLEX® Program Reports reporting year goes from April 1^st through March 31^st. Given the test center closures, the 2019-2020 reporting year ended on March 16^th, 2020, 15 days earlier than prior years. The 2020-2021 reporting year began on March 25^th, 2020, the day testing resumed using the Modified NCLEX® Examination which was 6 days before the usual commencement of the NCLEX® Program Reports reporting year. See Table 3 for details on the impact of the changes to the reporting year for each edition of the 2019-2020 and 2020-2021 NCLEX® Program Reports.

Table 2. Modified NCLEX® Program Reports Editions.

Although the reporting periods were different than prior years, the changes were small, representing, at most, a 16% reduction in the number of days in which testing was available. Noting that March is a month where very few graduates typically complete their NCLEX® Examination, the impact of these changes on the size of samples used in the NCLEX® Program Reports was globally small. Nonetheless, there was a slight decrease in the number of candidates testing in Q2 and a slight increase in the number of candidates testing in Q3 which may be attributed to the limited number of seats available for testing during Q2. Mountain Measurement recommends that programs using the Q2 and Q3 Editions of the 2020-2021 NCLEX® Program Reports exercise caution when interpreting these editions of the Reports. The volume of test-takers for all other 2020-2021 reporting periods to date was similar to prior years. Given that the odd reporting year samples were similar in size to the samples from prior years, it is reasonable to assume that the characteristics of the comparison group samples were also similar to prior years.

Despite the similarity in the sample of comparison group candidates, there may be differences in the sample of graduates from a particular program that completed the Modified NCLEX® Examination in 2020-2021. Mountain Measurement recommends that programs validate the sample size and demographics of their graduates that tested during the impacted reporting periods before making longitudinal comparisons with NCLEX® Program Reports data.

Table 3. NCLEX® Examination Volume and Pass Rate by NCLEX® Program Reports Reporting Quarter[3]

Overall Performance Report

Given that the representativeness of the sample of test-takers in the comparison groups, the ability estimates of the test takers, and the passing standard for the NCLEX® Examination was not impacted by the modifications to the NCLEX® Examination, then from a psychometric perspective, passing rates should not be impacted. Programs experiencing a change in the passing rate should attribute this to changes in their program’s offerings, the competencies of their graduates, the characteristics of the sample of graduates from their program that tested, and the impact of COVID on their graduates' lives. Passing rate changes should NOT be attributed to the changes in the specifications of the Modified NCLEX® Examination.

NCLEX® Test Plan Report

With the Modified NCLEX® Examination, candidates with ability estimates close to the passing standard take fewer items than they would have with the Pre-COVID NCLEX® Examination. Given that they are administered fewer items, the precision of Modified NCLEX® Examination NCLEX® Test Plan area ability estimates is less for candidates who would have taken at least 131 items on the Pre-COVID NCLEX® Examination.

Although the individual test plan area ability estimates for some candidates are not as precise, they are not biased, meaning that the accuracy of NCLEX® Test Plan ability estimates derived from the Modified NCLEX® is the same as the ability estimates derived from the Pre-COVID NCLEX® Examination. Moreover, recall that the NCLEX® Program Reports utilize only the median of the NCLEX® Test Plan ability estimates for each program. Therefore, any potentially small changes in the accuracy of individual test plan ability estimates that might have occurred due to undetectable bias are likely to be neutralized when only the median ability estimate is taken as a proxy for the program’s typical graduate. Therefore, it is safe to conclude that the percentile ranks reported in Section 4 are not impacted.

Content Dimension Reports

Since content dimension performance is calculated in the same manner as NCLEX® Test Plan performance, the precision of individual content dimension ability estimates for Modified NCLEX® Examination candidates with ability close to the passing standard will be less than they would have been for the Pre-COVID NCLEX® Examination for the same reasons stated above. Despite this similarity, the data supporting the Content Dimension Reports are not constrained by the NCLEX® Test Plan. The Content Dimension Reports provide topic area performance for only programs that had at least ten candidates complete at least three items from the topic area. When there are less than ten candidates that satisfy this criterion, there is not enough information to calculate a stable measure of central tendency, and the percentile rank is not provided meaning that blank space is found where there normally would be the bar of a bar chart.

Given the decrease in the number of items administered with the Modified NCLEX® Examination, the number of blank bars is expected to increase. In addition, given the reduction in the number of items administered, the number of extreme scores (when a candidate answers all of the items of a content dimension correctly or all items incorrectly) is expected to increase. The precision of ability estimates for extreme scores is far less than non-extreme scores thereby exacerbating the reduction in the precision for individual topic area ability estimates. As with the NCLEX® Test Plan ability estimates, although precision is impacted, the estimates are theoretically not biased thereby allowing one to infer that the accuracy of the content dimension ability estimates is also not impacted.

To estimate the size of the impact of the changes to the Content Dimension Reports, hypothetical Modified NCLEX® Examinations were created using the actual NCLEX® Program Reports data for US candidates completing the NCLEX® Examination between April 1 and June 30, 2019. Table 5 provides the change in the percentage of missing, extreme, and non-extreme bars found on the NCLEX® Program Reports between the two samples (Modified NCLEX® Examination and Pre-COVID NCLEX® Examination). The largest change reported in this table was slightly less than 5% and the median of the absolute value of the changes was 0.25% suggesting that the size of the impact was quite small.

The largest increase in missing bars came in Stress, Adaptation, and Coping (5% for RN programs and 3.3% for PN programs) followed by Stages of Maturity (3% for RN programs and 1.9% for PN programs). For RN programs, the largest change in bars with at least one extreme ability estimate was in Stages of Maturity. For PNs the largest change in bars with at least one extreme ability estimate was in Human Functioning.

Table 4. Changes in the Percentages of Non-Extreme, Extreme, and Missing Bars by Functional Area.

Figure 1. Changes in the Percentages of Non-Extreme, Extreme, and Missing Bars by Functional Area.

Test Duration Report

The Test Duration Report provides information about the number of items completed and the amount of time that candidates spent taking the NCLEX® Examination. Values reported in this section of the NCLEX® Program Reports have changed due to changes in the maximum number of items and the maximum test administration time with the introduction of the Modified NCLEX® Examination. For the semi-annual fall reporting period (Apr 1 – Sept 30), the percentage of all first-time US test-takers administered the maximum number of items was at least 10% greater for candidates taking the Modified NCLEX® Examination than the Pre-COVID NCLEX® Examination (See Table 6).

Table 5. The average number of items administered & time spent and percent of minimum and maximum length test-takers from all first-time US test-takers in Fall 2019 and Fall 2020.

The average time that candidates spent taking the Modified NCLEX® Examination was about 30 minutes less than the Pre-COVID NCLEX® Examination. Fewer candidates are expected to run out of time as the mean item response time needed to complete the Modified NCLEX® Examination is approximately 1.8 minutes, an increase of 24 seconds per item from the Pre-COVID NCLEX® Examination.

Programs should avoid making comparisons with their Test Duration Report information across different versions of the NCLEX® Examination. Instead, comparisons should be made between a program’s test duration and the comparison groups listed on page 6.4 (e.g., comparing the average completion time for one program NCLEX® Examination to the average for all programs in the nation).

Conclusion

In response to the COVID-19 pandemic, NCSBN developed modified versions of the NCLEX-RN® and NCLEX-PN® Examinations intending to provide a licensure exam that protects the public while maintaining the psychometric features of the pre-COVID NCLEX® Examination. Mountain Measurement conducted analyses to determine the impact of the NCLEX® Examination changes to the NCLEX® Program Reports. Although these analyses were conducted using US data, the results apply to both the US and Canadian editions of the NCLEX® Program Reports.

The Modified NCLEX® Examination was released to impact the end date for 2019-2020 and the start date for the 2020-2021 NCLEX® Program Reports reporting years. Although this change did not impact the size of the comparison groups for most reporting periods, there was a slight decrease in the volume in 2020 Q2 and a slight increase in 2020 Q3, likely attributed to a delay in testing based on limited availability as test centers reopened in the Spring and Summer. Programs should exercise caution when making comparisons using the Q2 and Q3 Editions of the 2020-2021 NCLEX® Program Reports. In addition, programs should compare the number of graduates from their program testing during a reporting period to the number of graduates in prior years to ensure the comparability of the sample of their graduates in each cohort. This is particularly important when trending the performance of a program’s graduates over time.

• Overall Performance as reported in the Summary Overview will not be impacted.
• Test Plan Performance as reported in the Test Plan Reports will not be impacted.
• Content Dimension Performance will be impacted in the following ways:
  • A small increase in the percentage of “blank” bars is expected.
  • A small increase in the percentage of candidates having extreme scores in at least one content dimension is expected. Given that the median ability estimate was used to describe a program’s typical candidate, little to no impact is expected.
• Changes are expected in the Test Duration Report
  • The percentage of candidates taking the maximum length exam will increase.
  • The average amount of time that candidates spend taking the NCLEX® Examination will decrease.
  • The percentage of candidates running out of time will likely decrease.

References

• The latest information from NCSBN on the impact of the modified exam and other COVID-19 news
• Schwartz, J. A., Betts, J., and Griffin, M. (2020). COVID-19 & the NCLEX: Educator Webinar
• Schwartz, J. A. (2020). Review of NCLEX Modifications Effective Oct 1, 2020
• 2019 NCLEX Pass Rates
• 2020 NCLEX Pass Rates

Abstract

NCSBN began administering the Modified NCLEX® Examination to entry-level nurse candidates on March 25^th, 2020 as a means to increase the safety of the testing experience. The purpose of the study was to investigate the impact of the Modified NCLEX® Examination on the information provided in the NCLEX® Program Reports. The results largely indicated that there was little to no impact on the completeness and accuracy of the information provided and the interpretation of the information contained in the NCLEX® Program Reports.  Projections indicate that there will be a slight increase in the number of missing percentile ranks reported in the Content Dimension Reports. Nursing programs observing a change in their program's performance should not attribute this change to the change in the specifications of the Modified NCLEX® Examination.

[1]Schwartz, J. A., Betts, J., and Griffin, M. (2020). COVID-19 & the NCLEX: Educator Webinar

[2]Schwartz, J. A. (2020). Review of NCLEX Modifications Effective Oct 1, 2020

[3]2019 NCLEX Pass Rates

[4]2020 NCLEX Pass Rates