Blog 008: How has/will COVID-19 Impact Nursing Program Evaluation?
In order to express gratitude to all of the nurse educators preparing new nurses to save lives during the COVID-19 pandemic, Mountain Measurement, Inc. hosted a panel discussion titled “How has/will COVID-19 Impact Nursing Program Evaluation?” on Thursday, May 28, 2020.
The panel featured the following distinguished leaders in nursing program evaluation:
Nell Ard, PhD, RN, CNE, ANEF, Director, Accreditation Commission for Education in Nursing
The Panel has agreed to provide answers to the questions that were not addressed during the web-based panel discussion. Those responses will be posted here when they become available.
“How has/will COVID-19 Impact Nursing Program Evaluation?”
Questions from the audience
Q: Nancy - In NJ many of us have requested waivers for our programs to be on campus for skills and simulation labs in the event that the Governor institutes only online learning in the fall. We believe that this is needed especially for 1st-semester nursing students just learning skills. Do you think that this will set us up for liability?
A: As long as you follow the recommended guidelines for simulation, you should be fine. I think this is especially good for first semester students as it will prepare them for their actual clinical experiences. You might be interested in the nurse leader supported practice/academic partnership for when you can start clinical experiences again: https://www.ncsbn.org/Policy_Brief_US_Nursing_Leadership_COVID19.pdf. We are hoping that practice will begin to understand how essential student nurses are to the workforce (they should not be considered “visitors”) and that this will be a collaborative experience. Watch for our JNR supplement this summer for the results of 3 national studies we conducted to investigate nursing program quality indicators. Quality clinical experiences and simulation were identified as being a critical aspect of nursing programs.
Q: Karen -Are there any specific suggestions of topics we should be evaluating now?
A: It likely goes without saying, but I will reinforce, measures to address accreditation and regulatory expectations is an essential part - but not the all - of a program’s systematic plan for program evaluation. Beyond these core expectations, selection of topics to include in program evaluation are best generated from within the program, by its faculty and other key stakeholders. From my perspective, these topics must be meaningfully and practically approached with a backward design mindset.
Before we ask the “what” we should measure and the “how” we should measure, ask yourself “what” do you want to know and “why”. Topic suggestions will generate from how you and your program answer these questions. In an effort to address this question more specifically, let’s use this framework, and ask:
- What do you want to know about your program's performance during this time of COVID-19?
- Why? What insights do you want to gain?
Rather than adding new topics, consider examining the impact of the disruptions of COVID-19 on existing program performance as revealed in its measures.
Q: Nancy - What is the NCSBN doing or thinking in regards to the required number of hours for students at the bedside in states where clinical sites are extremely limited?
A: As above, you might be interested in our JNR summer supplement on quality indicators in nursing programs. Besides conducting 3 national studies (a Delphi, a qualitative study of 5 years of approval site visits, and a quantitative study of 5 years of annual reports), we did a literature review. While we found how important clinical experiences are, we did not find evidence to support numbers of hours. In the U.S. the median number of hours ADN programs have is 573 hours, whereas it is 712 in BSN programs (Smiley, 2019, JNR), so BONs that require hours go with this, or lower. If you look globally, you’ll find the U.S. lags behind English-speaking countries (Canada, Australia, UK, and New Zealand) in numbers of clinical hours with patients (Hungerford, 2019, JNE). We do realize clinical hours are not always easy to find. Educators must be creative. They should look for off-hours and days (weekends, evenings, etc.) and also for clinical experiences in alternative settings, such as daycare settings, assisted living, clinics, etc.
Q: Karen - How will student feedback be included in future program evaluations? Will qualitative measures be considered?
A: Students are key stakeholders in nursing programs. As such, their feedback should already be considered as one of many data sources considered in program evaluation. As previously emphasized, method selection rests fundamentally on the questions about “what” you want to know (and why) together with the practicality of collecting the types of data needed to answer those questions and the time, resources, and skill set needed to carry it out.
Q: Nell - Our ACEN self-study is due next semester. How much information regarding the fall 2019 semester changes should we include in the document?
A: The Self-Study Report (SSR) is the program’s “story” about its compliance with the ACEN Standards and Criteria. The SSR typically covers the last three years of events. Therefore, the program’s response to Covid-19 would be part of the story. Since the SSR is due soon, instead of incorporating information throughout the SSR which may be in the finalization phase, the faculty can attach an addendum that addresses the program’s response to Covid-19 which primarily may affect Criterion 1.10, Criterion 2.10, Criterion 3.8 and Criterion 3.9, Criterion 4.7 if some evaluation methodologies changed, Criterion 4.9, Criterion 4.11, and Criterion 5.4. Programs are encouraged to contact the ACEN office to speak with a Director if they have any questions.
Q: Nancy, Rose - Have you thought about outcomes several years out? While we may not see changes in NCLEX pass rates from graduates this spring, what will the pass rate be for those that started their first semester of nursing school during COVID-19, i.e., the Spring 2022 graduates? How will step-outs for COVID-19 financial issues impact retention, 4-yr graduation rates, etc.?
A: The BONs have thought about long-term outcomes of this, and they will be collecting data on this in their annual reports. My perspective is that the alternative teaching approaches for a semester won’t make a big difference in pass rates. I do think it could affect the retention and enrollment of new students, which of course will affect the workforce. We just need to collect data so that we can make predictions for the future.
Q: Nancy, Rose - What are the presenter’s thoughts of COVID’s impact on pass rates?
A: I believe I answered this above. Also, I refer you (again) to the JNR summer supplement as it has approval guidelines for BONs. Remember, NCLEX pass rates are only part of what BONs look at.
Q: Nancy - Many Boards of Nursing have been focused on patient care and have not said much about schools of nursing. How will the Boards of Nursing address issues such as an increase in simulation activities, clinical hours, or accreditation?
A: Actually, most BONs have considered education during these unprecedented times. Please see this document for their requirements: https://www.ncsbn.org/Education-Requirement-Changes_COVID-19.pdf. If you have a question about your BON, please either contact the BON or contact me and I’ll be happy to get your question answered.
Q: Judie - I am wondering if a lot of schools waived some of the admission or application requirements due to the COVID-19 situation?
A: This is a very important consideration. I am not aware of any program that is waiving admission requirements, rather they are extending deadlines to apply and complete the application process. This will vary widely as each state has different requirements for gathering for the purpose of taking entrance exams; however, companies that provide entrance exams to schools of nursing have been able to adapt and provide for remote proctoring.
It would not be advisable to waive requirements for admission. Your program’s current admission requirements are based on data from student success and attrition. To waive any requirement for admission may negatively impact retention which is a trigger for regulatory and accreditation agencies to review the nursing education program.
It is my understanding that applications to post-secondary institutions are down everywhere. However, it is not clear if this is the case for nursing education programs specifically. We will just need to wait to see. There will likely need to be some consideration given to the final grades of applicants coming in from high school or transferring from other educational programs. Nursing programs are generally quite competitive and very strong applicants often get turned away so it is unlikely that any changes to admission requirements will impact the quality of students. However, this is something we definitely need to monitor in the years to come.
Q: Nell - Many of our End of Program Student Learning Outcomes are at least in part evaluated by scores on proctored standardized tests. We were not able to utilize these same tests because we could not offer proctored settings. We might additionally address these areas in our Mtn Measurement report data in place of that. In the future, we hope to offer proctored testing. Are we on track here or do we need to change our SPE?
A: Just like with any project where data is being collected, sometimes there are extraneous variables beyond the control of the program in collecting data - such as Covid-19. Therefore, the faculty should indicate in the systematic plan of evaluation that secondary to Covid-19, the data could not be collected. The faculty may then need to modify the frequency of assessment specific to the end-of-program student learning outcomes (SLOs) that now have missing data. Instead of waiting until the next scheduled time to review, the faculty should plan on collecting data for that end-of-program SLOs at the next opportunity along with those that are scheduled for this next year.
The faculty can certainly look at the information available related to the Spring 2020 graduating cohort by using post-graduation data such as the Mountain Measurements report. However, the faculty need to remember that this data is collected after graduation and other factors (e.g., work experiences) may have an impact on the data that would not have affected the data collected at or near the end of the program prior to graduation.
Q: Nancy - how does the COVID pandemic impact plans for the NexGen NCLEX implementation?
A: So far there has been no impact on the Next Generation NCLEX. Please keep up with any information on it here: https://www.ncsbn.org/next-generation-nclex.htm
Q: Rose - Do you think this is going to affect what our incoming class is going to look like 5 years from now?
A: COVID-19 is definitely going to have long-term impacts on nursing programs. There is no question that students entering universities and colleges in the next few years will have missed some of the academic content that was previously covered. However, schools are now adapting curricula and prioritizing what content is covered. Nursing programs will need to be sensitive to the needs of students entering their programs in the years to come and be prepared to address content that may have been previously addressed in secondary schools. It will be prudent for post-secondary institutions (including nursing programs) to communicate with secondary schools so that the expectations of students are realistic.
Q: Rose - How are programs going to evaluate their virtual simulated clinical experiences?
A: Like all learning activities, documentation of virtual simulated activities will be very important. Programs should use standard templates to track the quantity of simulation that is substituting clinical experiences and to connect simulation activities to program outcomes. We have much to learn about the effectiveness of virtual simulation and if it produces outcomes that are comparable to clinical practice or even face-to-face simulations. IN many ways linking evaluation data to virtual simulations will be much easier than linking it to clinical experiences. In virtual simulations, we can control student experiences and ensure every student has the same experience. The methods of evaluation do not necessarily have to change because the medium changed.
Q: Nell/Judie - Do you have suggestions for clock hour programs to keep track of student engagement/attendance?
A: Clock hour programs can require the student to keep a time log for the courses and/or clinical experiences. This log would then be submitted as a course requirement. Additionally, the program could plan some synchronous activities that would require all students to participate, and then faculty can account for those specific times/hours.
Many of the products that have become available will include information on the average length of time students will spend completing the assignment. Use this information as well to document the time students have spent on these activities.
Q: Nancy - Some states have implemented emergency policies to allow graduates of nursing programs to function in a nurse role with supervision without passing the NCLEX. Is there concern for / and ideas to ensure graduates complete the NCLEX exam in a timely manner?
A: There has been some concern about this, and most BONs have seen that students are taking the NCLEX. We know from our research that the sooner a student takes the NCLEX following their program, the better they will do on the NCLEX.
Q: Nancy - We are finding that virtual simulation is more time-intensive and rigorous in regards to constant contact compared to onsite clinical. There has been some documentation to support that virtual simulation is 2 hours compared to 1 hour of onsite clinical. Have the Boards of Nursing had any discussion about this?
A: BONs have definitely had a discussion on this. We are reviewing the literature on that now to see if there is supportive evidence. We encourage research in this area. Remember, NCSBN does have a grant program where we can award up to $300,000 for a proposal that meets our criteria. Here is a link to details about our grant program: https://www.ncsbn.org/10682.htm
Q: Nell - I think we can expect that many programs will submit substantive change reports based on our positive experiences during this period when there have been changes in our methods of delivery. If this should happen, will ACEN be able to manage a large influx of reports for January?
A: The ACEN processes substantive changes as they are received. The review process can take as long as eight weeks depending upon the volume of substantive change reports received. The ACEN has developed templates for programs to use when reporting specific types of substantive changes (e.g., implementation of distance education); the various templates are accessible at https://www.acenursing.org/for-programs/resources-reporting-substantive-change/. Ideally, substantive change reports are submitted no less than four months prior to the proposed change. Programs are encouraged to contact the Coordinator of Substantive Change if there are questions regarding the components required when submitting a substantive change. Incomplete submission of the required components can delay the overall review process.
When something is changed, like the SPE, clinical strategies, simulation, on-line didactic - always remember to document the rationale. Place these rationales in the minutes of meetings or when communicating with faculty, or boards of nursing, or SPE.
Q: Nell - Relative to the concern for continuing our ability to teach and ultimately evaluate clinical nursing courses: I believe that the outcome of a safe nurse in the real post-COVID health care environment is a real and mortal concern. In NYC, the COVID situation is still in the RED zone. In fall 2020, in our AAS we have a second clinical semester [2/4], who have not cared for patients in the live environment. We have been denied clinical placements for fall and expect we will be fully online in a virtual clinical environment for a second semester in a 4 semester AAS program. We are considering deferring the clinical experience until the clinical agencies will accept students. We maximized virtual opportunities for medical surgical clinical experiences this semester but we are deeply concerned that the virtual setting has its limitations in terms of evaluating safe delivery of care.
A: According to the ACEN Glossary definition, clinical/practicum learning experiences are: Direct, hands-on, planned learning activities with patients across the lifespan, interaction with the interprofessional team, and interaction with the patient’s family and friends that are sufficient and appropriate to achieve the end-of-program student learning outcomes, program outcomes, and/or role-specific professional competencies, and are overseen by qualified faculty who provide feedback to students in support of their learning. Clinical/practicum learning experiences are required for all nursing students enrolled in any undergraduate or graduate program, including all students enrolled in post-licensure undergraduate programs, graduate programs, all program options in any undergraduate and graduate programs, and/or certificate program options.
Therefore, programs will need to ensure that all graduates have some direct, hands-on experiences for each graduating cohort. Programs will also need to ensure that they are compliant with any state nursing regulatory agencies' requirements related to clinical learning. To ensure that all cohorts have these experiences, programs may be altering the sequencing of the courses and/or providing an incomplete grade until the clinical/practicum requirements can be met. Additionally, faculty are also encouraged to think outside of the box with regards to where clinical/practicum experiences may occur. If the end-of-program student learning outcomes can be met using alternative clinical sites to those typically utilized by a program, the faculty should consider those options.
- We have updated our simulation policy to adapt to virtual clinicals, etc.
- Virtual sims and/or simulations are intentionally designed while clinicals in the hospital can sometimes be opportunity-based (i.e., labor and delivery - no pregnant delivering moms or patients refuse students)
Continue reading this article
Sign up for a free account to access the full blog post and more.
Create Free Account Already have an account? Log in
