Blog 009: How much Cheating is Happening in Nursing Education Today and What Can We Do About It?

Jennie Miron, PhD, RN - Board of Directors for the International Center for Academic Integrity and Faculty at Humber College.

Rose Rossi Schwartz, PhD, MBA, RN - Associate Dean of Undergraduate Programs Widener University with expertise in secure exam delivery through an LMS

Lynn Webb, EdD - Psychometric consultant and auditor for assessment programs pursuing IEC/ISO 17024 accreditation through ANSI. Recently, Lynn was a secret shopper participating in mock test sessions that were proctored by the leading remote proctoring solutions available today. 

James Wollack, PhD - Editor of Quantitative Methods for the Detection of Cheating and faculty and the University of Wisconsin.

The Panel provided answers to the following questions that were not addressed during the web-based panel discussion. 

Questions from the audience

Q: Jennie: Have you implemented a policy for academic integrity in your learning environment and how long did it take you to see a difference in students and also faculty?

A: We have just moved from policies to academic regulations exclusively. They are still labeled as academic misconduct and can be viewed here.

Within my faculty, we have many different initiatives that seem to have made some difference (although I still believe we under-report). Some of the initiatives include: pledging statements signed by the students before tests/exams and handed in with all assignments, introduction to the values of academic integrity (honesty, trust, fairness, respect, responsibility, courage) through a student-led interactive exercise at orientation -- this year it was done virtually (scroll down to view the We Believe in Academic Integrity video).

We have an established process for following up on suspected acts of dishonesty with an online report form to make it easier for faculty, we just finished academic integrity modules -- which eventually will be an expectation for all 1st-year students to complete, and our specific faculty site has an academic integrity tab full of resources, available here.

Q: Jennie: Did your nursing program develop a separate policy for academic integrity from that of the larger institution? If so, how was that explained?

A: We have a process for test/exams, not a stand-alone policy, that goes through the responsibilities of students and faculty. We also have banners that we encourage faculty to use on their Learning Management Systems and emails that highlight the values. 

Q:  Jennie: What should be included in an academic integrity policy?

A: I just recently co-authored a paper that speaks to policy and I have included the reference here. I also include the link for an article by Bretag et al. that speaks to exemplary practices around academic integrity policy. In a nutshell, an academic integrity policy should be clear about expectations, the process for breaches of integrity, and consequences.

• Bretag T., & Mahmud S. (2016). A conceptual framework for implementing exemplary academic integrity policy in Australian higher education. In: Bretag T. (eds) Handbook of Academic Integrity (pp. 463-480). Singapore: Springer. https://doi.org/10.1007/978-981-287-098-8_24
• Bretag, T., Mahmud, S., Wallace, M., Walker, R., James, C., Green, M., East, J., McGowan, U., & Partridge, L. (2011). Core elements of exemplary academic policy in Australian higher education. International Journal for Educational Integrity, 7(2), 3-12.
• Stoesz, B. M., Eaton, S. E., Miron, J., & Thacker, E. J. (2019). Academic integrity and contract cheating policy analysis of colleges in Ontario, Canada. International Journal for Educational Integrity, 15(1), 1-18. https://doi.org/10.1007/s40979-019-0042-4

Q: Shawna: Are the AZ guidelines for testing in nursing education available publicly?
A:  The AZ guidelines are available here. These [guidelines] are currently being updated to include more on online testing.

Q: Jennie: As the environment becomes more secure (lockdown browser, video, etc.), do you find that students get more creative in ways to cheat? Or, does the more secure environment discourage them from cheating?
A: Bretag’s research identified trends that we are seeing in higher education that are contributing to increases in cheating:

• Competition
• Commercialization of degrees and diplomas
• Corruption
• Cost-cutting
• Casualization
• Credentialism

Q: Shawna: How are you handling student barriers to Exam Monitoring within ExamSoft (e.g., privacy concerns, security issues, or potential damage to hard/software) based on their device incompatibility or refusal to download?

A: Shawna - At my university, students understand that if the exam is placed in a locked-down browser, they are not permitted to take the assessment until they have followed all of the prompts (show picture ID, scan the room, etc.) to get them to the assessment. It, thankfully, has been a non-issue on my campus, specifically.

Q: We are experiencing more students stating that they have connectivity issues. How do you deal with those issues?

Q: Jennie: Have you had any issues with having students not being able to afford (and have access to) a second device for zoom monitoring?

A: Yes, we have had students with limited resources. Our school has been very accommodating and has come up with loaners, temporary financial assistance and other strategies to support students in need

Q: Shawna: If you have a student flagged for suspicious behaviors, do you give them a zero score? Do you take a percentage off the top? Do you spell this out in the syllabus?

A: Academic integrity violations are spelled out. Students understand that the video will be reviewed for any flags. If anything looks suspicious, a meeting is scheduled with the student. If the activity warrants a zero, the student is told this during the meeting. An academic integrity violation letter is issued to them for documentation. Of course, the student can appeal the decision if they so choose to do so.

Q: Jennie, Shawna: How strict are the consequences for cheating at your programs? Do you feel students should have more than one chance if found to be cheating?

A1: Jennie: At our school, we do consider the type of cheating, the number of cheating offenses, the intent, and the penalty is discussed with faculty and our Associate Deans. A lot of our discussion comes down to intent and whether this is repeat behavior. I am very interested in using restorative practices combined with educational requirements for the student to complete. This way the student is challenged to consider the harm they have done and hopefully will learn the skills they need to avoid departures in the future.

A2: Shawna: We do take into consideration the situation; it could be deemed as a “teachable moment” but is still documented.

Q:  Shawna: How do you get faculty to have a common approach towards cheating?  You can have a policy in place but faculty do not always have the same view on what constitutes cheating and how best to respond to it.

A: This warrants constant communication and discussion on the matter; everyone must be on the same page for a consistent message to the students.

Q: Shawna: There have been students testing from home who have family members, children, etc. at home, who have interrupted the online testing process. What do you suggest to improve this situation?

A: We know and understand as it’s the same environment that faculty are dealing with. We tell them to do the best that they can; find a bathroom or a closet with good lighting and wifi access if need be. Creativity is sometimes necessary.

Q: Jim, Lynn: Any thoughts of AI proctoring/reviewing vs live proctoring?

A1: Jim: Both AI and Live Online Proctoring (LOP) constitute significant advantages over honor code or merely trusting students to respond with integrity to exam questions. I recognize that LOP is significantly more expensive than an honor code; however, if the money is in the budget, at least until we understand more about AI, I think there are many advantages to using LOP, particularly for higher stakes exams like NCLEX, but even for classroom exams. As far as I’m concerned, the biggest advantage of LOP is that it provides a mechanism for preventing content from being stolen during the exam.  AI may be able to detect if someone is taking pictures or writing down test content during the exam, and it may be possible to cancel their score afterward; however, by then, the items may be on the Internet and you may have a much bigger problem on your hands. Otherwise, I feel like it is too early to fairly evaluate AI. The latest technological advances are always given an unfair advantage because there’s so much fancy technology available and so much fun science fiction out there that the average person on the street tends to be exceedingly different to technology. The fact remains that AI is sufficiently new, and we know very little about how well it works. What we know is that we can monitor certain variables during the test (e.g., eye movement), build distributions of those variables, and set cutoffs so that we flag individuals for whom we observe “extreme” results. So the fact that AI can “detect” anomalies is meaningless because every distribution will have values in the tails, even due to random error alone. To the best of my knowledge, there is no published work that links these variables or the flagging thresholds to actual cheating or cheating behaviors. And until there is, it’s going to be very difficult to use AI to make any cheating claims unless physical evidence of the cheating is recorded (e.g., the person can be seen on camera accessing prohibited materials). For sure, some of this will happen, but I think the much more common situation is that the behavior flagged will not be accompanied by clear visual evidence of wrongdoing. In these situations, I’m not comfortable with AI (yet?) making the inference that the odd behavior (like looking away from the computer screen too much) constituted cheating. I will also add that given the nature of the types of variables that AI is flagging, there is strong reason to suspect that students with disabilities (who are often more distractible, may get up and walk around, move their mouths or speak the words while reading, etc.) will be flagged at a much higher rate, creating possible equity issues.  Even if these cases are reviewed after the fact, we know that the existing AI flag is likely to create a confirmation bias. I do think that that there is a place for AI, but I would proceed with extreme caution in interpreting the results from AI flagging until we learn much more about how it works and would encourage faculty/testing programs to carefully investigate flagged individuals and bring in other data to help eliminate false positives as much as possible.

A2: Lynn: I agree with Jim that protecting the test questions is essential. Online proctoring is showing good results for high- and medium-stakes tests. AI, as Jim mentions, is newer and less established.

Q: Jim, Lynn: What percentage of the exam would you suggest changing each year to protect integrity?

A1: Jim: I think it depends somewhat on how long the test is, and on how secure you believe your content is. But, I think 50-75% new should probably be a goal. This way, enough of the test involves repeated content so that you can compare results with those from prior years and can build the test’s foundation using items that have a strong history and that you’ve found to work well. But, it involves enough new content that students can’t just learn the old stuff. With content split roughly 50-50, you’ll be in a strong position to compare students’ performance on the two “halves”, just to see if there are any students who perform very well on the reused content but significantly worse on the new content (hopefully after adjusting for the fact that the two sets of items may not be equally difficult in the aggregate).  

A2: Lynn: Refreshing tests is a highly effective way to ensure fairness across examinees and maintain the validity and reliability of test scores. Maintaining a strong portion of used items (50% or so) will allow for cheating analyses and decrease the burden of producing new test material.

Q: Do you suggest that we allow students to take breaks during testing?

A1: Jennie: We have shortened our tests/exams (we no longer have 3-hour exams but instead have shorter 1.5-hour exams). We did go through a phase where we accompanied students to the bathrooms but honestly, it felt strange to do this. Our biggest challenge now is the virtual world.

A2: Jim: I agree with Jennie that it’s good to avoid breaks if you can, because they provide opportunities for students to communicate, share/post content, and to look up information. If the test is too long and breaks are needed, I would recommend creating the exam in two “sections”, building in a scheduled break between them, and not allowing students the opportunity to return to the content completed prior to the break. Keep in mind that students with disabilities may need to take breaks, either because frequent breaks are a part of their accommodation or because the extended time accommodation makes it an unreasonably long time for them to go without. If there’s no way to prevent students from returning to previously seen test content, it would be nice to have a system that logs students’ clicks so that you can see if students immediately returned to test content after the break to change several answers (presumably from wrong to right).

Q: Brian: How do you give individualized exam feedback to students without compromising the integrity of the questions in online exams?

A: The first strategy is to code test items to useful frameworks such as the unit objectives and providing subscores (aka topic scores) based on the students’ performance on the items coded to each category of the framework. The second strategy is to select items that were problematic and provide clones of those items for the students to review. The third is to provide individualized feedback to the students on the topics that they missed on the exam. The last one seems quite labor-intensive to me. The other two seem reasonable.

Q: Jim: When using statistical analysis to detect cheating using past questions, how do you differentiate pre-knowledge cheating from genuine knowledge acquisition via group discussions?

A: This is a great question.  At the level of the individual item, there is absolutely no way to know, so we want to make sure to always base decisions on patterns observed over many items. This is one of the reasons that it is important to have quite a few repeated items on the exam. This helps get a reliable measure of the student’s performance on potentially compromised items. However, if we know that the student scored well on repeat items, we still do not know if they had prior access to those items or if they just knew the material. This is why the exam needs to also include a lot of new content. Assuming we have enough new items to get a reliable score, the new content should provide a reasonably clear measure of the student’s overall proficiency level. If the student really struggles with the new content and aces the old content, that is when things start to look suspicious.

Q: Jim: How do you access similarity statistics? What do these specific similarity stats look at? Is there a way to run them in ExamSoft?

A: Unfortunately, I’m unfamiliar with what tools may be available within ExamSoft to help with the detection of cheating. I know that cheating detection software (e.g., similarity analysis) is starting to get incorporated into more testing platforms, but I’ve always written my own code so I’m not very knowledgeable about what’s out there and how well it works. I do know that Assessment Systems markets a program called SIFT that will compute a number of similarity statistics, and I also know that there is a freely available R package (CopyDetect) that performs similarity. There are a lot of different similarity statistics available, and there’s been quite a bit of research on them. The best indexes are omega (ω), the generalized binomial test, and S2. These three have all been shown to be the best at identifying cheating when it exists, while also failing to falsely detect honest individuals at a rate similar to (but not exceeding) user-specified thresholds. The idea behind a similarity index is that it evaluates whether the number of answer matches between two examinees is significantly larger than would be expected due to chance alone, given the overall performance of those examinees and the characteristics of the items. What distinguishes all the similarity indexes is how the expected number of matches is calculated. However, because the general approach behind these methods is similar and because they’re sensitive to the exact same information (e.g., the number of matches), they tend to correlate very high. Therefore, even though these software programs have the means to compute multiple indexes, my recommendation is to select the one that you trust most (I’d strongly recommend one of the ones mentioned above) and look at that one only.  Because of the nature of the dependencies, computing 3-4 indexes will very rarely result in flagging a pair that wouldn’t have been flagged by the “best” index alone, and as tempting as it is to interpret data from 3-4 different indexes in a multiple-measures way and feel extra confident when a pair is flagged by multiple measures, because the indexes aren’t independent (and in fact are highly dependent), it’s very easy to become over-confident in having multiple flagging indexes.

Q: Does using statistical analyses to detect cheating deter cheating? What actions do you suggest taking once cheating is identified using statistical analyses?

A: I don’t know that there is good research on this, but my intuition says that doing statistical analysis does deter cheating, provided the examinees are aware that some post-exam data forensics will be done. As for what to do after flagging, I think it’s very important to understand that statistical analyses identify statistical anomalies that are consistent with cheating. However, people actually need to be the ones who actually make the decision that cheating occurred. So following statistical flags, I think it’s important to do an investigation, including speaking with the examinee in question or possibly other examinees, review all the test data (including response time data or other process data indicating what examinees clicked on and how they proceeded through the test), and review any available video/audio, proctor/AI reports, etc. The entirety of the evidence should be considered, including evidence in support of and not in support of cheating, and the faculty member should decide based on the strength of the evidence and their own personal tolerance for making false positive or false negative decisions. If it is decided that cheating occurred, I would look to your institution’s policy for direction on how to proceed.