Quality control for scanned multiple-choice exams | R-Bloggers

Step-by-step manual for the use of NOps_Fix () for solving problems that took place during the scanning of written NOps exams in R/exams.

Overview

The R/exams Package offers a workflow for generating (exams2nops()), scan (nops_scan()), and automatically evaluate (nops_eval()) Large-scale pen-and paper multiple-choice exams, see the corresponding Nops -Tutorial For more information.

Recently another function nops_fix() was added to the package that helps in finding and correcting problems that occurred when reading the scanned exams and that the automatic evaluation process could hinder. Typical problems include scans that have been played or with damaged scanner markings, answer courses that are not correctly checked or filled, or missing checks in the courses for the registration -IDs, among others. Therefore, nops_fix() the researcher helps to solve such problems afterwards nops_scan() and before nops_eval() To guarantee an accurate assessment.

In this self -study we emphasize the most important characteristics of nops_fix():

1. Automatic display of problematic sheets
2. Give rise to manual corrections
3. Specialized checks for unbelievable answer patterns
4. Manual assessment of specific exam leaves

These are illustrated by repairing the PDF scans of two demo exams: one with exercises with one choice (and only smaller problems) and one with multiple choice feeds (and more serious problems). Some more detailed comments about all functions are given at the end of this tutorial.

Demo 1: Single-Cheice exam with typical problems

Create exam

First we generate five PDFs (in a new subfolder “NOps”) of a demo exam with five single-choice exercises (all provided in the package: Swiss capital” Rogue” Deriv2″ Fruit2″ Hessian).

.

library("exams")
sc <- c("swisscapital.Rmd", "Rlogo.Rmd", "deriv2.Rmd", "fruit2.Rmd", "hessian.Rmd")
set.seed(403)
exams2nops(sc, n = 5, dir = "nops", date = "2024-10-07")

Perform exam and scan results

Suppose we have carried out the exam with three participants and scanned their exam leaves (first page), turned upside down (usually facilitating scanning because the top left corner is damaged by the staples).

Here you can download the PDF file scans-schoice.pdf:

Then we can read the information of the scanned sheets via nops_scan()resulting in a file nops_scan_*.zip. The * Place holder corresponds to the date/time in which you perform the code. The file contains the three scanned exam leaves that have been converted to PNG together with a text file Daten.txt With the information that R has taken from the scanned sheets.

nops_scan("scans-schoice.pdf", rotate = TRUE)

Recovery scan results and final evaluation

Subsequently, the function calls nops_fix() Without searching for any arguments for files with the name nops_scan_*.zip And checks on obvious errors, such as incorrectly entered registration -IDs in the demo -sheets provided.

nops_fix()

When you perform the code, R shows a part of the exam sheet and ask you to manually enter the correct registration number. In this case, the program cannot read the number nine because the cross was wrongly entered in the last column in the penultimate position.

After entering the registration number, the corrected data is stored in the updated ZIP file.

## Correct registration number (for 0000000, S0000002.PNG): 0157193
##   adding: Daten.txt (deflated 89%)
##   adding: S0000001.PNG (deflated 8%)
##   adding: S0000002.PNG (deflated 6%)
##   adding: S0000003.PNG (deflated 6%)

Moreover, we can check whether this is in this schoice Exam There is exactly one checked box per question. Irresponsible answer patterns without checked boxes or with more than one checked box can be real answers from the Examinandi, but can also be due to scan problems where controlled or filled boxes are not correctly detected. If the scanned answer needs correction, you can enter the correct answer as a letter (a” b…), like a number (1” 2…), or in binary coding (e.g. 00010). Just press to accept the scanned result .

First we check and (if necessary) correct cases with more than one checked box:

nops_fix(check = "schoice")

## Correct answer 3 (for 01010, S0000003.PNG): d
## Correct answer 4 (for 01001, S0000003.PNG): 
##   adding: Daten.txt (deflated 89%)
##   adding: S0000001.PNG (deflated 8%)
##   adding: S0000002.PNG (deflated 6%)
##   adding: S0000003.PNG (deflated 6%)

Secondly, we check for cases with missing answers:

nops_fix(check = "missing")

## Correct answer 2 (for 00000, S0000003.PNG): 
##   adding: Daten.txt (deflated 89%)
##   adding: S0000001.PNG (deflated 8%)
##   adding: S0000002.PNG (deflated 6%)
##   adding: S0000003.PNG (deflated 6%)

Finally, after solving the above problems in the nops_scan_*.zip File, the exam with one choice can be evaluated. This also requires the correct solutions of the exams2nops() Metain formation (produced above) and a list of the registered participants (with the registration number, the name and users -ID). The latter can be downloaded as participants-schoice.csv.

ev <- nops_eval(
  register = "participants-schoice.csv",
  solutions = "nops/metainfo.rds",
  scans = Sys.glob("nops_scan_*.zip"))

This produces the usual nops_eval.* output that are described in more detail in the Nops -Tutorial.

To clean up for the next demo, you can use the following code:

unlink("nops", recursive = TRUE)
file.remove(Sys.glob("nops_eval.*"))
file.remove(Sys.glob("nops_scan_*.zip"))

Demo 2: Multiple-Choice exam with serious problems

Create exam

library("exams")
mc <- c("capitals.Rmd", "switzerland.Rmd", "gaussmarkov.Rmd",
  "boxplots.Rmd", "scatterplot.Rmd", "ttest.Rmd")
set.seed(404)
exams2nops(mc, n = 5, dir = "nops", date = "2024-10-08")

Perform exam and scan results

Again, suppose we have carried out the exam with three participants and scanned their exam sheets (first page), turned upside down.

Here you can download the PDF file scans-mchoice.pdf:

Then we can read the information of the scanned sheets via nops_scan()resulting in a file nops_scan_*.zip.

nops_scan("scans-mchoice.pdf", rotate = TRUE)

In this case this leads to a warning message that there are errors in the scanned data and that running nops_fix() is necessary prior to nops_eval().

Recovery scan results and final evaluation

As above, calling the position nops_fix() Without searching for any arguments for files with the name nops_scan_*.zip And checks for clear errors. Here the third page is very heavily rotated and the scanner markings are all damaged so that the PDF cannot be read at all.

Due to the poor rotation and damaged markers, it is not possible for nops_fix() To reliably extract the individual fields. That is why it shows the entire scanned exam sheet in the browser, in addition to an interactive web form where all fields can be entered manually. Because even the ‘type’ field cannot be read, nops_fix() Do not know that there are only six exercises and therefore it shows 45 exercises, the maximum possible.

nops_fix()

After entering all fields, press the “OK” button in the browser (bottom right). This copies all the data entered in the clipboard (in a JSON sign series) and R will try to read it from there or you can stick it in the R -prompt. Reading the clipboard must always work on Windows, while on other platforms the R package clipr It should be installed to read reliably from the clipboard. In any case, you must ensure that the clipboard is not changed before you read or paste the clipboard again.

Instead of using the interactive shape in the browser, you can also use nops_fix(display = "browser") which only displays the scanned sheet in the browser and then all fields can be entered separately in the R -prompt (as in the previous examples).

We can then perform the extra precautionary checks of answer fields where all or none of the courses have been checked. In these three exams, however, we have none of these problems.

nops_fix(check = "mchoice")
nops_fix(check = "missing")

To demonstrate a handy characteristic of nops_fix()Let us assume that we know that those answer fields on the second scan must be more thoroughly inspected.

nops_fix(exam = 2, field = "answer")

This is repeated by all six answer fields, in this case a number of ambiguities that have to be resolved by the researcher.

## Correct answer 1 (for 00001, S0000002.PNG): 
## Correct answer 2 (for 10000, S0000002.PNG): 
## Correct answer 3 (for 01100, S0000002.PNG): 
## Correct answer 4 (for 01110, S0000002.PNG): 
## Correct answer 5 (for 10000, S0000002.PNG): 1
## Correct answer 6 (for 11010, S0000002.PNG): bd
##   adding: Daten.txt (deflated 88%)
##   adding: S0000001.PNG (deflated 8%)
##   adding: S0000002.PNG (deflated 5%)
##   adding: S0000003.PNG (deflated 6%)

Then the nops_scan_*.zip File is ready for the final evaluation and you only have to download the list of participants participants-mchoice.csv Before you are running nops_eval().

ev <- nops_eval(
  register = "participants-mchoice.csv",
  solutions = "nops/metainfo.rds",
  scans = Sys.glob("nops_scan_*.zip"))

Use to clean up the workbook after the demos:

unlink("nops", recursive = TRUE)
file.remove(Sys.glob("nops_eval.*"))
file.remove(Sys.glob("nops_scan_*.zip"))

Detailed

1. Automatic display of problematic sheets
   • If certain sheets are poorly scanned (eg incorrectly aligned or poorly scaled) or if the courses for registration -ID are not checked correctly (exactly one checked box per column), nops_scan() marks these sheets as incorrect and nops_fix() Presents them automatically for manual inspection.
   • The function shows the entire exam sheet in the browser or the section that needs attention in an R-plot and asks the user to enter the console, making step-by-step inspection and correction possible.
   • The most suitable display is automatically chosen or can be set to "plot" (Display scanned fragment in R -Plot), "browser" (display fully scanned image in browser), or "interactive" (Display fully scanned image together with interactive HTML form in browser). The standard is "plot" Unless errors took place during scanning where "interactive" is used instead.
2. Give rise to manual corrections
   • For each field In the exam sheet (both "type"” "id"” "registration"or "answers"),),, nops_fix() shows the detected content and asks the user for possible updates.
   • Only on Enter is the detected content accepted without changes.
   • When checking a certain answer field, the detected answer is displayed by a five -digit binary code (even if the question had fewer than five answer alternatives) 01001 (When the second and the fifth course were checked). To correct this, only say to the second box, the following formats are supported: binary (01000), Letters (b) or numeric (2).
3. Specialized checks for unbelievable answer patterns
   • To support examiners in finding potential Problems in the exam sheets before researchers complain, nops_fix() Offers checks for various unbelievable answer patterns.
   • Often such unbelievable patterns are errors on behalf of the Examinandi (and then simply must be stored), but when the errors are due to the incorrect detection of the checked boxes in the scanned leaf, the answers can be corrected.
   • nops_fix(check = "missing") Checks answers without checked boxes that are unbelievable, unless there are negative points for incorrect answers.
   • nops_fix(check = "schoice") Checks answers with more than one checked box that is often caused by not fully filling in a wrongly checked box.
   • nops_fix(check = "mchoice") Checks answers with all checked courses that are unbelievable because such patterns are usually assessed with zero points.
4. Manual assessment of specific exam leaves
   • Sometimes specific exam sheets require more attention, for example when special conditions are applied in the exam or when the magazine is not filled in correctly.
   • Say if this applies to the first scanned leaf, then nops_fix(exam = 1) Can be used to check every field of the leaf.
   • Such an assessment ensures that each answer is investigated, which is especially useful in institutions with high deployment or verification requirement institutions.

The full details for all arguments have been documented on the ?nops_fix Manual page that also contains more typical application examples.

Related