Expert Recommendations »Recommendation

Administering research-based assessments online

posted February 10, 2016 and revised April 20, 2023
by Adrian Madsen & Sam McKagan, PhysPort Assistant Director & Director

Research-based assessments, such as the Force Concept Inventory (FCI), have had a major impact on physics education reform. The FCI, a test of basic concepts of forces and acceleration, has been given to thousands of students throughout the country and the results show that research-based teaching methods lead to dramatic improvements in students’ conceptual understanding of mechanics. These results have inspired many physics instructors to try the FCI with their own students and to radically change their teaching methods based on the results. Similar assessments exist in nearly every topic in physics at many levels, and also for non-physics-content topics such as beliefs and attitudes about learning, problem-solving, lab skills etc. Giving a research-based assessment to your students allows you to measure the effectiveness of your teaching by comparing your students’ results to other students across the country.

Traditionally, research-based assessments have been administered in class using paper tests and scantrons. However, many faculty would prefer to administer them online, either to save class time, or to make it easier to analyze student responses, or both.

Warning: Most assessment developers do NOT recommend giving students unsupervised access to assessments of content online, and ask that you give assessments online only in supervised environments. This requirement can be relaxed for advanced undergraduate or graduate students. Research-based assessments that don't have a correct answer, such as surveys of students' attitudes and beliefs, can be given online without concern, as these are asking students to report on their own beliefs. Administering ungraded conceptual questions online outside of class can decrease response rates [Jariwala et al. 2016, Bonham 2008, Sayre et al. 2012] and compromise test security. If you choose to administer a concept inventory online, we recommend following these guidelines to increase response rates and maximize test security.

Ways to administer concept inventories online include:

Regardless of which system you use, you should make sure that it is set up with proper security so students can’t copy, save, or print test questions.

Increasing response rates: If you ask your students to complete a test outside of class, you will probably find that fewer students take the test than if you administer it in class. This can lead to problems in interpreting the results for your class, since the students who make time to take the test at home are typically not representative of the class as a whole.  Email reminders will increase the completion rate of the test [Bonham 2008, Jariwala et al. 2016]. Send several email reminders over the course of the week. If possible, send personalized reminders to students who have not yet completed the test [Bonham 2008]. Further, Jariwala et al. found that offering makeups, and offering extra credit for completion can increase participation rates. 

Maintaining test security: There are two big reasons to maintain test security. First, if your students consult each other or other resources while they take the test, you won’t get an accurate picture of their understanding. Second, if your students are able to share portions of the test (to their fraternity brothers or to online answer-sharing sites like Cramster), the test won’t be accurate for future users either. This will compromise the usefulness of these tests for researchers. For these two reasons, test developers require that you maintain test security as a condition of using their tests.

Where? You can give the test in class if there are computers available in the classroom, or ask the students to come to a proctored computer lab or even to your office to take the test on their own time. Some faculty do allow the students to take the test at home on their own time, but this is not recommended, especially for introductory students.

When? Make the pre-test available online during the first week of class and the post-test available during the last week of class.

How?

  • Use a secure online system to give the test. Make sure that the system is set up so that students can’t copy, save or print the test questions. No software can fully prevent students from copying and sharing the test online; it can only make illicit copying marginally more difficult.
  • Set your software so students can only access the test once and put a time limit on their access. A good rule of thumb is to double the recommended time for each assessment. That ensures that students will have enough time to complete, but won’t be able to leave their browser windows open indefinitely.
  • Don’t let students see the answers after submission.
  • Ask students not to use any other resources including help from any people, books, notes, websites, etc. Emphasize that you want to know what the students personally know and none of their responses will be marked wrong [Bonham 2008].

Keep questions and answer options in the original order. Some course management systems will let you scramble answer order or question order when you administer a test. This can increase security because it makes it harder for students at adjacent computers to copy from each other. However, these tests have only been validated for the original question order. Changing question orders can also affect how students answer subsequent questions, and changing answer orders can affect which answer choices students pick. Also, systems that scramble question order often export results in formats that are difficult for other systems, such as the PhysPort Data Explorer, to read. We don’t recommend scrambling orders because it will make it harder for you to compare results among your class and to other classes.