Guidelines for administering concept inventories online

posted February 10, 2016 and revised August 12, 2016
by Adrian Madsen & Sam McKagan, PhysPort Assistant Director & Director

Concept inventories such as the Force Concept Inventory (FCI) have had a major impact on physics education reform: The FCI, a test of basic concepts of forces and acceleration, has been given to thousands of students throughout the country and the results show that PER-based teaching methods lead to dramatic improvements in students’ conceptual understanding of mechanics. These results have inspired many physics instructors to try the FCI with their own students and to radically change their teaching methods based on the results. Similar assessments exist in nearly every topic in physics and at many levels. Giving a concept inventory to your students allows you to measure the effectiveness of your teaching by comparing your students’ conceptual learning to other students across the country.

Typically, concept inventories are administered in class using paper tests and scantrons. However, many faculty would prefer to administer them online, either to save class time, or to make it easier to analyze student responses, or both.

Warning: Most assessment developers do NOT recommend giving students unsupervised access to assessments online, and ask that you give assessments online only in supervised environments. This requirement can be relaxed for advanced undergraduate or graduate students. Administering ungraded conceptual questions online outside of class can decrease response rates [Jariwala et al. 2016, Bonham 2008, Sayre et al. 2012] and compromise test security. If you choose to administer a concept inventory online, we recommend following these guidelines to increase response rates and maximize test security.

Ways to administer concept inventories online include:

  • The LA Supported Student Outcomes (LASSO) website, produced by the University of Colorado-Boulder Learning Assistance Alliance, enables instructors to administer concept inventories online, including the FCI, FMCE, BEMA, CSEM, and CLASS.
  • The American Modeling Teachers’ Association (AMTA) has an online system for administering the FCI. Contact Colleen Megowan for more details.
  • WebAssign has the FCI, MBT, FMCE, and BEMA coded into their system and allows any instructor to use these assessments for free. Contact Matt Kohlmyer for more details.
  • It’s possible to administer tests online using a free survey tool, like Google Forms or SurveyMonkey, your institution’s course management system, like Blackboard, Moodle, WebCT, or a homework system, like Mastering Physics or WebAssign. However, using these systems might not be secure enough, and might make more work for you to set up or introduce inadvertent errors into the scoring process.

Regardless of which system you use, you should make sure that it is set up with proper security so students can’t copy, save, or print test questions.

Increasing response rates: If you ask your students to complete a test outside of class, you will probably find that fewer students take the test than if you administer it in class. This can lead to problems in interpreting the results for your class, since the students who make time to take the test at home are typically not representative of the class as a whole.  Email reminders will increase the completion rate of the test [Bonham 2008, Jariwala et al. 2016]. Send several email reminders over the course of the week. If possible, send personalized reminders to students who have not yet completed the test [Bonham 2008]. Further, Jariwala et al. 2016 found that offering makeups, and offering extra credit for completion can increase participation rates. 

Maintaining test security: There are two big reasons to maintain test security. First, if your students consult each other or other resources while they take the test, you won’t get an accurate picture of their understanding. Second, if your students are able to share portions of the test (to their fraternity brothers or to online answer-sharing sites like Cramster), the test won’t be accurate for future users either. This will compromise the usefulness of these tests for researchers. For these two reasons, test developers require that you maintain test security as a condition of using their tests.

Where? You can give the test in class if there are computers available in the classroom, or ask the students to come to a proctored computer lab or even to your office to take the test on their own time. Some faculty do allow the students to take the test at home on their own time, but this is not recommended, especially for introductory students.

When? Make the pre-test available online during the first week of class and the post-test available during the last week of class.

How?

  • Use a secure online system to give the test. Make sure that the system is set up so that students can’t copy, save or print the test questions. No software can fully prevent students from copying and sharing the test online; it can only make illicit copying marginally more difficult.
  • Set your software so students can only access the test once and put a time limit on their access. A good rule of thumb is to double the recommended time for each assessment. That ensures that students will have enough time to complete, but won’t be able to leave their browser windows open indefinitely.
  • Don’t let students see the answers after submission.
  • Ask students not to use any other resources including help from any people, books, notes, websites, etc. Emphasize that you want to know what the students personally know and none of their responses will be marked wrong [Bonham 2008].

Keep questions and answer options in the original order. Some course management systems will let you scramble answer order or question order when you administer a test. This can increase security because it makes it harder for students at adjacent computers to copy from each other. However, these tests have only been validated for the original question order. Changing question orders can also affect how students answer subsequent questions, and changing answer orders can affect which answer choices students pick. Also, systems that scramble question order often export results in formats that are difficult for other systems, such as the PhysPort Data Explorer, to read. We don’t recommend scrambling orders because it will make it harder for you to compare results among your class and to other classes.