Supporting physics teaching with research-based resources

Concept inventories are useful for assessing the effectiveness of your teaching, but only if your students take them seriously. You may be worried about how seriously your students are taking them. Here is what we know about students taking concept inventories seriously and some ideas for how you can encourage your students to do their best on these kinds of tests.

How seriously do students usually take concept inventories?

Overall, there is evidence that the large majority of students do take concept inventories seriously both in and outside of class. Henderson 2002 looked for signs of students’ lack of seriousness when taking concept inventories in-class that were graded or ungraded. One way he did this was to look answer patterns that indicated that the students were not taking the test seriously, for example, leaving lots of blanks, answering all A’s or B’s etc. He found that when the post-test what ungraded, 3.4% of students used these answer patterns and compared to 0.6% when the test was graded. So, the maximum percent of students who used answer patterns which indicated a lack of seriousness as a result of the test not being graded was 2.8%. Henderson looked at patterns of wrong answers as a way to estimate the number of students who were blatantly not taking the test seriously. It could be that students don’t think about their answers carefully but do not use one of the test patterns described above. To estimate the number of students who take the concept inventory somewhat seriously, we can compare scores when the test does and does not count toward students’ grades. Assumedly, when the test counts toward students’ grade, they will answer each question more thoughtfully. Henderson compared the overall scores on a popular concept inventory when it was given to the same group of students, once graded and once ungraded with a three-week winter break in between. He found that when the test was graded, students scored half of a test item better.

Research has also been done looking at how serious students take concept inventories when taking them online, outside of class. Bonham 2008 found only a small number of individuals who systematically used other web resources while taking the concept inventory. He also found very low rates of printing, saving or copying questions from the concept inventory. The research reassures us that most students are taking concept inventories, however, in both of these studies, the instructors took care to talk to their students about the concept inventories in a certain way. We discuss ideas for effective presenting the test to your below.

What kinds of incentives will help my students take the test seriously?

There are many kinds of incentives you could give your students to further encourage them to take your concept inventory seriously. Commonly used incentives include giving a small number of participation of extra credit points, giving students grades for their individual answers, incorporating into an exam or giving no incentive at all. We discuss the pros and cons of each below.

• Giving course credit for participation: Several instructors find that giving a small amount of participation points or extra credit for completing the test works well to motivate their students to take the test seriously [Adams and Wieman 2011, Redish 2003]. This incentive ensures a low-stakes testing environment where students are not worried about the correctness of their answers. Further, it is common practice give concept inventories in a low stakes testing environment and if there are systematic influences on students’ test scores as a result of the way the test is given, we assume that these effects are similar across students at that their scores can be compared
• Post-test as a review for the final: Several instructors have found that framing the post-test as a review for the final on the second to last day of class is an effective way to help the students take it seriously [Adams and Wieman 2011]. To do this, explain that the post-test covers some of the key concepts from the course that will be on the final. You can then go over the most commonly missed questions on the last day of class as a review for the final (Some believe going over questions in class can compromise the security of the test. see “Best Practices for Administering Concept Inventories” for a more thorough discussion of test security.)
• Giving course credit for correctness (not recommended): Concept inventories are meant to assess help you understand how effective your teaching is. In order for you to get the most meaningful data from these tests, students should answer the questions according to what they really think, not what they think you want them to say. Grading students’ tests creates a high-stakes environment that may cause students to take the test too seriously. In this environment, students are more likely to use test-taking skills instead of answering according to what they think and may be motivated to access the test questions ahead of time and memorize the answers [Adams and Wieman 2001]. This kind of testing environment tells you less about what your students really think about the concepts, and more about how resourceful they are. You might worry that giving a concept inventory ungraded will mean that students take it less seriously, but research suggests that most students do take the concept inventory seriously when it is ungraded (discussed above).
• Incorporating test into an exam (not recommended): Similar to giving grades for individual students’ tests, incorporating test questions into an exam creates a high stakes testing situation where students may take the test too seriously and not answer according to what they really think. This incentive does increase completion rates of the test [Bonham 2008], but the results will likely not tell you about your effectiveness as an instructor.
• No course credit: Some instructors believe that their students should complete coursework without the extra motivation of course credit, so they choose to give no grade incentive. In this case, the way you frame the concept inventory when giving it to your students is very important.

How do I talk to my students about the concept inventory to help them take it seriously?

The way that you present the concept inventory to your students can impact how seriously they take it. There are several different ways you can talk about the test with your students:

• Purpose of the test: Explain that the purpose of the test is for you to learn about how well your teaching works. You are looking at the scores of the class as a whole, and not each individual student, to figure out how much the class as a whole knows or has learned so that you can improve your teaching. You could also talk about how their test scores can help you make a case to your department that this is a good way to teach. We have found that students respond well to the test when you talk about how it is helping someone. This might be a research student who is collecting data for a project or you, their instructor, who is trying to improve their teaching.
• Pre-test: You want to create a low-stakes testing environment where students aren’t stressed about the test. To do this on the pre-test, assure the students that they are not expected to know the answers to the questions, but they should try their best to answer each question. You could also explain that you want to know what they know so that you can adjust the topics in your course and how thoroughly you cover each accordingly.
• Post-test: Similar to the pre-test, you want to encourage your students answer according to what they really think, not what they think you want them to say. Remind them that their test scores are important to you personally so that you can understand how well you taught them. This test is not about their individual performance.
• Giving the test online: If you are giving the test online, outside of class, you can also ask students not to use any other resources including help from any persons, books, notes, websites, etc. Emphasize that you want to know what the students personally know and none of their responses will be marked wrong [Bonham et al. 2003]. Getting your students to complete the test becomes more challenging when you give it outside of class. For ideas on how to do this, see Guidelines for administering concept inventories online

Image courtesy of PhET Interactive Simulations, University of Colorado Boulder