Expert Recommendations »Recommendation

Which astronomy research-based assessment should I use in my class?

posted April 10, 2021 and revised September 9, 2022
by Adrian Madsen, Sarah B. McKagan and Eleanor C. Sayre

This recommendation initially appeared as an article in the American Journal of Physics:  A. Madsen, S. B. McKagan and E. C. Sayre, Resource Letter RBAI-1: Research-Based Assessment Instruments in Physics and Astronomy, Am. J. Phys. 85, 4 (2017).

There are eight RBAIs for astronomy, and all are designed for use in the introductory astronomy course. Three of these, the Astronomy Diagnostic Test 2.0 (ADT2) (Hufnagel 2002), the Test of Astronomy Standards (TOAST) (Slater 2014), and the Astronomical Misconceptions Survey (AMS) (LoPresto and Murrell 2011), contain questions about a wide range of topics covered in an introductory astronomy course and can be used to assess the overall effectiveness of your course. Five of these, the Star Properties Concept Inventory (SPCI) (Bailey 2007), the Light and Spectroscopy Concept Inventory (LSCI) (Bardar et al. 2007), the Newtonian Gravity Concept Inventory (NGCI) (Williamson 2013; Williamson, Willoughby, and Prather 2013) the Lunar Phases Concept Inventory (LPCI) (Lindell and Olsen 2002), and the Greenhouse Effect Concept Inventory (GECI) (Keller 2006) cover a more narrow range of content, and can be used to assess your students’ understanding of specific content from your course. All of these RBAIs are multiple-choice. All astronomy assessments are summarized in Table 1.

Table 1. Astronomy assessments.

Title Content Intended Population Research Validation Purpose 

General Astronomy Assessments
Astronomy Diagnostic Test 2.0 (ADT2)

Seasons, lunar phases, motions in the sky, and size and scale

 Intro college Gold

To assess students’ conceptual understanding of introductory astronomy topics.
Test of Astronomy Standards (TOAST) General astronomy content knowledge

Intro college

 Silver

To measure students’ mastery of core concepts in a general astronomy course.
Astronomical Misconceptions Survey (AMS)

Misconceptions about introductory astronomy courses

Intro college

 Research-based

To identify misconceptions introductory students hold and measure the effectiveness of instruction to dispel these misconceptions.
Specific Astronomy Topics Assessments 
Star Properties Concept Inventory (SPCI)

Stellar properties, nuclear fusion, star formation

Intro college

 Gold

To measure student learning about the properties and formation of stars.
Light and Spectroscopy Concept Inventory (LSCI)

Light, waves, spectroscopy

Intro college

 Silver 

To measure students’ conceptual understanding of topics related to light and spectroscopy, and evaluate the effectiveness of instruction in introductory college astronomy courses.
Newtonian Gravity Concept Inventory (NGCI)

Gravity

Intro college

 Silver 

To assess student understanding of Newtonian gravity and effectiveness of instruction in general education introductory college astronomy course.
 Greenhouse Effect Concept Inventory (GECI)

Types of greenhouse gases, energy equilibrium balance, greenhouse effect mechanisms, global warming vs greenhouse effect

Intro college

 Silver 

To assess pre- and post-instruction conceptual understanding of the greenhouse effect focusing on the physics of energy flow through Earth’s atmosphere.
Lunar Phases Concept Inventory (LPCI)

Phases of the moon

Intro college

 Bronze

To assess college students’ mental models of lunar phases.

General astronomy assessments

Astronomy Diagnostic Test 2.0

The Astronomy Diagnostic Test 2.0 (ADT2) (Hufnagel 2002) is a multiple- choice conceptual pre/post-test for non-science majors taking an introductory astronomy course and covers content commonly found in the K-12 curriculum including seasons, lunar phases, motions in the sky, and size and scale. It was designed to help instructors assess their students’ initial knowledge coming into a college astronomy course, as the topics included were likely covered in K-12. The multiple-choice questions on the most recent version of the Astronomy Diagnostic Test (ADT), version 2.0, come from an earlier version of the ADT, which consisted of questions from several earlier astronomy tests.

Test of Astronomy Standards (TOAST)

The Test of Astronomy Standards (TOAST) (Slater 2014) is a multiple-choice broad conceptual assessment of general astronomy content knowledge that is built on and from earlier astronomy assessments (all the astronomy assessments included here). The content includes gravity, electromagnetic radiation, fusion and formation of heavy elements, evolution of the universe, star and stellar evolution, evolution and structure of the solar system, seasons, scale, yearly patterns, daily patterns, moon phases. The content on the TOAST was determined based on that which was deemed more important for introductory astronomy students as described in expert position statements from several professional organizations90,91 and later reviewed by 28 experts in astronomy. This makes it a unique astronomy RBAI, as the topics are broad, covering the whole intro course, and are chosen based on based on consensus documents from the astronomy community. Further, most of the questions are taken from other astronomy RBAIs.

Comparing the ADT2 and TOAST

The TOAST and ADT2 cover very similar content including phases of the moon, motions in the sky, seasons, scale, distances, sizes, properties, and lifecycles of stars, gravity, and the universe. There are several questions that are the same on both tests since the TOAST was created using ques- tions from other astronomy assessments. The TOAST contains questions about production of light (emission, absorption, etc.), while both tests ask about the relative speed of electromagnetic waves. There TOAST asks about the Big Bang, and the ADT2 does not. The ADT2 has one question about global warming, and the TOAST does not. Both tests are general assessments for introductory astronomy. They have both been well validated. The ADT2 been used widely in introductory astronomy courses across the US, so there is a lot of comparison data available. The TOAST is a newer assessment, so there is less comparison data available now, but this will likely change in the near future.

Astronomical Misconceptions Survey (AMS)

The Astronomical Misconceptions Survey (AMS) (LoPresto and Murrell 2011) is a pre/post-conceptual multiple-choice survey of common misconceptions in introductory astronomy, e.g., the phases of the moon are caused by the earth’s shadow or the seasons are caused by differences in the earth’s distance from the sun. There are two versions of the AMS: the true/false version and the multiple-choice version. The true/false version can be used to help instructors understand the misconceptions their students come to their course holding. The multiple-choice version can be given to students to help instructors understand the misconceptions their students have or to assess the effectiveness of different types of instruction at addressing these misconceptions. The questions on the AMS are not about a particular topic, but instead a variety of topics for which students have commonly held incorrect beliefs. The questions on the AMS were developed from a list of 25 astronomy misconceptions, which were based on previous research on misconceptions.

Because the AMS is a test of students’ misconceptions about astronomy, the topics covered and the focus of the questions is very different from the questions on the ADT2 and TOAST.

Specific astronomy topic assessments

Star Properties Concept Inventory (SPCI) 

The Star Properties Concept Inventory (SPCI) (Bailey 2007) is a multiple-choice pre/post conceptual assessment of stellar properties, nuclear fusion, and star formation for introductory astronomy courses. The SPCI questions were developed based on exam and textbook questions and the authors experience with teaching the content. It was developed in response to research on students’ alternative conceptions about stars (Bailey et al. 2011).

Light and Spectroscopy Concept Inventory (LSCI)

The Light and Spectroscopy Concept Inventory (LSCI) (Bardar et al. 2007) is a multiple-choice pre/post conceptual test about the electromagnetic spectrum and the nature of light and is meant for introductory astronomy courses. These specific topics have been chosen because they were found to be central topics common across most introductory astronomy courses. The more narrow range of topics means that there are multiple questions probing each. Students usually score near guessing (25%) on the pre-test, implying that the LSCI is testing material unfamiliar to students. That said, most instructors still give it as a pre- and post-test. The LSCI questions were developed based on expert opinions about the important core knowledge around light and the electromagnetic spectrum, and research on student ideas about light and quantum phenomena.

Newtonian Gravity Concept Inventory (NGCI)

The Newtonian Gravity Concept Inventory (NGCI) (Williamson 2013; Williamson, Willoughby, and Prather 2013) is a multiple-choice pre/post-conceptual assessment of gravity, a foundational topic in introductory astronomy courses. The questions probe four conceptual dimensions including the directionality of gravity, the force law, independence of other forces (e.g., gravity is not affected by rotation), and thresholds related to gravity (e.g., there is not distance for which gravity suddenly stops). The NGCI was developed for use in introductory astronomy courses, but can also be used in introductory physics. The questions are based on student ideas about gravity.

Lunar Phases Concept Inventory (LPCI)

The Lunar Phases Concept Inventory (LPCI) (Lindell and Olsen 2002) is a multiple-choice pre/post conceptual assessment of lunar phases concepts including cause and period of lunar phases, period and direction of the Moon’s orbit, and observational phenomena. It is designed to assess students’ mental models of lunar phases using a mathematical technique called model analysis theory (Lindell and Olsen 2002). The result of this analysis is the probability of students in a course answering with the correct model as well as the probability of answering with one of several incorrect models. The LPCI can also be analyzed and scored in the more common way of finding the percent correct on the pre- and post-test and then calculating the normalized gain. Furthermore, since the test content was developed based on students’ ideas about the lunar phases, as opposed to expert opinions about the most important content related to lunar phases, it is most appropriate to use the LPCI to understand your students’ thinking and mental models, instead of how well their ideas match expert conceptions.

Greenhouse Effect Concept Inventory (GECI)

The Greenhouse Effect Concept Inventory (GECI) (Keller 2006) is a multiple-choice pre/post-conceptual assessment about the physics of energy flow through Earth’s atmosphere. Topics include types of greenhouse gases, types of electromagnetic energy, energy equilibrium balance, greenhouse effect mechanisms, global warming versus the greenhouse effect. The GECI can be used in introductory astronomy courses that cover relevant content. The questions were developed based on extensive research on students’ beliefs about models of the greenhouse effect.

Because the content of these specific astronomy topic assessments is so different, we don’t compare them. 

Recommendations for choosing an astronomy assessment

Use the TOAST or ADT2 if you are making changes to your entire introductory astronomy course, and want to measure the effectiveness of the change. Use the TOAST if you want to assess students’ understanding of how light is produced in addition to other standard introductory concepts. Use the ADT2 as a pre-test if you want to understand the ideas your students bring to your course from their K-12 education. Use the AMS if you are particularly interested in understanding your students’ misconceptions about astronomy.

If instead you are making changes to a specific portion of your course, use an assessment of specific topics that match the content you are changing (SPCI, LSCI, NGCI, LPCI, or GECI). The developers of the LSCI point out that the topics covered on the LSCI (electromagnetic spectrum and the nature of light) are foundational and central in many astronomy courses, so you could use this test as a proxy for understanding the effectiveness of your instruction for your course, even though it covers only a subset of the material. Furthermore, if comparing your students’ scores to others is important to you, use either the ADT2 or LSCI, as there is a large amount of comparison data published. A list of articles with ADT2 and LSCI comparison data can be found on the research tab on their respective assessment pages on PhysPort. The TOAST is a newer assessment, so there is less comparison data available now, but this will likely change in the near future.

References