【MESA8220】Psychometrics I

Item Analysis Activities

Part A: Cognitive Test Item Characteristics

Using the data file provided, conduct an item analysis for mock field test results. Your analysis should provide information about each item’s characteristics. In addition, your analysis should identify the items you feel are problematic. For each “problem” item, describe what evidence led you to identify the item as problematic, propose possible reasons why the item may be problematic, and suggest what should be done with the item. Your analysis should conclude by identifying the sub-set of items you recommend be forwarded for possible use during operational test form construction.

Part B: Cognitive Test Scale Properties

This assignment builds on Part A.  Begin by conducting a reliability analysis for the full item set. Next, examine how the reliability changes when the items identified as problematic during Part A are removed from the item set.  Then, attempt to improve reliability further by removing additional items. However, when doing so, consider the impact that item removal has on content representation.  Your end goal is to select a set of items that: a) minimizes administration time; b) maximizes reliability; and c) maximizes domain representation.  Once your have selected your final dataset, provide a rationale for why you opted to select this item set.  Your final product should combine your analyses performed during Part A and Part B to present a case for: a) removing problematic items (Part A); b) your final item set; and c) your scaling method and its application to specified students.

Part C: Survey Scale(s) Technical Properties

Using the data provided, conduct a factor analysis for the items intended to form each survey scale. As part of the analysis, identify problematic items and make a recommendation for the final set of items that should be used to form each scale. Also conduct a reliability analysis for each scale.

Parts A, B, & C will be submitted as a single document due on April 24.

Scoring Criteria: The purpose of this

This course is designed to prepare graduate students to design and develop cognitive tests and survey instruments and to critically review instruments developed by others.  The course focuses on essential aspects of designing cognitive tests and survey instruments, authoring items, piloting an instrument, performing item and scale analyses, selecting and assembling items, and developing a validity argument.  Special attention is paid to developing valid and reliable measurement scales.  The course introduces statistical techniques and true-score theory, thus providing an introduction to psychometric theory.