Psychometric Values

Psychometrics has nearly a century of history of finding ways to ethically make decisions from student data. You will find a copy of the NCME/APA Joint Standards for Educational and Psychological Testing here. Familiarize yourself with the types of standards in this text:

• Standards 1.0-1.25, which describe the kind of validation measurements need before being used for decision making, and their impact on students (“validity”)
• Standard 2.0-2.20, which describe the accuracy necessary (“reliability”)
• Standard 3.0-3.20, which describe fairness and bias implications
• Standard 4.0-4.25, which describe transparency requirements, the design of tests, and scoring. Take a look at sections on automated algorithms (e.g. 4.19)
• Skip chapters 5 and 6
• Skim chapter 7, which describes docmentation are requirements
• Read, in-depth, chapter 8, which describes the rights of test takers.
• Quickly skim chapter 9, which describes requirements of test administrators and institutional users
• Chapters 10, 11, and 13 are out-of-scope for this workshop, but helpful, since they focus on psychological, workplace, and policy measurements, respectively.
• Chapter 12 gives requirements specific to education and educational decisions making

What are the equivalent rules we should use when using the kinds of educational measurements we see at AI-ED, EDM, and LAK? Such decisions rely on complex algorithms which often require a computer science Ph.D to understand. More modern algorithms – such as evolved deep learning systems – are black boxes to everyone.

Come up with an equivalent framework in the era of big data, data aggregation, machine learning, and especially, black box models.