While there is little doubt that improving undergraduate students’ critical thinking is an important higher education goal (Ennis, 1993; Kuhn, 1999; Facione, 2007; Facione, Facione, and Giancarlo, 2000), only recently have higher education institutions started exploring standardized testing to evaluate their success at achieving that goal. The Council for Aid to Education (CAE) has addressed the growing interest in measurement of higher education outcomes with the development of a constructed-response test of critical thinking, the Collegiate Learning Assessment (CLA). The CLA, which consists of mini work samples, tests a combination of high-level cognitive skills, and emphasizes value-added in its reports....

Standard settingis the process by which cut scores are established to classify scores on a test into different categories or standards for performance. Cut scores (or cut points) are the points at and above which a test score is considered to qualify as meeting a performance standard and below which it does not. Examples of standards for performance includepassingandfailing; expert, proficient, basic, andbelow basic; andunsatisfactory, satisfactory, andexceeds expectations.

The reasons for establishing standards on a test are varied. One use of standards is for passing or failing people on professional licensing tests. Examples...

As discussed in the previous chapter, the use of multiple PTs, multiple panels, and the sorting step at the end, is part of building a validity argument to show that the results can be interpreted meaningfully. We used a three-step process consisting of (1) setting standards individually, (2) arriving at consensus within a panel of four to five participants, and (3) participating individually in a sorting task where panel members (blind to the scores of responses) sort a random set of set responses into the different performance standards. The first two steps of the method are similar to the first...

In this portion of our report, we investigate several questions, and we present results for each separately. Because some questions addressed comparisons of PTs, while others addressed comparisons of locations, and still others addressed comparisons of the individual results to the group results, the data are summarized differently to satisfy each question. For example, if location and PT differences were not the focus of the question, the results were combined across PTs and across locations. For cases where locations or PT differences were the focus of the question, the locations were treated separately and/or the PTs were treated separately. (Tables...

We addressed several questions in the previous chapter that illustrate the type of questions schools should ask to ensure that the standard-setting procedure is functioning as intended. These questions are not an exhaustive list of questions that could be posed and answered about the standard-setting process and results. Instead, they were selected to illustrate the types of question that could and should be examined as part of a standard-setting process.

The first question (“Was there consistency across individuals in where they placed the cut points?”) provides an overall assessment of the amount of agreement among panel members. Among our panel...

In this study, we demonstrated a technique for the CLA that can be used for establishing uniform standards within a college or even within a department that are tailored to the institution’s goals and mission. Using the guidelines for evaluating a standard-setting study (as described in Chapter 2), our study was designed to build a validity argument that the results can be interpreted meaningfully. Specifically, the validity argument included the following:

The standard-setting method was carefully selected to address the unique features of the CLA.

The purpose of the standards was clearly specified and the method was designed to address...