Evaluation and Applied 

Research Consulting



PROJECT 1: Cultural Responsiveness in STEM Program Planning, Implementation, and Evaluation

The recent increase in STEM and science-based programs targeted to minority students must be met with appropriate culturally responsive practices (Johnson, 2005). The current project was designed to investigate ways in which program evaluators and staff implement culturally responsive practices. Evaluators across the country and program staff in New York, California, and Texas were invited to participate in a multiphase concept mapping project to 1) Brainstorm culturally responsive practices, 2) Sort or organize these statements according to themes of their own choosing, and 3) Rate each statement on importance and feasibility with respect to their practice. We summarize results of the structured conceptualization effort in comparison to the theoretical literature, discuss statistical differences between perceptions of Importance and Feasibility of practices, and suggest activities that consolidate and align practices as conceptualized by each group in a way that makes principles actionable.

PROJECT 2: Evaluating Causal Explanations of Historical Events (EEE)

       In the causal reasoning literature, Klahr and Dunbar (1998) propose a positivist perspective of scientific reasoning which consists of three components: hypothesis formation, experimental design, and evaluation of evidence. In this current research, we focus on the third component, evaluation of evidence. Traditional experiments on evaluation of evidence require that participants identify and/or rate pieces of evidence provided as potentially causal explanations for an occurrence or event (e.g. Amsel & Brock, 1996; Ruffman, Perner, Olson, & Doherty, 1993). Often, participants are then provided with anomalous information and are asked to rate each piece of evidence again in order to test for the relative strength of an individual’s beliefs/confidence in a given piece of evidence.

We use a similar experimental paradigm in this proposed research. In this research, unlike past research on scientific reasoning, we argue for a logical qulaification of Lipton (2000), that experience and background information situated in the cultural reality of some ethnic/racial groups affect an individual’s evaluation of evidence in particular ways. We believe this to be especially true when reasoning about social events or problems that have some flexibility in how they are interpruted (i.e. historical or social events whose occurrence can be viewed as the result of multiple causes depending on an individual’s socio-historical position.) Specifically, this study is designed to 1) show that membership in a cultural group effects what an individual perceives as relevant evidence in explaining causes of a social event and 2) investigate the relative strength of an individual’s confidence that a piece of information is relevant to the causal mechanisms for an event even when faced with an anomaly to that evidence.

PROJECT 3: An investigation of reasoning in professional evaluators' application of culturally responsive evaluation (CRE) principles

  A paradigm used in the field of scientific reasoning (Koslowski, 1996) is used to investigate how culturally responsive evaluation (CRE) professionals reason about culture in evaluation practice. Evaluators will be presented with six program scenarios, each paired with two options for conducting an evaluation. One of the options provided for each scenario will consist of integrated CRE principles derived from earlier research. Participants will provide ratings for each option, give a qualitative explanation of their ratings, and identify the approach "most likely" to result in desirable evaluation outcomes. In addition, participants will be provided with anomalous evidence regarding their "most likely" choice and will be asked to re-rate the option they chose as best. By adapting this research paradigm we intend to 1) determine the extent to which these CRE principles are applied across varying contexts, 2) determine the extent to which approaches that utilize CRE principles are discernable from other evaluation approaches, 3) investigate the role of culturally situated background knowledge in how evaluators reason about culture in their evaluation work, and 4) verify that CRE principles systematically derived in earlier research (Casillas & Trochim, in preparation) represent how evaluators think about culture in evaluation practice.



Recent Blog Entries