6
Research on Admissions

The final panel session of the day focused on research questions about admissions metrics and policies as well as ideas for translating or adapting programs to use in different contexts and at different scales. The four speakers were Teri Reed (University of Cincinnati), Bruk Berhane (Florida International University), Michael Bastedo (University of Michigan), and Julie Posselt (University of Southern California). After the presentations, Theresa Maldonado (University of California President’s Office) moderated a discussion among the speakers.

QUANTITATIVE APPROACHES TO IDENTIFY BIAS AND CHANGE ADMISSIONS CRITERIA

Purdue University’s College of Engineering has been working to increase representation of women in its first-year class for many years, said Teri Reed, who was a faculty member there before moving to the University of Cincinnati. She showed the stages of admissions (Figure 6-1), and reported that when Purdue began working to increase the number of women engineering students, she and her colleagues were “advised to increase our applicants, which was successful between 2006 and 2010,” when the engineering college saw a 46 percent increase in the number of women applicants—but only a 24 percent increase in the number of women admitted.

Analysis of the metrics used in admissions decisions showed that female applicants, compared to male applicants, had higher high school GPA, class rank, and verbal SAT scores, but there was no statistically significant difference between women’s and men’s average math SAT scores. A similar pattern existed for those admitted to engineering: women had higher average class rank and GPA than men, while the latter had higher average math SAT scores than admitted women. Although admitted men’s average math SAT scores were higher than women’s, Reed noted that the lowest score on every metric was submitted by a White man; the lowest math SAT score was above 400 for admitted women, while the lowest SAT score among admitted men was below 400. Similarly, no woman with a GPA less than about 2.4 was admitted, while men were admitted with a 2.0 GPA.

When Reed and her collaborators examined the metrics for students who were denied admission, women again scored, on average, higher than men who were denied admission on every metric except SAT math scores. One glaring finding was that the highest 25 percent of women denied admission had higher overall GPAs than the lowest 25 percent of the men admitted to Purdue’s engineering college. Reed noted that if the university’s admissions process were unbiased, there would be no statistical differences in the metrics of the admitted populations.

Given the copious amount of research showing that SAT and ACT scores are not predictors of postsecondary academic achievement or graduation rates, that high school metrics are a better predictor of first-year college grades than SAT scores, and that there is a consistent gender bias in standardized tests, there were three possible conclusions to draw from Purdue’s experience:

1. Only the highest-ability women are encouraged and/or self-select to apply to the College of Engineering, and men with a much wider range of academic ability are encouraged and/or self-select to do so.
2. Women applicants are held to a higher standard than men regarding their high school performance.
3. Admissions counselors put more weight on test scores than high school performance.

It turned out that the third was true, as admissions counselors were using SAT math scores as a first-pass filter. This points to the existence of institutional bias, said Reed.

Reed and her collaborators turned to colleagues who were developing a model of student success using the Student Attitudinal Success Inventory instrument (Reid 2009), which incorporates 14 psychosocial measures and a variety of other factors to predict retention and academic performance. The important factors predicting first-year retention were different for women and men. Semesters of math classes were important for men but not women, while leadership skills and semesters of science classes were important for women. For graduation in five years, semesters of English classes were important for men, while deep learning ability was the most important factor for women.

After presenting these findings to various levels of the administration at Purdue, Reed and her colleagues were given the go-ahead to change the engineering school’s admissions factors. These changes included emphasizing the SAT verbal and/or writing components and no longer using SAT math scores as a first-pass filter; counting semesters of science, English, and math; science grades; and assessing traits reflecting creativity, leadership, social relevance, openness to diverse thinking, deep learning ability, academic motivation, and persistence. By 2020, after implementing these new admissions criteria in 2010, women accounted for 26 percent of the incoming class of engineering students, compared to 15 percent in the early 2000s.

ASSESSING THE EFFECTS OF HOLISTIC REVIEW

Bruk Berhane presented research that examined the holistic engineering first-year student admissions review led by the university enrollment management team in partnership with the college of engineering at a large East Coast university. The “target group and the population of students that this holistic review project and process were designed to…impact were historically underrepresented populations,” said Berhane. He noted that this was not so much a formal research study as a practitioner’s approach, grounded in some of the research conducted by Reed and colleagues that “would lead to policy changes, ideally leading to higher numbers of women, higher numbers of students of color, higher numbers of first-generation college students.”

Berhane explained that the important part of working with enrollment management was that the office had a framework in place that enabled staff to look at factors such as leadership, community service, extracurricular activities, letters of recommendation, essays, trends in grades (e.g., improvements in students’ high school grades as they progressed through high school), and unique circumstances and challenges. He noted that until 2012 the institution in question was still using SAT scores as a proxy for admissions even after enacting a holistic review process, and that over the study period of 2012 to 2018, admission of women grew from 24 percent to around 33 percent, although admission of students from the other target populations remained at approximately 15 percent. He noted that retention and graduation rates for students of color and first-generation students were not tracked for this study, but one-, two-, and three-year retention rates and four-year graduation rates for women were consistent with those prior to the policy change. Five- and six-year graduation rates for women increased from approximately 90 percent to 94 percent in five years and 96 percent for six years.

IMPROVING CONTEXTUALIZED HOLISTIC REVIEW WITH ADMISSIONS DASHBOARDS

When Michael Bastedo asked over 300 people how they defined holistic review, there were three common answers. The “whole file” answer meant that admissions decisions were determined by reading all parts of an application, but “there wasn’t anything more than that about what they’re doing with the information in that file, and a lot of times the decision making is actually pretty formula driven, regardless of the fact that they’re saying that they do holistic review.” A “whole person” response meant that admissions decisions considered the applicant as having unique individual characteristics and achievements and assessing whether they could make a contribution to the school and community. And a “whole context” response meant “reviewing the individual in the context of the opportunities that are available in their environment, in their family background, considering extenuating circumstances and educational opportunities.” This third view, said Bastedo, is what most admissions leaders talk about when they discuss holistic review, but it accounted for only about 30 percent of the responses.

One of the challenges in conducting holistic review is related to the information sheets high schools provide to admissions offices. These sheets have a lot of information but not all of it is useful for the admissions process (for example, Bastedo showed a slide of a high school profile sheet that described the school in “a picturesque community of rolling hills and century-old dairy farms centered around the quaint village”), and only 10 percent of them provide class rank information. In fact, he said, wealthier and private schools rarely provide class rank information because of pressure from parents worried about their child’s “ability to get into more

selective colleges” and the feeling among the high schools that class rankings can create a toxic competitive environment.

One question Bastedo wanted to answer in his research was whether he could demonstrate in a randomized laboratory experiment the cognitive biases that are supposed to be common in admissions decisions. He and his collaborators recruited 311 admissions officers who regularly read admissions files and randomly assigned them to one of two conditions (Bastedo and Bowman 2017). One group received somewhat limited information of the type that could be gleaned from a high school profile sheet, such as graduation rate, whether the high school was public or private, number of students, and information about parental education as a surrogate for socioeconomic status of the family. The second group received detailed information that included college enrollment statistics, average test scores, percentage of students on reduced or free lunch programs, percentage of students with limited English proficiency, number of AP classes, and percentage of students with a score of three or more on AP tests.

Results showed that more students from lower-socioeconomic-status families were given a positive admissions recommendation by the group that had detailed information than by the group with limited information. There was no difference in admission of students from higher-socioeconomic-status families regardless of whether the student was middle achieving or high achieving. Bastedo noted that the results did not differ depending on how much experience the admissions officers had, their title, or whether they worked at a more or less selective school. One significant difference was that the admissions officers who said context was important to them, when they were initially asked to define their idea of holistic review, were more likely to admit an applicant from a low-socioeconomic-status family. To Bastedo, that suggests that giving admissions officers more robust contextual information could move the needle in terms of admitting students from low-income families into engineering programs.

Bastedo and colleagues (2017) then created a dashboard similar to the College Board’s Landscape dashboard,¹⁵ which provides consistent contextual information for each applicant in the form of key indicators of their high school and its neighborhood. When the admissions officers reread applications using the dashboard, there was no difference for those from an institution using a formula-driven admissions process, but those from a private university that employs a robust philosophy of holistic review were about “20 percent more likely [to] admit students with the highest adversity in their pool,” said Bastedo, adding that this resulted in increased racial diversity of the admitted class without any change in average SAT scores, so “it wasn’t any kind of access-diversity trade-off that some people have concerns about.”

In conclusion, Bastedo said that contextualizing admissions uses high-quality school and neighborhood information for every student, regardless of whether they are in state or from a school that the admissions officer knows. Achievement, he added, can be evaluated in a holistic, contextualized manner without guessing, but training might be useful so that it becomes the norm for admissions officers to interpret the data and look for context in every application.

HOLISTIC REVIEW: A USEFUL TECHNOLOGY THAT REQUIRES CAREFUL IMPLEMENTATION

When Julie Posselt began studying the role of merit, diversity, and faculty gatekeeping in graduate admissions, she wanted to understand why admissions officers rely on criteria that tend

___________________

¹⁵ The Landscape tool is available at https://professionals.collegeboard.org/landscape.

to undermine their institutions’ diversity goals. To answer that question, she interviewed 85 faculty members at 10 highly ranked PhD programs in nine fields at three public and three private universities, and observed 22 hours of multiple rounds of admissions review in six of the programs (Posselt 2016; Posselt et al. 2019).

From her ethnographic study, she identified clear patterns in which the main definition of merit that shapes who gets admitted differed from the definition of merit used in the initial screening process for the admissions shortlist. “The way that faculty in this study conceptualized what merit means when they were doing their initial screening was largely in terms of conventional academic achievement, identifying students who they believed to have a low risk of attrition,” Posselt explained, “and they tried to contextualize numbers” by considering the prestige and rigor of the curriculum in the undergraduate institution. “But a very different definition of merit was used when they were trying to judge from among all the highly academically qualified individuals on the short list. Here faculty were thinking much more about the future of their discipline and how they could use admissions to help craft the future of the discipline that they imagined.”

From her research, Posselt (2020) has developed a framework for equity-minded holistic review. It is not enough, she said, to change the criteria or sequence of the review process; it is how admissions officers and faculty make sense of information that makes review holistic. “Like any technology, it is not the claim to using holistic review, but rather the way it is designed and implemented that determines how well it can contribute to an agenda of organizational activity toward diversity and equity,” she said. From the design standpoint, her research describes four main characteristics of equity-minded holistic review:

• comprehensiveness in terms of accounting for a variety of criteria that represent the student and their potential (e.g., coursework and grades, research experience, alignment with current faculty research interests, prior contributions to equity and inclusion, and socioemotional skills competencies);
• contextualization, in terms of both thinking about how different pieces of information are related to one another to form a landscape of opportunity and recognizing the error inherent in statistics such as grade point averages or test scores so they are not used “in overly precise ways”;
• being systematic rather than ad hoc in the review process, relying on shared and structured protocols; and
• being mindful about equity throughout the process.

In practice, Posselt said, conducting equity-minded holistic review calls for being both race conscious and aware of systemic inequities as well as willing to account for them when making sense of a student’s achievement. This review is also asset based, looking for opportunities to highlight a student’s strengths and not eliminate them because they are missing some particular trait or experience, and paying attention to a variety of elements in an applicant’s file to provide context about the candidate as a whole person. Equity-minded holistic review also means asking questions about the extent to which patterns of representation are tracked across the different phases of review. Perhaps most fundamentally, said Posselt, institutional responsibility for equity-minded holistic review means being willing to ask critical questions about practices and processes.

Posselt then reviewed some of the lessons she and her collaborators in the California Consortium for Inclusive Doctoral Education have learned as institutions implement equity-minded holistic review by establishing new routines and delegitimizing some of the older,

standard routines. One lesson is that implementation entails regular discussions about what is legitimate (e.g., about GRE scores) versus what is simply inherited admissions policy. Another is the need to develop and implement rubrics in a way that defines shared norms for file review routines among reviewers. Finally, it is important for individual reviewers to develop a routine of contextualizing information in an application and how that information relates to systemic inequities.

In closing, Posselt invited everyone to consider these processes as tools that are not inherently good or bad but as things to be designed, honed, and improved to achieve specific ends. The diversity of the designers of these processes is important, she added, so it is crucial when developing holistic review processes to have a diverse group of people at the table.

DISCUSSION

Theresa Maldonado opened the discussion by observing that the panelists had touched on the vulnerability associated with transitions, and the relevance of high school profiles as guides for students transitioning to undergraduate programs and of undergraduate success to predict transitions to graduate school.

She then asked the speakers to highlight some of the pain points that inhibit successful transitions and how they might inform research questions they would like to address. Berhane replied that getting the review process done without artificial intelligence or data-driven software and having the actual hands to do the work was challenging for the institutions he has examined. In one case, for example, there were 15 to 20 volunteer reviewers from the college of engineering who received a very small gift for volunteering. He commented that without some kind of built-in algorithm, it will always be necessary to properly train reviewers to get optimal results.

The single pain point that Posselt sees in PhD program admissions is the perceived tradeoff between getting through the large volume of applications and doing a deep and thoughtful review of a complete file. She noted that some institutions are implementing multirubric review processes to deal with a large volume of applications. If designed correctly, these rubrics can make the initial review process more efficient by identifying the important factors to look for in an application.

Reed said her institution has created a scholarship committee charged with using the information that she and her colleagues have produced that illustrates what makes for a successful student. This made a difference that resulted in increasing the number of scholarships that went to women applicants. It was important, she noted, to provide annual training for the reviewers, and even recruiters, about what it takes to be a successful engineer.

Bastedo commented that faculty are risk averse when it comes to graduate admissions because they worry that the wrong decision will force them to work for several years with a student who was not qualified. He added that undergraduate admissions officers worry more that students will fail if those responsible for admissions do not rely on traditional metrics.

Part of the problem, Bastedo said, is that data are not available for students who were not admitted and would have succeeded even though they did not meet certain standards on the traditional metrics. Without those data, he said, it is difficult to convince admissions officers otherwise. He suggested some small-scale experiments in which different types of students are admitted, supported, and tracked to produce the data. Posselt agreed, noting that holistic review and admissions cannot occur in a vacuum without data to support those types of review.

Systemic change will occur, she said, only when an equity-minded approach is brought to bear not just on admissions but on mentoring, academic and social support, and financial aid, rather than “relying on a single lever to do the work.”

Asked to discuss some of the rubrics they are using to ensure comprehensiveness in the review process, Posselt explained that being comprehensive does not mean considering everything but being intentional about what data are used in the decision-making process. Most of the evaluation rubrics she has seen include five to eight criteria and they are each operationalized so that there is room for flexibility, individual judgment, and nuance. She noted that no rubric that evaluates humans is complete without having a space for comments where the reviewers can justify their assessments and explain where and why they are deviating from some convention.

An audience member asked the speakers to talk about how to conduct holistic reviews for foreign STEM students seeking admission to US graduate schools. According to Posselt, this is challenging because contextualization is difficult. Some graduate schools are working with the services that provide transcript translations to identify comparable courses of study at US institutions. She is a proponent of using the Test of English as a Foreign Language scores, since current research suggests that its predictive validity for international student performance is significantly greater than that of the GRE and keeping it in the mix will not hurt domestic student opportunities.

Another question focused on whether making the evaluation process more complicated makes it hard for students who do not have coaching to compete for admission. Bastedo acknowledged that that can be an issue, and posited that the key is to have criteria that are not coachable or subject to manipulation. For example, test scores, which are often a primary decision-making factor, can benefit from coaching, and students can be coached about the importance of engaging in extracurricular activities (although that does not necessarily help them seek out and engage in those activities). He advised attendees to consider “what your criteria are, how coachable they are, and how connected they are to wealth and privilege.” He also suggested considering “the degree to which people do not have that wealth and privilege. How can they succeed on the metrics that you are using? What does it look like when students do not have wealth and privilege? What do they look like? What can we imagine? And can we treat that as just as valuable in the process—as just as much of an asset—as we do the students who have wealth and privilege?”

Berhane agreed with that idea, and elaborated on it by making the case for pulling out data from first-generation college students or students from underresourced neighborhoods as their own data points to compare apples to apples. Posselt added that when thinking about how various metrics, such as participation in extracurricular activities or leadership skills, might apply to applicants from less privileged backgrounds, it is critical to have that conversation with students who come from those backgrounds and embody those perspectives. As an exemplar, she pointed to Fisk-Vanderbilt’s master’s to PhD bridge program, which created metrics from these other perspectives to recognize different manifestations of desired student qualities. The result, she said, has been to significantly increase the participation of Black students in the school’s astronomy, physics, and some engineering graduate programs. Maldonado mentioned that North Dakota State University has established a cultural advisory council to gain those perspectives.

An audience member asked the speakers if any of them had examined the admissions process used by the military academies or military promotion boards for lessons that could be applied to undergraduate or graduate admissions. Posselt replied that she has not but is excited

about the prospect of research in that area. Bastedo said that he is studying military hiring, but is not sure what questions he would ask that would pertain to college admissions.

An audience member asked Posselt if she could comment on why Black students do not do as well in admissions to PhD programs in physics. She replied that it starts with the programs’ emphasis on a standard cutoff GRE score as a primary filter, which excludes most Black applicants. She also noted that she heard “discussions of hairstyles, hobbies, hometowns—things that are racialized in admissions in ways that I don’t think the physics community is aware of.” A third factor, she said, is that most of the Black PhD applicants are from Historically Black Colleges and Universities (HBCUs) and other institutions from which graduate physics programs are less likely to recruit students.

As a final question, Maldonado asked the speakers for their ideas on research questions to identify how the COVID-19 pandemic disrupted admissions and the plans of many students. Bastedo said that he would like to see what the effect will be from so many schools going to a test-optional or test-blind admissions process in the wake of the pandemic, which implies that they will use a more holistic review process. Berhane seconded that idea and also suggested looking at the implications for people who do not enter university or college immediately after high school and where they fit in the admissions review process.

REPORTS FROM THE BREAKOUT DISCUSSIONS ON FUTURE RESEARCH NEEDS

After the day’s three panel sessions, the attendees distributed themselves in two virtual breakout rooms to spend 45 minutes to discuss the following three questions and then report back to the assembled workshop attendees:

• What barriers do you anticipate in implementing the practices and research you heard about today?
• How would you overcome these barriers?
• Where should future research on admissions focus?

Tom Farris reported that his breakout group talked at length about the military promotion process as an analogy for admissions, specifically the collection of broad and deep subjective information about individuals. Because of the subjective nature of the information as well as the stakes involved in military promotions, evaluators receive a great deal of training on how to process that information to make decisions. This group also discussed how artificial intelligence and machine learning could be applied to the admissions process, as well as the utility of an adversity index and conducting interviews to assess leadership potential to improve the process.

Tanya Ennis’s group discussed barriers such as the lack of data and the desire to gather more data on students and how they perform. In terms of research, the group wanted to study (i) how to create and maintain an asset-based framework for admitting students to an alternate engineering track and (ii) who gets to make the decisions for admissions. Research could also identify effective ways of changing the mindset about who belongs in engineering, a longstanding issue. Related to that was the importance of determining how to not just incentivize faculty to adopt new ways of considering admissions but also hold them accountable for making the changes that will make engineering—and all of STEM, for that matter—more inclusive.

This group also discussed creating both a culture that supports the use of high-impact practices that are known to benefit students and avenues for faculty to better welcome students to campus and get to know them by name. The group wondered how faculty could both organize into groups to serve as a resource for entering students, particularly those who are the first in

their family to attend college, and work to break down the elitism among many engineering students relative to one another based on engineering discipline. It is important, said Ennis, to disrupt the hierarchy of disciplines that students can adopt and take with them as they mature in their profession.

In terms of research, the group raised the question of how to track and collect data on students through graduation, and expressed interest in research on the interplay among admissions, financial aid, retention, and graduation, particularly as that interplay pertains to students from underresourced communities. There was a suggestion to conduct research to see whether preengineering programs or programs that offer alternative college major options for students who are not directly admissible to an engineering program are successful at increasing diversity, equity, and inclusion. And the group considered the utility of research to identify which parts of the holistic review process are most important for predicting success for historically minoritized students and to understand the predictors of success based on students’ applications.

This page intentionally left blank.