Connecting Efforts to Support Minorities in Engineering Education: Proceedings of a Workshop (2023)
Chapter: 7 Data Collection and Sharing: Technical, Legal, and Policy Issues
7
Data Collection and Sharing: Technical, Legal, and Policy Issues
Isabel Cardenas-Navia (Workcred), workshop planning committee member, introduced and moderated the panel. She started by describing her own pathway from a STEM-focused summer program in middle school to being a mentor herself in graduate school for girls in elementary and middle school. She described her experience of secondary, undergraduate, graduate, and postdoc education in four different states both to provide a sense of the scale of educational pathways and to invite the audience to consider how data can be used to support individuals along their educational pathways to an engineering career.
Afet Dundar (National Student Clearinghouse, NSC) started by giving an overview of NSC, a nonprofit organization “created in 1993 with a mission to serve the education and workforce communities and all learners with access to trusted data, related services, and insights,” she said. “We make sure that how we collect the data, how we share the data, how we use the data is in a manner that is trusted and secure” and FERPA-compliant.52 The data received by NSC are mostly from colleges and universities, but also from high schools and industry credential providers.
The NSC’s Enrollment Reporting53 data cover 97.3 percent of all enrollments at Title IV degree-granting institutions, 99.5 percent of those at public 2- and 4-year institutions, 95.6 percent at private 4-year nonprofit institutions, 82.4 percent at private 4-year for-profit institutions, and more than 90 percent of enrollments in each of the 50 states. “The institutions submit data to us voluntarily and at no cost to share the data. In some cases, there is a low cost when they are looking at outcomes of their own students, such as when students transfer to other institutions,” she explained.
NSC’s DegreeVerify54 contains more detailed information about degrees and certificates—“like degree award date, major”—sent by colleges and universities, to help individuals with degree and certificate verification for employment and other purposes.
Dundar then explained other types of data sources, such as the Postsecondary Data Partnership (PDP),55 a relatively new NSC service with over 500 participating schools. “Through PDP, colleges and universities send us a lot more granular data like course level data, credits, grade data, and demographic characteristics. We use the data to produce key performance indicators for the institutions themselves and share those key performance indicators with a lot of organizations with the institutions’ permission,” said Dundar.
___________________
52 Family Educational Rights and Privacy Act (FERPA): https://www2.ed.gov/policy/gen/guid/fpco/ferpa/index.html
53 https://www.studentclearinghouse.org/colleges/enrollment-reporting/
54 https://www.studentclearinghouse.org/colleges/degreeverify/
NSC’s Industry Credentials56 is a service with six participating credential providers that collects comprehensive information about learners, and High School Diploma Data57 covers almost half of all US high school graduates. “High schools send this data to receive postsecondary information about their students,” she said.
With the data collected from all types of institutions, NSC is “able to measure outcomes over multiple years for different cohorts at the national, state, or regional levels,” said Dundar. She noted that since 2016 outcomes can be disaggregated by race and ethnicity, and robust program-level data have also been available since 2015. NSC’s Research Center analyzes various types of collected data to look at educational pathways to benefit institutions, the education community, policymakers, and students. The center publishes reports based on its data analysis58 and “is also leading the Clearinghouse’s new initiative on Equity in Data, Research, and Analytics,” which examines what equity means in data, what biases are present in data, and how the biases are or could be mitigated.
Speaking about how the data are made available to others for use, Dundar said that “in most cases the data can be used in the aggregate, but…an organization or institution…can get [data] for those students who are enrolled in their college or organization” to show students’ enrollment and completion outcomes.
Using NSC’s StudentTracker59 service, colleges, universities, and high schools can track the outcomes of their participating students, and the data can also be used to improve an institution’s ability to recruit and retain students.
Dundar explained that a major goal of the PDP program is to enable organizations working with colleges and universities to improve student success by using PDP key performance indicators. She assured that, “in all these cases, colleges and universities give permission to the Clearinghouse to share their data with these third-party student success initiatives.”
Last, she said that NSC also does custom research,60 which involves aggregate data only, for educational institutions as well as other organizations.
Claus von Zastrow (Education Commission of the States, ECS) began with some background about this “national nonpartisan, nonprofit that helps education leaders make effective education policy. We work with roughly 350 commissioners in every state and the District of Columbia,” he said, adding that “commissioners are typically people like governors and their staff, state legislators, leaders of state education agencies, state board members, and higher education officers, among others.” ESC works to inform these people by, for example, conducting research on state policy, reporting on policy trends, convening state leaders, and offering tailored counsel and guidance.
“In the past 10 years,” von Zastrow said, “most states have adopted K-12 science standards that incorporate engineering” and programs like Project Lead The Way61 and Engineering is Elementary62 have expanded. In addition, “over the same period, state data systems have become much more sophisticated in K-12 as well as in other areas. This offers the
___________________
56 https://www.studentclearinghouse.org/workforce/industry-credentials/
57 https://www.studentclearinghouse.org/high-schools/
58 https://nscresearchcenter.org/publications/
59 https://www.studentclearinghouse.org/colleges/studenttracker/
60 https://nscresearchcenter.org/customresearch/
62 https://www.eie.org/stem-curricula/engineering-grades-prek-8/engineering-is-elementary
prospects of better information about the kinds of opportunities students actually can receive in K-12 engineering.”
But there are difficulties in determining “what students have opportunities to learn engineering in or out of K-12 schools or if those opportunities are at all widespread or equitable,” he said. “Data from national surveys suggest that, for example, a girl of color is much less likely to get exposure to engineering in K-12 than a White male student.” He observed that there are likely more available data than people think, and that all kinds of data can be helpful.
As a case study, von Zastrow used the California Arts Education Data Project63 to draw parallels to engineering education. This public dashboard has information on access and participation in arts education in California schools, including whether arts classes are available to students in every district and school across the state. “It also allows visitors to measure inequities in access and participation by race, ethnicity, and family income.” He noted that these data were already in the state system, but it took work to present them in an effective way.
Explaining the types of information available in the state data systems, von Zastrow explained that “you can find out, for example, who has access to engineering classes, who participates in these classes, and what kinds of credentials do teachers have to teach them.” He went on to say that the state data systems would likely not yield much information on engineering at the elementary level, but that most states have data on middle and high school engineering course titles and enrollment data, as well as somewhat less reliable data on teacher credentials.
In terms of how stakeholders in K-12 engineering education can get access to such data or make them public, von Zastrow said that advocates can take “steps to have collaborations with states to understand state data systems and have attention to the quality of the information because it might not be perfect right out of the gate.” He added that “some kind of capacity to analyze large datasets and the ability to create visualizations” are also useful.
Giving the example of the Arts Education Data Toolkit,64 created by ECS with the National Endowment for the Arts, von Zastrow said that “it takes you step by step through the process of extracting, analyzing, and visualizing state data on education” in the arts. He suggested that a resource like this could be helpful for engineering education as well.
Regarding data on out-of-school-time K-12 engineering education, he acknowledged that “these data are harder to find—but not impossible,” and cited ArtLook Map65 to show what is possible in this space.
“Data on the quality of programming or the quality of K-12 instruction are not really available in the state data system, and they are very hard to collect at scale.” von Zastrow said. He elaborated that “small informal engineering programs have little capacity for data collection. It’s important to protect students’ privacy, so you cannot necessarily get data on individual students if you are a teacher or that student’s guardian.” However, “you can get a sense of what’s happening in your community, what’s happening in your school. Do students of color who attend these schools have access to these courses? Do they have access of opportunities?” The answers to these questions provide “a foundation of information and a context” for more examination of the data.
He concluded with an important message: “the bottom line is that the data exist. They are not super easy to get your hands on, but it is possible. Collaboration with states can allow you to
___________________
63 https://createca.org/california-arts-education-data-project/
make these kinds of information public so that people can use them and promote equity in engineering.”
Louis Soares (American Council on Education, ACE) reviewed results from the council’s Race and Ethnicity in Higher Education report66 showing that the US population has grown more educated and more racially and ethnically diverse over the past two decades, in large part due to the growth of the Hispanic/Latine population. Nonetheless, “great differences exist by race, ethnicity, and gender in where students go to college and what they study, signaling an uneven playing field in the labor market and a threat to the opportunity for intergenerational upward mobility.” Specifically, Soares reported that a larger percentage of students from historically minoritized populations, compared to students from other populations, attend for-profit institutions, “which tend to have lower performance rates.” Moreover, field of study for degrees varies across racial and ethnic groups at all educational levels—and students from historically minoritized populations are less likely to enter STEM fields.
Highlighting another finding from the report, Soares said that, “among all positions and seniority levels, faculty, staff, and administrators remain less diverse than the student body,” and many of the higher education jobs held by individuals from historically minoritized populations “tend to be outside of the classroom and leadership, meaning students of color are more likely to see people from similar backgrounds in clerical, technical, and service staff positions.” In short, “we have a lot of work to do in terms of diversifying STEM generally, in terms of both gender and race or ethnicity.”
Discussing how onboarding students, providing them with a supportive community of peers, and engaging family members, faculty, and staff can support students throughout their education, Soares provided some examples of successful programs that ACE has been involved with. These include the ADVANCE Program67 at Florida International University, Meyerhoff Scholars Program68 at the University of Maryland, Baltimore County, and the STEM Collaboratives Project69 at California State University.
In closing, Soares described ACE’s Shared Equity Leadership Model70 “that affects critical consciousness at the individual level of faculty and leaders on campuses, informs values, and then informs the practices to create a more inclusive environment.”
DISCUSSION
Cardenas-Navia started the discussion session by taking a question from the audience about whether the panelists’ organizations partner with ASEE or other organizations for collecting and reporting data. Dundar answered that, while the NSC “Research Center collaborates with many organizations that, for example, focus on college access and success outcomes of particular groups of students,” it does not collaborate with ASEE.
Cardenas-Navia then asked the panelists how their organizations’ data and features differ from those of the Department of Education’s National Center for Education Statistics (NCES).71
___________________
66 https://www.equityinhighered.org/
68 https://meyerhoff.umbc.edu/
69 https://www.calstate.edu/impact-of-the-csu/research/stem-collaboratives
70 https://www.acenet.edu/Documents/Shared-Equity-Leadership-Work.pdf
Dundar explained that there are differences in how the two entities receive data; for example, NSC’s “Research Center uses the data to produce publications,” but because they “don’t have first-time indicator in enrollment data, we use definitions to come up with first-time cohort.” She noted differences depending on decision making and definitions in publications. von Zastrow added that “NCES data, at least in engineering, in the K-12 arena tend to be survey-based,” such as the eighth-grade National Assessment of Educational Progress (NAEP)72 in engineering, which provides representative samples but not data at the state, district, or school levels. He recognized that “state administrative data systems can be helpful,” as they provide information about “who has access to what kinds of courses.” Soares commented that it is helpful “to make the data actionable in a way that speaks to the community stakeholders.”
In response to a question from Cardenas-Navia about using qualitative data or a mixed-methods approach to better understand student experiences in STEM, von Zastrow replied that “I am confining my conversation more to the K-12 level, which is an emerging data field. My sense is that the quantitative data are easier. The qualitative data are harder.” He used the ArtLook Map example to explain how an interesting qualitative picture can be obtained, but admitted that this is very hard to do at scale.
Responding to a question about how statistics and data can have an impact on policymaking, Soares made the case for “understanding the motivations of [federal] policymakers and how they receive the data,” such as viewing engineering education through lenses of innovation capacity, social mobility, or economic competitiveness. At the precollege level, von Zastrow said, data do not affect policy because they are not generally widely disseminated, other than the NAEP scores, which he said did not drive policy decisions. Returning to the example of arts education, he elaborated that data analysis can help with real action, rather than assuming that all students have access to similar education programs.
Finally, Cardenas-Navia asked the panelists what data on students and programs are collected but not yet fully explored. Dundar noted an “increased interest in looking at smaller groups of student populations in program-level data,” and mentioned an NSC publication73 “looking at credit accumulation rate, credit completion ratio in the first year, because that is related to how they will progress and how they complete down the road.” Soares added that “I do think that part of the next evolution of research in this space is going to more deeply integrate the available quantitative information with qualitative” to examine small group differences.
Cardenas-Navia commented on “this idea of crowdsourcing as we think about getting the reflections of students who are going through those pathways, and faculty, and the technology has helped in terms of collecting quantitative data in different ways.”
___________________
72 https://nces.ed.gov/nationsreportcard/tel/
73 https://nscresearchcenter.org/yearly-success-and-progress-rates/
This page intentionally left blank.
