Appendix B

Survey and Interviews

OVERVIEW OF SURVEY AND INTERVIEW FINDINGS

This document highlights some themes that emerged looking across both the survey and interview findings. [Note that the interviews were meant to both provide examples of some of the activities and collaborations of engineering societies, as well as to gather additional information not addressed on the survey (e.g. evaluation strategies) to round out the picture of societies’ education efforts].

OVERALL THEMES

All of the societies are primarily concerned with the professional development and continuing education of their membership. There is also a commitment (particularly among discipline-focused societies) to elevating the status of engineering and ensuring its future.

Engineering education at the undergraduate level is not a priority area of focus for all societies, but it ranks highly. Student chapters represent the most common strategy for connecting to higher

___________________

The survey was conducted by Inverness Research, which analyzed the results and prepared this report.

education. Pre-college education, while important to many, is not a priority—whether that is due to a lack of resources, or know-how, is not clear. However, those not involved in the pre-college engineering education believe it is critical to ensuring quality candidates for the engineering pipeline.

Nearly all interviewees expressed an interest in the results of the study and learning more about education efforts across the field.

Areas Where Societies Report Being Strong in Their Education Efforts

• Disseminating practices within their own membership through conferences, meetings, newsletters, etc.
• Strengthening the field through professional development and continuing education of their membership
• Creating greater awareness of engineering (or their particular discipline within engineering) through student chapters and other outreach efforts
• Making the case for the importance/relevance of their discipline
• Reaching out to and supporting groups or populations that are underrepresented in engineering
• Partnering with outside organizations or other societies
• Providing certification (discipline-based societies)

Areas Where Societies Could Strengthen Their Education Efforts

• Tracking the details of their education efforts (e.g. budget, investment, human capacity, etc.)
• Evaluating the impact of their programs in more coherent and rigorous ways (counting participants is the most common approach)
• Engaging in precollege engineering education efforts (not all societies are interested in this but they all believe it is important)

• Disseminating their engineering education efforts (interviews revealed dissemination is limited to traditional channels within societies’ membership)
• Connecting across societies (there are some examples of strong connections, but there are several barriers as well)

Possible Topics for Discussion at the Workshop

• What are some efficient strategies for societies to engage in collaborative education efforts, or how can societies collaborate and communicate efficiently?
• What are some alternative dissemination strategies, so that promising practices and innovative programs and examples can be shared across the field?
• What types of evaluation approaches make sense for societies to adopt? How can evaluation be designed so that programs improve and others can learn from the outcomes?
• What are some ways societies can address communication barriers across societies? Are meetings and conferences the best way? Are there others?
• How can societies engage faculty and encourage them to change their practice, or introduce innovation into engineering education at the undergraduate level?
• Is there a way to go beyond student chapters as a way to address undergraduate engineering education?
• For those that are interested, and have the financial and human capacity, how can societies support more and better engineering education at the pre-college level?

NAE ENGINEERING SOCIETIES STUDY – SURVEY RESULTS

This document provides highlights of the NAE Engineering Societies survey data collected by Inverness Research in the spring and summer of 2016.

Method

After developing and implementing a pilot survey that included 10 societies, Inverness Research drafted a final survey that was reviewed and approved by the project committee. The NAE and IR then drafted an email invitation that was sent to 121 societies, in some cases to multiple people at a given society. The email asked for societies’ participation in the study, and in particular, for recipients to decide who was the best person to complete the survey for their society.

After NAE sent two reminder emails, a total of 58 surveys were completed, for a return rate of 48%. There were eight cases in which two surveys were completed for a single society. In these cases, we decided to either contact the individuals and ask which survey to include in the study, or opted to include the survey of the individual who also participated in an interview, if applicable.

The Sample

The surveys were completed by a range of leaders within the organization, with the most prevalent position being the Executive Director or the President. Respondents with other roles included:

• Board of Directors member
• Director (various areas, such as education, outreach, etc.)
• General manager
• Vice President

The breakdown of responding societies, according to number of members was:

Because there was a relatively even spread of societies in these initial size groupings, we conducted sub-analyses of the survey

questions based on the size bands above. We also conducted sub-analyses according to whether a society was discipline-focused, or an affinity society. In the following summary, we indicate where there were statistically significant differences between these groups.

After an initial review of the survey findings, we conducted an additional sub-analysis to explore the relationship of societies’ priority education goals, activities, and audiences, and their rating of education as a priority weighed against other society priorities.

General Findings

• Overall, engineering (or engineering-related) societies are engaged in a range of education activities that target a range of audiences. Nearly every category of goal, activity type, and audience is represented in the work of the societies who responded to our survey.
• Supporting and growing membership is a high priority goal for all of the societies, which means that activities that both increase membership numbers and contribute to the professional growth of (practicing) members are important. Societies are less concerned with influencing policies related to engineering education, or in addressing pre-college engineering (for the most part). A sub-analysis revealed that discipline-based societies are more apt to focus on improving curricula and materials than affinity societies. Further, affinity societies are more apt to focus on culture change than discipline-focused societies are.
• The sub-analysis by membership size did not reveal many significant differences. One notable exception is level of investment in education endeavors, where small- and medium-sized societies are more apt to say their level of investment has stayed the same, while larger societies say their funding for education has increased in the past two years.
• The majority (87%) of societies face some kind of barrier in their engineering education work. The most common barriers include: communication; improving engineering curricula; incentives; as well as issues related to time, resources, and

• funding. A sub-analysis revealed that affinity societies are more apt to report facing barriers than discipline- focused societies.

• About ¾ of all respondents are engaged in partnerships with outside organizations, and 86% use connections to at least some extent with other engineering societies. Over half believe these connections to other engineering societies are useful to a good or great extent. There is a wide range of organizations and societies that are engaged in these partnerships.
• Societies’ investment in engineering education has not decreased in the last two years. Annual budgets, industry, and university-based faculty are counted as resources for engineering education work for most societies. Further, 77% of respondents said that engineering education is just as or more important than other society priorities.
• A majority (85%) of societies consider themselves leaders in the field. However, half of the societies rated their overall capacity to plan and implement education work as either low or some. 38% rated their capacity as high, and 12% rated it very high.
• Leadership Development is a higher priority for participants who said engineering education was “more or much more important” in the scope of their society’s goals and activities compared to those who said engineering education was “less or much less important.” The same holds for Continuing Education and Engineering Education Issues/Trends Research - these activities are high priority for those who said engineering education is more or much more important than other society activities. There were no statistically significant differences for target audiences.

Education Goals

Over half of all societies count supporting professional development, leadership development, and increasing diversity as high priority goals. Professional development leads with 90% of societies reporting it as a high priority. Fostering policy changes, and improving curricula and materials are lower priority goals.

A sub-analysis revealed differences between discipline-focused societies and affinity societies. Discipline-focused societies are more apt to focus on improving curricula and materials, even though it is a lower priority overall. Further, affinity societies see culture change as a higher priority than disciplinary societies. There were no significant differences based on society membership size.

In our analysis of the relationship between goals and commitment to education, we found that societies who rated education as important to their society were more likely to identify leadership development as a priority goal.

Education Activities

The majority of all societies are involved in professional development for their membership (82%). Sixty-five percent have student chapters, and 61% provide continuing education. Over half (58%) are working on partnerships with industry, and 51% are concerned with women in engineering. Strong or medium priority activities include mentoring and academic partnerships. Low priority activities include programs for veterans, fellowships, employability training, and student competitions.

A sub-analysis revealed that affinity societies’ priority goals are more apt to include fundraising, programs to promote diversity, and pre-college engineering education than discipline-focused societies. However, discipline-focused societies are more apt to include certification as a high priority goal. There were no significant differences in the analysis by society membership size.

In our analysis of the relationship between education activities and commitment to education, we found that societies who rated education as important to their society were more likely to identify Continuing Education and Engineering Education Issues/Trends Research as priority activities.

Target Audiences for Education

Undergraduate students (63%), graduate students (57%), industry (68%), and government agencies (56%) are high priority audiences for over half of societies’ education efforts. University faculty (37%) and high school students (33%) are a close second. Low priority audi-

ences include pre-K teachers and students, elementary school teachers and students, middle school teachers and students, and deans/department chairs. There were no significant differences by size, and there were no significant relationships between their rating of the importance of education and their target audiences.

Partnerships

Just over three quarters (77%) of societies said they are engaged in some kind of partnership with an outside agency or organization for their education work.

Professional and technical societies, academic organizations/institutions, government agencies, STEM organizations, and industry were the most commonly cited. At least one society also partnered with the following kinds of organizations:

• Diversity organizations
• International development organizations
• Engineering education organizations
• Accrediting bodies
• Private organizations
• Humanitarian organizations
• Consulting Engineers
• Museums
• Manufacturing Institute
• Mentoring organizations
• Girl-serving organizations
• Non-profit
• Media outlets
• Other standards developing organizations (SDOs)
• State affiliates
• Company that focuses on webinars and other distance-learning for environmental issues and engineering geology

General Program Information

Not all respondents answered all of the following questions. We indicate the number for each question.

Numbers served (N = 54)

41% of respondents did not know how many people were served by their education efforts. Of those who answered, the majority (43%) serves up to 10,000 people.

Numbers of volunteers participating in education work (N=54)

Just under 1/3 did not know how many volunteers participate in education work. Of those who answered, the 50% have up to 500 volunteers.

Annual budget for education (N=50)

34% (17) did not know the annual budget for education. Of those who answered, there was a range of numbers, indicated below:

Student Chapters (N=41)

Just under ½ of those responding to this question have less than 50 student chapters. Most of the remaining societies have over 50, upwards of 200. Seven societies have over 200 student chapters.

Program administration (N=51)

Societies’ central offices administer around 75% of programs for just under ½ of those who answered this question. Local chapters and/or divisions administer up to 50% of programs for just over half of the societies. Student chapters administer up to 25% of programs for just under half of societies. For 33% of societies, student chapters do not administer any of their education programs.

Level of investment in engineering education

The level of investment in education for societies has either increased (51%) or stayed the same (44%) for most societies. It has decreased for 4%, and 2% did not know.

A sub-analysis of the data by membership size revealed a significant difference with respect to recent change in level of investment in

engineering education. Small and medium societies’ level of investment in engineering education has stayed the same, whereas larger societies have increased their funding for education in the last two years.

Capacity for engineering education

Societies’ capacity to plan and implement education work is essentially split—half of the societies rated their overall capacity to plan and implement education work as either low or some. Forty-one percent rated their capacity high, and 9% rated it very high. There were no differences based on membership size.

Resources for Education

The large majority of support for education comes from societies’ annual operating budget (93%). Following that, resources include: membership-industry (70%), corporate sponsorship (72%), membership-faculty (67%), membership—college/academic department leaders (52%). Student members (46%), foundations (41%), internal research and/or evaluation results (30%), literature (24%) and NSF funding (24%) are also used. NAE and ASEE publications were resources for just 20 societies of the total sample. There were no differences based on membership size.

Connections with Other societies

Eighty-seven percent of respondent use connections with other engineering societies or organizations at least a little. Eleven percent (6 societies) said they use them “a lot.” Of those who use them at least a little, 43% believe that these connections are currently beneficial to some extent. Thirty-four percent believe they are useful to a good extent, and 23% to a great extent. All of those who do not currently use connections to other societies believe they could be beneficial to at least some extent. There were no differences based on membership size.

Barriers

Eighty-seven percent of societies face some kind of barrier in their education work to at least some extent. The majority of examples of barriers described by respondents fell into the following categories:

• communication issues (e.g., connecting members to educators; meetings)
• curriculum related issues (e.g., challenges with changing accepted curricula)
• incentive issues (e.g., getting faculty to change their practice)
• time, resource, funding issues (e.g., lack of resources to be able to scale local programs into a repeatable framework for national level use)

Other barriers were more specific to the discipline or particular mission, such as needing background checks for members to work in schools, lack of identity of a specialty, finding the right partners, improving student access, finding speakers, and the like.

Sub-analysis revealed that affinity societies are more apt to face barriers to their education efforts than discipline-focused societies. There were no significant differences based on membership size.

Leadership in Education

Eighty-five percent of societies consider themselves leaders in engineering education to at least some extent, and 17% of those to a great extent. For many of them, this refers to their particular specialty (e.g. conservation engineering). With respect to the relative importance of engineering education in their society, 78% of respondents believe that education is at least as important than other goals or activities pursued by their society, with 36% saying it is more or much more important. Only 23% (12 societies) said education is much less or less important.

NAE ENGINEERING SOCIETIES STUDY— INTERVIEW RESULTS

The following is a high-level summary of the 30 interviews of engineering societies conducted by Inverness Research. Most of the individuals we interviewed were Executive Directors or Presidents, although there were a few who held other leadership positions. It is worth noting that many of the interviewees began by saying that they did not see their society as an “engineering society,” per se, but one that influences and is influenced by engineering, and includes engineers in the membership.

Goals and Activities

Professional development for practicing engineers is a common focus for societies. Supporting and growing membership is also a goal. Across the sample, it is evident that societies engage in a range of activities depending on the needs of their membership, their available expertise and capacity, and funding. Further, societies may organize geographically and provide different kinds of opportunities for different, local audiences.

For the smaller or specialty societies, a common priority goal is to spread awareness of their particular branch of engineering. They also have some expertise in outreach to pre-college and undergraduates to educate engineers and others about their particular specialty.

Other goals and activities that interviewees mentioned include: engaging students in the community and service learning; ongoing professional development for practicing engineers; ensuring the next generation of engineers is prepared to practice engineering; expanding and protecting the reputation of engineering; professionalizing the engineering field; providing a forum for engineers to interact; promoting quality engineering education through ABET; facilitating career transitions; providing research-based design standards; working with regulators, legislators, and policy makers; and providing more application-based programs.

Concerns about the extent to which students graduating from engineering programs are ill-prepared to work in industry were shared by several societies, as exemplified by this statement:

The biggest complaint we get from our membership of 56,000 engineers is that engineers graduating from engineering programs are not well equipped on an application basis to participate in our industry. There are lots of reasons behind that. So we feel a duty to pick up that mantel and really focus on teaching application, how do you actually do what these folks are asked to do in our industry?

Also important for many societies is retaining and supporting diversity in engineering. One interviewee recalled her own experience as an engineering student to explain how important her society’s goals are for many:

Another interviewee described their goals as:

Activities of societies include: ensuring that degree programs are preparing students for engineering jobs/practice; creating awareness of the discipline (e.g., environmental engineering); providing mentor programs for high school and undergraduate students; providing education for certification programs; providing internship programs to support transitions to work; providing curriculum or support for curriculum at the undergraduate level; providing scholarships; creating and offering webinars and workshops for continuing education; offering accreditation through ABET; offering an early career faculty program; endorsing existing programs such as FIRST Robotics and Project Lead The Way; offering courses and workshops at annual meetings; and partnering students with practicing professionals.

One large society is addressing both the faculty and student experience: it has begun a program for early career faculty to provide them with resources (2-4 hour workshops, networking, mentoring) that will better prepare them for teaching at the university level, and a student program that helps them learn the “difficult to learn” subject matter.

A few societies are more focused on the technician—education programs aimed at high school and community college students who would not likely complete a 4-year engineering degree but aim to work in a job providing engineering or technical support to the engineer.

Overall, most activities of societies are designed to meet the needs of the membership. There are a couple of societies with longer-range views. Finally, there is a growing emphasis on providing programming virtually—through online courses, webinars, and the like.

Evaluation

Most of the societies attempt to collect feedback from participants in their most significant programs but most interviewees noted that they would like to do more to evaluate their work. They tend to collect numbers of participants and programs as indicators of success. As one interviewee said, their evaluation is:

Many also conduct member satisfaction surveys (i.e. people vote with their feet and wallets), or surveys that help them understand the professional development needs of their membership.

Some interviewees mentioned wishing that they did more to evaluate the longer-term impact of their work. One interviewee said, “Everyone struggles with that and it takes lots of resources to figure out what are the right metrics.”

DISSEMINATION

Most societies disseminate their work, education or otherwise, through annual meetings/conferences, journals, websites, member newsletters, and the like. One interviewee said that a formal venue for disseminating or sharing work related to undergraduate engineering education does not exist, but should.

CONNECTIONS WITH OTHER SOCIETIES

Most societies have at least some connections with other engineering societies, although the connections may or may not involve their work in education. Many invite other societies to their meetings, or attend others societies’ meetings.

However, few have made substantial connections that have resulted in collaborative projects. Mostly, the connections among societies are about sharing information.

Notable examples of connections among societies include: participation in a network of Executive Directors from other societies; consulting with others when developing curricula; having MOUs with several societies to work in three focal areas (membership reciprocity, curriculum development, and access to training and licensure courses); and joint professional development workshops or seminars.

One society organized a large coalition of organizations in April of 2016—83 people from 42 different organizations to work towards the mission of producing fifty-thousand underrepresented engineers by 2025. This is a striking example of a coordinated and purposeful effort. The representative interviewee said:

As another example, a society benefited from an influx of foundation funding over 15 years ago to advance the educational mission of the domain. The funding allowed the society to organize and offer fellowships, early career awards, professional development for department chairs, and leadership development.

The funding also supported two education summits where people across different disciplines exchanged best practices, curricula, lab activities, and courses. These summits provided the benefit of contributions from multiple perspectives for a multidisciplinary field. Unfortunately, the program is now defunct. The society has been able to hold one summit since then and is hoping to do another.

Almost all of the societies interviewed expressed a desire to be more connected to other societies.

GAPS IN ENGINEERING EDUCATION

Most of the individuals we interviewed believe that many of the gaps in engineering education offerings they see are being addressed in some way, somewhere in the landscape. However, there were a few areas they felt need more concerted effort, such as: faculty preparation to be instructors in engineering; addressing the lack of hands-on/application experiences for undergraduates; a re-focus on the design side of engineering; an emphasis on the business side of engineering (such as financial and general business acumen); support for how to integrate new teaching technologies into the engineering classroom and for preparing engineers for new technologies; getting industry more involved in ABET; turning more attention to the two-year programs and preparedness for the workforce; support for preparing doctoral students for teaching; and preparing students in general for team-work.

One interviewee said there is generally a need for a better understanding of the 1st year experience, “We need to change the thinking about designing for failure. We need to make student success a focus.” A few interviewees mentioned a gap in engineering education aimed at the K-12 level. One simply said: “We don’t have the resources for k-12.” One notable exception has a program that places high school students in labs in an effort to encourage them to pursue a technical field. They also provide resources, content, and pedagogical support to teachers at the middle school and high school levels who are teaching the subject but do not have a degree in it.

One society is very involved in filling the pre-college gap, particularly the competencies needed for high school students to be successful in college engineering. Along these lines, another interviewee voiced the need for earlier (than college) exposure to engineering and the diversity within it:

Similarly, another interviewee expressed the need to reach younger learners if the engineering pipeline is to stay filled:

Another interviewee noted that while they don’t do any work in the pre-college realm, it is “on the list” as an area they would like to get involved in, particularly teachers.

One Executive Director lamented the current state of undergraduate engineering education:

Finally, an Executive Director felt that the key missing pieces to the development of new engineers is the provision of mentors and real life education. She said:

FINAL NOTES

There are some interesting examples of societies that are either trying new initiatives to reach new audiences (such as a focus on early career faculty) and larger, bold initiatives that involve multiple societies and set out ambitious goals (such as the large coalition). Perhaps these societies could facilitate conversations about their experiences and stimulate thinking around innovative new programs.

It is important to keep in mind that collaborative efforts of any kind take time to both plan and get traction. They take patience and time to build trust and a collective vision. One Executive Director made this important point: