Assessing 21st Century Skills: Summary of a Workshop (2011)
Chapter: Appendix A: Agenda and Participants for the January Workshop
Appendix A
Agenda and Participants for
the January Workshop
Assessment of 21st Century Skills
Workshop
January 12–13, 2011
University of California, Irvine
Beckman Conference Center
Huntington Room
AGENDA
Wednesday, January 12
| 9:30-9:40 |
Opening Remarks Welcome |
|
Stuart Elliott (Director, Board on Testing and Assessment) |
|
|
Bruce Fuchs (National Institutes of Health, cosponsor of the project) |
|
| Overview of Workshop | |
| Joan Herman (CRESST, Chair of Workshop Steering Committee) | |
| 9:40-12:15 | Session 1: Background Information |
| Moderators: Joan Herman and Pat Kyllonen (ETS and | |
| Workshop Steering Committee) | |
| (9:40-10:00) Why Are 21st Century Skills Important? | |
| Richard Murnane (Harvard University) |
| This presentation will address the following issues: | |
|
• What is unique in the 21st century that makes these skills especially valuable in the labor market and/ or in other life domains (learning, family life, civic engagement)? |
|
|
• How does the growing use of computers and technology affect the labor market and the demand for 21st century skills? |
|
|
• What does more recent research suggest about the skills needed to be successful in the 21st century? |
|
| (10:00-10:15) How Will You Know If Your Students Are 21st Century Ready? | |
| Deborah Boisvert (Boston Area Advanced Technical Education) | |
| The presenter will respond to the opening presentation, reflecting her work with employers to define, teach, and assess 21st century skills of computer technicians. | |
| (10:15-10:45) The Teaching and Learning of 21st Century Skills | |
| Eric Anderman (Ohio State University) | |
| This presentation will address the following issues: | |
|
• What is known about the extent to which the three skill clusters and/or the skills within them can be taught and learned? |
|
|
• To what extent are learning, teaching, and assessment of the three skill clusters domain specific or domain general? |
|
| (10:45-11:00) Discussion | |
| Moderators will lead a question-and-answer session with the presenters and audience members. | |
| 11:00-11:15 | Break |
| (11:15-11:35) Approaches to Developing Assessments of 21st Century Skills | |
| Deirdre Knapp (HumRRO, Workshop Steering Committee) | |
| This presentation will address the following issues: | |
|
• What are the different approaches to assessment of these skills and what steps are involved in carrying out these approaches? |
|
|
• What processes are used for identifying the skills to be measured, operationalizing the skills through the test blueprint, and creating assessment tasks and scoring procedures? |
|
|
• How should the intended uses of the assessment results guide the test development process? |
|
|
• What steps should be taken to ensure that the assessments are reliable and valid? |
|
| (11:35-11:55) Unique Challenges and Opportunities in the Assessment of 21st Century Skills | |
| Steven Wise (Northwest Evaluation Association, Workshop Steering Committee) | |
| This presentation will address the following issues: | |
|
• What are the unique challenges and opportunities for defining and measuring these constructs, when compared to more traditional academic skills and knowledge? |
|
|
• How might the results of these assessments be used? Should they be used for high-stakes purposes? |
|
|
• What issues may arise in relation to the validity, reliability, and fairness of assessments of these skills? |
|
| (11:55-12:15) Questions and Discussion | |
| Moderators will lead a question-and-answer session with the presenters and audience members. | |
| 12:15-1:15 | Lunch in the Beckman Center Cafeteria |
| Continued discussion of ideas presented during the morning sessions |
| 1:15-3:45 | Session 2: Assessing Cognitive Skills |
| Moderators: Greg Duncan (University of California, Irvine, Workshop Steering Committee) and Paul Sackett (University of Minnesota, Workshop Steering Committee) | |
| (1:15-1:45) Defining and Measuring Cognitive Skills | |
| Nathan Kuncel (University of Minnesota) | |
| This presentation will address the following issues: | |
|
• What are 21st century cognitive skills? To what extent do they differ from each other and from general cognitive ability? What are the conceptual differences that are proposed to exist between these constructs? |
|
|
• What are the existing measures of these constructs, and to what extent do these existing measures match their conceptual specifications? |
|
|
• What are the relationships between the existing measures of these constructs? |
|
| (1:45-2:00) Questions and Discussion | |
| Moderators will lead a question-and-answer session with the presenters and audience members. | |
| (2:00-3:15) Panel Discussion: Examples of Assessments of Cognitive Skills | |
| For each example, the panelists will address the following issues: | |
|
• What skill or skills are measured? Why are these skills important? |
|
|
• What is the purpose of the assessment? |
|
|
• What strategies were used to develop the assessment and why were these selected? |
|
|
• What assessment methods are used and why were these selected? |
|
|
• How is the assessment scored? What data are available on the technical quality of the assessment, including validity, reliability, fairness, and comparability across administrations? |
|
|
• What data are available on the cost and practical feasibility of the assessment? |
| (2:00-2:20) Interactive Problem Solving for PISA 2012 | |
| Joachim Funke (University of Heidelberg), by video conference | |
| (2:20-2:40) Operation ARIES!: Learning Critical Thinking about Science with Intelligent Conversational Agents in a Game Environment | |
| Art Graesser (University of Memphis) and Heather Butler (Claremont McKenna College) | |
| (2:40-3:00) Intrusive and Unobtrusive Assessment of Entrepreneurial and Technical Skills through Simulation and Gaming | |
| John Behrens (Cisco Systems) | |
| (3:00-3:20) Assessment of Critical Thinking and Problem Solving on the Multistate Bar Exam | |
| Susan Case (National Conference of Bar Examiners) | |
| 3:20-3:30 | Break |
| (3:30-4:00) Moderated Discussion | |
| Moderators will explore the following issues with panelists and audience members: | |
|
• What are the implications of the presentations (and examples) for the design of 21st century assessments for K-12 and higher education? |
|
|
• Do common themes or approaches emerge from the examples? How might the noneducation examples generalize to education? |
|
|
• How might 21st century assessments be incorporated into current research efforts, such as the development of assessment systems by the two-state consortia? What functions can/should the assessments serve? How might the results be used? |
|
|
• What equity and accessibility challenges do these assessments raise? |
|
|
• What barriers might slow development and/or use of assessments of 21st century skills? How might they be overcome? |
| 4:00-5:00 | Synthesis of Key Ideas |
| Moderator: Joan Herman | |
| (4:00-4:20) Discussion: Eva Baker (CRESST) | |
| (4:20-4:40) Discussion: Richard Murnane (Harvard University) | |
| Discussants will reflect on the day’s discussions and offer their synthesis of the ideas presented. Audience members will be invited to ask questions and share their ideas as well. | |
| 5:00 | Conclude Formal Agenda for Day 1 |
| Joan Herman | |
| 5:30 | Working Group Dinner (at Beckman Center) |
| Plan for the second day of the workshop | |
| Thursday, January 13 | |
| 9:00-11:45 Session 3: Assessing Interpersonal Skills | |
| Moderators: Deirdre Knapp and Juan Sanchez (Florida International University, Workshop Steering Committee) | |
| (9:00-9:30) Defining and Measuring Interpersonal Skills | |
| Steve Fiore (University of Central Florida) | |
| This presentation will address the following issues: | |
|
• What are 21st century interpersonal skills and why are they important? |
|
|
• How are these skills typically assessed? What are the challenges in assessing them? |
|
|
• What types of assessments are currently available to evaluate these skills? |
|
| (9:30-9:40) Questions and Discussion | |
| Moderators will lead a question-and-answer session with the presenters and audience members. | |
| (9:40-11:00) Panel Discussion: Examples of Assessments of Interpersonal Skills | |
| For each example, the panelists will address the following issues: | |
|
• What skill or skills are measured? Why are these skills important? |
|
|
• What is the purpose of the assessment? |
|
|
• What strategies were used to develop the assessment and why were they selected? |
|
|
• What assessment methods are used and why were these selected? |
|
|
• How is the assessment scored? What data are available on the technical quality of the assessment, including validity, reliability, fairness, and comparability across administrations? |
|
|
• What data are available on the cost and practical feasibility of the assessment? |
|
| (9:40-10:00) Online Portfolio Assessments of the 4 Cs | |
| Bob Lenz (Envision Schools) | |
| (10:00-10:20) 21st Century Skills in STEM Workforce | |
| Training Assessments | |
| Louise Yarnall (SRI) | |
| (10:20-10:40) Using Situational Judgment Tests for Medical School Admissions | |
| Filip Lievens (Ghent University, Belgium), by video conference | |
| (10:40-11:00) Assessment Centers 2011: Fifty Years of Best Practice and Today’s Innovations | |
| Lynn Gracin Collins (SH&A/Fenestra) | |
| 11:00-11:10 | Break |
| (11:10-11:45) Moderated Discussion | |
| Moderators will explore the following issues with panelists and audience members: | |
|
• What are the implications of the presentations (and examples) for the design of 21st century assessments for K-12 and higher education? |
|
|
• Do common themes or approaches emerge from the examples? How might the noneducation examples generalize to education? |
|
|
• How might 21st century assessments be incorporated into current research efforts, such as the development of assessment systems by the two-state consortia)? What functions can/should the assessments serve? How might the results be used? |
|
|
• What equity and accessibility challenges do these assessments raise? |
|
|
• What barriers might slow development and/or use of assessments of 21st century skills? How might they be overcome? |
|
| 11:45-12:45 | Working Lunch in the Beckman Center Cafeteria |
| Continued discussion of ideas presented during the morning sessions | |
| 12:45-3:30 | Session 4: Assessing Intrapersonal Skills |
| Moderators: Pat Kyllonen and Steven Wise | |
| (12:45-1:15) Assessment of Self-Regulation and Related Constructs: Prospects and Challenges | |
| Rick Hoyle (Duke University) | |
| This presentation will address the following issues: | |
|
• What are 21st century intrapersonal skills and why are they important? |
|
|
• How are these skills typically assessed? What are the challenges in assessing them? |
|
|
• What types of assessments are currently available to evaluate these skills? |
|
| (1:15-1:30) Discussion |
| (1:30-3:30) Panel Discussion: Examples of Assessments of Intrapersonal Skills | |
| For each example, the panelists will address the following issues: | |
|
• What skill or skills are measured? Why are these skills important? |
|
|
• What is the purpose of the assessment? |
|
|
• What strategies were used to develop the assessment and why were these selected? |
|
|
• What assessment methods are used and why were these selected? |
|
|
• How is the assessment scored? What data are available on the technical quality of the assessment, including validity, reliability, fairness, and comparability across administrations? |
|
|
• What data are available on the cost and practical feasibility of the assessment? |
|
| (1:30-1:50) Integrity Testing for Employee Selection | |
| Paul Sackett (University of Minnesota, Workshop Steering Committee) | |
| (1:50-2:10) Targeting Context-Specific Self-Regulated Learning (SRL) Processes: An Overview and Illustration of SRL Microanalysis | |
| Tim Cleary (University of Wisconsin–Milwaukee) | |
| (2:10-2:30) Assessing Behavioral Problems That Predict Poor Educational and Life Outcomes | |
| Candice Odgers (University of California, Irvine) | |
| (2:30-2:50) Out of the Maze? In Search of Skills for Emotional Intelligence | |
| Gerald Matthews (University of Cincinnati) | |
| 2:50-3:00 | Break |
| (3:00-3:30) Moderated Discussion | |
| Moderators will explore the following issues with panelists and audience members: | |
|
• What are the implications of the presentations (and examples) for the design of 21st century assessments for K-12 and higher education? |
|
|
• Do common themes or approaches emerge from the examples? How might the noneducation examples generalize to education? |
|
|
• How might 21st century assessments be incorporated into current research efforts, such as the development of assessment systems by the two-state consortia? What functions can/should the assessments serve? How might the results be used? |
|
|
• What equity and accessibility challenges do these assessments raise? |
|
|
• What barriers might slow development and/or use of assessments of 21st century skills? How might they be overcome? |
|
| 3:30-4:00 | Session 5: Reflection and Synthesis |
| Moderated discussion led by workshop steering committee | |
| 4:00 | Closing Remarks, Adjourn |
| Joan Herman |
PARTICIPANTS
Eric Anderman, Ohio State University
John Behrens, Cisco Systems
Lola Berber-Jimenez, California Polytechnic Science Project
Paul Bloomberg, Transformative Inquiry Design for Effective Schools and Systems
Deborah Boisvert, University of Massachusetts, Boston
Liane Brouillette, University of California, Irvine
Christopher Brown, Pearson Foundation
Peggy Burke, Transformative Inquiry Design for Effective Schools and Systems
Heather Butler, Claremont McKenna College
Susan Case, National Conference of Bar Examiners
Tim Cleary, University of Wisconsin–Milwaukee
Sara Clough, ACT, Inc.
Lynn Gracin Collins, Sandra Hartog & Associates/Fenestra, Inc.
Emily Dalton Smith, Gates Foundation
Tran Dang, University of California, Irvine
Greg J. Duncan, University of California, Irvine
Steve Fiore, University of Central Florida
Dennis Frezzo, Cisco Systems
Bruce Fuchs, National Institutes of Health
Joachim Funke, University of Heidelberg
Tracy Gardner, General Educational Development Testing Service
Nicole Gerardi, University of California, Los Angeles
Art Graesser, University of Memphis
Valerie Greenhill, e-luminate
Erika Hall, Pearson Foundation
Joan Herman, National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles (committee chair)
Rick Hoyle, Duke University
John Jackson, National Science Foundation
Stuart Kahl, Measured Progress
Deirdre J. Knapp, HumRRO (committee member)
Art Kramer, University of Illinois
Brandi Kujala, Educational Policy Improvement Center
Nathan Kuncel, University of Minnesota
Patrick Kyllonen, Educational Testing Service (committee member)
Robert Lenz, Envision Schools
Filip Lievens, University of Ghent
María Alicia López Freeman, California Science Project
Tim Magner, Partnership for 21st Century Skills
Michael Martinez, University of California, Irvine
Gerald Matthews, University of Cincinnati
Mick McManus, University of Queensland
Beth Miller, Nellie Mae Education Foundation
Julia Rankin Morandi, Los Angeles Education Partnership
Richard Murnane, Harvard University (committee member)
Suzanne Nakashima, California Science Project
Paul Nichols, National Center for the Improvement of Educational Assessment
Candice Odgers, University of California, Irvine
Cornelia Orr, National Assessment Governing Board
Pamela Paek, National Center for the Improvement of Educational Assessment
Jason Ravitz, Buck Institute for Education
Michael Russell, University of California, Irvine
Paul Sackett, University of Minnesota (committee member)
Andrea Saenz, U.S. Department of Education
Juan I. Sanchez, Florida International University (committee member)
Mary Seburn, Educational Policy Improvement Center
Brian Stecher, RAND
Christine Tell, Achieve
Cathy Tran, University of California, Irvine
Bernie Trilling, Oracle Education Foundation
Jerry Valadez, California State University, Fresno
Marjorie Wine, General Educational Development Testing Service
Steven Wise, Northwest Evaluation Association (committee member)
Louise Yarnall, SRI
Raymond Yeagley, Northwest Evaluation Association
Linda Zimmerman, Pearson
Doron Zinger, Olive Crest Academy
