Page  66 Michigan Journal of Community Service Learning Fall 1996, pp. 66-71 An Assessment Model for Service-Learning: Comprehensive Case Studies of Impact on Faculty, Students, Community, and Institution Amy Driscoll Barbara Holland Sherril Gelmon Seanna Kerrigan Portland State University A comprehensive case study model of assessment developed at Portland State University responds to the need to measure the impact of service-learning on four constituencies (student, faculty, community, and institution). The case studies blend quantitative and qualitative measures in order to determine the most effective and practical tools to measure service-learning impact and to provide feedback for continuous improvement of practice. Insights from the design process and preliminary results have potential value for institutions with similar agendas for service-learning and community partnerships. In this time of dramatic transformations in higher education, one very visible change on many campuses is the expansion of partnerships between colleges and universities and community agencies, organizations, and other constituencies. Those partnerships take many forms from campus to campus, but a typical connection is service-learning - the integration of community service with the academic content of course work. Service-learning responds to the call for higher education to improve the quality and productivity of instruction and to "become more engaged in addressing the nation's many problems" (Edgerton, 1995). As more and more educational institutions heed the call, the need to evaluate and interpret both the outcomes and the impacts of service-learning has grown. At Portland State University (PSU) servicelearning has long been present in the curriculum, but in fragmented forms with scattered visibility. When we revised our general education curriculum in 1993, our commitment to broad integration of service-learning became focused and supported, and clearly connected to our university mission. The first year (1994) of deliberate campus-wide focused service-learning was marked with high levels of enthusiasm and faculty claims of exciting impact. Aware that our enthusiasm and claims must give way to hard data and demonstrated outcomes, faculty and administrators held a series of meetings to develop an assessment plan uniquely targeted to service-learning courses. We began by searching for other models of assessment for ser vice-learning and found that program evaluations dominated the literature (Shumer, 1991). We soon became aware that we were part of a larger national community seeking to ameliorate the "scarcity of replicable qualitative and quantitative research on the effects of service-learning on student learning and development, the communities in which they serve, or on the educational institutions" (Giles, Honnet, & Migliore, 1991, p.2). This paper describes our efforts to study and document the impact of service-learning and to develop an assessment model that contributes to service-learning practice. We were also committed to establishing a "culture of evidence" at Portland State University (Ramaley, 1996) to document our reform efforts. Literature Review We began our conceptualization process by reviewing the theoretical and development literature on service-learning. Like PSU faculty, the proponents of service-learning in journals and other publications have been enthusiastic about its potential. Claims for its success include enhanced relevance of course content, changes in student attitudes, support for community projects and needs, and increased volunteerism (Erlich, 1995; Giles & Eyler, 1994; Harkavy, 1992). Those same supporters also acknowledged the gaps in our knowledge about the effects of service-learning and the difficulty in measuring those effects. As Eyler and Giles (1994) point out, the outcomes of service-learning have not been clearly conceptual 66

Page  67 An Assessment Model for Service-Learning ized, nor is there agreement about the intent of service-learning. Such dissention and lack of clarity have contributed to the lack of significant progress in the development of assessment measures. We decided to address the lack of clarity of outcomes as we began our assessment plan. Another challenge to the assessment of servicelearning is that the benefits are spread among different constituencies: students, faculty, the community, and the institution. Colleges and universities have typically struggled with the assessment of student learning and institutional impact. Currently there are and have been multiple projects focused on student outcomes (Bringle & Kremer, 1994; Cohen & Kinsey, 1994; Giles & Eyler, 1994; Hesser, 1995; Markus, Howard, & King, 1993; Wechsler & Fogel, 1995), but the profession has concentrated little effort toward assessing faculty impact, and has only begun thinking about the process of assessing community impact. The issue of multiple constituencies is a major challenge to the task of assessing service-learning if institutions are to effectively evaluate the full ramifications of a commitment to integration of service-learning in the curriculum. This is especially important to the partnership concept that PSU embraces as the essence of its urban mission. Thus, the commitment to assessing the experiences and impact for multiple constituencies was a guiding principle of this study. Context for Development Before describing the conceptual development of our assessment plan, it is important to acknowledge the context in which we worked. Portland State University, an urban institution, had recently reformed the undergraduate curriculum in an effort to fulfill our mission, to better accommodate our non-traditional student population, and to attend to research on effective teaching and learning. Service-learning was integrated throughout the new curriculum in freshmen experiences, servicelearning courses, and in graduation requirements. This comprehensive approach to the integration of community service influenced the design of an assessment model. The newness of our servicelearning integration and its comprehensive impact across campus called for an exploratory and formative assessment approach. This meant that our model would have to ensure the collection of assessment data that could provide feedback for continuous improvement and sufficient breadth to serve the diverse forms of service-learning in our curriculum. The design would also have to honor PSU's commitment to mutually beneficial partnerships with the community, and therefore, provide data of value to our community partners. Conceptual Development of an Assessment Model In response to the paucity of assessment approaches in the literature and with attention to our campus-wide service-learning approach, we decided to test the use of comprehensive case studies as a structural approach to the assessment and description of our service-learning courses. An additional objective was to develop assessment strategies that would be adaptable to other community service activities throughout our general education curriculum. The comprehensive case studies were designed not only to assess and describe our service-learning courses, but to pilot multiple forms of assessment instruments. We needed to explore many mechanisms for measuring the impact of our courses in order to determine which approaches and tools would provide the best and most informative data. We were reminded by Giles and others (1991) that there was a "myriad of potential effects to be derived from combining service and learning in the educational enterprise" and by Hesser (1995) that the "variables to be controlled are almost infinite," (Hesser, 1995) so our intent was to be as comprehensive as possible for the draft of our case study model. The first step in designing the case study model, that of defining purposes, attended to our commitment to a comprehensive approach directed by well-defined goals. Our purposes were: 1. To describe and assess the impact of service-learning courses on multiple constituencies. 2. To develop and pilot an exploratory case study model that integrates continuous improvement with educational assessment theory and practice, that measures a maximum number of impact variables for multiple constituencies, and that tests a broad range of potential measurement tools. 3. To monitor both data collection and data analysis to determine the most effective assessment approaches and tools to measure service-learning in order to develop a practical and valid assessment model for future use. 4. To consider the lessons learned from the comprehensive case studies in order to develop assessment models for other community service activities on campus. 67

Page  68 Driscoll, Holland, Gelmon, and Kerrigan As we proceeded from these purposes to the articulation of hypotheses for our study, we encountered the need to define outcomes of our community-based learning courses. Just as the literature described, our courses did not have clear or specific outcomes regarding effects of servicelearning on participants. Much of our development work became the task of defining desired impact. If we claimed that service-learning courses had an impact on students or community or other constituencies, what did the impact look like? How could we establish that there was an impact? Before designing measures, a comprehensive definition of impact was needed for each of the constituencies. A set of potential impact variables for each constituency was developed in a participatory fashion that considered each group's perspective. We conducted a series of reviews of the impact variables with members of the four constituencies (students, faculty, community, institution) and made recommended revisions until there was agreement on their inclusiveness. An example of an impact variable for students is "awareness of community." To measure the impact variables, we developed indicators and drafted appropriate tools to capture the existence of an indicator or measure changes in an indicator. Building on the previous example for "awareness of community," indicators were determined as "knowledge of community history, strengths, problems, and issues," as well as "definition of community." Our design suggested that those indicators could be measured by means of interviews, journal analysis, focus groups, and surveys. Figures 1, 2, 3, and 4 display the variables, indicators, and appropriate measurements for each of the four constituencies. Once the range of impact variables for all four constituencies was determined, the case studies were designed to make a broad assessment of a maximum number of impact variables for all constituencies. Indicators and appropriate measurement of each impact variable directed the case study design to blend quantitative and qualitative approaches. Further, it was anticipated that the case studies could demonstrate the potential for linking teaching, research, and service. With the impact variables providing measurement direction, the resulting hypotheses of our comprehensive case study research were: 1. Participation in service-learning courses will have an impact on students. 2. Participation in service-learning courses will have an impact on faculty. 3 Participation in service-learning cour ses will have an impact on community. FIGURE 1 Student Variables, Indicators and Measurements Variables Indicators Measurements Awareness of cor- Knowledge of Interview, journal munity community history, analysis, focus strengths, problems, groups, survey definition Involvement with Quantity/quality of Interview, surveys, community interactions, attitude journal analysis, toward involvement focus groups Commitment to ser- Plans for future Surveys, focus groups vice service Career choices Influence of commu- Surveys, interview, nity placement job focus groups opportunities Self awareness Changes in awareness Surveys, interview of strengths, limits, direction, role, goals Personal development Participation in Interview, journal additional courses, analysis, focus extracurricular groups, survey activities Academic Role of community, Interview, survey, achievement experience in grades, focus groups understanding and applying content Sensitivity to Attitude, understand- Journal analysis, diversity ing of diversity, com- reflections, survey, fort and confidence interviews Autonomy/ Learner role Interview, class independence observation Sense of ownership Learner role Class observation, interview Communication Class interactions, Class observation, community community interactions observation FIGURE 2 Faculty variables, indicators and measurements Variables Indicators Measurements Involvement with Quantity/quality of Logs, surveys, intercommunity interactions/contacts view, journals Awareness of com- Definition of commu- Interview, written munity nity, knowledge of comments, journals history, strengths, problems Level of volunteerism Valuing personal Vita, interview, volunteerism, actual survey volunteerism Professional develop- Influence of commu- Vita, interview, jourment nity-based learning in nals conference/seminar attendance Scholarship Influence of Vita, artifacts community-based learning in articles., presentations, etc. Teaching methods Influence of commu- Class observation, nity-based learning in journals, surveys, class format, organi- teaching and learning zation, interactions continuum Faculty/student inter- Content, variety, fre- Class observation, action quency, direction teaching and learning continuum Philosophy of teach- Faculty/student roles, Interview, class ing/learning outcomes, pedagogy, observation, syllabus curriculum analysis, journals, teaching and learning continuum Role in community- Self perceptions of Log, interview, surbased teaching role vey, journals 68

Page  69 An Assessment Model for Service-Learning FIGURE 3 Community Variables, Indicators and Measurements Variables Indicators Measurements Nature of partnership Present and future Interview, syllabus activities Involvement with Contribution to com- Interview, survey, community munity, achievement focus groups of goals of the agency and course Perceived capacity to Number of clients, Interview, focus serve clients services, value added groups, survey Economic benefits Cost of services pro- Interview, survey vided by faculty/students, funding opportunities Social benefits New connections, Interview networks New insights about Changes in goals, Interview operations/activities activities, operations Awareness of PSU Changes in image, Interview, focus confidence, knowl- groups, CAE log edge of programs Establishment of Changes in levels, Interview, focus ongoing relationships nature, breadth of groups contacts, future partnerships Identification of Actual hirings Interview, survey prospective employees Satisfaction with PSU Level of communica- Interview, Survey interactions tion/interaction with students/faculty FIGURE 4 Institutional Variables, Indicators and Measurements Variables Indicators Measurements Role in community Numbers of types of CAE log, IRP reports, requests for assis- IASC interview tance from community, changes in enrollment and transfer patterns Orientation to teach- Number of faculty CAE log, survey ing and learning involved in commu- (NG), content analynity-based learning, sis of grants, dissertafocus/content of pro- tion, class observafessional develop- tions ment activities, focus/content of dissertations, enrollment and transfer patterns Resource acquisition Contribution levels, CAE log, Currently, site visits by other IRP reports campuses, grant proposals and awards related to service, changes in enrollment/transfer patterns Image in community Number of media CAE log, Currently, (local, state, national, reports, number of PR reports int'l) site visits by other campuses, number of publications, conference presentations, contributions 4. Participation in service-learning courses will have an impact on the institution. 5. Service-learning courses will transform the teaching and learning para digm of the university from a traditional instruction model to an interactive learning model. Our hypotheses were intentionally broad to support our comprehensive approach and the wide range of impact variables. The last hypothesis resulted from earlier exploratory observations in classrooms in which service-learning was integrated with course work. Study Methodology As indicated earlier, the broad range of variables, indicators and appropriate measurement tools and approaches demanded a blend of quantitative and qualitative approaches. Some of the approaches were to be used in a pre-post format, others were to be used for ongoing assessment throughout a course, and others were to be used for a one-time measurement. An overview of the indicators and appropriate measurement revealed three major categories of mechanisms or data collection procedures. The categories are illustrated in Figure 5. They include: in-person assessment; independent reflection measures; and review of existing documentation. The in-person assessment is composed of: interviews of students, faculty, and community representatives; focus groups to be conducted with students and community groups; and bi-weekly classroom observations of service-learning courses. The independent reflection measures are meant to capture journalized reflections of faculty and students, and pre-post surveys of students, faculty, and community representatives. The review of existing documentation will include analysis of syllabi, review of faculty vitae, analysis of institutional reports (admissions data, alumnae surveys, etc.), and activity/contact logs. Pilot Study During Winter quarter 1996 the comprehensive case study model was piloted in four service-learning courses at Portland State University. These courses were selected to ensure diversity of disciplines, faculty with previous experience in servicelearning, and variation in the kind of service. The courses being studied include a graphics design course in the School of Fine and Performing Arts, a public health course on programs for children and families in the School of Urban and Public Affairs, a technical writing course in the College of Liberal Arts and Sciences, and an introductory education course in the School of Education. Graduate research assistants assisted the project team with classroom and community observations, interviews of faculty, students, and community members, and 69

Page  70 Driscoll, Holland, Gelmon, and Kerrigan FIGURE 5 Mechanisms to measure impact Mechanisms to Measure Impact In-Person Focus Classroom Interviews Groups Observations Independent Reflection Journals Surveys Documentation Syllabus Review Existing Activity/ Analysis of Vitae Reports Contact Logs focus groups with students and community members. Students, faculty, and the research assistants maintained reflective journals during the entire quarter. In the process of data collection, it was apparent that most of the assessment strategies were documenting impact for the four constituencies as well as providing formative assessment information, that is, feedback for continuous improvement of service-learning courses. Our classroom observations began to reveal a non-traditional paradigm of teaching in some of the classes. Faculty and student reflections in interviews and focus groups indicated affirmation of the value of service-learning experiences. Those general trends were immediately obvious in the process of collecting data, but there is an enormity of data to be analyzed before drawing final conclusions. At the time of this writing, data analysis is only partially completed. Preliminary findings from student interview data show support for all of the predicted student impact variables, especially awareness of and involvement with community, self awareness, personal development, academic achievement, sensitivity to diversity, and independence as a learner. Community interviews also support the predicted variables, especially perceived capacity to serve clients, economic benefits, social benefits, new insights about operations, and awareness of PSU. There is strong support for the variable, satisfaction with PSU interactions, and additional related community impact variables emerged from the data. Analysis of faculty vita indicate that community service teaching experiences have begun to influence scholarship in the form of research, conference presentations, and publications. It would be premature to draw any conclusions at this time, but we are encouraged by results from the partial analysis. We predict that some of our assessment strategies will provide useful and significant information, and that some of our strategies will not. Our immediate intent is to use the results of our data analysis to refine the case study model for use with more courses during the 1996-97 academic year. Our long-term goal is to produce a practical and valid package of assessment strategies that can be embedded in all of our service-learning courses and adapted for other community service activities. Summary A comprehensive case study model of assessment developed at Portland State University responds to the concerns and questions about the impact of service-learning, accommodates the range of constituencies influenced by servicelearning, and seeks to address the paucity of approaches for measuring service-learning outcomes. In addition, for the PSU community, the case study approach has the potential to support and inform institutional efforts to monitor the role of service-learning in the fulfillment of the urban mission. The model was developed by a team of faculty and administrators, with input from students and community representatives. The case study design is a blend of quantitative and qualitative measures to assess the impact of a servicelearning course on faculty, students, community, and institution. Although analysis of the case studies is in progress, insights from the design process and from preliminary results have potential value for institutions with similar agendas for servicelearning and community partnerships. Note Funding for this project was provided by the Corporation for National Service and the Center for Academic Excellence at Portland State University. The authors wish to acknowledge the assistance of MJ Longley, Carolyn Martin, and Amy Spring. References Bringle, R. G., & Kremer, J. F. (1993). An evaluation of an intergenerational service-learning projects for 70

Page  71 An Assessment Model for Service-Learning undergraduates. Educational Gerontologist, 19, 407 -416. Cohen, J., & Kinsey, D. (1994). "Doing good" and scholarship: A service-learning study. Journalism Educator, 4-14. Edgerton, R. (1995). Crossing boundaries: Pathways to productive learning and community renewal. AAHE Bulletin, 48 (1), 7-10. Erlich, T. (1995). Taking service seriously. AAHE Bulletin, 47 (7), 8-10. Eyler, J., & Giles, D. E. (1994). Research and evaluation in community service: The higher education agenda. Racine: Proceedings from the Wingspread Service Learning Conference. Giles, D. E., & Eyler, J. (1994). The impact of a college community service laboratory on students' personal, social, and cognitive outcomes. Journal of Adolescence, 17, 327-339. Giles, D. E., & Eyler, J. (1994). The theoretical roots of service-learning in John Dewey: Toward a theory of service-learning. Michigan Journal of Community Service Learning, 1 (1), 77-85. Giles, D. E., Honnet, E., & Migliore, S. (1991). Setting the agenda for effective research in combining service and learning in the 1990's. Raleigh: National Society of Experiential Education. Harkavy, I. (1992). The mission of the Center for Community Partnerships. Almanac, 4-5. Hesser, G. (1995). Faculty assessment of student learning: Outcomes attributed to service-learning and evidence of changes in faculty attitudes about experiential education. Michigan Journal of Community Service Learning, 2, 33-42. Markus, G. B., Howard, J. P. F, & King, D. C. (1993). Integrating community service and classroom instruc tion enhances learning: Results from an experiment. Educational Evaluation and Policy Analysis, 15, 410 -419. Ramaley, J. (1996). Personal communication. Shumer, R. (1991). Setting a research agenda for service learning. Unpublished article. Weschler, A., & Fogel, J. The outcomes of a servicelearning program. National Quarterly Society for Experiential Education 20 (4), 6-7, 25-26. Authors AMY DRISCOLL is Director of Community/ University Partnerships in the Center for Academic Excellence at Portland State University. She is a Professor of Education who specializes in teacher education and instructional strategies. BARBARA HOLLAND is Executive Director of Community Relations at Portland State University. She specializes in the study of urban universities and institutional cultures. SHERRIL GELMON is Associate Professor of Public Health Education in the School of Urban and Public Affairs at Portland State University. She specializes in the study of health professions education, assessment, and improvement strategies. SEANNA KERRIGAN is Capstone Coordinator at Portland State University. Her specialties include student leadership and community partnerships. '71