20 How Mature Teaching and Learning Centers Evaluate their Services

Types of Services

Services offered by the thirty-three TLCs were similar to those reported in Chism and Szabo’s (1997) and my (Hines, 2009) studies. The majority provided seminars, workshops, brown bag sessions, conferences, and orientations, in addition to a variety of consultation services, online resources, and grant programs. Unlike the centers in the previous studies, over half of the TLCs surveyed sponsored faculty learning communities (FLCs). Approximately one-third designed customized programs such as faculty inquiry groups, academic fellowship programs, faculty writing programs, course revision programs, early and midcareer teaching programs, teaching enrichment series, interactive theater programs, and department-specific support programs.

Staff Conducting Program Evaluation

Most TLCs dispersed program evaluation duties among all staff members. The three TLCs with full-time staff assigned exclusively to evaluation indicated these were essential and fairly recent additions to their program. One director supported this recent appointment by saying, "If you’re trying to figure out what’s working, what’s not working, and where to invest time and money, you need one." Another indicated that the need for evaluation staff was related to projects funded by government grants. All three reported significantly higher levels of program evaluation activity than the other TLCs in the study.

Individuals outside the TLC were also involved in program evaluation. Approximately 20 percent of TLCs hired outside consultants to perform periodic program reviews. Several tapped staff from their university’s office of assessment. A small number recruited their advisory committees to review their physical and online teaching and learning resources.

Prevalence of Evaluation

All TLCs engaged in some routine evaluation, although disparities appeared in the types of services being evaluated (Table 20.1). A high percentage of TLCs evaluated, at least occasionally, user satisfaction and impact on teaching resulting from their events and activities, consultation services, and mentoring programs. Almost half of the TLCs offering grant programs, consultation services, or large resource events made some effort to measure the impact these services had on student learning. There was little interest in gathering satisfaction or impact data relating to publications and resources.

Evaluation Methods

Evaluation methods most frequently consisted of satisfaction surveys, participation data, formal self-reported changes in teaching, grant reports, and formal teacher-reported changes in student learning, respectively (Figure 20.1). A variety of other methods were also implemented, although significantly less frequently.

USAGE

Participation and usage were commonly tracked. Attendance data included the department or school in which the participant taught. Usage data for online resources were frequently tracked using Google Analytics.

SATISFACTION

Satisfaction was measured through the routine use of paper or electronic surveys administered after an event or service. Almost all participating TLCs reported the use of postevent meetings to debrief on satisfaction data and plan program adjustments. Three TLCs administered an annual survey, one administered a survey every four years, and two did an end-of-term survey for consultations only. Anecdotal satisfaction data were rarely used. All respondents indicated a moderate to high level of satisfaction with their services. One TLC director was able to use an activity report listing annual services and events offered, combined with satisfaction, participation, and usage data, to persuade state legislators to avoid funding cuts that would have had a negative impact on the center.

IMPACT OF SERVICES ON TEACHING

Self-reports of changes in teaching were commonly gathered by embedding specific questions in satisfaction surveys. Typical questions were, "What will do you differently as a result of this program?" and, "What did you learn? Did you apply it?" Half of the twenty-four TLCs offering grant programs required recipients to report pedagogical changes that occurred as a result of the funded project. Many asked grant recipients to share their new instructional insights in a seminar, workshop, or poster session. Experimental data demonstrating instructional gains were collected if the design of the funded project produced such data. Evidence of pedagogical changes resulting from high-impact programs were gathered through focus groups, one-on-one interviews, and a review of instructorcreated products resulting from program participation.

Besides self-reports from follow-up surveys, teaching impacts from consultation services were often evaluated through follow-up classroom observations, if appropriate and permitted by the faculty member. Student evaluations were used with similar conditions. If a faculty member requested a small group instructional diagnosis, a follow-up was sometimes performed to gather student reports of changes in teaching.

The evaluation of mentoring programs relied heavily on self-reported changes in teaching solicited through e-mail inquiries and follow-up surveys. TLC directors also reported gathering feedback from mentors and mentees through focus groups, one-on-one interviews, pre-post examinations of syllabi, and pre-post reviews of student evaluations and in-class feedback. One TLC asked mentees to write "critical account" analyses. Two designed a formal experimental study using a control group (those not in the program) and an experimental group (those in the program) and compared gains, with one using tenure ratings as a measure.

The impact of FLCs on teaching was most commonly measured through self-reported changes solicited through e-mail inquiries, focus groups, project reports, and presentations. One director conducted a retroactive faculty survey inquiring into the impact of FLCs on teaching careers over the past twenty years. A few used evidence-based measures, including teaching portfolios, classroom videos, and experimental studies, to measure gains in teaching.

IMPACT OF SERVICES ON LEARNING

Eleven of the fifteen respondents who measured impacts of events on learning solicited teacher-reported changes in student learning through surveys and interviews. FLCs were often evaluated in this manner as well. One respondent administered a five-year follow-up survey to 650 FLC participants inquiring into the perceived impact of FLC participation on student learning.

More robust evaluation efforts, reported by four respondents, targeted high-impact events where evidence demonstrating the return on the investment was required. Methods used to measure changes in learning varied by program. Programs designed to improve specific student skills (writing, for example) typically measured qualitative changes in products of student learning such as e-portfolios, writing samples, and capstone projects. Programs focused on changing instructional methods, such as large course redesigns, active learning initiatives, and cluster teaching projects, used pre-post quantitative measures of student course performance (for example, test scores, homework scores, drop-withdraw-fail rates) or overall academic success (for example, retention, persistence, grade point average). Instructional technology programs gathered reports of changes in learning through case studies, student self-reports, and student surveys. Combinations of these methods were included in reports from faculty who received instructional grants.

Program Evaluation Purpose and Strategy

Reasons for evaluation varied in frequency and type. All thirty-three TLCs studied evaluated program services for purposes of improvement: twenty-seven did so to document success, twenty-six wanted to see if their goals were met, fifteen reported that their administration required program evaluation, thirteen desired to do so, and one wanted to model evidence-based practices.

The production of an annual report summarizing program activity (participation, usage, and satisfaction) and linkage to program goals was the most common summative program evaluation practice, reported by twenty seven of the thirty-three participants. Seven of the thirty-three participants commissioned a periodic program review conducted by individuals outside their center. Four of thirty-three improved efficiency and focus, with staggered evaluation across individual program offerings from year to year.

The most systematic practices were reported by a director who developed a staggered and staged approach to program evaluation. A three-year evaluation plan, staggered program by program, was staged to measure three outcome levels: participation, implementation, and impact. The director first tracked participation data, noting, "It’s not possible and there’s no point to measure impact on student learning and teaching if participation is not present." After adequate participation became evident, evidence of implementation was gathered. Once the data indicated implementation, then impact on learning was measured. Implementation data were gathered through the diligent creation of "a one page case study (like a health record) with pre- and post-assessment data to look for improvement and holes in the process." Three weeks in July are set aside to analyze the data and write up the annual report, during which time all program activities and most services cease. The direc or readily admits this approach "is very hard work and very time-consuming" and emphasizes the crucial need for automation and customized databases to make this work, especially with limited staff. This unique approach captured valid evidence of significant impact and yearly progress that was published in the annual report.

Reported Reasons for Gaps in Program Evaluation

Evaluation of events and activities was performed to various extents at all responding TLCs. However, events seen as informal, infrequent, irregular, or lightly attended reportedly did not justify evaluation, Consultation services were not routinely evaluated due to the desire to maintain confidentiality and also the perceived lack of time and resources. A small number believed consultation evaluation to be too difficult or unnecessary, or the services too irregular, to justify assessing. Participants reporting a lack of evaluation for their occasional services, such as online and physical resources, grant programs, and mentoring programs, most commonly cited a lack of time and resources as the cause. Other respondents indicated that evaluation was too difficult to do, it was low on the priority list, there was no good process, or the informality of the services did not justify evaluation.

The most common reason reported for gaps in evaluation regarding the impact of services on teaching was a lack of time and resources. Accompanying comments such as, "We want to, though, so we can show our dean for funding," and, "The scientist in me says it’s a good approach, but between being a scientist or helpful, it’s better to be helpful," highlight the conflict between desire and resource constraints. Others attributed the evaluation gap to the inherent difficulty and fear of causing survey fatigue.

The most common reason for not evaluating impacts on learning was a lack of time and resources. Several indicated that the presence of multiple confounding variables make the evaluation of impact on student learning very difficult. One respondent summed it up this way: "Being a psychologist with training in assessment, I know the level of effort it takes to do this well; anything less is a crapshoot or just a political tool."

At the conclusion of the interviews, many respondents remarked there was no reason to institute more rigorous program evaluation practices since administration already supports their work, suggesting that time, resources, and energy should go toward providing, rather than justifying, their services. An equal number of respondents reiterated the need for more staff and funding in order to develop more rigorous evaluation practices. For some, the absence of an institutional culture of assessment or leadership inconsistencies reduced their desire to improve program evaluation efforts. Others indicated that a lack of knowledge and absence of models for developing quality program evaluation plans greatly hindered their evaluation work. Disciplinary knowledge played a role and was reflected in comments such as, "We lack the knowledge in the staff. The director has a Ph.D. in social science, so there’s a high standard for quality assessment with rigorous methods which we’re [the rest of us are] unable to do" and "I have a Ph.D. in English, not stats. I would like to know how to do it."

Several other parting comments suggested strong interest and support for continued work in program evaluation:

Staffing

Build staffing and institutional collaboration to support program evaluation efforts. Distribute data gathering among staff or, ideally, assign it to a full-time staff specialist hired spe ifically for program evaluation. In addition, collaborate with the university’s office of assessment to design evaluation plans, provide readily available institutional data, and combine survey efforts.

Systemization

Create a comprehensive plan to systematically gather data for evaluating the program. Determine the goal of the program, the outcomes to be measured, the methodology and timing for data collection, and the schedule for analyzing, reviewing, and implementing the findings. Customize the plan to fit the resource limitations. Where possible, use technology such as online survey software, content management servers (such as SharePoint), database software, and student response systems (clickers) to automate the collection and analysis of data. Simplify survey distribution by standardizing surveys, using preexisting institutional data, and combining survey efforts with other institutional assessment efforts. Embed evaluation in program planning as part of standard practice along with annual reports or fact sheets to track and report program trends and success.

Staggered Evaluation of Programs

Evaluation of the entire TLC does not need to occur at one time. Stagger the evaluation of individualized programs or services on an annual basis. Create a three- to five-year plan outlining the staggered evaluation of each component of the TLC. For example, in year 1, evaluate consultation services; in year 2, evaluate the teaching certificate program; and in year 3, evaluate the mentoring program.

Staged Outcome Evaluation

Take a staged approach to the evaluation of outcomes of various programs. For example, for any given program, track participation only until a significant number is achieved. Then gather data to determine if participants are implementing the new skills. Finally, me sure the impact on student learning once significant implementation is seen. Another approach could be to tailor evaluation to selected outcome measures most appropriate to the intended impact of the individual programs or services. For example, satisfaction data may suffice for ad hoc workshops, data about impact on teaching may be needed for mentoring programs, and data concerning impact on student learning may be important and feasible for grant-funded teaching projects. In other words, collect data that will add value ro the center’s work.