Chapter 12 Reported Long-Term Value and Effects of Teaching Center Consultations

The authors would like to acknowledge Jennie Dorman, Kate Dunsmore, Irina Gendelman, Jason Hendryx, Margy Lawrence, Lana Rae Lenz, and Riki Thompson for their assistance in developing the survey and analyzing responses.

Since the emergence of teaching and learning centers in the latter half of the twentieth century, scholars increasingly have emphasized the importance of self-assessment as an essential component of faculty development programs (Centra, 1976; Frantz, Beebe, Horvath, Canales, & Swee, 2005; Plank, Kalish, Rohdieck, & Harper, 2005; Sorcinelli, 2002). Purposes include both summative (such as documenting achievement and justifying the use of resources) and formative (such as improving services and setting examples that can promote institutional accountability). Sorcinelli (2002) suggests that faculty developers can use program assessment not only to demonstrate “that we do what we say we do” (p. 16) but also to satisfy demands for accountability to administrators.

In an effort to understand more about the assessment of faculty development programs, Chism and Szabo (1997–1998) conducted a survey with a sample of two hundred institutions in the United States and Canada. They concluded that faculty developers have begun to heed calls to assess their own services, particularly with the use of participation numbers and satisfaction surveys to assess the success of faculty development events. The researchers noted, though, that fewer than 20 percent of their respondents always or usually evaluated the impact of their services on users’ teaching.

Brinko and Menges (1997) note that instructional consultants in centers rarely seek to assess the impact of their consultations. Certainly, there are those who assess consultation services, usually through measures of satisfaction obtained immediately upon completion of the consultation process (Fink & Bauer, 2001) or through occasional assessment of the use of consultations (for example, Wilson, 1986). However, any long-term effects of consultations are rarely assessed.

This chapter describes our attempt to begin addressing this challenge of identifying the long-term effects of our teaching center’s consultation services. We designed a client survey that attempts to go beyond user satisfaction by asking former clients to report the value and ongoing effects of their consultations with our staff. We begin with a brief overview of the work of the Center for Instructional Development and Research (CIDR), followed by a description of the survey and a discussion of responses we received from former clients.

CIDR Services, Programs, and Resources

The CIDR has been a resource for teaching and learning at the University of Washington (UW) for nearly twenty-five years. Primarily a consulting center for UW faculty and TAs, CIDR also provides a wide range of additional programs and services for the UW teaching community: we foster campus dialogue on teaching issues through our Quarterly Forum on Teaching and Learning; we play a central role in hosting the Annual Teaching and Learning Symposium, which features scholarship of teaching and learning projects by UW faculty and graduate students. In addition, each fall, we host the Annual TA Conference on Teaching and Learning for 700 to 750 new graduate TAs, as well as work annually with 120 to 150 international graduate students through CIDR’s International TA Program. CIDR staff members also play central roles in a number of the UW Graduate School’s Preparing Future Faculty initiatives. Finally, we have developed an extensive set of resource materials and Web guides to help faculty and TAs address their teaching and learning questions.

Though we are well known at our institution for these programs and resources, the core of CIDR’s work has always remained individual consultations for faculty, TAs, and departments. We annually consult with individuals representing 100–110 different academic units at our institution on a wide range of questions related to teaching, learning, and assessment. During the 2006–2007 academic year, approximately 50 percent of our nearly 700 consultation services were for faculty members, and approximately 35 percent were for graduate TAs. The remaining 15 percent were for administrators and other university leaders. Just over 60 percent of our consultations focused on the individual faculty member or TA’s course; 30 percent focused on departmental questions, such as curriculum development or program assessment, and the remaining 10 percent focused on university-wide or off-campus issues.

Because of the central role of consulting in our work, CIDR has made it a regular practice to ask for feedback on our individual consultations each quarter. This practice allows clients to provide anonymous feedback to our staff and provides a valuable client perspective on how we can continue to develop our services for the UW teaching community. Response rates to requests for feedback in the past have typically been 30 to 40 percent and have been as high as 75 percent since we moved the feedback survey form online in 2005.

Faculty and TA responses to these quarterly feedback surveys have given us confidence that many find our consultations highly valuable. Our 2006–2007 rating for the overall value of consultations averaged 4.8 (on a 0 to 5 scale, with 5 = “highly valuable”), and responses to open-ended questions on our quarterly feedback form reveal more specifically what clients value about CIDR’s individual consultation services (see Table 12.1).

Though we learn a great deal by regularly asking clients for feedback on their recent work with us, we also realize this feedback is limited. Our hope is that consultations make a lasting contribution to a client’s teaching, but other than anecdotal reports from clients who returned to consult with us again, we have little basis for understanding the extent of longer-term contributions we might be making. Thus, to begin identifying these effects, we decided in spring 2007 to survey individuals who had consulted with CIDR during the preceding five years (autumn 2001 through spring 2006) and ask for their current perceptions of the value and effects of their earlier work with us.

Survey and Sample Preparation

CIDR staff worked together to develop a set of survey questions to address our long-term feedback concerns. Our goal was to keep the survey brief and direct, on the assumption that people would be less likely to complete a lengthy survey and more likely to provide a higher level of detail in their responses if they had fewer questions to address.

We limited our questions to topics that could be answered by any of our clients, independent of discipline, academic rank, or CIDR service. We were also careful to develop questions that could be answered meaningfully through retrospective, reflective self-report. For example, we did not ask respondents to rate their own teaching effectiveness (which would require data of other types beyond reflective self-report) but rather, to rate the effects of our services on their teaching (which could be answered based on the client’s memory of work done with CIDR).

We chose to request a combination of numerical ratings and open-ended responses. Ratings questions asked respondents to rate the extent to which consulting with CIDR had affected their teaching, the value of their work with CIDR, and the likelihood that they might recommend CIDR to a colleague. Open-ended questions were paired with each of the ratings questions, asking respondents to identify examples of effects on their teaching, specific benefits of their work with CIDR, and both recommendations and further opportunities for CIDR. Finally, since we received significantly higher response rates after moving our quarterly client feedback form online, we chose to administer the survey entirely online (see Appendix A for the printed version). We identified all consultation clients in our database between autumn 2001 and spring 2006. From this group, we removed all clients who were also working with CIDR during the current academic year (2006–2007) in order to avoid confusing effects of current CIDR consultations with effects of consultations in earlier years. We also removed clients who had consulted with CIDR on departmental issues. We are highly interested in the long-term effects of our services to departments, but we determined that a survey with this goal would require different questions, and it was not clear in every case that the contact person for the service in our database would always be the best person to comment on departmentwide effects. Because the survey addressed effects of individual consultation services, we also chose not to address the effects of CIDR events, programs, or resources.

Because clients over the five-year period included graduate students, many of whom had since completed their degrees and left the university, we located current e-mail addresses for all individuals on the survey list. After removing clients from the current academic year, departmental clients, and clients for whom we could not locate current e-mail addresses, we had a list of 784 former individual consultation clients to contact for the survey.

Finally, we created five identical versions of the survey (one for each of the five academic years represented in the sample) and sent each former client a unique e-mail linking them directly to the version of the survey that corresponded to the most recent year they had worked with us. The e-mail invitation to complete the survey was sent out in March 2007, followed by a reminder e-mail request ten days later.

Results

We received a total of 170 responses, giving us an overall response rate of 22 percent. The response rate for the most recent year (2005–2006) was 29 percent and varied from 18 to 22 percent across the four earlier years. Nearly 60 percent of respondents indicated that they had consulted with CIDR one to three times; 20 percent indicated consulting four to six times, and 20 percent indicated seven or more times.

The profile of our respondents represented the full range of academic disciplines and ranks that CIDR typically works with in an academic year. In comparison with the disciplinary distribution of CIDR’s individual clients for 2006–2007 (see Table 12.2), humanities clients are underrepresented among the respondents, while clients in the sciences and health sciences are somewhat overrepresented. The proportion of tenure-track faculty is higher than it is among CIDR’s individual clients for 2006–2007, while the proportion of graduate students at the time of service is somewhat lower (see Table 12.3).

Discussion

We found this survey helpful for learning more about the ongoing effects of our services, including both reported effects on our clients’ teaching and unanticipated effects that went beyond direct effects to their classroom practices. We found that those who gave CIDR the highest ratings are those who also reported changes in their thinking or in their teaching practices as a result of their work with CIDR, and many of these described the effects of those changes as ongoing: present-tense practices or perceptions are attributed to past-tense work with CIDR. In many ways, these responses confirmed what one might hope to hear from faculty and TAs who came to a teaching center: people who perceived that our services affected their teaching valued their work with us.

However, ratings also raised additional, unexpected questions for us. First, clients rated the value of our services more highly than they rated the effect of our services on their teaching. What, in addition to effect on their teaching, did clients value about our services? Second, clients rated their likelihood to recommend us more highly than the value of our services or the effect on their teaching. What, in addition to value and effect, would motivate them to recommend us? Responses to open-ended questions suggest at least five values added by teaching centers that go beyond direct effects on classroom practices.

First, some respondents noted that they simply didn’t teach very much and so had little basis for identifying effects on their teaching. However, they still found it valuable to meet with a consultant, discuss their questions, and gain additional perspective on teaching. For instructors with limited teaching roles, teaching center consultations provide immediate access to information, resources, and support for addressing their questions about teaching.

Second, others found our services valuable for the connections We helped them form—connections both to a community of people who share an interest in teaching and to the work that members of this community are engaged in. Respondents did not always identify direct or immediate effects of these connections, but they reported finding them valuable nonetheless. Through this survey we have learned that one important role for teaching center consultations is to provide faculty and TAs with the opportunity to learn indirectly from the teaching innovations of other faculty and TAs who work with us.

A third value many found was in the additional perspective gained through their work with CIDR. As one client noted, working with CIDR helped make it possible to “focus on the joy of sharing ideas with my students”; for another, it helped make it possible to see teaching “not as a burden that takes away from research.” For others, work with CIDR was valuable not necessarily because it solved their problems but because it helped them feel “no longer alone” in trying to solve their problems. This added appreciation of teaching—that it can be enjoyed, that not all problems can be easily solved—was another highly valued contribution of teaching center consultations.

Fourth, even among clients who identified changes in teaching practices as a benefit of their work with CIDR, some expressed the perception that the departmental climate for teaching negatively affected their classroom practices. Clients who see attention to teaching as “counterproductive to promotion,” as one of our respondents reported, find value in the opportunity to consult with like-minded colleagues outside their departments, even though they feel the contexts they are working in limit their ability to put changes fully into practice.

A fifth highly valued benefit identified by respondents was the opportunity to consult with experts. In our case at CIDR, we place considerable emphasis on our clients’ expertise and ownership of their teaching, and we rarely present ourselves as “experts.” And yet, survey results show that respondents highly value our expertise in student learning, our knowledge of what other instructors and programs have done, our understanding of teaching and learning in the disciplines, and our ability to listen and adapt to individual instructors’ needs—as one respondent put it, “dealing with real-life teaching situations that the literature overlooks.” In the institutional culture of a research university that highly values expertise, clients highly value having a teaching center as a place to consult on questions about teaching that fall outside their own areas of expertise.

This variety of reasons that clients value CIDR consultations raises one important final question for us. Because consultations on teaching are so central to our work, we typically represent ourselves on campus in terms of our contributions to teaching. However, direct effects on teaching are only one thing that clients value about their work with us, and findings from this survey have caused us to question whether we might be underrepresenting our value by focusing primarily on the changes we bring about in classrooms. This wider range of effects is by no means any easier to measure or document than direct effects on teaching, but we would not want to overlook this wider range of effects in communicating what we offer to the teaching community at our university.

Conclusion

Because this was our first attempt to identify long-term effects of our consulting practices, we chose to survey broadly, but future efforts to identify longer-term effects might benefit from a more focused approach. For example, because we wanted survey questions to be relevant to all clients, we did not ask specific questions related to particular CIDR services or instructor needs. A followup assessment of particular groups, such as clients who requested a Small Group Instructional Diagnosis (SGID), participants in Preparing Future Faculty (PFF) initiatives, or faculty preparing for tenure review, would allow us to ask more detailed questions related to the specific intent and effects of the consultations. We also chose to ask respondents only for information about academic identities (rank, discipline, institution), not other personal or social identities. What might we learn by looking at consultations in relation to other dimensions of client identity such as race, gender, or age?

Furthermore, we also chose to focus only on the long-term assessment of individual consultations. We would potentially learn a great deal by surveying former clients regarding longer-term effects of other CIDR activities, such as services for departments, materials disseminated through our Web site, and university-wide programs and events.

Finally, this survey raises a number of additional questions for further study that will require additional steps beyond surveying clients for their self-reported reflections on the effects of their work with us, and we conclude by suggesting these as possible next steps for those of us who are interested in examining the roles of teaching and learning centers and the effects we have at our institutions. First, other than the self-reported reflections collected through this survey, we have little documented evidence of the effects of specific CIDR services. For example, how are clients’ classroom practices affected by particular services, such as collecting student feedback or consulting on a teaching portfolio?

Second, a number of respondents commented on the influence of departments on individual instructors’ teaching efforts. CIDR currently works with departments in a variety of ways, and we may be able to learn a great deal by examining how some of our ongoing departmental collaborations have influenced both the climate for teaching and the awareness and perceptions of CIDR in those departments.

Last of all, what initially motivates faculty and TAs to seek out CIDR? We now have information about what clients found helpful after they came to CIDR, but we have not yet investigated what originally motivated clients to seek us out, or what obstacles might be hindering others from seeking us out. Closer examination of these questions would help all of us at teaching centers to demonstrate and increase both our value and our impact on our campuses.

Appendix A

References

• Brinko, K. T., & Menges, R. J. (Eds.). (1997). Practically speaking: A sourcebook for instructional consultants in higher education. Stillwater, OK: New Forums Press.
• Centra, J. A. (1976). Faculty development practices in U.S. colleges and universities. Princeton, NJ: Educational Testing Service.
• Chism, N. V. N., & Szabó, B. (1997–98). How faculty development programs evaluate their services. Journal of Staff, Program, and Organization Development, 15(2), 55–62.
• Fink, L. D., & Bauer, G. (2001). Getting started in one-on-one instructional consulting: Suggestions for new consultants. In K. G. Lewis & J. T. Povlacs Lunde (Eds.), Face to face: A sourcebook of individual consultation techniques for faculty/instructional developers (pp. 21–44). Stillwater, OK: New Forums Press.
• Frantz, A. C., Beebe, S. A., Horvath, V. S., Canales, J., & Swee, D. E. (2005). The roles of teaching and learning centers. In S. Chadwick-Blossey & D. R. Robertson (Eds.), To improve the academy: Vol. 23. Resources for faculty, instructional, and organizational development (pp. 72–90). Bolton, MA: Anker.
• Plank, K., Kalish, A., Rohdieck, S., & Harper, K. (2005). A vision beyond measurement: Creating an integrated data system for teaching centers. In S. Chadwick-Blossey & D. R. Robertson (Eds.), To improve the academy: Vol. 23. Resources for faculty, instructional, and organizational development (pp. 173–190). Bolton, MA: Anker.
• Sorcinelli, M. D. (2002). New conceptions of scholarship for a new generation of faculty members. In K. J. Zaborski (Ed.), New directions for teaching and learning: No. 90. Scholarship in the postmodern era: New venues, new values, new visions (pp. 41–48). San Francisco: Jossey-Bass.
• Wilson, R. C. (1986). Improving faculty teaching: Effective use of student evaluations and consultants. Journal of Higher Education,57(2), 196–211.