7 Evaluating Teaching Workshops: Beyond the Satisfaction Survey

Workshops are a prevalent approach to fostering instructional development for both teaching assistants (TAs) and faculty. Frequently we evaluate workshops by asking participants to fill out a satisfaction-oriented survey at the end. To what degree do such surveys evaluate adequately the workshop’s long-term effect on participants’ learning? The authors explicate earlier investigative work on transfer of training, and present the results of a follow-up survey to two groups of TA workshop participants designed to assess the degree to which conditions theoretically conducive to the transfer of training exist at their institution.

INTRODUCTION

Teaching assistants (TAs), both native and nonnative English speakers, are frequently prepared for their university teaching duties through voluntary, and sometimes mandatory, attendance at workshops (Lambert & Tice, 1993); our university is no exception. A standard means of evaluating workshop design is through end-of-training satisfaction surveys that often ask for both qualitative and quantitative responses from participants. Rarely are the participants’ learning outcomes assessed through follow-up measures of classroom data. Workshop designers and presenters might benefit more from knowing whether there is either any carryover of workshop content to actual classroom practice, or if conditions are present in the participants’ home departments to encourage the classroom implementation of new ideas and teaching methods introduced in workshops. It is the second of these questions that this chapter focuses on.

We surveyed two groups of workshop participants: the first group, one month after participating in the workshop, and the second group, six months later, to determine whether conditions were present in their daily lives that would encourage transfer of the training experience to their classroom practice. We used many questions from a study conducted in 1998 (Notarianni-Girard, 1999), in which our institution was one of the 12 original colleges included in the database. We found some conditions that Notarianni-Girard’s study defined as facilitating the transfer of training present on our campus.

The term “transfer of training” evolved from business and industry training literature, where it was originally expressed as “transfer of learning” (Huczynski, 1978; Huczynski & Lewis, 1980; Noe, 1986). By the mid-1980s, the notion of conditions that facilitate or inhibit the transfer of training outcomes beyond the training experience was documented in the K-12 staff development literature (Showers, Joyce, & Bennett, 1987). Notarianni-Girard defines transfer of training as “the degree to which trainees continually and effectively apply the knowledge, skills, and attitudes gained in a training context to the job” (1999, p. 120). A simpler definition states that “transfer is the degree to which behavior will be repeated in a new situation” (Detterman & Sternberg, 1993, p. 4).

STUDY BACKGROUND

Our study had its beginnings in 1996, when Deborah Notarianni-Girard, then a graduate student in the University of Connecticut’s Department of Educational Leadership, asked our university, along with 11 other institutions, to participate in a study dealing with transfer of training for graduate students. Out of a total of 75 graduate students on our campus who received her survey, approximately 61 students returned it, giving her a response rate of over 80% for our campus. Notarianni-Girard’s study was designed to address the following four research questions through a post-training survey.

• 1) To what extent are there facilitating factors present in the work environment of teaching assistants trained in a TA training program?

• 2) To what extent are there inhibiting factors present in the work environment of teaching assistants trained in a TA training program?

• 3) To what extent did/do teaching assistants perceive supervisor support before, during, and after training?

• 4) To what extent did/do teaching assistants perceive peer support before, during, and after training?

Her study asked graduate students to rate all training experiences, whether at the university or departmental level. In contrast, our survey was concerned solely with a university-level workshop series. The purpose of our study was to determine the degree to which conditions conducive to the transfer of TA workshop training were present at our university, and to use the results for guiding our strategic planning efforts and improve the effectiveness of the TA development program.

DESCRIPTION OF THE STUDY

In the spring of 2000, a survey was conducted of TAs who took workshops in either fall 1999 or spring 2000. Of the 198 surveys mailed, 69 were returned after an emailed follow-up reminder. Twenty-two graduate students participated in the fall 1999 workshop series, while 25 attended the spring 2000 workshops (two students did not declare which semester they participated), yielding a 34.8% return. The responders represented five colleges and 31 academic departments; 31 were male, 36 were female, and two did not declare their gender. Their previous teaching experience ranged from zero to six semesters.

The survey consisted of two parts. Part one included 17 statements paralleling Notarianni-Girard’s earlier survey. The respondents were asked to rate the degree they agreed or disagreed with the statement. The statements were then grouped into the four areas representing her earlier research questions.

The first group of survey items was designed to determine the extent of facilitating factors present in the work environment of teaching assistants who attended our workshop series (see Appendix 7.1, Research Questions Group I).

The next set of statements were designed to determine to what extent inhibiting factors are present in the work environment of teaching assistants who attended our workshop series (see Appendix 7.1, Research Questions Group II).

Notarianni-Girard’s third research question, “To what extent did/do teaching assistants perceive supervisor support before, during, and after training?” was addressed through the next set of survey items (see Appendix 7.1, Research Questions Group III).

The fourth research question answered in Notarianni-Girard’s earlier study was addressed through the fourth group of survey items (see Appendix 7.1, Research Questions Group IV).

Part two of the survey consisted of 15 questions soliciting demographic information, suggestions for new workshop topics and formats, and other comments. Questions on demographics asked for students’ gender, native language, college of enrollment, academic department, which workshop series they attended, how many semesters they had been teaching, whether their participation in the workshop was before or during their teaching assistantship, the perceived usefulness of the workshop both before and after attending, their motive for taking the workshop, whether the workshop was required or voluntary, when they started their graduate program, and when they anticipated finishing.

RESULTS

Response totals for survey questions one through three are reported in Table 7.1.

Questions 4 through 17 were tabulated according to their mean response on a 5-point scale, with 1 indicating strongly disagree and 5, strongly agree. Their results are reported in Figure 7.1.

Figure 7.1 Response Means for Survey Questions 4–17

Survey items 2, 3, 4, 7, 10, and 17 had to do with the degree to which factors that facilitate transfer of training exist at our university. One obvious factor is the availability of information on training workshops. This factor is present at our university, evidenced by the percentage of respondents (50 out of 69 respondents, or 72%) who said that their department provided them with information about the workshops. Most respondents agreed (mean score 4.2 out of 5.0) that innovations or ideas they have about teaching are supported within the course they teach, and that they feel they have the freedom to conduct their teaching as they wish (3.8 out of 5.0). In addition, most respondents agreed (3.6 out of 5.0) that their TA schedule allowed them time to apply what they learned in the workshops. Finally, most agreed (3.7 out of 5.0) that their department provided sufficient resources (equipment, mentors, secretarial help) for them to be successful in carrying out their instructional responsibilities.

One item indicated an inhibiting factor in transfer training effects. Most respondents (80% of those responding to this question) reported either “no” (31 out of 56) or “don’t know” (14 out of 56) when asked if TAs who receive low performance evaluations are provided with supplemental teaching assistance.

The responses to items 6, 8, and 9 have to do with the degree to which factors inhibiting the transfer of training exist at our university. Most respondents disagreed (1.9 out of 5.0) that their ideas for enhancing teaching were discouraged. Most were neutral or unsure of whether their department preferred they use familiar teaching strategies, while most disagreed (2.1 out of 5.0) that departmental rules and administrative details make it difficult for new ideas they have to receive consideration.

Items 1, 5, 11, 13, and 15 had to do with the degree to which teaching assistants receive supervisor support before, during, and after training. Most respondents (59 out of 66, or 89%) did not meet with their supervisors after training to discuss how they might have their teaching evaluated. Most respondents agreed (3.6 out of 5.0) that their supervisor was enthusiastic about their participation in the training workshops. Most did not meet with their supervisor (2.3 out of 5.0) to discuss ways to implement ideas they learned in the workshops into the classroom, nor to discuss the quality of their teaching (2.6 out of 5.0). However, most TAs perceive that their supervisor listens to their suggestions for implementing the ideas acquired during the workshops into their course (3.7 out of 5.0).

Items 12, 14, and 16 surveyed the extent to which the TAs perceived peer support before, during, and after the workshops. Most TAs did not meet regularly with other workshop TAs to discuss implementing new teaching ideas (2.0 out of 5.0), nor did they have TAs who previously attended workshops as mentors to discuss implementing new ideas (1.9 out of 5.0) A slight majority (3.2 out of 5.0) felt it was a strength having TAs from the same department attend the training with them.

SURVEY CONCLUSIONS

In general, teaching assistants agreed that factors facilitating the transfer of their workshop training to the classroom were present at our university. They were informed about the workshops beforehand, they perceived a supportive environment in their courses for innovative ideas, they felt they were free to conduct their teaching as they wished, their schedules permitted them time to apply what they learned in training to the classroom, and they felt their departments provided sufficient resources for the successful carrying out of their jobs. However, a sizable group (31 out of 69) felt that their peers who received low evaluations did not receive supplemental teaching assistance.

These results are consistent with the respondent data that Notarianni-Girard obtained from TAs at our university in her original study conducted the year before, namely that the consensus was agreement to the presence of facilitating factors existing in the TA work environment (Notarianni-Girard, 1999). Her data showed the same pattern for TAs who received poor evaluations: They apparently received no additional training.

In general, TAs agreed that conditions inhibiting the transfer of training effects were not present in their working environment. They did not feel discouraged from pursuing ideas to enhance their teaching, they were neutral or not sure if their department preferred they use familiar teaching strategies, and they felt their new ideas were considered and not hindered by department rules or administrative details. These results again were consistent with the earlier data collected from TAs at our university by Notarianni-Girard, although in her sample, most respondents felt their department preferred familiar teaching strategies.

The most serious indication for lack of transfer of training factors came from the data on supervisor support. The majority (89%) of respondents did not meet with their supervisor after training to discuss implementing the ideas they learned or the quality of their teaching, even though they felt their supervisors were enthusiastic about their involvement in training and listened to suggestions on how to implement workshop ideas. These results also echoed the data from Notarianni-Girard’s study, which showed that most TAs did not meet with their supervisor to discuss implementing training or to discuss teaching quality before, during, or after training.

Questions 15 and 16 reflect a difference between males and females. Females (3.9 out of 5.0) were more likely to report that their supervisor listened to their suggestions on how to implement ideas acquired from the training workshops than their male peers (3.4 out of 5.0). Females were also more likely (3.5 out of 5.0) to find having TAs from the same department attend the training workshops with them a strength than their male peers (2.6 out of 5.0). This may reflect that females are discussing their training more with their departmental peers and supervisors than males.

Data on peer support also suggested a lack of transfer of training factors: most were not meeting with their peers to discuss implementing new ideas or being mentored by TAs who previously attended workshops. Only a slight majority agreed that having TAs from the same department attend the training with them was a strength. This trend was also prevalent in Notarianni-Girard’s earlier data.

We also found a difference in the response means between the TAs who taught in English as a second language on items 14 and 16 (2.4 and 2.3, respectively) versus TAs teaching in English as a first language (1.68 and 3.5, respectively). Apparently, international TAs are more likely to meet with TAs who previously attended our workshops to discuss implementing teaching ideas than do their native English speaking counterparts. On the other hand, international TAs are less likely to consider attending training workshops with peers from their own department a strength than do native English speaking TAs. They may be grouping with compatriots from their native cultures outside their departments.

IMPLICATIONS FOR PROGRAM AND EVALUATION REFORM

From these data we are investing efforts in three areas to facilitate the transfer of training for graduate teaching assistants on our campus. First, to encourage TAs to discuss their teaching and training ideas regularly after our one-day training workshops which happen at the beginning of each semester, we will begin hosting more follow-up sessions throughout the semester. Additionally, we are developing a cohesive network of TA facilitators who assist with and coteach our university-wide workshops and college-based workshops to foster informal, departmental-based discussions of teaching and learning. The facilitators can act as liaisons with their local departments and colleges to keep us informed of evolving issues and needs.

Secondly, we are exploring ways of getting TAs to participate with their supervisors in events that support the ongoing discussion and review of teaching. We have initiated a discussion group with TAs and their supervisors who meet during the semester after the training workshops. These have been well attended by both TAs and faculty. We are developing materials that workshop participants can take back to their supervisors to support the transfer of training. These materials will help supervisors to clarify teaching goals, monitor skill development, and provide ongoing, evaluative feedback.

Finally, we are revising the ways in which we evaluate TA training programs to include more post-training data-gathering. Currently, TAs have the opportunity to have midterm student evaluations of teaching tabulated with follow-up individual consultation. Because midterm evaluation and consultation is optional, only a subset of workshop participants benefits from these. We plan to be more proactive in the future by providing workshop participants information and encouragement for follow-up evaluation after the workshops.

One outcome of our study is a revision of our working definition of transfer of training to mean that the specific skills and learning outcomes conveyed in the training will be exhibited in the appropriate situation after the training. Thus, teaching skills will be manifested in the classroom, during office hours, or in the ways that TAs interact with students. Behaviors can be directly observed by an instructional developer or supervisor or recorded on videotape.

Some training outcomes may not involve observable behaviors, but changes in the way TAs think. This may be monitored by peer TAs who participated in the workshops at the same time, experienced TAs who can mentor newer teaching assistants, or by faculty supervisors. Through discussions and meetings both before and after the workshops, specific goals and learning outcomes for teaching assistants can be identified and periodically monitored.

REFERENCES

• Detterman, D. K., & Sternberg, R. J. (Eds.). (1993). Transfer on trial: Intelligence, cognition, and instruction. Norwood, NJ: Ablex.
• Huczynski, A. A. (1978). Approaches to the problems of learning transfer. Journal of European Industrial Training, 2 (1), 26-31.
• Huczynski, A. A., & Lewis, J. W. (1980). An empirical study into the learning transfer process in management training. Journal of Management Studies, 17, 227-240.
• Lambert, L. M., & Tice, S. L. (1993). Preparing graduate students to teach. Washington, DC: American Association for Higher Education.
• Noe, R. A. (1986). Trainees’ attributes and attitudes: Neglected influences on training effectiveness. Academy of Management Review, 11 (4), 736-749.
• Notarianni-Girard, D. (1999, Spring). Transfer of training in teaching assistant programs. Journal ofGraduate Teaching Assistant Development, 6 (3), 119-147.
• Showers, B., Joyce, B., & Bennett, B. (1987). Synthesis of research in staff development: A framework for future study. Educational Leadership, 45, 77-87.

Contact:

• David G. Way

• Center for Learning and Teaching

• Cornell University

• 420D Computing and Communications Center

• Garden Avenue

• Ithaca, NY 14853

• (607) 255-2663

• (607) 255-1562 (Fax)

• Email: [email protected]

• Web: www.clt.cornell.edu

• Virleen M. Carlson

• Assistant Director for Instructional Support

• Center for Learning and Teaching

• Cornell University

• 415 Computing and Communications Center

• Garden Avenue

• Ithaca, NY 14853-6601

• (607) 255-8425

• (607) 255-1562 (Fax)

• Email: [email protected]

• Susan C. Piliero

• Associate Professor of Mathematics Education

• Director of the Center for Learning and Teaching

• 420 Computing and Communications Center

• Cornell University

• Ithaca, NY 14853

• (607) 255-6122

• (607) 255-1562 (Fax)

• Email: [email protected]

David G. Way is Director of Instructional Support within the Center for Learning and Teaching at Cornell University. His work involves teaching development programs, including seminars and workshops on teaching, individual instructional consultation, and advising college administrators on effective teaching evaluation practices for improving the university learning and teaching climate.

Virleen M. Carlson holds a PhD in Curriculum and Instruction and wrote her dissertation on TA training. She works for the Center for Learning and Teaching at Cornell University, a centralized support unit, where she serves as the Assistant Director of Instructional Support.

Susan C. Piliero is Director of Cornell’s Center for Teaching and Learning, an omnibus center that includes the Office of Instructional Support for teaching assistants and faculty, the International TA Training Program, the Learning Strategies Center, and Student Disabilities Services. She is also Associate Professor of Mathematics Education, contributes to Cornell’s Teacher Education program, and chairs the University’s Educational Policy Committee.

APPENDIX 7.1