Implementing a University Wide Change Initiative

Institutional Profile

The University is an urban, public, regional institution with a commitment to the development of human capital through exemplary practices in teaching, research, and community service. It was founded in 1963 and is located in the southern region of the United States. Current enrollment is approximately 15,000 students. It serves many students, including many who otherwise may not have had the opportunity to pursue an undergraduate, graduate, or professional degree. Seventy four percent of students are full time, first generation college students and underserved minorities. Over 90% of the students receive some type of financial assistance with 75% of these students receiving grants and 54% receiving loans.

Initial Project Planning

During the summer of 2011, the vice president began the planning stage by studying SACS standards and creating a Leadership Team to guide the QEP process. Members included the University President and Vice Presidents, and the Faculty Senate President. The Leadership Team met several times to discuss SACS accreditation requirements and ensure a development process that would address each accreditation standard.

In the fall of 2011, the Leadership Team appointed a Concept Development Committee to coordinate a participative process to identify approaches that, if implemented, would improve student learning. The Concept Development Committee consisted of deans, associate deans, professors, and students. The Committee solicited and evaluated ideas for approaches that could be implemented to fulfill the accreditation requirement. Each of the suggested approaches was evaluated based on the extent to which it: (a) showed promise for achieving meaningful improvements in student learning, (b) had been validated by previous assessment findings regarding student learning, (c) was manageable in terms of scope, (d) was amenable to assessment, (e) was affordable, and (f) was consistent with the University Mission.

Data sources used in evaluating the intervention included graduation rates, persistence, Educational Testing Service proficiencies, the National Survey of Student Engagement scores, graduating seniors’ perception of academic preparation, and the 25 courses with the highest failure rates. Based on a review of this data, the Concept Development Committee concluded that two areas of student performance merited special attention: critical thinking and the ability to work collaboratively with others. Moreover, student retention and graduation rates were identified as lower than those of peer institutions. The Committee then solicited ideas from stakeholders for strategies to address these areas and created a rubric to assess the extent to which they: (a) supported the University Mission, (b) fit the student population, and (c) promoted student learning. The Committee then sought feedback from multiple constituencies through (a) an e mail survey, (b) a Facebook page, (c) a rotating panel on the University website, and (d) a series of forums with different University groups. Through these efforts, the Committee formulated a list of approaches for the initiative that were then evaluated. Finally, the Concept Development Committee evaluated the proposals and submitted two recommendations to the Leadership Team. These were (a) increasing the use of collaborative learning and (b) focusing on critical thinking. The Leadership Team chose to integrate the two recommendations by promoting the campus wide adoption of TBL (Michaelsen et al., 2004).

TBL was selected for a variety of reasons. One of the most important is that TBL is amenable to assessment due to the fact that it employs the conceptual model called “Backward Design” (Wiggins & McTighe, 1998, 2005). Backward Design is a technique used by instructors in which the first step in developing lesson plans is determining the goal (end result) and designing activities to assess whether that goal has been achieved. Instructors then work backward to develop specific learning activities. In addition, TBL automatically provides assessment data points that would be helpful when assessing the overall impact of the project. Furthermore, adopting TBL enabled the Council to require faculty who participated in the QEP pilot to create a Target Mastery Report in which they first identified overall course goals in terms of student learning outcomes, matched them with assessments, and established benchmarks or mastery targets.

Another reason that TBL is amenable to assessment is that, because it utilizes a very specific set of instructional practices (e.g., see Michaelsen, Davidson, & Major, 2014), it is possible to ensure treatment fidelity (i.e., whether or not faculty are actually implementing the approach). These practices include the following: (a) intentional selection and permanence of student teams; (b) a readiness assurance process, and an empowering procedure for students to challenge answers determined by the instructor; (c) peer evaluation; and (d) application activities that are based on the 4 Ss, Significant problems, the Same problem, students make a Specific choice, and teams reporting their choices Simultaneously (Sweet & Michaelsen, 2012). TBL includes real life experiences with clear applications to course content that helps students understand course concepts, allows them to work on complex intellectual tasks, and offers them the opportunity to move beyond their individual capabilities. It can transform traditional content with application and problem solving skills, while also developing interpersonal skills (Knight, 2004; Opatrny, McCord, & Michaelsen, 2014) and has been shown to improve student achievement by increasing students’ reasoning, problem solving and critical thinking skills, encouraging more scientific thinking, and developing a deeper understanding of course content (Dunaway, 2005; Haidet, MacCormack, & Kubitz, 2014; Koles, Nelson, Stolfi, Parmelee, & Destephen, 2005; McInerey & Fink, 2003; Perkowski & Richards, 2007; Vasan & DeFouw, 2005; Zgheib, Simaan, & Sabra, 2010).

Working Toward Implementation

During the spring of 2012, the Leadership Team appointed an Implementation Team, consisting of students, faculty, department chairs, and deans from across campus. Some members of the Leadership Team previously served on the Concept Development Committee.

The Implementation Team was charged with two responsibilities. The first was fostering interest in the initiative. The second responsibility was providing the Leadership Team with a recommendation in regard to the individual to be hired as the Project Director. The Project Director’s primary role would be to oversee the development and implementation of the QEP.

Ongoing responsibilities of the Project Director would include day to day management and delivery of the project, including participant recruitment and selection, professional development, fund management, and oversight of QEP classrooms. The Director would also be charged with compiling and presenting annual project reports.

The first activity initiated by the Project Director was an assessment of the university climate for change. This assessment began before the Project Director was actually hired. The initial impressions came from questions asked during the interviews that occurred during his campus visit (e.g., How will you motivate instructors to participate in the QEP?) and continued after his appointment during informal conversations with key faculty members and administrators. Based on this data, the Director concluded that, as in any organization, successful implementation of the project was as much about understanding and working through the dynamics of change as it was about the quality of the change itself.

Conceptual Framework

The overriding conceptual framework selected by the Project Director for the implementation of the initiative was based on the planning principles and practices developed by Todd Littman (2013). These principles and practices influenced all aspects of the project. They were as follows: comprehensive—all significant options and impacts were considered; efficient—the process did not waste time or money; inclusive—all university stakeholders were represented; informative—results were communicated to, and understood by, stakeholders; integrated—project decisions supported strategic, long term goals; logical—each step lead to the next; and transparent—all participants understood the process.

Kotter’s (1996) Leading Change and Rogers’ (2003) Diffusion of Innovation were also integrated into the change initiative as a personalized model to guide day to day implementation of the project. Their integrated model consisted of the first six of eight steps posed in Kotter’s Leading Change (1996). These steps included the following: (1) creating a sense of urgency, (2) developing a coalition, (3) developing a shared vision, (4) communicating the vision, (5) empowering action, and (6) celebrating success. As the project is still in its early stages, continue momentum (step 7) will be addressed as the project progresses, and institutionalization of the change (step 8) will be addressed in the closing years of the project (years 4 and 5).

Utilization of a Change Strategy

A performance plan based on the integrated work of Littman (2013), Kotter (1996), and Rogers’ (2003) was developed by the Project Director to be used as a road map to guide the project. The components of the plan and specific actions were associated with the creation of a sense of urgency relative to change, building a sound project foundation through a strong coalition of faculty, creating and communicating a project vision, and empowering faculty to reach self actualization. These components are summarized in the subsequent paragraphs.

Professional Development

Haar (2001) identified quality ongoing professional development as an essential component of faculty growth. Faculty skills and knowledge, obtained through professional development, have a great influence on student learning and achievement and result in personal self actualization. The two day summer faculty development workshop conducted by Dr. Larry Michaelsen was used as a yearly kickoff for the project. The workshop was intentionally designed to demonstrate (as opposed to talking about) the fundamental elements of an actual TBL class. After the summer workshop, professional development sessions focusing on instructional topics connected to TBL were delivered on multiple days and times during the fall and the spring. The themes of professional development sessions included the assessment of higher order thinking, course redesign using Backward Design (Wiggins & McTighe, 1998, 2005), construction of student learning outcomes around higher order thinking, and instructional technology. Specific workshop topics included Crafting Multiple Choice Questions addressing Higher Order Thinking, Flipping Classrooms using TBL, Designing Application Activities using Case Studies, Collegial Coaching, Reciprocal Questioning, Using SMART Boards to Support TBL, Using iClickers with iRATS, Using the Comprehensive Assessment of Team Member Effectiveness (CATME) for Peer Evaluation, Use of Sakai (the University’s Learning Management System) to Support TBL, and Developing Student Learning Outcomes addressing Higher Order Thinking. After the initial session, follow up sessions were conducted to discuss the effectiveness of the implementation of the topic addressed. In all, there were 202 attendees at these follow up sessions.

Building Collegiality

Faculty empowerment will not occur in an environment of failure. Moreover, change initiatives coupled with coaching, professional development, and other forms of support have a 90% rate of successful implementation (Demonte, 2013; Denning, 2005; Drucker, 1999; Lappan, 1997; Williams, 2001). The coaching occurred in two ways. One was through numerous one on one visits by the Project Director. The other was through a formal method of collegial coaching called Learning Walks (Guilott & Parker, 2012). Learning Walks focus on classroom ecology (student to student and student to instructor interactions) and student engagement. Their objective, in this project, was creating a community of faculty learners who would together build a common understanding of practice around the delivery of TBL. In addition, they also helped build professional culture and opportunity for professional reflection.

The Learning Walk communities consisted of three to four instructors, who had participated in TBL training sessions and who were heterogeneously grouped across disciplines to promote conversations focused on pedagogy as opposed to content. The Learning Walk process consisted of four steps: (a) preparation, (b) classroom visits to observe each other’s TBL teaching practice, (c) team debriefings, and (d) closing conversations. Because of the large number of faculty participants, the Project Director trained selected faculty to serve as leaders (mentors) for Learning Walks. These mentors were veteran instructors who were involved with the project, beginning with the pilot.

Celebrating Success

Project success was celebrated in an ongoing manner with formal “Celebration of Success” events conducted each fall and spring. Project participants were honored at events, a Fall Ice Cream Social and Spring Celebration of Success, through the presentation of various awards aimed at recognizing their accomplishments and providing credentials useful when going through the third year review, promotion, and tenure process. To celebrate faculty success, awards were presented during the spring. They included the Certificate of Team Based Pedagogy (a matted, framed certificate awarded to those who: (a) attended no less than five professional development sessions during the academic year, (b) conducted a refereed presentation, submitted a manuscript or grant proposal based on their TBL implementation experience, and (c) developed no less than three complete application activities, with iRAT/tRATs); an Academic Affairs Letter of Commendation (a letter of commendation written by the Senior Vice President of Academic Affairs each semester to participating faculty); Professional Development Fellow status (granted to those who attended no less than five professional development sessions); QEP Collegial Coach Certificates (awarded to instructors who lead Learning Walks); and the QEP Educator of Distinction Award (open to instructor nomination and selected by an anonymous QEP faculty committee using posted selection criteria). Permanent plaques were also hung in the main Team Based Classroom to recognize project participants who received awards. In addition, improvement of student learning was also celebrated through the presentation and discussion of project assessment data.

Assessment Instruments

Assessments were built around critical thinking, collaboration, engagement, and persistence/retention (see Figure 1). The Student Learning Target Mastery Report consisted of three to six student learning outcomes that were matched with assessments and a target mastery level or benchmark established by the instructor. Student learning outcomes were based on higher order thinking and defined using Bloom’s revised taxonomy (Anderson, Krathwohl, & Bloom, 2001). The California Critical Thinking Skills Test (CCTST) was a standardized multiple choice test, and the other assessments consisted of Likert scale and open ended questions.

Figure 1. Goals and Connected Assessments

Data Analysis

Convenience sampling was used to determine frequencies that were reported for items in the Target Mastery Level Report, Student and Faculty Satisfaction Surveys, the Teamwork Interaction Faculty Observation Report, the Critical Thinking and Collaboration Pre and Post Tests, the CCTST, and the student persistence report. An analysis of variance (ANOVA) treatment was used to determine statistical significance of items in the Critical Thinking and Collaboration Pre and Post Tests, the CCTST, and Persistence report. Descriptive data, generated though open ended questions in Student and Faculty Satisfaction Surveys, was analyzed by using a selective coding technique to develop topical categories for each qualitative response set and a nominal ordinal method recording the relative frequency for each response category to quantify responses (Trochim, 2006). Persistence was determined by calculating the number of withdrawals from classes where TBL was utilized, as compared with identical, or when not available, similar classes where TBL was not. At the end of the fourth year of the project, retention will be calculated by comparing the graduation rate of students who were enrolled in TBL classes and those who were not.

Discussion of Findings

The outcomes from the initiative have been overwhelmingly positive. Regarding critical thinking, (a) 82% of Student Learning Outcome Mastery Targets were met; (b) mean scores were 2.7% higher on student critical thinking post tests when compared with the pre tests, and there were statistically significant differences in the evaluating and analyzing domains; (c) scores of items pertaining to deeper understanding and problem solving were higher than the mean score for all other items on the student satisfaction survey; (d) the “TBL strategies increased critical thinking” item score was higher than the mean score on the faculty satisfaction survey; and (e) percentile and mean scores of the CCTST were higher for students enrolled in QEP classes when compared with those who were not, and there were statistically significant differences in all CCTST constructs between QEP and non QEP student scores (Induction, Deduction, Analysis, Inference, Evaluation, Interpretation, and Explanation). For collaboration, (a) mean scores were 5.3% higher on the post test as compared with the collaboration pre test, and there were statistically significant differences in all items; and (b) the “TBL strategies increased collaboration” item score was higher than the mean score on the faculty satisfaction survey. Pertaining to engagement, (a) the “TBL strategies helped increase student engagement” item score was higher than the mean score on the faculty and student satisfaction surveys; and (b) faculty members commented that student engagement facilitated constructive criticism and dialogue.

Comments provided by students and faculty produced further insights into data collected concerning project effectiveness. Students were asked the following open ended question in the Student Satisfaction Survey: “What was the most beneficial aspect of Team Based Learning?” Selected responses included:

• “Taking tests as a team”

• “Reasoning within the group”

• “Being able to ask your group for help instead of the professor”

• “Being able to exchange knowledge in a fluid and constructive way.”

Figures 2 and 3 contain a quantified compilation of coded responses using a technique to develop categories for each response set and a nominal ordinal method recording the relative frequency for each response category (Trochim, 2006). Comments related to collaboration, critical thinking, and deeper understanding of content were the most common student responses identified in Figure 2.

Figure 2. Coded Student Responses

Figure 3. Coded Faculty Responses

Faculty members were asked the following open ended question in the Faculty Satisfaction Survey: “What was the most beneficial aspect of Team Based Learning?” Selected responses included:

• “The active and engaging environment … is both effective AND inspiring”

• “Students learning from each other”

• “The use of the iRAT and tRAT [because] students who made errors in their iRAT found out instantly from their teammates where they had gone wrong when they completed the tRAT”

• “Students were no longer passive learners.”

Figure 3 contains a comprehensive compilation of faculty responses. Comments related to class preparation and student engagement were the most common responses.

There were also higher percentages of As and Bs in QEP courses (n = 60) when compared with the same number of identical or similar non QEP courses. The most important finding may be related to persistence as the student withdrawal rate from non QEP courses (7.8%) was twice as high as student withdrawal rate from QEP courses (3.6%), and there were statistically significant differences in student withdrawals when comparing QEP and non QEP courses.

Faculty and Student Growth

The number of classes and students grew from the pilot conducted during the fall of 2012 and spring of 2013, through the first year of the project, the fall of 2013, and spring of 2014 (see Table 1). Various sizes (6–90) of freshman through graduate level classes from multiple disciplines across the university were represented.

As well, faculty participation in the QEP has grown. A total of 11 faculty participated in the pilot, 60 in year 1, and 150 are currently enrolled for year 2. The percentage of total university faculty has also grown from 1.4% in the pilot year to 8% in year 1, and to 21% in year 2 (enrolled). The majority of faculty participants have been from STEM and STEM related (STEM/SR) disciplines.

Targeted Areas for Improvement

There will be three areas targeted for improvement during the next year of the project. They are student peer evaluation, collegial coaching, and professional development. These areas involve strategies and techniques focused on improving student learning by facilitating the utilization of TBL. Each area targeted for improvement will address ways to improve instructional delivery by facilitating solutions to common problems noted during the first year of the project.

Student Peer Evaluation

The larger the class, the more difficult it is to provide meaningful and timely feedback regarding peer evaluation. While paper and pencil methods will still be used, training will be provided to instructors of large classes on the use of a web based system called the Comprehensive Assessment of Team Member Effectiveness (CATME). The system, developed through a National Science Foundation grant administered by Purdue University, involves a sophisticated peer evaluation system by which students rate themselves and their team. Upon conclusion, they are provided with a diagnostic report that identifies areas in need of improvement with specific tips to address each of these areas. Instructors are also provided quantitative data that may be used for grading peer evaluation, a component of TBL. In addition, the Teamwork Interaction Faculty Observation Report will be introduced to provide a direct assessment of peer evaluation by faculty.

Collegial Coaching

Over 25% of project participants mentored and coached each other during the first year of the project. When student data from their classes, preciously discussed, was compared with student data from other project classes, their findings indicated higher assessment scores in the classes where instructors participated in mentoring/coaching. However, other instructors expressed an interest in participating in mentoring/coaching, but could not because of the distance between classrooms across campus. As higher levels of student achievement were found in the classes of instructors who participated in mentoring/coaching, the Project Director will mitigate the problems presented by distance by grouping faculty into teams by college. This will be possible because of the increased number of project participants and will greatly reduce the distance between classes as many are found within the same building. There will also be a link created on the project website for small group blogging to facilitate mentoring/coaching between instructors.

Professional Development

Online professional development sessions will be offered during the next year of the project to ease the conflict with faculty schedules. These sessions will be crafted around mastery and subsequent application of subject matter related to TBL. Faculty members who receive credentialing credit for completion of a session will be required to complete a brief online content assessment and submit an application of the session topic to the Project Director for feedback. Credentialing credit will also be granted to instructors who coordinate service learning opportunities for their students to gain expertise in the development of application activities linked to real world scenarios.

With the expansion of online delivery systems, professional development sessions will be administered online to assist instructors with the delivery of TBL strategies.

Many instructors are conducting courses that are using asynchronous technology for instructional delivery in a totally online environment. With the use of sophisticated video conferencing technologies, it is possible to employ the principles of Michaelsen’s TBL in blended environments using synchronous and/or face to face pedagogical structures in conjunction with those that are asynchronous.

Conclusions

One conclusion that can be drawn from this study is that by properly designing the initiative, it is possible to successfully bring about a culture change. Based on the experience described in this paper, the formula for replicating this type of project is based on the what and how of change. An organization must first discover what needs to be done to solve a given problem after building a consensus fashioned around a sound rationale based on input from multiple constituents using relevant data. After the what needs to be done is determined, stakeholders using the same process must endorse a corresponding solution. The solution to the problem is the bridge between the what and the how. In many cases, the how to get it done is the most difficult step for organizational growth as they are traditionally more apt to identify the problems they face and potential solutions, but for various reasons they have difficulty getting to the how to make what needs to be done become a reality.

The journey to discovering the what needs to be done described in this paper involved the creation of multiple committees charged with the identification of the most critical problems facing their students and the solution to those problems. After a great deal of time and resources, it was agreed that there were two areas pertaining to student learning most in need of improvement. These areas were the ability of students to work together (collaborate) and to think critically.

As described above, the how occurred in three stages. The first was a decision to promote the campus wide use of TBL. The second was hiring a Project Director to develop and implement a strategy for increasing the use of TBL. The final and, by far, the most challenging stage was actually getting it done. The challenge was compounded by the fact that the Project Director came from another university to lead the project less than a year before the reaffirmation visit.

The Project Director initiated the implementation stage by involving faculty in the QEP by utilizing a strategy constructed around the integration of three change models, Rogers’ Diffusion of Innovation (2003), Kotter’s Leading Change (1996), and Littman’s Planning Principles and Practices (2013). For the implementation of the QEP, maybe the two most critical aspects of the how to get it done phase were the selection of TBL as the intervention and the guidance provided by Rogers’ Diffusion of Innovation for deciding how to proceed. The practicality and proven success of TBL was critical in the recruitment of Early Adopters and without their involvement and demonstrated success, the project would not have been embraced at such a considerable level across campus.

The second conclusion that can be drawn from this study is that TBL was a highly effective strategy for achieving the desired outcomes. Student persistence significantly increased as did the number of students who earned C or higher grades (i.e., course withdrawal rates were cut in half and the number of Ds and Fs was reduced by 38%). Data gathered from the project indicated that students were more likely to utilize critical thinking skills that went beyond teacher introduced knowledge. Students exhibited higher cognitive processing, increased problem solving skills, and collaboration. They also better understood the material, stayed focused longer, and enjoyed many other benefits as documented in this paper. It is for these reasons that the use of TBL is a highly attractive option for faculty developers and university administrators who are looking for ways to improve educational outcomes on their campus.

In closing, the success of the project has been acknowledged across campus at Dean’s Council meetings, through the QEP newsletter, at student government meetings, and on the QEP website. The success has not only impacted students, but professors as well. Professors have conducted over 30 professional conference presentations reporting the impact of TBL on learning in their classes. Eighteen have received Certificates of Pedagogy in TBL, 5 have received Certificates of Collegial Coaching, and 34 have been recognized for attending no less than five QEP professional development sessions during the academic year. Two have received grants totaling $36,500 and six have received “Top Prof” Awards sponsored by the Azalea Chapter of Mortar Board, the National Senior Class Honor Society, for exceptional contributions to the University and their service to students. It is the intent of the Project Director to help sustain the momentum of the project through recognition of those who have been responsible for its success.