Cultivating and Sustaining a Faculty Culture of Data-Driven Teaching and Learning:
A Systems Approach

Carnegie Mellon University as a Case Study

Carnegie Mellon University (CMU) is a research-intensive, private institution with approximately 14,000 graduate and undergraduate students and 1,400 faculty. Roughly 1,000 faculty teach at least one course annually. CMU has a long-standing CTL—the Eberly Center for Teaching Excellence & Educational Innovation. The Eberly Center has a history of providing seminars, workshops, and one-on-one teaching consultations. Annually, half of the Eberly Center’s work supports faculty and staff with instructional responsibilities. The other half supports graduate students. During the 2018–2019 academic year, the CTL served more than 600 faculty and over 600 graduate students. Services included consulting individually with 362 faculty and staff members with educational responsibilities, representing 280 courses and 342 instructor-course combinations, approximately one in three instructors teaching courses. These clients came from all seven CMU schools/colleges and all faculty ranks and appointment types.

The Eberly Center is currently part of the provost’s office, reporting directly to the vice provost for education. The CTL employs staff specializing in evidence-based pedagogy, educational technology, and/or formative assessment of student outcomes. Currently, it employs 23 full-time staff, including logistical support personnel. However, as little as six years ago, the CTL had eight employees. One should note that we piloted many approaches described below with fewer staff. In part, these successful pilots are responsible for the Eberly Center’s rapid growth.

A prominent goal of our CTL over the past several years has been supporting faculty to regularly reflect and act on learning data collected in their own courses. We started with faculty who were already making notable changes in their courses and curious about the efficacy of those changes. Our initial efforts attempted to reduce barriers to leveraging data to inform and refine these teaching innovations. We then expanded our target audience to include those who want to understand where student learning currently stands in order to devise targeted interventions.

In the next four sections, we describe some of the activities we established to implement the four steps of our framework. Each section begins with a general explanation of the corresponding step in the framework, including possible implementation strategies. Then, each section highlights one or more of the specific strategies we have used, illustrated through actual scenarios.

Given Competing Demands on Faculty Time, What Incentives Might Persuade Faculty to Engage in Data-Informed Teaching and Course Design?

The first component of the MEFD framework involves motivating faculty to collect data as they teach. There are multiple ways to approach this, such as through financial and other direct incentives or activities that implicitly communicate the value an institution ascribes to collecting and using data in one’s teaching. A common financial incentive involves creating a faculty grants program (e.g., Cook et al., 2011) or providing special stipends and/or teaching relief for projects that involve collecting and using data to improve education. Another approach involves changing the reward structure, such as criteria for promotion and tenure and/or education-related awards. Though effective, these strategies can be tricky and costly to implement effectively. We have explored modest- and moderate-cost versions of these strategies in combination with appealing to faculty members’ intrinsic motivations—for example, to understand what their students are learning well (and not)—to ascertain whether their recent teaching innovation actually made a difference and to guide their future teaching efforts.

In 2014, CMU’s administration launched a Faculty Seed Grants Program (ProSEED) that encouraged faculty who wanted to innovate in their teaching via technology-enhanced learning to also collect data measuring the innovation’s impact. Once or twice per year, nine faculty projects, on average, have been funded. Note that this was not just a grants program for developing teaching innovations, as has become somewhat common across higher education, but one focused on (a) developing evidence-based teaching innovations and then (b) leveraging data to iteratively improve the teaching innovations. As such, selection criteria evaluate the strength of evidence suggesting that the innovation will enhance learning (Does previously conducted education research suggest it will work?) and the quality of the data and assessment plan (Are direct measures of student learning and performance included, rather than just surveys of students’ perceptions?; see Appendix). For each award, the university provides $15,000 to be spent directly by the faculty for summer support or materials for themselves and/or graduate students working on the project. An additional $15,000 may be “spent” on technical, pedagogical, and data support personnel from the CTL to help design a study, develop appropriate measures, analyze data, and so forth. In this sense, not only do the selection criteria promote data-driven innovation, but awardees are expected to leverage the in-kind, matching support of personnel resources to collect and analyze data relevant for assessing and improving their innovations. Over time, we have experimented with proposal writing help of various kinds, including a grant writing workshop, online resources, and ultimately pre-submission consultations while faculty colleagues are envisioning their projects.

Based on reviewer scores, the quality of grant proposals has risen significantly in terms of the proposals’ assessment plans. Specifically, we compared all proposals submitted in spring 2014 and fall 2018, before versus after MEFD framework implementation, respectively. On a 5-point scale, mean rubric scores for the criterion “collect and analyze data on student learning outcomes” were significantly higher in fall 2018 (M = 3.82) than spring 2014 (M = 3.12). A two-tailed t test suggested that the mean difference of 0.7 was significant, 95% CI [.168, 1.156], t (59) = 2.56, p < .05, d = .78. We tentatively attribute the observed change to concerted efforts to communicate the importance of direct measures of outcomes, above and beyond student self-reports and satisfaction surveys. Articulating a plan for collecting data to inform iterative refinement at key project milestones is required in the application and highlighted as a key award-selection criterion. Additionally, all proposal writing support mechanisms described above distribute resources to educate applicants regarding the pros and cons of potential data sources on student outcomes. Of course, we cannot rule out other causes for the observed increase. For example, we lack an appropriate comparison group to use as a control for ambient changes in proposal quality over time. We also acknowledge that reviewer scores have not been calibrated across years or raters. Thus, we simply interpret the positive change in rubric scores as suggestive and in the predicted direction.

Another award program that our CTL administers is the Wimmer Faculty Fellows program. This program is also selective and application based. However, it focuses on junior faculty developing new courses or teaching strategies and offers a comparatively modest $3,000 stipend to up to five faculty annually. We have gradually increased the degree to which data collection is encouraged, expected, and required as part of the program. Currently, participants are required to submit an evaluation plan as part of their fellowship application and may request a consultation to inform their plan prior to submission. Each faculty fellow is paired with a CTL support team, including teaching, assessment, and educational technology consultants, to facilitate the design, implementation, and evaluation of their proposed project. Teaching consultants provide a required Small Group Instructional Diagnosis (see Finelli et al., 2008; Finelli et al., 2011) in each fellow’s course to gather mid-semester student feedback on how students are experiencing the new teaching approach in time to make adjustments on its implementation. Additionally, assessment consultants help fellows collect, analyze, and interpret at least one direct measure of student learning outcomes. Consequently, all fellows now use multiple sources of data to refine their projects, and some have engaged in full-on educational research projects focused on direct measures of student outcomes (e.g., Christian et al., 2019).

Money is not always available as an incentive, but an award program can still leverage prestige and/or special treatment to motivate faculty who are willing (with help) to collect data on student outcomes. For example, the CMU Teaching Innovation Award (TIA) recognizes faculty members’ educational innovations, based on both their transferability to other teaching contexts and effectiveness for student outcomes. While many teaching awards focus on lifetime achievement, the TIA recognizes innovations that are fine-grained in scope and scale within a specific course. Busy faculty do not have to overhaul an entire course to be recognized. Instead, the TIA celebrates small changes, such as a single assignment, classroom activity, or teaching technique, that result in measurable differences in student learning or engagement. Consequently, the award promotes a data-valuing culture of teaching among faculty who are too busy or not motivated for large-scale course transformations. In contrast, the aforementioned ProSEED and Wimmer Faculty Fellows programs provide special treatment in the form of in-kind staff support to motivate faculty engagement. CTL consultants provide support for identifying data sources, designing and implementing assessments, and analyzing and interpreting data. By providing critical just-in-time support to faculty who already have a full plate, this special treatment lowers barriers to engagement.

In addition to their motivating aspects, prestige and special treatment also impact other steps of the MEFD framework. Teaching awards can help disseminate transferable, high-impact strategies among faculty colleagues through award ceremonies and publicity. Likewise, special treatment in the form of in-kind support can help educate and facilitate, increasing faculty skills regarding classroom research and sustaining projects that would otherwise falter during implementation, respectively.

The Framework in Action (Motivation)

Sarah Christian is an assistant teaching professor in civil and environmental engineering who was selected to participate in our Wimmer Faculty Fellows program. Her project revised the cookbook-style labs in her Material Properties course. Exam data showed that students were not transferring concepts from labs to exams. She wanted help to implement a more effective teaching strategy and to assess how it was working. Her collaboration with our CTL started as a teaching consultation focused on inquiry-based learning (IBL) as the evidence-based pedagogy of choice (Pedaste et al., 2015). The consultation revealed that her primarily “cookbook” labs actually included a lightweight component of IBL, making a prediction and testing it by following a recipe-style procedure provided to students (referred to as “structured IBL” by Tafoya et al., 1980). She worked with a teaching consultant to build a more open-ended form of “guided IBL” (Tafoya et al., 1980) into one lab. This guided IBL approach was more tightly aligned with her learning objectives regarding application of concepts and thinking critically like an engineer. She also worked with an assessment consultant to (a) revise exam questions to best measure the learning outcomes associated with each of the three labs and (b) set up a study design in which the impact of structured versus guided IBL could be measured by comparing exam performance on questions assessing the lab content encountered via structured versus guided IBL. By enhancing assessments already embedded in the course, she could directly and rigorously measure learning without creating extra course-related work for her or her students. Students performed 10% higher on the exam questions from the guided IBL lab compared to the more cookbook, structured IBL labs (Christian et al., 2019). Christian is now adding guided IBL to the other labs and continuing to collect data for ongoing iterative refinement. She repeatedly communicated to her consultants that the Wimmer Fellowship motivated her to pursue a project that she would not have done on her own, primarily by providing a formalized support structure, recognition for her teaching efforts via multiple campus events and websites, and mentorship for engaging in SoTL.

How Can Faculty Acquire (or Strengthen) the Skills Required to Navigate the Challenges of Designing, Conducting, and Interpreting Classroom Research?

A common goal of CTLs is to provide professional development for faculty members in their roles as educators. The educate step in our framework aims to design professional development that meets faculty where they are, in terms of interest and readiness to engage in data-informed teaching. CTLs often provide professional development through workshops and individual consultation with faculty colleagues. However, faculty members do not always partake. Many CTLs design their programming based on revealed interests of the faculty. For example, if there is an upswell of faculty asking about the “flipped classroom,” it is time to create workshops on this topic. It is also known from the institutional change literature that there is value in leveraging early adopters because, if they are opinion leaders, they can promote ideas and values to their peers (Rogers, 2003). These strategies have guided our approach.

A few years ago, at our CTL, we noticed a trend of numerous faculty consultations that involved issues around data collection and testing the impact of teaching innovations. So, in fall 2014, we hosted a faculty special interest group (SIG) on collecting data to inform teaching that gathered together faculty members who were interested in or already giving classroom-based research a try. Compared to developing a whole new workshop series around TAR, this discussion-based SIG not only required fewer of our resources but also gave us a chance to interact with faculty who were pushing this envelope so we could learn about their goals and struggles while offering support, guidance, and a venue for peer discussion, learning, and feedback. Our SIGs usually run for a single semester, with 12 to 15 faculty meeting three to four times. This SIG was immensely successful in that it drew over 25 participants, representing all seven of CMU’s schools/colleges, and the participants requested en masse that we continue hosting discussions—for four semesters in a row!

The success of our faculty-driven SIG gave us a sense of the key issues from the faculty perspective and an early read on how this topic could potentially draw a crowd. As with most CTL programs, we wanted to move beyond the early adopters to faculty who could become the next wave of interested adopters. Therefore, we launched the Teaching as Research Institute. Our goal for this program was to provide more structured educational support for colleagues who were new to collecting data in their courses while leveraging the resources we had already developed for the SIG audience. We also needed to think about what would interest a new audience in this work and how we could scale our support to address a broader set of research/assessment projects as well as faculty newer to this work.

Now an annual event, our TAR Institute engages a group of over 20 faculty per year in a four-day program to learn about study design, data sources, and ethical research practices. However, the institute is pitched as a chance to both do new pedagogical development—incorporate active learning into your course—and collect data to analyze and improve student learning. Participants leave this program ready to implement a new, evidence-based teaching strategy and simultaneously to collaborate on a classroom-based research project with CTL support. Faculty need not come with a research question in mind. As we engage with the pedagogical theme, we present the extant body of associated research, highlighting a menu of questions that have not yet been investigated or fully answered. Faculty then work one-on-one with consultants to select and refine a question aligned with their interests and teaching context.

To date, over 80 faculty and staff have attended a TAR Institute, jump-starting over 50 data-driven teaching projects. Some clients wished to publish from the outset. Others simply engaged to better inform their future teaching. Given our goal is promoting data-driven teaching, publication rate is not necessarily a valid measure of impacts, whereas the number of CTL clients collecting data to inform teaching is (e.g., see Table 1 and associated text). Additionally, experimental null results can be quite informative for course design but can be difficult to publish. Regardless, nine TAR projects (and counting) have led to peer-reviewed publications and/or presentations at disciplinary conferences.

The Framework in Action (Educate)

Copious research suggests that infusing active learning into traditional lectures leads to greater learning outcomes and reduces DFW rates (Freeman et al., 2014). However, questions remain about how best to implement active learning to maximize student outcomes. Does the way one debriefs an activity impact learning gains? Three information systems faculty—Mike McCarthy, Marty Barrett, and Joe Mertz—who attended our TAR Institute decided to explore this question in their Distributed Systems course, which enrolls over 100 students. Their students discuss concept-based multiple-choice questions in small groups and then vote on the correct answers. The faculty opted on a classroom research design to test whether debriefing all answer choices added value beyond debriefing the rationale for the correct answer alone (a less time-consuming approach). They did not enter the institute with this question, but the institute helped them formulate a data-driven inquiry to refine their implementation of active learning. Results showed that active learning enhanced average exam performance by 13% compared to a traditional lecture format (McCarthy et al., 2019). Furthermore, although debrief mode did not impact exam performance on recall questions, average exam performance on application questions increased more than 5% when instructors debriefed all answer choices. Consequently, following application-focused classroom activities, the instructors now debrief all answer choices rather than only the correct answer.

Are the Logistics of Conducting a Study a Dealbreaker for Faculty to Adopt Data-Driven Teaching?

In our experience, they most certainly are. For example, to conduct a classroom research study, a university’s IRB must review and approve a faculty member’s research plan. To the uninitiated, the IRB process can be opaque, convoluted, and time-consuming, despite the best efforts of well-meaning IRB offices. Typically, even though many faculty wish to use similar types of interventions and data sources to test their classroom research questions, they have to individually reinvent the wheel as they learn to navigate the IRB system. To lower this barrier, our CTL collaborated with colleagues in the IRB office to establish a broad protocol for classroom research focused on improving the design and delivery of specific courses. The broad protocol allows instructors-of-record to collaborate closely with our consultants to conduct classroom research, implement particular study designs, and collect certain types of data to conduct research within an approved protocol. As long as particular eligibility requirements are met, this piece of infrastructure eliminates the need for each faculty member interested in educational research to navigate the IRB system on her own. Our approach is similar to that pioneered at other CTLs (e.g., Wright, 2008; Wright et al., 2011).

Augmenting the IRB broad protocol, our CTL’s Teaching as Research consulting service also provides tangible support for several other steps of the research process that faculty may not be comfortable with or have time to do. This service includes help with study design, background research, assessment design, and data analysis and interpretation. Consultations focus on how best to implement an intervention, including comparison groups and accounting for potential confounding factors. Background research may entail finding relevant literature to inform the study design as well as identifying and vetting appropriate assessment instruments. Assessment design may involve collaboratively workshopping an instructor’s assessments to align with learning objectives and embedding valid, reliable, direct measures of learning outcomes into coursework. After data collection, we often function as a data broker by analyzing data for faculty lacking necessary quantitative or qualitative skills or time. We also coordinate the matching of course-level data with other student data from the registrar. Instructors don’t necessarily have permission to see registrar data, and our CTL consultants are not allowed to see student identifiers in connection with student data from courses or the registrar. Thus, we developed an automated process by which we can remove identifiers from the data and yet still match data from instructors and the registrar without anybody being able to see or reconstruct the full data set with student identifiers.

The Framework in Action (Facilitate)

Hakan Erdogmus, teaching professor in electrical and computer engineering, participated in our TAR Institute and then benefited from the above services. He devotes the majority of class time to active learning in order to engage students. He was committed to an innovation, flipping the classroom, but did not know how best to optimize its implementation. After the institute, he wanted to leverage his new skills to investigate how best to support students’ pre-class learning via instructional videos. Consultations helped him identify data sources and an appropriate study design to test the effectiveness of inserting guiding questions into the instructional videos to focus student attention. We also helped him analyze and interpret the data, including registrar data, to inform his future course design choices and to prepare a paper to disseminate his findings at an international engineering conference (Erdogmus et al., 2019). Because students performed significantly better on exam questions related to videos containing guiding questions, Erdogmus now requires students to submit answers to guiding questions for all videos.

How Can Teaching and Classroom Research Help Build a Community of Practice Rather than Remain Solitary Endeavors?

CTLs can effectively position themselves as a clearinghouse of resources on learning and teaching as well as a nexus for fostering dialogue, community-building, and a culture of teaching on campus (Cook & Kaplan, 2011). To this end, our CTL provides a number of pathways to “close the loop” on TAR projects, disseminate findings, and create community. For instance, annually, we host a university-wide teaching and learning conference. This Teaching & Learning Summit provides a venue for teacher-researchers to share their work with one another and the broader university community. Through posters, roundtable discussions, demonstrations, and interactive presentations, instructors share both their teaching innovations and the data they have collected. Each year, roughly 200 faculty and graduate students attend, with over 40 instructors presenting their work. Although we do not have an exact count, many presentations include quantitative or qualitative data on student outcomes, which is encouraged as part of the event’s request for proposal (RFP) process.

Seeing the work of colleagues at this event has motivated other instructors to join this community of practice and take advantage of the TAR programs and services described above. For example, one instructor commented, “I had no idea my colleague was doing that, and that there was support available. If she can do it, maybe I can, too.” This outcome is one we hope for—for the summit to be a driver of interest and a mechanism connecting instructors with the multiple entry points for motivation, education, and facilitation in the MEFD framework.

Our Teaching & Learning Summit is not the only pathway for disseminating TAR findings. We opportunistically highlight rigorous faculty work at other CTL events on campus. We also feature TAR projects prominently on our CTL website (Eberly Center, 2019). Some instructors wish to disseminate beyond our university by publishing in relevant journals focused on DBER and SoTL and/or presenting at disciplinary conferences. Our TAR consulting service encourages and supports these goals. Often, faculty have no idea where to disseminate their work or the norms for doing so. We help them find appropriate venues and audiences. Additionally, our consultants guide faculty in writing up the work, provide templates and examples to follow, and provide feedback on drafts.

The Framework in Action (Disseminate)

Alexis Adams is the instructor for a section of Reading and Writing in an Academic Context, a first-year writing course. In addition to teaching written communication skills, this course seeks to develop students’ oral interactional competence, primarily via classroom discussions of assigned readings. Students come from a wide range of disciplines. Many speak English as a second language. Frustrated by the quality of student participation during discussions, Adams piloted several targeted interventions to scaffold development of interactional competence. Our CTL helped design her study, including an observation protocol to measure student engagement. During eight class sessions, a human observer recorded the number and type of rhetorical moves used by each student during a 20-minute discussion. At the 2018 Teaching & Learning Summit, she presented on her interventions and study design. Afterward, a number of attendees requested her innovative educational materials. In 2019, she will return to the summit, presenting on the data collected and lessons learned. In this way, the summit provides instructors an opportunity to get low-stakes feedback on their work while networking with instructors, helping to convert teaching from a solitary to a community endeavor. Our CTL continues to support Adams as she prepares a SoTL manuscript for publication.

Data on CTL Teaching Consultations: Is the Systems Approach Working?

After implementing the MEFD systems approach, we observed significant changes in the frequency of data-driven consultation services requested by faculty (Table 1). We define a service as a substantive consulting interaction provided by a CTL staff member to an instructor-course combination. A single service may (and frequently does) include multiple face-to-face meetings between consultant(s) and client(s). Data-driven consults explicitly address the systematic, direct measurement of student outcomes via quantitative and/or qualitative research methods. During the last five years, both the total number of course-level consultation services and data-driven services increased steadily, by a factor of 1.5 and 6.4, respectively. With the exception of the ProSEED Grants, established in 2014, we implemented the key elements of the MEFD framework during academic year (AY) 2016–2017, including the TAR Institute, TAR consultation service and broad IRB protocols, Teaching & Learning Summit. Before versus after AY 2016–2017, the frequency of course-level, data-driven services increased dramatically, by an average of 20%. While the annual proportion of data-driven services has remained somewhat stable after implementing the MEFD framework, the impact on absolute numbers of faculty and courses continues to increase. Prior to AY 2016–2017, almost all data-driven consults focused on leveraging learning analytics from educational technology tools. Few, if any, clients collected other types of data on direct measures of learning outcomes. Projects resulting from the TAR Institute (educate) and faculty grant and fellowship programs (motivate) alone do not account for the observed increases, suggesting that other steps of the framework (facilitate and disseminate) may also be playing a role.

The data in Table 1 have obvious limitations. They are observational, and we lack data from an appropriate comparison group, such as faculty not self-selecting to request CTL services. Nevertheless, we find no other plausible explanation for the observed patterns. Consequently, we believe these data suggest our CTL’s systems approach is contributing to greater adoption of data-driven teaching and course design on our campus. The large jump observed in AY 2016–2017 may also suggest that our campus’ faculty seed grant program alone may not have been enough to move the needle on faculty adoption of data-driven teaching.