Sutherland, T., & Guffey, J. (1997). The impact of comprehensive institutional assessment on Faculty. In D. DeZure (Ed.), To improve the Academy, Vol. 16 (pp. 151-164). Stillwater, OK: New Forums Press and the Professional and Organizational Development Network in Higher Education. Key Words: Assessment, Faculty Development

In this age of accountability, colleges and universities are being called on to provide evidence of their effectiveness. As a result, comprehensive assessment initiatives are being implemented on most campuses, requiring increasing numbers of faculty to become involved. Beginning with an overview of a faculty-driven assessment model, this article describes specific roles faculty play and the results of a study in which faculty describe how their involvement influences their teaching and professional development. The primary purpose of faculty development is to improve the learning environment. Faculty participation in institutional assessment efforts enhances that environment. The results of the study provide compelling evidence of the benefits of faculty involvement in institutional assessment initiatives.

In an age of greater accountability, institutions are being called on to provide evidence of quality and effectiveness in the delivery of higher education. The influence of accreditation associations and state legislatures has brought some form of assessment program to nearly every college and university campus. As both the need for and the scope of institutional assessment grow, more faculty will be called upon to participate in the process. Interesting challenges grow out of this need. Should faculty be involved in institutional assessment? Why is their participation important? What benefits await those who do participate? To provide a context for readers to understand the experiences faculty have had as they participate in assessment activities, the article begins with an overview of Truman State University’s (formerly Northeast Missouri State University) faculty-driven assessment model. The second half of the article describes specific faculty roles in the assessment program and the results of a study in which faculty discuss how their involvement has influenced their teaching and professional development

A History and Overview of Assessment at Truman State University

Truman State University is a public liberal arts and sciences university located in rural northeast Missouri with an enrollment of 6,000 primarily residential undergraduates. In order to monitor its progress toward achieving its educational goals, Truman has developed an extensive assessment program intended to fulfill three primary purposes:

  1. to measure student growth and development;

  2. to assess whether our graduates are nationally competitive; and

  3. to focus on quality rather than quantity as a measure of institutional success.

A critical element in realizing these purposes is the deliberate fostering of a sense of trust within the university community with regard to how the data will be used. Data is never used punitively or to create comparisons between academic units; it is instead used for making historical comparisons within academic units and for evaluation of the effectiveness of the university as a whole. The focus is on continuous improvement and greater understanding of our students and institution. In this kind of environment, data can contribute to frank and meaningful discussions concerning decision-making, goal-setting, and planning.

Truman’s assessment model is based on a model of “triangulation,” which brings to bear multiple measures to better understand how our students learn and develop while attending the university. While the overall program is pervasive and ambitious, the average student spends about 16 hours on assessment activities dming their college career. The program has three major components, each designed to gather information about a different aspect of university effectiveness with regard to student outcomes:

  1. value-added—to provide a pre-test/post-test assessment of general education outcomes;

  2. comparative—to monitor whether or not graduates are nationally competitive in their fields; and

  3. attitudinal—to describe students’ attitudes, values and experiences.

Over the past 20 years, the University has adopted a number of strategies to develop a comprehensive program which has lead to a culture that expects the use of data in its decision-making and planning processes.

The Comparative Component

Assessment began with its comparative component at Truman in 1973 when President Charles McClain, interested in discovering how our students stacked up nationally, asked graduating students to volunteer to take senior exams. Beginning in 1974, all graduating seniors sat for a comparative exam in the major. While various exams were used, major programs administered nationally-normed instruments whenever possible. A few academic programs, such as home economics and agriculture, used locally written exams in the absence of a suitable externally-normed instrument. Many academic programs now use the Major Field Achievement Test (MFAT) as their senior test. Others administer instruments like the National Teacher’s Examination (NTE) and board exams for professional degrees, such as accounting and nursing.

The Value-Added Component

Value-added assessment began in 1975 with the Sequential Test of Educational Progress (STEP). In 1978, the University began utilizing the ACT as a pre- and post-test to measure growth in the basic skill areas. In the spring of 1981, the University initiated the use of the ACT College Outcome Measures Project (COMP) to measure value-added. The Collegiate Assessment of Academic Progress (CAAP) was put into place in 1990, with 50% of the freshman class taking COMP and 50% taking CAAP as the pre-test. Each student would then retake the same exam as a post-test after the completion of 45 semester hours. In the 1993, the post-test was shifted to be taken after the completion of 60 semester hours, and in 1994 the Academic Profile (AP) was piloted in place of the COMP. The effects of shifting the time of post-test and the use of the AP exam are being monitored and reviewed.

The Attitudinal Component

Attitudinal assessment began at about the same time as the value-added testing initiative. Truman developed the Summer Orientation Student Survey (SOSS) for incoming freshman and the Institutional Student Survey (ISS) for currently enrolled students. The Graduating Student Questionnaire (GSQ) was adapted from a survey developed by the National Center for Higher Education Management Systems. The University triennially administers a Survey of Alumni and a Survey of Employers developed by ACT to provide an assessment of effectiveness of our students’ performance after graduation. Truman currently uses each of these instruments with the exception of the SOSS. It has been replaced by the Cooperative Institutional Research Project (CIRP), sponsored by UCLA.

The early history of assessment at Truman State University is discussed in Astin’s Why Not Try Some New Ways of Measuring Quality? (1982) and McClain’s Assessment Produces Degrees with Integrity (1987). Truman’s implementation of assessment within an academic unit, nursing, is presented in Cornell’s The Value-added Approach to the Measurement of Educational Quality (1985).

Since 1984, Truman’s assessment program has expanded to include a variety of qualitative measures of effectiveness. This expansion evolved from a tradition of developing multiple measures of quality and student growth, particularly in complex areas for which objective tests were believed to be inadequate, such as higher-order thinking skills, scientific reasoning and data analysis, writing ability, and overall student growth over time. Portfolios were initiated in 1988 to develop a local assessment of the liberal arts and sciences curriculum and to promote self-assessment by students. The Sophomore Writing Experience began in 1979 and was mandated in 1984 for all students to assess the effectiveness of writing use across the curriculum and student growth as writers. An interview project began in 1993 with a random sample of juniors to gather more in-depth information about their classroom experiences than can be determined through written surveys. That project currently focuses on second-semester freshmen and their experiences in the first year. Finally, capstone experiences have been designed for academic majors so that each student has the opportunity to see several years of study come together as a unit and program faculty can holistically evaluate individual students as well as the major program.

Clearly, what began in 1973 as an assessment project has grown into a pervasive assessment culture which expects the use of multiple measures to produce data that illuminate answers and generate more questions about accomplishing the university’s educational purposes. These multiple measures have evolved into a wide range of instruments that provide both quantitative and qualitative information about the educational program at Truman State University. This evolution comes at a price, however. As the program includes more qualitative measures, it requires much more effort by the faculty to gather and assimilate the data. At the same time, we are discovering that substantial benefits await those involved in the assessment process—both for the university and for the faculty themselves.

Faculty are Central to the Assessment Enterprise

The assessment initiative at Truman is faculty driven. Faculty coordinate the direction of the program and the compilation and analysis of the data, and they make recommendations about the dissemination of information related to institutional effectiveness. This effort is directed by the Advisory Committee on Assessment to the Vice President for Academic Affairs, which is led by a faculty chair and made up of faculty from all academic divisions, students, and staff from critical areas.

The assessment committee oversees the implementation of qualitative assessment measures, including the Sophomore Writing Experience, Portfolio Assessment, Capstone Experiences, and an annual Interview Project. These projects are some of the most time-intensive, each requiring groups of faculty from across campus to spend time together collecting data and discussing its implications. An overview of the activities faculty engage in as part of these initiatives builds an understanding of how faculty might benefit by their involvement.

The Sophomore Writing Experience

Students participate in this evaluation after completing the requirement for freshman composition (Composition I) and before enrolling in Composition II. They receive guidelines and prepare an essay on one of two possible topics. The essays are evaluated holistically by teams of faculty who meet on various Saturdays for day-long grading sessions. Due to the length of the day, faculty who participate receive a small stipend. Each session begins with training in holistic grading and faculty work at tables coordinated by experienced readers. This process typically involves in-depth conversations about what constitutes good technique as well as voice, risk-taking and flair in the evaluation of writing. Each essay is evaluated by at least two readers. The process is brought to closure as each student has an individual conference with a trained faculty member to discuss his or her essay and the student’s development as a writer.

Portfolio Assessment

The purpose of the portfolio assessment is to evaluate the effectiveness of our general education program. Students develop portfolios according to guidelines provided by the assessment committee. Some of the types of entries requested include ones which reflect: (a) interdisciplinary thinking, (b) scientific reasoning, and (c) two entries which demonstrate growth as a thinker. While most entries are papers written for various classes throughout the student’s college career, other types of submissions have included videotapes, audiotapes, and slides. Portfolios are completed and turned in during senior seminars or capstone courses. The portfolios are then rated by groups of faculty who meet for one week during the period between the spring and summer semesters. These faculty receive a small stipend for the week.

Similar to the description of the Sophomore Writing Experience, the assessment of each entry in the portfolio is preceded by discussion of what constitutes an appropriate submission. All readers experience times of comfort and discomfort as they encounter a variety of submissions from students all across the university. Mathematics faculty reading poetry and literature faculty reading laboratory reports find themselves talking to colleagues about the content and worth of the work before them. Faculty often discuss afterward their rediscovery of the difficulty and importance of clearly defining goals of courses and individual assignments. Many find the portfolio reading a rich opportunity to get new ideas about how to present good writing prompts or class projects. They discover ways to interconnect their class with others, both near and far, around campus. An openness to asking questions and having productive dialogue are critical to the effectiveness of these exchanges.

Capstone Courses

Capstone experiences are designed to give seniors the opportunity to demonstrate and build on knowledge in their chosen disciplines. While the portfolio assessment is intended to evaluate students’ breadth in general education, the intention of the capstone is to measure their depth in understanding and knowledge of their field of study. As a requirement grounded in the specifics of a discipline, each academic program’s requirement is different. Depending on their major, a student might present an article review, write and defend a formal thesis, design a research project, or sit for a locally-developed comprehensive exam.

In contrast to portfolio assessment which encourages faculty to connect with colleagues across disciplines, developing capstone experiences requires faculty to connect within the discipline. Faculty in each academic program design their capstone based on a shared vision of the kinds of knowledge, skills and attitudes graduates from that major should display. Defining that vision requires faculty to work together in new and unaccustomed ways. No longer does each class stand on its own; colleagues must review the content and sequence of courses, determining how they fit together to create an integrated whole. These interactions lead to developing consensus about how students can best learn disciplinary content and how that learning will be assessed.

Interview Projects

The annual interview project gathers data beyond that collected in the institutional surveys. Each year the assessment committee identifies an area of particular interest and develops an interview protocol focusing on some element of students’ learning experiences. Students from a random sample are interviewed individually by a team of one faculty member and one student. The interviews are short: approximately 30 minutes long. Following each interview, the faculty-student team compiles notes taken individually onto a common protocol sheet. All interviewing teams participate in a debriefing either just before or just after they complete their interview schedule.

Discussions between the faculty-student interview teams are always lively as they work together to reflect accurately the perspectives of their interview subjects. Even more animated are the discussions during the debriefing sessions held over lunch just before or just after students are interviewed. The sessions provide an opportunity for interviewers to consider and make meaning from the data they have gathered. A strength of interview data is the powerful way it captures student voices, and interviewers are always eager to share the things they have learned from their subjects.

Opportunities to talk with colleagues in an extended and intense way are rare in the busy world of most faculty. Each of the qualitative assessments provides a rich environment in which faculty learn from each others’ perspectives and experience and debate the true desired outcomes of students’ educational experience at the University.

What Faculty Say About Participating in Institutional Assessment Activities

While the institution gains valuable information from its assessment initiatives, the programs take a toll on busy faculty. Is it worth the time faculty must commit to keeping this assessment initiative afloat? Does participating in assessment activities provide any benefits for faculty? After several years of hearing colleagues describe faculty development benefits gained while participating in assessment activities, the Director of Faculty Development at Truman initiated a study to discover exactly what they meant.

The goal of the project was gaining in-depth understanding of a narrow range of faculty experience. Toward that end, an in-depth interview method was selected, allowing participants to express themselves in their own voices. A sample of “extreme cases” was used (Skrtic, 1985, p. 105), targeting only faculty with involvement in at least one assessment project to gain information available only from those with that particular experience. An interview guide addressed three open-ended questions, asking faculty to describe how participating in Truman’s assessment program has affected their teaching practices, professional activity and personal development. To date, twenty-one faculty have been interviewed. The interviews were taped and transcribed, and the data analyzed in the constant comparative style (Lincoln & Guba, 1985, p. 339) to allow categories to emerge from the data. As certain pieces or “chunks” of data, in this case quotations from interviews, were grouped by a common theme, initial categories were identified. While coding continued, each new chunk was compared to those already identified with a particular theme to determine parameters of the category and best fit of the data. Several initial themes quickly emerged

Faculty:

  1. described having a better understanding of students’ abilities as a result of their opportunities to read and evaluate student work in assessment projects;

  2. valued the chance to have interdisciplinary conversations and to develop interdisciplinary connections with colleagues;

  3. described developing a greater understanding of and commitment to institutional goals after gaining an understanding of the “big picture” of the University; and

  4. judged the work worth the time and effort. (Not a single participant described regret over time spent in assessment initiatives.)

One of the great benefits of conducting a qualitative study is the richness of the data which reflects the actual voices of study participants. Their comments communicate some of the real flavor of their experiences. The quotes below are representative of faculty perspectives in each of the emergent theme areas and provide insights into the benefits they derive from being involved in Truman’s assessment initiative.

Better Understanding of Students’ Abilities

This category emerged as faculty described gaining a better understanding of how students in their academic programs compared with those in other majors, and how students across the university perform in terms of broader institutional goals. One colleague addressed the advantages of seeing and comparing student work, not only in her own field but also in other fields, on other kinds of tasks:

...you have a much greater feel for what your students are doing. I would always pay attention to when [our majors’] portfolios came through... Not that I thought they weren’t doing good work in other areas too, but you see that this person obviously put some time in on this English paper or you get a better feel for the whole student; a sense of a deeper dimension of the student…. You see that there’s more to the student than just [my field]—what I see [as a teacher]. Sometimes they’d have some personal things in there and you’d get a deeper insight there. You see these math majors who write beautiful poems, making cross-connections. I guess I became really more aware of the talent our students have in…diverse areas.

A scientist reported that involvement in assessment not only provided a better understanding of students’ abilities overall but also enhanced his teaching by fostering greater sensitivity to the diversity of students and their skills and talents. He talked about how much he learns from colleagues:

The other thing that’s really good is reading students’ work in other areas besides [mine]. I never read history papers and nursing plans and things like that in my job. So it gives me a better perspective on what our students can do. You know I see all kinds of those students who do those things well, in my [science] class, where they struggle. And it’s good to be reminded that everyone has their skills and talents and I think it affects how you approach students in a class like that where you have a mixed group of people [mixed majors]... It’s been a long time since I took an English class, and for me to read something and think, “Okay, I think that’s good; it’s well written, it’s well organized”...And then have someone like [an English professor] say, “Yeah, but...” and then talk about the things that are missing. I learn from that. I learn to be more critical, and what things to look at.

Interdisciplinary Connections with Colleagues

With its mission as a public liberal arts university, Truman places an emphasis on students making connections across disciplines. In developing their general education portfolios, one of the things students are asked to demonstrate is “interdisciplinary thinking.” Interestingly, the opportunity to make those same kinds of connections with colleagues emerged as a theme when participants in the study talked about the benefits of their involvement in the substantial conversations that are part of an assessment culture. One participant described the importance of time spent sharing goals and expectations with a cross-disciplinary group of colleagues as they reviewed student portfolios:

It’s the opportunity. It’s what goes on it that little room. It’s the opportunity to sit for eight hours a day with colleagues from across disciplines and talk about student learning and what they expect and what they perceive; and what they think is good work; and what they think is bad work; and what they think we should be doing for our students; and all the sharing that goes on. The most important thing was the faculty sharing experience.

Another talked about how participation bridged the isolation that can be the by-product of a heavy class load and typical disciplinary structure and also provided broader perspectives:

...I find it very interesting to know what’s going on in other parts of the university, because I feel pretty isolated myself at times, not having very much time... That’s one of the things about participating in the assessment that I appreciated most—was just seeing people from other divisions, face-to-face, for several days in a row, that I don’t usually get to see, and hearing them talk about what they’re trying to do.

Institutional Commitment Via Understanding the “Big Picture”

One of the challenges of our complex organizational structures is the difficulty of faculty finding ways to feel connected to larger institutional issues and goals. It can seem impossible to communicate with faculty across the university about the things that are of greatest importance. This study provides some evidence that participation in assessment leads faculty to a better sense of the university’s aspirations and concerns. A business professor reported gaining a greater feeling of belonging to the institution and interest in all its students:

I’m not just a member of my department; I feel more a part of the University and have an allegiance to the University. I am interested in political science students and fine arts students; they’re mine—just like business students are mine.

One of his colleagues went on to say that assessment helped him to develop an awareness of different disciplines’ perspectives and how that understanding can build morale:

The biggest thing university wide was that it gave you a different viewpoint. You saw science’s viewpoint and math’s viewpoint. It was a wonderful thing to be involved in and find out other people’s points of view and it gave you an appreciation for the university as a whole. And I think that’s why there is a gang of people who are crazy about this university; is that they still have a sense that everybody works together.

Worth the Effort

While each participant in the study spent some time talking about a question or concern they had about Truman’s assessment initiative, not a single person said it was not worth the time faculty spend on the project. In fact, many made a point of recognizing the time required and its value to faculty and to the university. One of many quotes articulates the opinion that faculty work in assessment is important for students:

I’m sold that it’s a special thing we do that has value, and that students gain a benefit from being here because we have this assessment culture. I’m pretty much sold on that.

A senior professor with many years of assessment experience reflected on the personal and professional benefits found through years of involvement:

It’s satisfying to know that the University can look at itself and attempt to correct problems that are found. I get a kick out of going through those processes, of trying to discern peoples’ opinions... I find satisfaction in it; personal as well as institutional. A kind of pride of belonging.

A final remark shows recognition of the time required, but also of the benefits of the learning opportunities for faculty involved in institutional assessment:

For me it’s very much been useful. As a professor it has been very worthwhile. It did take up some time... But I think you learn a lot, you gain a lot seeing the breadth of what the students experience here. I think it’s very much worthwhile.

The primary purpose of faculty development is to improve the learning environment of the University. It may be surprising news for many faculty that participation in institutional assessment efforts can in fact enhance that environment. The voices of the faculty at Truman provide a compelling argument for faculty involvement in institutional assessment initiatives.

As one faculty member summarized the benefits of assessment for both students and the university:

...it’s so powerful seeing the work your students do in your class compared to similar classes... I really believe that assessment does cut both ways—that it is a way to assess students, but, on the other hand, [it helps us assess] whether or not we’re doing a good job in the classroom.

References

  • Astin, A. A. (1982). Why not try some new ways of measuring quality? Educational Record, 63, 10-15.
  • Cornell, G. R. (1985). The value-added approach to the measurement of educational quality. Journal of Professional Nursing, 1, 356-363.
  • Lincoln U. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.
  • McClain, C. (1987). Assessment produces degrees with integrity. Educational Record, 68, 47-52.
  • Northeast Missouri State University. (1984). In pursuit of degreeswith integrity:A value-added approach to undergraduate assessment. Washington, DC: American Association of State Colleges and Universities.
  • Skrtic, T. M. (1985). Doing naturalistic research into educational organizations. In Y. S. Lincoln (Ed.), Organizational inquiry: The paradigm revolution (pp. 185-220). Beverly Hills, CA: Sage.

  • Contact:

  • James Guffey

  • Truman State University

  • Division of Mathematics & Computer Science

  • Kirksville, MO 6350l

  • (816) 785-4554

  • [email protected]

Tracey Sutherland is Director of Faculty Development at Truman State University where she works with a representative Faculty Development Committee to support her colleagues in their efforts in teaching, research and service. She is particularly interested in assessment as a vehicle for faculty development, student learning styles, and investigating the effectiveness of various approaches to encouraging active learning.

James Guffey is associate professor of mathematics in the Division of Mathematics and Computer Science at Truman State University. From 1994 to 1997, he served as chair of the Faculty Development Committee, an advisory committee to the Director of Faculty Development. He has also been active in institutional assessment for a number of years. Other professional interests include nonhomogeneous Poisson processes, reliability theory, and the use of technology in the classroom.