Assessment Literacy in College Teaching:
Empirical Evidence on the Role and Effectiveness of a Faculty Training Course

Assessment in Postsecondary Education

Student assessment in PSE continues to undergo reforms as constructivist teaching methodologies are more widely adopted, as accountability requirements intensify, and as assessment is increasingly recognized as having potential to improve teaching and learning (Taras, 2005, 2007). Requirements from accreditors and other agencies have increased institutional efforts to measure student learning, making assessment more important than ever. It is not likely that the focus of accreditation requirements on assessment will diminish any time soon, so institutional efforts to document, understand, and improve student learning must continue (Krzykowski & Kinser, 2014). Additionally, earlier empirical research has documented the importance of assessment as part of the teaching and learning process and its key influence on student learning (Fernandes et al., 2012; Struyven et al., 2005; Webber, 2012). As the demand for assessment literacy among postsecondary instructors increases, there is a greater need to understand how new faculty members develop their competency in using effective and contemporary classroom assessments within their instructional practice.

Advances in cognitive science have enhanced our understanding of how students learn and have had significant implications for assessment practices. Research over the last several decades has supported an increased focus on constructivist teaching methods that help students to actively assimilate and accommodate new material as a result of active engagement in authentic learning experiences and feedback-driven learning tasks (Black & Wiliam, 1998; Gardner, 2006). Compared to traditional instructional approaches, in which students are seen as passive receivers of information, current thinking about how students learn elevates assessment beyond its summative means to a central component of active learning. In this view, students are provided with ongoing feedback on their learning in relation to defined learning outcomes. These ongoing assessments are formative (i.e., not graded) and used as the basis for remediation or acceleration of learning. In addition, students are encouraged to take ownership of their own learning trajectories by using assessment information to direct their study. This contemporary view of assessment has required institutions to promote new assessment practices and thus support faculty members in designing and integrating constructivist assessment methods aimed at enhancing student learning, not just measuring it (Shepard, 2000). In PSE, instructors must balance this newer formative conception of assessment with traditional summative functions in order to support as well as evaluate student learning. In the following section, we delineate various conceptions of assessment that faculty members must negotiate as they aim to integrate assessment throughout their instructional practice.

The Role of Instructional Development

Until only recently, the predominant assumption with regard to preparing PSE faculty for their role as teachers has been that a trained and skilled researcher or subject matter expert would naturally be an effective teacher. Seen as dichotomous by some, the enduring struggle between research preparation and teaching preparation still lingers today; however, it is now generally acknowledged that at least some direct preparation for teaching is essential (Austin, 2003; Hubball & Burt, 2006). As a result, intentional efforts to offer training to faculty members to help them improve their teaching competencies have become increasingly prevalent within PSE institutions. While terms such as professional development and faculty development typically refer to initiatives concerning the entire career development of faculty (Centra, 1989), including activities designed to enhance one’s teaching, research, and administration activities (Sheets & Schwenk, 1990), the more specific term instructional development refers to activities explicitly aimed at developing faculty members in their role as teachers (Stes et al., 2010).

Several factors have persuaded PSE leaders in the last few decades that instructional development programs are worthwhile initiatives. Specifically, evidence has been accumulating of the ineffectiveness of the traditional, teacher-centered ways of teaching (Gardiner, 1998). Lectures, for example, have been shown to only support lower levels of learning, and students do not retain even this knowledge very long (Bligh, 2000). To help faculty members adopt contemporary pedagogical strategies and improve their teaching practice, a variety of instructional development programs and activities have been designed and implemented at colleges and universities. These activities have included formats such as workshops and seminars, newsletters and manuals, consulting, peer assessment and mentoring, student assessment of teaching, classroom observations, short courses, fellowships, and other longitudinal programs (Prebble et al., 2004; Steinert et al., 2006). Many studies have emerged investigating the effects of instructional development initiatives, and the majority report positive outcomes. Research reviewed by Stes et al. (2010), together with more recent studies, have demonstrated positive impacts on faculty in terms of their attitudes toward teaching and learning (Bennett & Bennett, 2003; Howland & Wedman, 2004), conceptions of teaching (Gibbs & Coffey, 2004; Postareff et al., 2007), learning new concepts and principles (Howland & Wedman, 2004; Nasmith et al., 1997), and acquiring or improving teaching skills (Dixon & Scott, 2003; Pfund et al., 2009). Previous research has also examined the impacts of instructional development on students and have demonstrated positive changes in student perceptions (Gibbs & Coffey, 2004; Medsker, 1992), behaviors (Howland & Wedman, 2004; Stepp-Greany, 2004), and learning outcomes (Brauchle & Jerich, 1998; Condon et al., 2016; Perez et al., 2012; Stepp-Greany, 2004).

Studies have consistently demonstrated that faculty development programs are generally assessed using only superficial means (e.g., Beach et al., 2016; Chism & Szabó, 1998; Hines, 2009). Findings from Beach et al. (2016) indicate that as the complexity of the assessment approach increases (e.g., measuring the change in teaching practice or student learning), the percentage of use declines. Specifically, the reporting of the effects of faculty development programs more often takes the form of tracking participation numbers and participant self-report satisfaction and rarely includes data on an increase in the knowledge or skills of participants or a change in the practice of participants, even more rarely paying attention to changes in the learning of the students served by participants and/or changes in the institution’s culture of teaching. The recent work of Condon et al. (2016) offers a rare example of a sustained effort to provide measurable evidence that faculty development leads to improved instruction and improved student learning.

Conceptions of Assessment

Assessment serves a variety of purposes and can be implemented in many forms. As an umbrella term, understandings of assessment vary, depending upon how faculty members see the role of assessment in relation to student learning. Within PSE contexts, there is a core difference in conceptions of assessment when learning is situated within an “assessment culture” versus a “testing culture” (Birenbaum, 1996, 2003). The traditional testing culture is based on the behaviorist model of learning, focusing on objective and standardized testing with the practice of testing as separate from instruction (Shepard, 2000). In this testing culture, multiple-choice and close-ended assessments that focus on memorized knowledge are typical test formats. In contrast, an assessment culture shifts assessment toward a socio-constructivist learning paradigm in which assessments are opportunities for learning as mediated by other students’ learning and teachers’ responsive instruction (Shepard, 2000). In such a culture, assessments are used both formatively and summatively but are reflective of the enacted curriculum and students’ constructed understandings (Birenbaum, 2003).

Given these paradigmatic views of assessment, faculty members may espouse specific orientations toward assessment within their context of practice. Drawing on the work of DeLuca et al. (2013), we explicate four conceptions of assessment that shape faculty members’ practices. These conceptions are (a) assessment as testing, (b) assessment as format, (c) assessment as purpose, and (d) assessment as process.

The conception of assessment as testing is characterized as regarding assessment primarily as measuring the success of knowledge transmission. This conception of assessment is common among educators who regard teaching as simply the transmission of knowledge and are therefore likely to view assessment as an activity that follows learning, rather than as an integral and formative element of learning itself (Parpala & Lindblom-Ylänne, 2007). Testing instruments typically used in PSE include written tests given at the end of a term, chapter, semester, or year for grading, evaluation, or certification purposes. Such assessments often include closed-ended questions, such as multiple-choice, true/false, and fill-in-the-blank questions, as well as open-ended response questions with distinct and narrow correct answers. Such testing practices often focus largely on memorizing and recalling facts and concepts and are used primarily for evaluation in which there is limited or no feedback beyond the reporting of a grade or score. The conception of assessment as testing is rooted in the dominant paradigm of the 20th century, which Shepard (2000) associates with behaviorist learning theories, social efficiency, and scientific measurement, all of which negate the formative role of assessment in teaching and learning.

In contrast to an understanding of assessment as testing, an assessment as format conception is based on the understanding that assessment can take on a variety of forms, beyond the narrow view of testing. The methods used by faculty to assess student learning in PSE has expanded considerably in recent decades. Once characterized only by multiple-choice tests, traditional essays, and research papers (Sambell et al., 1997), assessment in the PSE context now often includes portfolios, projects, self- and peer-assessment, simulations, and other innovative methods (Struyven et al., 2005). If assessment tasks are representative of the context being studied and both relevant and meaningful to those involved, then they may be described as authentic. The authenticity of tasks and activities is what sets innovative assessments apart from more traditional and contrived assessments (Birenbaum, 2003; Gulikers et al., 2004). Flexibility in assessment formats can be seen as a first step toward a more student-led pedagogy that engages students in the assessment process (Irwin & Hepplestone, 2012).

When instructors move beyond conceptualizing assessment as strictly for summative purposes, they begin to understand assessment as serving multiple important functions. An assessment as purpose orientation encourages multiple uses for assessment within teaching, learning, and administrative processes. Following Bloom et al. (1971), a distinction is typically made between formative and summative assessment. Summative assessment is concerned with determining the extent to which curricular objectives have been received, whereas formative assessment is conceptualized as the “systematic evaluation in the process of curriculum construction, teaching, and learning for the purposes of improving any of these three processes” (Bloom et al., 1971, p. 117). The distinction between formative and summative assessment, however, is often blurred (Boud, 2000). For example, assessments activities such as course assignments are often designed to be simultaneously formative and summative. Activities can be formative because the student is provided feedback from which to learn, and they can be summative because the grade awarded contributes to the overall grade for the course.

The final conception, assessment as process, is closely linked to assessment as, for, and of learning and conceptualizes assessment as a process of interpretation and integration. Pellegrino et al. (2001) state that “assessment is a process of reasoning from evidence” (p. 36). While recognizing that assessments used in various contexts, for differing purposes, often look quite different, this conception of assessment focuses on the fact that all assessment activities share the common principle of always being a process of reasoning from evidence. To conceive of assessment in this way is to understand that without a direct pipeline into a student’s mind, the process of assessing what a student knows and can do is not straightforward. Various tools and techniques must be employed by educators to observe students’ behavior and generate data that can be used to draw reasonable inferences about what students know (Popham, 2013).

Although there are other taxonomies for categorizing and conceptualizing assessment in PSE, these four conceptions––assessment as testing, assessment as format, assessment as purpose, and assessment as process––highlight several dominant conceptions that drive teaching practice in universities and colleges. In this research, we used these four conceptions of assessment as an analytic heuristic in order to analyze faculty members’ changing conceptions of assessment in response to an instructional training course.

Confidence in Assessment

Arguably, to apply one’s understanding of assessment effectively, a postsecondary instructor must also be confident in their assessment knowledge and skills. The construct of self-efficacy in social cognitive theory (Bandura, 1986) suggests that when an individual feels competent and has confidence in completing a task, they will choose to engage in it. In contrast, when an individual feels incompetent and lacks confidence in a task, they will avoid engaging in it. It is therefore critical that faculty members not only develop complex and more nuanced conceptions of assessment but also feel confident in their assessment abilities in order to fully engage in the process of assessment. As Bandura (1986) explains, several factors such as practice, exposure, and application contribute to how confident an individual is when using or applying a skill to a new context. This means that when college instructors have a solid understanding of how to construct a grading rubric that both communicates expectations to students and guides instructional decisions, for example, they are more likely to be successful in their use and application of assessment. Therefore, to create assessment-literate educators, assessment confidence must be addressed.

Although not yet widely explored in the context of PSE, assessment confidence is a concept considered in K–12 education literature, often understood as a teacher’s self-perceived confidence in administering various approaches of assessments (Ludwig, 2014; Stiggins et al., 2004). In their study investigating the assessment confidence of preservice teacher candidates, DeLuca and Klinger (2010) conceptualized three domains of assessment confidence: —confidence in assessment practice, assessment theory, and assessment philosophy. DeLuca et al. (2013) identified two principal components of assessment confidence: confidence in using practical assessment approaches to measure student learning and confidence in engaging in assessment praxis. Praxis, as differentiated from practice, is the dynamic interplay between thought and action involving a continuous process of understanding, interpretation, and application.

Context

This research is based on an instructor training course required for all newly hired faculty at the college and was also open to other faculty members at their department chair’s request. This semester-long course focused on teaching and learning styles, strategies to enhance student engagement and motivation, and various approaches to student assessment. Faculty participants attended a two-hour face-to-face class session each week that was facilitated by two faculty development officers. These class sessions involved active participation with numerous opportunities for participants to practice teaching and learning techniques and receive feedback from peers and facilitators. Additionally, the course included a variety of weekly assignments, which participants accessed and completed online. These activities included readings, quizzes, discussion forums, and practical assignments such as designing instructional materials (e.g., syllabi, lesson plans, and assessment instruments).

Participants

After obtaining institutional review board (IRB) approval, all 35 instructors enrolled in the summer 2013, fall 2013, and spring 2014 administration of the instructor training course were invited to participate in this study. Of the 35 instructors, 27 agreed, for a 77% response rate. Twenty-two (82%) of the participants were male, and five (18%) were female (see Table 1). These 27 instructors represented a cross-section of the college’s instructional departments, with representation from programs such as welding, building construction, culinary arts, computer graphics, landscaping, electronics, and aviation. Most of the participants had no prior teaching experience, but eight reported some teaching experience including some combination of postsecondary (7), K–12 (3), or other teaching (4), such as having worked as a military instructor, dance teacher, or flight instructor.

Data Collection

Data were collected from participants through two linked questionnaires: one administered during the first day of the instructor training course and one administered on the last day of the course. The questionnaires, adapted from the instrument used by DeLuca et al. (2013), consisted of both open-ended and close-ended items. Open-ended items asked participants to describe their conceptions of assessment (e.g., When you think about the word assessment what comes to mind?). Close-ended items asked participants to identify their confidence in (a) using practical assessment approaches to measure student learning (e.g., testing, performance assessment, portfolios, and observation) and (b) engaging in assessment praxis as represented through various assessment scenarios. Each paper-based questionnaire included 25 items, and the pre-questionnaire also had a seven-item demographic section. Pre- and post-questionnaires were completed by participants in a familiar classroom setting.

Data Analysis

Results were analyzed in relation to participants’ changing (a) conceptions of assessment and (b) confidence in assessment. Data analyses were modeled after processes implemented by DeLuca et al. (2013). In order to determine participants’ changing conceptions of assessment over time, their open-response questionnaire responses before and after the instructional development course were coded through standard, deductive coding (Patton, 2002) based on the theoretical definitions of conceptions of assessment. Specifically, data were coded based on participants’ primary and, when available, secondary conceptions of (a) assessment as testing, (b) assessment as format, (c) assessment as purpose, and (d) assessment as process. Data were independently coded by two raters with an inter-rater reliability of 97%. Frequencies for primary and secondary conception codes were based on the number of individual participants who mentioned a conception rather than the total number of times a conception appeared in the questionnaire data (Namey et al., 2008).

A primary code frequency table matrix for both pre- and post-questionnaire responses was constructed. Codes were then assigned numerical values in order of increasing complexity (testing = 1, format = 2, purpose = 4, and process = 8), and each participant received a numerical score representing assessment conception on pre- and post- open-response questionnaire items. As appropriate, dual codes were combined to create a composite assessment conception score. Pre- and post-assessment conception scores were summarized in a frequency table. Dependent t tests were conducted to determine significant differences between participants’ pre- and post-questionnaire assessment conception composite scores.

Descriptive statistics were conducted on fixed-response questionnaire items pertaining to two scales of assessment confidence in practical assessment approaches and assessment praxis. Pre-questionnaire items pertaining to each assessment confidence scale were then analyzed through principal components analysis using oblique (promax) rotation to determine subscales. A minimum subscale loading was set at 0.40, and Cronbach’s alpha values were calculated as internal consistency estimates for each subscale. Means for each identified subscale were then calculated for pre- and post-questionnaire responses and compared using dependent t tests to determine significant differences between participants’ responses before and after the instructional development course. Open-response items pertaining to assessment confidence were analyzed to enrich understanding of these fixed-response items.

The small sample size limits the reliability of the measure, the internal consistency of subscale component loadings, and the resulting effect sizes. These limitations reduce the generalizability of results from the principal component analyses and t tests. However, despite the small sample, this study contributes validation evidence of trends revealed in the work of DeLuca et al. (2013) and provides an initial basis for examining faculty members’ changing conceptions of and confidence in assessment as a result of an instructional development course.

Conceptions of Assessment

Participants’ conceptions of assessment developed from an initial view of assessment as testing to more complex conceptions of assessment as purpose and process. Table 2 summarizes the frequency distribution of participants’ conceptions of assessment for pre- and post-questionnaire responses. The frequency of assessment as testing decreased most from pre- to post-questionnaire implementation (19 to 3), whereas the frequency of assessment as process increased most (0 to 9). Assessment as format was not mentioned as a primary concern on either implementation of the questionnaire, and assessment as purpose was the most frequently reported conception on the post-questionnaire (15).

In pre-questionnaire open responses, participants’ descriptions of assessment were simplistic and concise in nature, emphasizing the summative role of assessment. Pre-questionnaire responses generally described assessment as a means to “test students,” “measure learning,” and “evaluate what students know.” Post-questionnaire responses consistently demonstrated participants’ richer and expanding conceptions of assessment that included both purpose and process. Furthermore, all participants’ post-questionnaire responses shifted beyond primarily summative conceptions of assessment to include the formative role of assessment as well. As one participant explained on the post-questionnaire, assessment serves the purpose of “examining student learning strengths and needs, telling students how to improve, and guiding teachers’ next steps.” Another participant described assessment on the post-questionnaire as a process that allows educators to “evaluate students’ understanding of lessons, evaluate teacher effectiveness, and identify areas for improvement in teaching and learning processes.” Overall, these results show that by the end of the instructional development course, participants’ primary conceptions of assessment had shifted to emphasize assessment as a formative process that supports both teaching and learning.

A frequency distribution of participants’ composite scores for conceptions of assessment is reported in Table 3. The most frequent composite score on the pre-questionnaire was associated with the conception of assessment as testing, whereas assessment as purpose was the most frequent conception on the post-questionnaire. Specifically, the most frequently mentioned purpose of assessment on the post-questionnaire was formative. Interestingly, few participants reported multiple conceptions of assessment: assessment as testing and purpose (3 and 5 on pre- and post-questionnaires, respectively) and assessment as format and purpose (2 on pre-questionnaire). This illustrates that despite participants’ increasingly complex conceptions of assessment, they were not systematically describing assessment as a multifaceted concept that encompasses testing, format, purpose, and process.

At an individual level, 25 of 27 participants’ assessment conception composite scores increased over time while two remained the same. A dependent t test revealed that changes in participants’ conception of assessment composite scores over time were statistically significant from pre- (M = 2.37, SD = 1.88) to post-questionnaire (M = 5.56, SD = 1.83; t (26) = -9.20, p < .05) (see also Table 5). Qualitative questionnaire responses elaborated on this finding. One participant initially described assessment as “a college requirement to measure learning.” On the post-questionnaire, the same participant explained that assessment serves to “support the learning process, inform teaching practice, and provide constructive feedback to students.” Another participant’s responses shifted from “finding out if students get it” to “evaluating students’ learning of content, and supporting the learning process by providing feedback to students and teachers.” These examples illustrate an important shift as participants began to view assessment as less of an instructor-led measurement requirement and more of an interactive process among instructors and students to guide teaching and learning.

Confidence in Assessment

Principal component analyses conducted on each confidence scale—practical assessment approaches (11 items) and assessment praxis (12 items)—revealed three subscales in each scale with eigenvalues greater than 1 (see Table 4). Internal consistency values associated with each subscale were determined to be adequate (α = .68 to .84). Items related to the practical assessment approaches scale were named instructor-led assessments (α = .84), interactive assessments (α = .69), and portfolio assessments (α = n/a; 1 item). Items related to the assessment praxis scale were named: rationalizing assessment practices and decisions (α = .81), explaining and using assessment theory (α = .81), and aligning assessment to curriculum and standards (α = .68).

Mean values and dependent sample t tests for participants’ overall confidence in assessment across both confidence scales before and after the instructional development course, as well as confidence in practical assessment approaches and assessment praxis, are reported in Table 5. There was a significant change in participants’ overall confidence from pre- (M = 2.78, SD = .75) to post-questionnaire administration (M = 4.19, SD = .68; t (26) = -7.11, p < .05). All individual assessment confidence subscales also exhibited significant increases (see Table 5).

Open-response items pertaining to assessment confidence were analyzed to enrich understanding of these fixed responses. It is interesting to note that increases in assessment confidence differed based on participants’ previous teaching experience. For example, the three participants with previous K–12 teaching experience and four of the participants with PSE teaching experience reported that the instructional development course had the greatest impact on their ability to explain and use assessment theory but had minimal impact on the other subscales related to assessment approaches and praxis. These participants began the course with relatively high levels of confidence on all practical assessment approach subscales (instructor-led, interactive, and portfolio assessments) and two of three assessment praxis subscales (rationalizing assessment practices and decisions and aligning assessment to curriculum standards). Although the small overall sample size and even smaller size of this subgroup limits any deeper analysis of the effect of previous teaching experience, these results may highlight the importance of differentiating assessment training initiatives for college instructors based on previous teaching experience.

The following figures illustrate how participants’ conceptions of assessment changed in relation to their increasing confidence in practical assessment approaches (see Figure 1) and assessment praxis (see Figure 2). As participants’ conceptions of assessment increased in complexity, their confidence in assessment approaches and praxis also increased. Taken together, these figures suggest the importance of instructional development courses designed to foster more complex conceptions of assessment in order to positively impact college instructors’ confidence in both practical assessment approaches and assessment praxis.

Figure 1. Comparison of Change Patterns in Participants’ Assessment Conceptions and Confidence in Practical Assessment Approaches

Figure 2. Comparison of Change Patterns in Participants’ Assessment Conceptions and Confidence in Assessment Praxis