Page  118 ï~~ 118 THE MICHIGAN BOTANIST Vol. 50 RELIABILITY OF WISCONSIN OAK SPECIES IDENTIFICATION BASED ON HERBARIUM SPECIMENS Jeffrey Lee Ralston 663 Yorkshire Road Neenah, WI 54956 ABSTRACT Identification of oak species is an important part of natural resource science and field work, but there exists little quantitative information about the reliability of individuals to make these identifications. This study examines the reliability among forestry students and faculty at The University of Wisconsin-Stevens Point when identifying oak species from herbarium specimens. Percent agreement ranged from 10 to 100 percent and inter-rater reliability was consistently fair. Both measures were highest for graduate students, the group with consistent field experience identifying oak species and a relatively high self-perceived level of competence. These results have implications when designing experiments or comparative studies, which require observers to identify oaks to the species level, and for forestry monitoring programs, which generate trend data using different staff over long time periods. KEYWORDS: Quercus section Lobatae, Fleiss' kappa, forestry students, percent agreement INTRODUCTION Researchers and workers in natural resources often identify oak trees to the species level (Johnson 1975; Stambaugh and Guyette 2008; Mujuri and Demchik 2009) and the Forest Service conducts annual surveys of forest resources that often rely on accurate identification of oak species (USDA Forest Service 2011). Quality assurance is a key component of these surveys. The reliability of oak species identifications has implications for interpretation of research findings, particularly those that rely on multiple observers and longitudinal studies. Currently there is no information in the published literature providing data regarding the agreement among observers identifying oak species. Within North America there are three sections of the genus Quercus L. (Nixon 1993; Jensen 1997): section Quercus L., section Protobalanus (Trelease) A. Camus, and section Lobatae G. Don. Within section Lobatae, five distinct species are recognized in Wisconsin: scarlet oak (Quercus coccinea Mtinchh.), northern pin oak (Quercus ellipsoidalis E.J. Hill), pin oak (Quercus palustris Mtinchh.), northern red oak (Quercus rubra L.), and black oak (Quercus velutina Lam.) (Fewless 2004; Freckmann 2010). Fertile hybrids between members of the same section, many of which are named, are known to occur (Palmer 1942; Jensen and Eshbaugh 1976a; Cottam et al. 1982, p. 47). The presence of trees with intermediate characters, or combinations of characters, supports the presence of putative hybrids (Jensen and Eshbaugh 1976a; Jensen and Eshbaugh 1976b). The treatment in Flora of North America (FNA) suggests that these species

Page  119 ï~~ 2011 THE MICHIGAN BOTANIST 119 can be readily distinguished if mature sun leaves, twigs, and mature buds are all observable (Jensen 1997). It is recommended that multiple characters be used to make species determinations because oak species are known to demonstrate wide character variability (Jensen 1988). While many species may have distinct characters, characters may be intermediate within sympatric populations (Palmer 1942; Knops and Jensen 1980; Jensen 1988). Many authors have lamented the difficulty of separating species because characters are variable and often form a continuum of size, shape, color, and pubescence (Palmer 1942; Jensen 1977; Hipp and Weber 2009). SPECIES IDENTIFICATION Scarlet oak is rare in Wisconsin. The University of Wisconsin-Stevens Point (UWSP) herbarium has two putative specimens, one from Rock County and one from Oconto County. Similarly, the University of Wisconsin-Madison (UWMadison) herbarium has three records (Cameron 2005). Scarlet oak hybridizes with black oak, producing Q. x fontana Laughlin. Scarlet oak also hybridizes with pin oak and northern red oak (Jensen 1997). Scarlet oak "has large buds that are pubescent on the upper half, (the) inner surface of the acorn cups is glabrous to sparsely pubescent and the acorns have concentric rings at their apices" (Jensen 1977). According to FNA (Jensen 1997) scarlet oak and northern pin oak have similar characters, however, in scarlet oak the acorn cup scale margins are strongly concave, rather than straight or slightly concave, the nut is subglobose versus ellipsoid, and the nut has one or more concentric rings of pits at the apex. Northern pin oak is common throughout Wisconsin. Northern pin oak "has small, sparsely pubescent buds, striate acorns, and the inner surface of the acorn cap (is) glabrous" (Jensen 1977). E. J. Hill (1899), in his original description of northern pin oak, stated that the species, in many characters, "suggests Q. velutina", and he considered the possibility of hybridization between scarlet, black or pin oak. While pin oak is commonly planted as a street tree in southern Wisconsin, in natural settings, it is a rare tree found in wet to moist habitats. Relative to northern red oak and black oak, pin oak leaves are small and the winter-buds are small and glabrous. Compared to other Wisconsin members of section Lobatae, the nut is small and short with a shallow cup and small, tightly appressed scales. The inner surface of the acorn cup is sparsely pubescent (commonly only around the scar where the nut was attached), and dense tufts of fine hairs occur in the main vein axils of the lower leaf surface (Palmer 1942; Jensen 1977). Northern red oak is found throughout Wisconsin. Some authorities, as noted by Jensen (1997), recognize two varieties: trees with large nuts with only onefourth of the nut covered by the cup are treated as Quercus rubra var. rubra and trees with smaller nuts and one-third of the nut covered by the cup are treated as Q. rubra var. borealis (F. Michx.) Farw. Northern red oak hybridizes with pin oak (Quercus x columnaris Laughlin), black oak (Q. x hawkinsiae Sudw.), and

Page  120 ï~~ 120 THE MICHIGAN BOTANIST Vol. 50 northern pin oak (Burns and Honkala 1990; Jensen et al. 1993). Northern red oak has large, glabrous winter-buds with a tuft of tomentum at the apex, ovate leaf lobes, and the inner surface of the acorn cups is glabrous to sparsely pubescent (Jensen 1977). It often has slightly drooping leaves that differ from other members of the section by the dull green upper surface (Palmer 1942). Black oak is restricted to the southern half of Wisconsin. Black oak hybridizes readily and named hybrids include Q. x fontana Laughlin (hybrid with Q. coccinea), Q. x palaeolithicola Trel. (hybrid with Q. ellipsoidalis), Q. x vaga Palmer & Steyerm. (hybrid with Q. palustris), and Q. x hawkinsiae Sudw. (hybrid with Q. rubra). Black oak has bright yellow inner bark, large winter-buds with grayish, pubescent scales, acorn cups with loose scales, and uniform pubescence on the inner surface of the acorn cups" (Palmer 1942; Jensen 1977). STUDY OBJECTIVE The objective of this study was to assess the percent agreement and the interrater reliability among forestry students and faculty at UWSP when identifying oak species. Students and faculty were asked to identify oak species from herbarium specimens that demonstrated an array of characters considered necessary to make an accurate identification. MATERIALS AND METHODS Ten-herbarium specimens were selected from the Robert W. Freckmann Herbarium on the campus of UWSP (Freckmann 2010). These ten specimens included five with multiple, distinct characters consistent with one of the five members of Quercus section Lobatae determined to be extant species in Wisconsin; one with multiple, distinct characters of a very familiar member of Quercus section Quercus, (bur oak, Q. marcrocarpa Michx.); and four with intermediate characters of the members of Quercus section Lobatae. Bur oak was included as a readily identifiable species to confirm that participants were acceptably engaged in the process. Characters were defined as distinct or intermediate based on a review of the literature (Palmer 1942; Jensen 1977; Jensen 1997; Hipp and Weber 2009). Each herbarium specimen had multiple characters represented, including abaxial and adaxial leaf surfaces, twigs with buds, and acorns (Figures 1 and 2). Each specimen included the county, date, and habitat of the collection site. All undergraduate forestry students at UWSP are required to complete FOR232 and FOR432. The former, Dendrology and Silvics, covers the ecology and classification of common upper Midwest trees and shrubs, while the latter, Silviculture, is designed to teach the principles governing establishment, treatment, and control of forest stands. FOR232 is a prerequisite for FOR432 (UWSP 2011). I attended all laboratory sections of FOR232 at the end of the 2010 fall semester and invited students to participate in the study during their assigned laboratory period. I invited all FOR432 students to participate in the study during two evening sessions at the end of the 2010 fall semester. These participants attended these sessions outside of their normally scheduled class time. I attended one UWSP forestry faculty meeting and invited all forestry faculty to participate in the study. All College of Natural Resource graduate students at UWSP with an emphasis in forestry were invited to participate. Faculty and graduate student participants attended one of the two evening sessions along with the FOR432 students or were provided additional opportunities at their discretion. Participants were provided with a data sheet and asked to identify each herbarium specimen, by sight, to the species level. References and keys were not available. Data sheets included the names of all six possible species. If a specimen demonstrated intermediate characters, participants were

Page  121 ï~~ 2011 THE MICHIGAN BOTANIST 121 2011 THE MICHIGAN BOTANIST 121 FIGURE 1. Herbarium specimen of Quercus rubra with multiple distinct characters. Percent agreement among graduate students was 100 percent. asked to select the species most representative of the specimen therefore each species name could potentially be recorded multiple times. Because of the similarity in common names, participants were asked specifically to avoid confusing northern pin oak and pin oak and were directed to use scientific names. Prior to starting identification, participants were asked to rate their self-perceived level of competence. Competence was on a scale of one to ten, relative to their peers, with ten being the most competent. No time limit was placed on participants. Prior to initiation, this research was reviewed and approved by the UWSP Institutional Review Board for the Protection of Human Subjects.

Page  122 ï~~ 122 THE MICHIGAN BOTANIST Vol. 50 122 THE MICHIGAN BOTANIST Vol. 50 FIGURE 2. Herbarium specimen with intermediate characters. Percent agreement among graduate students was 33 percent. STATISTICAL ANALYSIS Percent agreement was calculated for the six specimens with multiple, distinct characters consistent with one species. Percent agreement of 100 percent means that all participants identified the specimen to the species assigned based on distinct characters. The Fleiss kappa statistic (Fleiss 1971) was calculated for all participants, for students completing FOR232, for students completing FOR432, for graduate students, and for forestry faculty. Fleiss' kappa statistic is a measure of inter-rater reliability, corrected for chance agreement (Gwet 2010,

Page  123 ï~~ 2011 THE MICHIGAN BOTANIST 123 TABLE 1. Landis and Koch kappa benchmark scale. Kappa Level of Agreement <0.0 Poor 0.0 to 0.20 Slight 0.21 to 0.40 Fair 0.41 to 0.60 Moderate 0.61 to 0.80 Substantial 0.81 to 1.00 Almost perfect p. 26-28). Inter-rater reliability was calculated for all ten specimens and for the six specimens with multiple, distinct characters consistent with one species. A kappa coefficient of 1.00 means perfect agreement and a value of 0.00 means agreement equal to chance alone. The benchmark scale proposed by Landis and Koch (1977) was used to characterize the degree of agreement (Table 1). RESULTS Based on the entire population available for this study, participation was high among three of the four groups (Table 2). Percent agreement for the six specimens with distinct characters ranged from 10 to 100 percent (Figure 3). Percent agreement between graduate students was at or above 75 percent for five of the six specimens. Fifty percent of graduate students identified the northern pin oak (qe) specimen as black oak (qv) (Figure 4). By accepting black oak (qv) or northern pin oak (qe) for the specimens assigned northern pin oak (qe) and black oak (qv), respectively, the percent agreement increased substantially for most cases. FOR432 students identified the northern pin oak and the black oak specimens as pin oak (qp) 47 percent of the time. The specimen identified as pin oak (qp) was identified as scarlet oak (qc) 47, 47, and 37.5 percent of the time by FOR232, FOR432, and faculty, respectively. The specimen identified as scarlet oak (qc) was identified as black oak (qv) by 33 percent of FOR432 students and TABLE 2. Fleiss' kappa statistic for assessing inter-rater reliability, corrected for chance agreement. Raters assigned a species name to each of ten specimens. A list of six possible species names was provided to the raters. Six of the ten specimens had multiple characters consistent with one species and are listed as "specimen with good characters". Self reported competence, relative to peers, is on a scale of one to ten with ten being the most competent. Number of Percent All Specimen with Self perceived Rater status raters participation specimen good characters competence kappa kappa median range All 106 0.27 0.33 6 1-9 FOR 232 79 80 0.28 0.34 6 1-9 FOR 432 15 21 0.27 0.24 5.5 3-9 Graduate students 4 100 0.39 0.66 7 5-8 Forestry faculty 8 73 0.33 0.42 5 1-9

Page  124 ï~~ 124 THE MICHIGAN BOTANIST Vol. 50 124 THE MICHIGAN BOTANIST Vol. 50 E 100 90 80 70 60 40 30 20 10 \\\1 *FOR 232 I Specimen identification FIGURE 3. Percent agreement is the percent of raters in each group that correctly identified the six specimens with multiple characters consistent with that species. Participants were stratified as faculty, graduate students (Grad), students completing FOR 432 (FOR 432), and students completing FOR 232 (FOR 232). Species are bur oak (qm), northern red oak (qr), northern pin oak (qe), black oak (qv) pin oak (qp), and scarlet oak (qc). E 10' 70 60 20 40 11l I.AO:$3$ q qo qv 2 qvor q Specimen identification FIGURE 4. Percent agreement is the percent of raters in each group that correctly identified the specimen. Participants were stratified as faculty, graduate students (Grad), students completing FOR 432 (FOR 432), and students completing FOR 232 (FOR 232). Species are northern pin oak (qe), and black oak (qv). By accepting qv or qe for specimen qe and qv, respectively, the percent agreement increased substantially for most cases.

Page  125 ï~~ 2011 THE MICHIGAN BOTANIST 125 50 percent of faculty. Inter-rater reliability was consistently fair (Table 2). Interrater reliability for the six specimens with distinct characters was moderate for forestry faculty and substantial for graduate students. Graduate students had the highest median self-perceived competence and forestry faculty had the lowest (Table 2). DISCUSSION The Cohen kappa statistic was introduced to measure the degree of agreement between two observers for nominal scale data (Cohen 1960). Fleiss extended this statistic to the measurement of agreement among multiple observers (Fleiss 1971). Initially applied to psychiatric diagnosis, the Fleiss' kappa coefficient is commonly used to assess the inter-observer reliability of various classification systems in the medical literature (Brumback and Jones 1994; Malek et al. 2006; Maripuri et al. 2008). Orthopedic surgeons, who are highly trained professionals, categorize fractures based on accepted radiographic criteria. The mean kappa value among orthopedic surgeons classifying a common fracture was 0.61 (Malek et al. 2006). The kappa coefficient has also been used in natural resources, e.g., to evaluate inter-observer agreement between volunteers and experts for amphibian point counts conducted in Ontario, Canada. The kappa coefficient was approximately 0.75, which the authors interpreted as good to strong agreement among observers (Shirose et al. 1997). More commonly, natural resource investigators have reported percent correct or percent agreement based on a standard (Genet and Sargent 2003; Pierce and Gutzwiller 2007). Volunteers doing frog and toad surveys in Michigan were evaluated by mail survey which included a questionnaire and an audio CD. Twelve sites were represented on the CD and volunteers correctly identified species greater than 80 percent of the time for most sites. However, on six of the twelve sites responses significantly differed from correct identifications. Ability to make correct identification was not related to experience (Genet and Sargent 2003). Two trained observers in Texas agreed on the species identification during 269 frog call surveys 79 percent of the time (Pierce and Gutzwiller 2007). The reliability of species identification in forestry has not been examined closely. Data collection by volunteers in Brookline, MA has been studied as part of an urban forestry initiative (Bloniarz and Ryan III 1996). Ninety-seven volunteers were recruited from the community and participated in a twelve-hour training program. Volunteers documented the species of 11,250 trees. Two certified arborists re-examined 473 randomly chosen trees. Percent agreement between the volunteers and the arborists was greater than 90 percent at the genus level and ranged from 46 to 96 percent at the species level (70 percent for Quercus spp.). Genera included were Acer L., Fraxinus L., Quercus, Platanus L., Gleditsia L., and Tilia L. In the winter of 2010/11, I examined the entire collection of Quercus section Lobatae in the Robert W. Freckmann Herbarium (Freckmann 2010) on the campus of UWSP (unpublished data). Protocol requires that the curator review all

Page  126 ï~~ 126 THE MICHIGAN BOTANIST Vol. 50 specimens prior to placement in the herbarium. I examined 406 specimens and 46 were annotated by me based on the literature reviewed in the "species identification" section above. The most common reclassification, 25 specimens, was from northern pin oak to black oak. The identification of oak species can be complicated by many factors and multiple characters may be necessary to make the identification (Jensen 1997). Oak leaves show substantial differences in sun and shade leaves (Barnes and Wagner Jr. 2007, p. 9-11), and hybrids may have intermediate characters (Palmer 1942; Jensen and Eshbaugh 1976a; Jensen 1988; Jensen et al. 1993). Site conditions may also contribute to heterogeneity. In the field certain characters may not be present at the time of identification or may be inaccessible (Barnes and Wagner Jr. 2007). During vegetative monitoring, individuals of Quercus section Lobatae are often treated as one category because of difficulty distinguishing the species (Bray and Curtis 1957; Curtis 1959, p. 338; Cassidy 2007). Results from this investigation demonstrate that percent agreement among observers was generally low when identifying oak species from herbarium specimens despite the presence of multiple characters. Most participants correctly identified bur oak, the only representative of section Quercus. Despite the presence of multiple, distinct characters, northern red oak was misidentified by more than 20 percent of participants. Distinguishing northern pin oak and black oak can be challenging and undergraduate students did poorly. Agreement improved when either species was accepted as correct; however, percent agreement remained at or below 50 percent for undergraduate students. The agreement between observers for all specimens was fair. If specimens with intermediate characters were removed from the analysis, agreement improved for all observers, although undergraduate student agreement remained poor. The four graduate students have each had at least one full season in the field identifying oak species. This was reflected in the narrower range of selfperceived competence and in the substantial agreement between the graduate students in categorizing species based on distinct characters. UWSP forestry faculty have various areas of interest, including forest ecology, forest biometry, resource management, urban forestry, economics, and forest soils. Experience identifying trees is variable and is reflected in the broad range of self-perceived competence and the moderate strength of agreement among observers. While pin oak is planted on the UWSP campus, students and faculty have little experience with pin oak or no experience with scarlet oak. Although, the UWMadison herbarium (Cameron 2005) does house several collections of scarlet oak, the existence of this species in Wisconsin is in dispute (Rich Hauer, pers. comm., UWSP, Nov. 10, 2010). Students have considerable exposure and experience with northern red oak and northern pin oak. Despite being relatively common in southern and central Wisconsin, black oak is not emphasized in FOR232, thus differences in education and field experience may partially account for the results reported. Because participants in this study were not chosen randomly from a larger population, these results apply to the participants only and should be generalized with caution. The relatively low level of participation among FOR432 students

Page  127 ï~~ 2011 THE MICHIGAN BOTANIST 127 reflects the requirement that these students attend one of two sessions outside of their assigned class times, while FOR232 students could participate during their normally scheduled class time. Many FOR432 students incorrectly identified several specimens as pin oak. While cautioned to avoid confusing northern pin oak with pin oak it is possible that this error was made. These results have implications when designing studies that require observers to identify oaks to the species level. This may be particularly relevant for longitudinal studies that rely on the ability of multiple observers, spaced over long periods of time, to make similar identifications. Supervisors of monitoring programs such as the Forest Inventory and Analysis Program, which generate trend data using different staff over long time periods, may need to be cognizant of these findings. Future studies could include forestry professionals and identification of trees in the field. Selecting a random group of individuals from a larger population would allow for less limited generalizations. ACKNOWLEDGEMENTS I thank Richard Hauer, Ph.D. for his support and interest. Rich welcomed me into the Dendrology and Silvics laboratory, which accounts for the high percentage of participation of students in this study. I also thank Emmet J. Judziewicz, Ph.D. for generously offering use of herbarium specimens. LITERATURE CITED Barnes, B. V. and Wagner Jr., W. H. (2007). Michigan trees: A guide to the trees of the Great Lakes region. The University of Michigan Press. Ann Arbor, MI. 447 pp. Bloniarz, D. V. and Ryan III., H. D. P. (1996). The use of volunteer initiatives in conducting urban forest resource inventories. Journal of Arboriculture 22(2): 75-82. Bray, J. R. and Curtis J. T. (1957). An ordination of the upland forest communities of southern Wis consin. Ecological Monographs 27(4): 326-349. Brumback, R. and Jones, A. (1994). Interobserver agreement in the classification of open fractures of the tibia. The results of a survey of two hundred and forty-five orthopaedic surgeons. Journal of Bone and Joint Surgery Am 76(8): 1162-1166. Burns, R. M. and Honkala, B. H. (1990). Silvics of North America: 1. Conifers; 2. Hardwoods. Agri culture handbook 654. U.S.D.A., Forest Service, Northern Research Station. Washington, DC. Cameron, K. (2005). Wisconsin state herbarium. Available online at http://www.botany.wisc.edu/ herbarium/; last accessed Mar. 07, 2011. Cassidy, D. S. (2007). Impacts of a tornado event on the structure and composition of the herbaceous layer of an oak-pine barrens plant community in Quincy Bluff State Natural Area. M. Sc. thesis, University of Wisconsin-Madison. Madison, WI. 217 pp. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Mea surement 20(1): 37-46. Cottam, W. P., Santamour, F. S., and Tucker, J. M. (1982). Oak hybridization at the University of Utah. State Arboretum of Utah Publ. No. 1, Salt Lake City, UT. 82 pp. Curtis, J. T. (1959). The vegetation of Wisconsin. The University of Wisconsin Press. Madison, WI. 657 pp. Fewless, G. (2004). Key to the trees of Wisconsin. Available online at http://www.uwgb.edu/ biodiversity/herbarium/trees/tree-key049b.htm; last accessed Jul. 27, 2010. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin 76(5): 378-382. Freckmann, R. W. (2010). Robert W. Freckmann Herbarium University of Wisconsin - Stevens Point Available online at http://wisplants.uwsp.edu/; last accessed Jul. 20, 2010. Genet, K. S. and Sargent, L. G. (2003). Evaluation of methods and data quality from a volunteer based amphibian call survey. Wildlife Society Bulletin 31(3): 703-714.

Page  128 ï~~ 128 THE MICHIGAN BOTANIST Vol. 50 Gwet, K. L. (2010). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among observers. Advanced Analytics, LLC. Gaithersburg, MD. 197 pp. Hill, E. J. (1899). A new biennial-fruited oak. Botanical Gazette 27(3): 204-208. Hipp, A. L. and Weber, J. A. (2009). Taxonomy of Hill's oak (Quercus ellipsoidalis: Fagaceae): Ev idence from AFLP data. Systemic Botany 33(1): 148-158. Jensen, R. J. (1977). A preliminary numerical analysis of the red oak complex in Michigan and Wis consin. Taxon 26(4): 399-407. Jensen, R. J. (1988). Assessing patterns of morphological variation of Quercus spp. In mixed-oak communities. American Midland Naturalist 120(1): 120-135. Jensen, R. J. (1997). Fagaceae: Quercus section Lobatae. Available online at http://www. efloras.org/florataxon.aspx?florajid=1&taxonid=302020; last accessed Jul. 20, 2010. Jensen, R.J. and Eshbaugh, W. H. (1976a). Numerical taxonomic studies of hybridization in Quer cus. I. Populations of restricted areal distribution and low taxonomic diversity. Systematic Botany 1(1): 1-10. Jensen, R. J. and Eshbaugh, W. H. (1976b). Numerical taxonomic studies of hybridization in Quer cus. Ii. Populations with wide areal distributions and high taxonomic diversity. Systematic Botany 1(1): 11-19. Jensen, R. J., Hokanson, S. C., Isebrands, J. G., and Hancock, J. F. (1993). Morphometric variation in oaks of the Apostle Islands in Wisconsin: Evidence of hybridization between Quercus rubra and Q. ellipsoidalis (Fagaceae). American Journal of Botany 80(11): 1358-1366. Johnson, P. S. (1975). Growth and structural development of red oak sprout clumps. Forest Science 21(4): 413-418. Knops, J. F. and Jensen, R. J. (1980). Morphological and phenolic variation in a three species com munity of red oaks. Bulletin of the Torrey Botanical Club 107(3): 418-428. Landis, J. R. and Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics 33(1): 159-174. Malek, I. A., Machani, B., Mevcha, A. M., and Hyder, N. H. (2006). Inter-observer reliability and intra-observer reproducibility of the weber classification of ankle fractures. Journal of Bone and Joint Surg Br 88(9): 1204-1206. Maripuri, S. N., Rao, P., Manoj-Thomas, A., and Mohanty, K. (2008). The classification systems for tibial plateau fractures: How reliable are they? Injury 39(10): 1216-1221. Mujuri, E. and Demchik, M. C. (2009). Northern pin oak stump sprouting frequency on scrub oak sites of central Wisconsin. Northern Journal of Applied Forestry 26(2): 83-85. Nixon, K. (1993). Infrageneric classification of Quercus (Fagaceae) and typification of sectional names. Annals of Forest Science 50(Supplement): 25s-34s. Palmer, E.J. 1942. The red oak complex in the United States. American Midland Naturalist 27(3): 732-740. Pierce, B. A. and K.J. Gutzwiller, K. J. (2007). Interobserver variation in frog call surveys. Journal of Herpetology 41(3): 424-429. Shirose, L.J., Bishop, C. A., Green, D. M., Cameron, J. M., Brooks, R. J., and Helferty, N. J. (1997). Validation tests of an amphibian call count survey technique in Ontario, Canada. Herpetologica 53(3): 312-320. Stambaugh, M.C. and Guyette, R. P. (2008). Prescribed fire effects on the wood quality of three com mon oaks in the ozark region, Quercus coccinea, Q. velutina, and Q. alba. University of Missouri, Department of Forestry, Columbia. USDA Forest Service. (2011). Forest inventory and analysis national program. Available online at http://fia.fs.fed.us/; last accessed Mar. 07, 2011. UWSP (2011). University of Wisconsin-Stevens Point course catalog. Available online at http://www.uwsp.edu/news/uwspcatalog/forestry.htm; last accessed Mar. 07, 2011.