spobooks5621225.0001.001 in
    Improved budgeting with improved usage forecasts

    We first estimate an upper bound on how much the libraries could benefit from better usage data. We analyze each institution's accesses to determine what would have been its optimal bundle if it had been able to perfectly forecast which material would be accessed. We then calculate how much this bundle would have cost the institution, and compare this perfect foresight cost with the institution's actual expenditures. Obviously even with extensive historical data, libraries would not be able to perfectly forecast future usage, so the realized efficiencies from better usage data would be less. Below we analyze how the libraries used the information from 1998 to change their purchasing decisions in 1999.

    We present these results by access product in Table 6.10. We found that actual expenditures were markedly higher than optimal purchases in 1998. In particular, institutions in the Red and Blue groups purchased far more traditional subscriptions than would be justified if they had perfect foresight. Most institutions purchased more generalized subscriptions than would have been optimal with perfect foresight. We believe that much of the budgeting "error" can be explained by a few factors:

    • First, institutions overestimated demand for access, particularly for journals for which they purchased traditional subscriptions.[21]

    • Second, institutional practices, such as "use it or lose it" budgeting and a preference for fixed, predictable expenditures, might have affected decisions. A preference for predictable expenditures would induce a library to rely more heavily on traditional and generalized subscriptions, and less on reimbursed individual article purchases or interlibrary loan.[22] However, Kantor et. al. (2001) Kantor et al. (this volume) report the opposite: that libraries dislike bundles because they perceive them as forcing expenditures for low-value items.

    • Third, because demand foresight is necessarily important, libraries might want to "over-purchase" to provide insurance against higher than expected usage demand. Of course, per-article purchases (possibly reimbursed to users) provide insurance (as does an interlibrary loan agreement), but at a higher cost per article than pre-purchased generalized subscription tokens, or than traditional subscriptions.

    Table 6.10: Actual versus optimal expenditures per access product for 1998-1999
    Access Product Totals
    Traditional Generalized Per Article
    Year Instid Actual Optimal Actual Optimal Actual Optimal Actual Optimal $ Savings % Savings
    1998 3 25,000 17,000 2,740 3,836 7 133 27,747 20,969 6,778 24.43%
    5 N/A 0 15,344 6,576 0 169 15,344 6,745 8,599 56.04%
    6 N/A 0 0 548 672 0 672 548 124 18.45%
    7 N/A 0 24,660 12,604 0 0 24,660 12,604 12,056 48.89%
    8 N/A 0 13,700 2,740 0 0 13,700 2,740 10,960 80.00%
    9 0 556 13,700 6,576 0 56 13,700 7,188 6,512 47.53%
    10 4,960 323 8,220 7,672 0 483 13,180 8,478 4,701 35.67%
    11 70,056 5,217 2,192 13,700 0 84 72,248 19,001 53,247 73.70%
    12 2,352 107 2,192 1,096 0 98 4,544 1,301 3,243 71.37%
    13 28,504 139 N/A 0 952 1,120 29,456 1,259 28,197 95.73%
    14 17,671 0 N/A 0 294 504 17,965 504 17,461 97.19%
    15 18,476 0 N/A 0 0 1,176 18,476 1,176 17,300 93.63%
    1999 3 12,500 10,528 2,740 1,096 84 0 15,324 11,624 3,699 24.14%
    5 N/A 0 8,768 2,740 0 399 8,708 3,139 8,708 63.96%
    6 N/A 0 0 548 686 0 686 548 138 20.12%
    7 N/A 0 10,960 9,864 0 511 10,960 10,375 585 5.34%
    8 N/A 0 6,028 5,480 0 462 6,028 5,942 86 1.43%
    9 0 278 7,124 6,576 7 182 7,131 7,036 94 1.33%
    10 2,480 1,401 8,768 6,576 0 210 11,247 8,187 3,060 27.21%
    11 0 576 4,384 2,740 427 532 4,559 3,848 711 15.60%
    12 0 0 1,644 548 0 539 1,644 1,087 557 33.88%
    13 9,635 7,661 N/A 0 19,964 7,175 29,599 14,836 14,763 49.88%
    14 0 0 N/A 0 623 623 623 623 0 0%
    15 8,992 1,058 N/A 0 511 1,694 9,502 2,751 6,751 71.04%
    Table 6.11: Predicted vs. actual direction of expenditure change for traditional and generalized subscriptions (by institution, 1998-99).
    Change in expenditure 1998-99
    Traditional Generalized
    Institution Predicted Actual Predicted Actual
    3 - 0 + +
    5 N/A N/A - -
    6 N/A N/A + 0
    7 N/A N/A - -
    8 N/A N/A - -
    9 + 0 - -
    10 - 0 - +
    11 - - + +
    12 - - - +
    13 - 0 N/A N/A
    14 - 0 N/A N/A
    15 - + N/A N/A
    NOTE: Predicted change direction is based on whether institution over- or under-purchased that product in 1999.
    "0" indicates no change; "N/A" indicates the access product was not available to that institution; "+" and "-" indicate an increase and decrease, respectively.

    We also analyzed changes in purchasing behavior from the first to the second year of the project. The PEAK team provided participating institutions with regular reports detailing usage. We hypothesized that librarian decisions about purchasing access products for the second year (1999) might be consistent with a simple learning dynamic: increase expenditures on products under-purchased in 1998 and decrease expenditures on products they over-purchased in 1998. For each institution we compared the direction of 1998-99 expenditure change for each access product to the change we hypothesized.[23] We present the results in Table 6.11.

    Six of the nine institutions adjusted the number of generalized subscriptions in a manner consistent with our hypothesis.[24] Fewer adjusted traditional subscriptions in the predicted direction. Two of the seven institutions that purchased more traditional subscriptions in 1998 than was ex post optimal then decreased the number purchased in 1999. Indeed, only three of the eight institutions made any changes at all to their traditional subscription lineup. This suggests an inertia that cannot be explained solely by direct costs to the institution. Perhaps libraries see a greater insurance value in having certain titles freely available through traditional subscriptions than from having generalized subscription tokens available that can be used on articles from any title. Generalized subscription tokens are also more expensive per article than traditional subscription prices, so the libraries are purchasing more potential usage with their budgets. Another explanation might be that libraries were more cautious about purchasing generalized subscriptions because it was a less familiar product.

    Table 6.12: Estimation results for forecast error
    Independent variable Coefficient (standard error)
    Year 1999 -35.7*
    (9.3)
    Green 54.6*
    (10.0)
    Red 53.3*
    (8.1)
    Blue 85.8*
    (9.2)
    Sample Size 24
    R2 0.85
    NOTE: Dependent variable is forecast error (in percent).
    No constant term is included in the regressions.
    Standard errors are shown in parentheses.
    * Significant at the 99% level.

    We performed a regression analysis to assess the differences between apparent over-purchasing in 1998 and 1999. Our dependent variable was the difference between the perfect forecast expenditure and actual expenditure, which we call the "forecast error". In Table 6.12 we report the effects of learning (the change in the error for 1999) and the average differences across experimental groups. The perfect foresight overspending over the life of the project averaged between 53% (Red) and 86% (Blue). However, the overspending was on average 36 percentage points lower in 1999. This represents a reduction of about one-half in perfect foresight overspending.[25]

    We also considered other control variables, such as the institution's level of expenditures, fraction of the year participating in the experiment and number of potential users, but their contribution to explaining the forecast error was not statistically significant. The between-group variation and the 1999 improvement account for about 85% of the variation, as measured by the R2 statistic.