New Zealand Experiences with Network Traffic ChargingSkip other details (including permanent urls, DOI, citation information)
This work is protected by copyright and may be linked to without seeking permission. Permission must be received for subsequent distribution in print or electronically. Please contact firstname.lastname@example.org for more information. :
For more information, read Michigan Publishing's access and usage policy.
Presented at MIT Workshop on Internet Economics March 1995
Since 1990 the University of Waikato has operated a single Internet gateway to the United States on behalf of New Zealand universities and research institutions, charging users by volume to recover the costs. This has been very successful, allowing a steady growth in link speeds from 9600 bps to 512 kbps over four years.
The Internet backbone within New Zealand is provided by Tuia, an incorporated society of Research and Education users. Cost recovery for the backbone network within Tuia's membership is handled by several 'management groups,' with capacity-based settlements between the groups. This has proved to be a simple and effective scheme.
The New Zealand Universities are linked by the Kawaihiko network, one of Tuia's management groups. Initially Kawaihiko used a simple 'equal shares' cost recovery scheme for internal traffic, which has worked well to date but is now somewhat limiting. Kawaihiko is moving to a new scheme, allowing each site to pay directly for its required traffic capacities. We believe this will encourage the development of new, bandwidth-intensive services.
New Zealand's Internet Gateway
New Zealand's link to the Internet began as a joint development project with NASA and formed part of the Pacific Communications program, PACCOM. The gateway itself was known as 'the PACCOM gateway,' but we have since renamed it 'NZGate.' NZGate is managed by Waikato University on behalf of Tuia, which is an incorporated society whose mission is to further the networking interests of the Research and Education community within New Zealand. Tuia is a Maori word which means "bound together."
NZGate began in April 1989 with a 9600 bps analog cable link to Hawaii. Connectivity inside New Zealand was provided by PACNET (Telecom's public X.25 service) and Colored Book software at each of the universities. PACNET access speeds were mostly 9600 bps.
One slightly unusual aspect of NZGate was that although NASA provided generous support for the costs at the U.S. end of the link, no subsidy whatever was provided by the New Zealand government, which meant that all the link costs had to be recovered by charging the users. To get the project established, six of the universities agreed to pay 1/6 each of the startup and on-going costs.
Shortly after this the universities agreed that volume-charging would be a better approach to sharing the costs. The underlying goals were to:
- Measure traffic in both directions through NZGate for each participating site and charge for it by volume; i.e. for the number of Megabytes moved in and out each month
- Charge enough to cover actual costs, plus a percentage for development
- Use the resulting development funds to buy more capacity as demand grows
These goals provide an effective charging scheme for shared use of our single common link, but it is rather simple-minded. To make it usable we had to address several other matters, the first of which was predictability. Sites wanted to know well in advance how much they would have to pay for the NZGate service so that they could budget for it.
To provide predictability we adopted the notion of 'committed traffic volume' per month. The charging scale has large steps, and the price per Megabyte decreases as the volume increases. Each site made an initial choice of its committed volume, and thus monthly charge. The actual traffic was monitored month by month and reported back to all the sites. If a site's traffic falls into a different charging step for more than one month, that site's committed volume is changed to the actual rate. This allows a site to have a single unusual month, and it gives at least a month's warning of a change in the charge. It is simple to administer, since the committed volumes are changed automatically by the NZGate Management.
Another potential problem was that of paying for unsolicited incoming e-mail. As an example, a user at Auckland had a colleague in the United States who e-mailed him a 200 kbyte file. The remote system kept aborting after sending about 150 kilobytes, and did this every half hour over a three-day weekend. By the time we realized what was happening many tens of Megabytes had been received, which we had to pay for. We don't have a good answer for this—after all we have no control over systems outside New Zealand—but in more than four years of operation it hasn't happened often enough to be worth worrying about.
Volume charging for NZGate was begun late in 1990 using metering software developed at Waikato University, running on a 286-based IBM PC. The metering has been progressively improved since then, and now runs on a Sun SPARCStation. The charging scheme also has been refined, with discounts to encourage off-peak use. A summary of the charge rates is given as an appendix to this paper.
Traffic volumes have risen steadily, and the link capacity has been increased as follows:
Table 1. Timetable of NZGate Link Specifications
|April 1989||9600 bps||analog cable to Hawaii|
|November 1990||14.4 kbps||analog cable to Hawaii|
|September 1992||64 kbps||satellite link to NASA Ames|
|February 1992||128 kbps||satellite link to NASA Ames|
|March 1994||256 kbps||digital cable to NASA Ames|
|July 1994||512 kbps||digital cable to NASA Ames|
Providers of intercontinental data circuits provide them as 'half circuits,' i.e. each end is separately billed. In our early stages PACCOM paid for the U.S. half circuit and NZGate paid for the New Zealand one. For some time now, however, NZGate has paid the charges for both ends. The New Zealand access circuit is provided by Telecom New Zealand; over the years we have used various intercontinental telecommunications providers.
Most of the above growth can be attributed to a steadily increasing user population, but there were also several events that caused sudden increases in traffic. The first of these was the setting up of the Kawaihiko network in April 1991, linking together all seven of the New Zealand universities. Kawaihiko is a Maori word, derived from kawai (a branching structure, like tree roots) and hiko (electricity). It began with Cisco routers at each university, linked with 9600 bps leased lines which were later upgraded to 48 kbps. It was an IP network (at last), which made it possible to use TCP/IP services such as FTP and greatly simplified access to UseNet News.
Another event was the setting up of the Tuia network, which provided links between Kawaihiko, DSIRnet and MAFnet on an informal basis. This provided Internet access to all of New Zealand's government-funded research establishments, thereby increasing the number of sites using NZGate.
A third event occurred in July 1992, when the Department of Scientific and Industrial Research (DSIR) and Ministry of Agriculture and Fisheries (MAF) were restructured by the government and split into 11 Crown Research Institutes (CRIs). The Tuia Society was created, and a new network using a Frame Relay backbone was set up to link the Universities, CRIs, National Library, and Ministry of Research, Science and Technology. The Frame Relay backbone provided significantly higher link capacities, which in turn boosted the total NZGate traffic.
During the formative phase of our shared use of the NZGate service, charging by volume has provided a cost-sharing mechanism which is visibly fair for each of the participating sites. The notion of 'committed volumes' has smoothed out short-term usage transients, and thereby helped the budgeting process at each site.
The steady growth in transport volumes demonstrates the success of the service, and of its charging scheme. This is in stark contrast with the Chilean experience of charging. [The Chilean Internet Connection or I Never Promised You a Rose Garden, Ricardo Baeza-Yates, Jose M Piquer, Patrico V. Poblete, Proceedings of INET '93]
Management of the Tuia Network
Before 1992 there were three Research and Education networks in New Zealand: DSIRnet (a leased-line network linking DSIR sites), MAFnet (a private X.25 network linking MAF sites) and Kawaihiko (a TCP/IP network linking the universities). These three national networks were interlinked so as to provide a single TCP/IP network known as Tuia. This network had no formal management structure, relying instead on occasional coordinating meetings of the three participants, but it nonetheless provided reliable, effective communication between the sites, and (via NZGate at Waikato University) access to the Internet.
When the Tuia network backbone was set up in 1992 it linked 13 sites: five universities, six Crown Research Institutes (CRIs), the National Library of New Zealand, and our Ministry of Research, Science and Technology (MoRST). The locations of the TuiaNet backbone sites are shown in Figure 1; the universities are labeled in italics and the CRIs in plain text.
Tuia is an incorporated society, providing its members (the CRIs, Universities, etc.) with a legal entity and a formal structure within which to set up and run a single backbone network, known as TuiaNet. Rather than set up a new management structure for TuiaNet we decided to continue with the old groupings of the sites, which became 'management groups' within Tuia. Thus Kawaihiko is the Tuia management group which co-ordinates inter-university networking, a while Industrial Research Limited (IRL) and AgResearch co-ordinate groups of CRIs which correspond with the old DSIR and MAF. The remaining sites (National Library and MoRST) could have joined one of the three groups, but chose not to, which effectively makes them single-member groups.
TuiaNet uses a Frame Relay backbone provided by Netway Communications, a subsidiary of Telecom New Zealand. Netway's monthly charges for each site have two components, an access cost (determined by the line speed, 64 kbps to 768 kbps) and a set of Committed Information Rate (CIR) costs (reflecting the maximum continuous data rate for virtual circuits to other sites). CIR charges have been a flat rate of $8.75 per kbps since we began using Frame Relay. Netway's monthly bills are paid by each of the management groups, and each group then recovers these costs from its own users as its management requires.
For sites that are entirely within a management group, e.g. the University of Auckland, the site pays its share of the costs to the group; as explained for Kawaihiko (see below).
Where a site provides services for more than one management group (e.g. Massey University provides backbone access for IRL and AgResearch), simple settlements have been reached. These usually have each group paying a fixed share of the access charge, and each group paying CIR costs to other sites within the group. A special case of this occurs when virtual circuits run between management groups, e.g. between Waikato (Kawaihiko) and Gracefield (IRL). We have found that the traffic between members of a group is much greater than their traffic to other groups, so groups needing virtual circuits to other groups are happy to pay for the CIR at both ends.
The settlements are therefore capacity-based, in the sense that the amount each group pays is determined purely by the traffic capacity they agree to. They make no attempt to charge by volume, or for different network services; these possibilities are explored in the next section.
The simple arrangements for cost-sharing within Tuia have worked well. Having the management groups reduces the number of parties to settlements to a manageable number, allowing us to quickly reach agreement on the simple settlements. It seems unlikely that a flatter management structure, such as a single group which would then negotiate settlements between every pair of sites, would have been as effective.
Late in 1990 we began planning the Kawaihiko network, to link all the universities with their own IP network. Each site had a Cisco router, and the sites were interconnected with Digital Data Service (DDS) links. The topology was a central triangle joining Waikato, Victoria and Canterbury, with the others connected to it (as shown in diagram).
The network was installed and running in April 1990, providing greatly improved communications between the universities. Its on-going costs were Telecom's monthly access and transmission charges for the links. Following the success of volume charging for NZGate we resolved to implement volume charging to recover Kawaihiko's on-going costs. It was not, however, possible to measure traffic volumes at that time, so instead we agreed—as an interim measure—to share the costs in fixed proportions, which were 1/13 for Lincoln and 2/13 for the other six sites.
This simple arrangement continued for about two years, during which time most of the links were upgraded to 48 kbps. Waikato continued to measure traffic volumes through NZGate for each site. So as to provide metering of traffic between the Kawaihiko sites (rather than just the traffic on each of the Kawaihiko links), work was begun at Auckland in 1992 on an implementation of the Internet Accounting model. [RFC 1272, Internet Accounting Background]
The Kawaihiko network was re-engineered in July 1992 when the Universities and Crown Research Institutes implemented TuiaNet, using Frame Relay as the dominant transport technology, but continuing to use DDS links where Frame Relay would have been too expensive. The effect of all this for Kawaihiko was to make it more obviously part of a larger network, with some of the Kawaihiko sites providing connectivity for some of the CRIs. Obviously a more elaborate cost-sharing scheme was called for.
A site requesting a frame relay connection must decide on its required access speed, and the Committed Information Rate (CIR) it wants to all the other sites. The access rate should be greater than the sum of the CIRs, and there are good reasons to have it much larger. One of these is that access speed determines the maximum burst rate, another is that it can be expensive to change access rates (especially if such a change involves changing the underlying technology, for example from DDS to fractional E1).
Since access rate had to be determined by each site separately, we agreed that each site would pay its own access cost. Access costs for sites providing common access for CRIs were divided using a set of percentages agreed locally at each site. Initially we had to have a fully connected network, with at least 2 kbps of CIR between every pair of sites. We therefore agreed to share the costs using our '1/13ths' formula, until we were able to measure the traffic volumes between sites.
The overall effect of these changes was that the Kawaihiko network was treated as being made up of links with access costs paid by individual sites, and transport costs shared between all sites—regardless of the type of link. Effectively, we have tried to visualize the New Zealand transport network as comprising local (and locally-funded) elements plus a single shared element whose costs were shared on the basis of traffic volumes, just as for the single international link element (NZGate).
Since 1992 work on the Internet Accounting project has continued, resulting in the public-domain release late in 1993 of an Accounting Meter, NeTraMet, and its Manager/Collector program, NeMaC. (NeTraMet is available via anonymous FTP from [formerly ftp.auckland.ac.nz], directory pub/iawg/NeTraMet).
There have been three further releases of NeTraMet in 1994, prompting considerable interest from other sites looking for a way of metering network traffic. Kawaihiko sites are now deploying NeTraMet meters on their gateway networks, and these will allow us to produce matrices showing traffic volumes between the sites, subdivided into a number of traffic types. If volume charging continues to be appropriate for all or part of the cost-sharing, these will provide the raw data for it. They will, in any case, provide data for other purposes, such as verifying that traffic flows don't exceed their agreed levels in the long term.
Initial work with the NeTraMet meters has classified traffic by site into traffic to and from 'local,' 'other Kawaihiko sites (or networks connected to them)' and 'world.' Traffic types of particular interest so far include smtp, nntp, telnet, ftp, gopher, www and dns.
Cost Sharing within Kawaihiko
A charging scheme based on 'fixed shares' has worked well so far, but as our traffic volumes grow and new services develop the need for a better algorithm becomes steadily more apparent. Charges based on amounts of data actually moved have been helpful throughout our 'start-up' phase, during which institutions struggled to obtain funding for even a minimum level of service. This is essentially a cost-minimization regime for the participating sites, and most appropriate for 'queued' transfers whose transit time is not important. Not unnaturally, these have been dominant so far.
One of the requirements of a charging algorithm is that it should handle higher-layer services well. Some of these, such as Domain Name Service (DNS), and Network Time Protocol (NTP) are an essential part of the TCP/IP infrastructure. They must therefore be shared as 'common good' elements of the network. Network News is somewhat similar in that all sites want access to it, and it is essential to minimize the number of times any news item is replicated. Overall it seems better to charge separately for transport (including 'common good' services) and other 'Value-Added' services. Service providers can be charged for their transport costs, and then recover them from their end users.
As an example of this, consider a site wishing to offer a World Wide Web (www) cache as a chargeable service. Transport costs for traffic to and from the cache could be measured and paid for by the cache provider, who would set charges for the service so as to cover the transport charges both within New Zealand (to the customers) and to NZGate (source of the cached Web pages).
Another requirement is the emerging need to provide higher-bandwidth pipes within Kawaihiko which are site-specific, e.g. pairs of sites such as Waikato and Victoria would like 128 kbps capacity between them for experiments with packet video. These 'dedicated capacities' should clearly be paid for by the sites requiring them, and not be a shared cost for all of Kawaihiko. Charging directly for capacity provides predictable costs for sites, and allows for special link needs. Specified capacities appear to be the most effective way to provide for the continued growth of Kawaihiko, and are the basis of our proposed new charging algorithm, as follows.
Each site will nominate the required capacities for its links to every other site. From the lists of link capacities we will produce a full matrix—not necessarily symmetric—of actual link capacities (CIRs or line speeds) which will be sufficient to meet the stated requirements. In most cases these will be the CIRs we need to meet our current traffic levels plus any anticipated needs for the next year or so.
Each site will pay its own access costs, whether they are connected via Frame Relay or DDS lines. Sites will continue to make their own arrangements with distal (i.e. locally connected) sites. For these aspects, cost recovery simply continues our existing practice.
Transport costs will be paid by the sites. For each link or virtual circuit the costs will be shared between the two sites involved. This cost sharing may be equal, which amounts to having each site paying all its CIR costs directly. On the other hand it may be unequal, e.g. if an Auckland distal (locally connected) site wants 128 kbps of CIR to Waikato, Auckland could pay Access and 128 kbps CIR costs for both ends of the virtual circuit, then recover these from the distal site.
Over the past four years New Zealand has had a very effective Internet Gateway, the speed of which has increased steadily to meet users' traffic demands. Because we have charged users by volume for their traffic, users have always perceived that their payments were closely related to the benefits they derived. This perception was enhanced by our ability to develop our Internet gateway so as to handle the increasing traffic volumes.
The overheads of charging are significant. Operation and development of the NZGate traffic monitoring and usage billing software occupies about one half-time person. Development of NeTraMet is an ongoing research activity, which takes about one-fifth of my time. The management of Kawaihiko takes one person about half a day each week. The benefits provided by charging are, however, well worth their cost to us.
Within New Zealand we have used 'fixed share' algorithms to recover costs, which has worked well within our simple management structures. Kawaihiko, the universities management group, has invested a great deal of effort in developing traffic accounting and cost sharing methods, and is currently implementing a 'specified capacity' algorithm for sharing their traffic costs. We believe this will more accurately reflect our users' expectations, allowing us to handle the newer, more bandwidth-intensive services they demand.
The author records his appreciation to the New Zealand Vice Chancellors' Committee for its support of networking within our universities, to NASA for its enthusiastic support of our international Internet gateway in its early stages, and to the network support staff at all the New Zealand sites, especially to John Houlker (email@example.com). It was John who made our first contact with NASA and PACCOM, implemented the gateway, and has managed it so effectively ever since.
Nevil Brownlee (firstname.lastname@example.org) holds an M.Sc. and a Ph.D in Atmospheric Physics from the University of Auckland. Since 1975 he has been Deputy Director of the University of Auckland's Computer Center, where he is responsible for a campus network with about 2,500 connected hosts. He is manager of Kawaihiko and chairman of Tuia's Technical Working Group. He has been active in the IETF since 1992, and is currently co-chair of the Operational Statistics and Internet Accounting Working Groups. His hobbies include backstage Theater (especially Lighting Design), photography and music.
Reprinted with permission from ConneXions, Volume 8, No. 12, December 1994.
Connexions—The Interoperability Report is published monthly by:
Free sample issue and list of back issues available upon request.
Appendix: NZGate Charging Rates
The NZGate International Internet Charging Rates as of 8 Apr 94 are given below. More information can be obtained from email@example.com
Uncommitted rate $6.00 per Mbyte and committed rates are shown in the following table:
|1,000 - 1,250||2.90||3,625|
|1,250 - 1,500||2.80||4,200|
|1,500 - 2,000||2.60||5,200|
|2,000 - 2,500||2.50||6,250|
|2,500 - 3,000||2.50||7,500|
|3,000 - 3,500||2.50||8,750|
|3,500 - 4,000||2.50||10,000|
|4,000 - 4,500||2.40||10,800|
|4,500 - 5,000||2.40||12,500|
|5,000 - 5,500||2.40||13,200|
|5,500 - 6,000||2.30||13,800|
|6,000 - 6,500||2.30||15,000|
|6,500 - 7,000||2.30||16,000|
Committed rate buffer period:
If the committed rate is exceeded for a single month, the charge step remains unchanged (and similarly if it is not reached for a single month).
Low priority discounts:
Low priority traffic (currently FTP and MAIL) is discounted by 30%. Normal priority traffic is discounted by 15%. The full rate is charged for high priority traffic (currently TELNET and FTP commands) These are applied before the committed rate is det ermined.
Off peak discount:
An off peak discount of 80% applies to all traffic between 8pm and 9am. This is applied after the committed rate is determined.