Arguably one of the greatest innovations within the Global BGP routing system in the last decade is one that you probably never have to deal with. In 2007, the IETF published RFC 4893 which defines a proposal for expanding the available Autonomous System number space from its original 16-bit limit of 65535, to the lofty limits of 32-bit-ness.
This was no mean feat. The Autonomous System Number is central to the operation of the BGP protocol, appearing in opening neighbour negotiation messages and BGP routing updates at least, so it wasn’t as simple as re-compiling with new field sizes. In contrast to IPv6 deployment, which was initially conceived with a ships-in-the-night style transition plan in mind, RFC 4893 worked within the limits of the existing BGP4 protocol to ensure that existing 16-bit-only BGP implementations could communicate on an almost even par with newer 32-bit capable implementations.
All was not quite perfect, however. While some extraordinary lengths of detail were considered to guarantee the operational transparency necessary for such a fundamental extension to the Global Internet Routing system, one crucial detail did go amiss.
BGP Communities was a feature added to BGP in the relatively dark ages of 1997, and indeed RFC 1997. It provided a simple method for labelling routing information with any meaning relevant and appropriate to the autonomous system operator. Despite its apparent simplicity, the feature rapidly became a coming-of-age milestone for networks achieving the operational maturity and scale associated with large ISPs, since BGP’s extensive abilities to implement policy filters became so much more manageable and scalable if one could write policy in terms of route type or class, rather than the actual route prefixes themselves.
There was one problem: RFC 4893 didn’t address the BGP Community attribute. No doubt the authors themselves and the tireless members of the IDR working group within the IETF understand the significance of the omission now; but it certainly wasn’t as obvious then regular BGP Community attributes, as defined in RFC 1997, couldn’t operate with the convenient delegated namespace pattern in the face of 32-bit AS numbers. The BGP Community attribute itself was only 32-bits wide.
Original RFC 1997 BGP Community Attribute
It’s not quite true to say that there were no considerations in place for this problem, however. MPLS VPNs brought with them an Extended Community attribute format in order to implement Route Targets and Site Origin identifiers (arguably equivalent to the Internet application of RFC 1997 attributes). Unfortunately, Extended Communities, while indeed being extended, aren’t sufficient to hold 32-bit AS numbers either.
Similar initiatives in Flexible and Wide Communities aimed to solve the problem once and for all by providing a multi-data type attribute, defining a flexible scheme of atoms classes and containers, even including a way to communicate UTF-8 strings (perhaps allowing the propagation of worthy IETF-style ASCII art!)
While these other initiatives were comprehensive and thought-provoking, they lacked the raw pragmatism and simplicity of the original RFC 1997 Community attribute. The focus on extensibility and ability to be applied to as-yet-unforeseen applications detracted from the here-and-now requirements. And those requirements had become ever more here-and-now once the regional Internet registries started dispensing 32-bit ASNs to networks.
Ignas Bagdona described the problem lucidly in the spring 2016 meeting of the NetNod IXP and advocated a simplistic effort to re-do RFC 1997 style communities, but “just make them bigger”. His experiences attracted the attention and enthusiasm of energetic engineer, Job Snijders, responsible for development at NTT. Spurred on by comments from Internet die-hard and IETF veteran Ruediger Volk that even a fast-track development would take five years, Snijders seized the mantle. He quickly managed to enlist the help of Jakob Heitz, a Cisco developer, to co-author a draft and commit to an implementation much more simple in structure than the existing efforts.
BGP’s Large Community attribute was born, building on the simplicity of the original RFC 1997 model and bringing parity to those network service providers who wanted to make use of BGP Communities but who didn’t possess a traditional 16-bit ASN.
Large BGP Community Format and Example Usage
The road to standardisation wasn’t the easiest however. The IDR working group notionally responsible for developments within the BGP protocol area already had several committed efforts at tackling the same problem, so the first challenge was in gaining acceptance that the lack of pragmatism/perfection balance in those other efforts was proving counter-productive to immediate-term requirements.
There was a significant support base from the operator community within the IDR working group however, sufficient that the effort couldn’t be ignored and Snijders pursued the development effort through the established channels, consulting working group members and seeking consensus while continuously reminding contributors of the basic aims: simplistic RFC 1997 parity and 32-bit ASN scope.
The latter was perhaps the smallest lie since the Large Communities draft had managed to slip in an extra ASN-sized field. As Snijders put it to me: “My aim is to give eight bytes to the operators. How they use it is really up to them.” His own efforts at NTT, however, had seen the establishment of a pretty concrete policy example where customers could use a BGP Community attribute quoting a peer ASN in order to effect change in the announcement behaviour, by NTT, to that peer.
But even with infamous IETF hum-like consensus, the path was not to run smoothly. Following successful lobby of OpenBGPd and BIRD – Open Source BGP implementations commonly used at IXPs – Snijders offered his own personal domestic LAN segment as a route prefix to carry a special beacon for the newly assigned BGP path attribute that was to represent the Large Community attribute. One evening he discovered that his Internet connectivity was seriously impaired.
On detailed investigation, it emerged that implementors within Huawei had unknowingly camped upon Snijders’ officially assigned attribute path code-point and associated their own internal actions with it. Furthermore, they’d allowed this development-grade code out into production by accident, meaning that many Huawei-powered networks were unlikely to be able to work with Large BGP Communities unless they were upgraded.
Before any rocks were even thrown at such apparent anti-social behaviour, Cisco also quickly stepped forward, rather red-faced, to admit similar transgressions.
It was becoming apparent that one of the most positive aspects about BGP – its modularity and extensibility through enumerated attributes and capability negotiation, which enables its use across multiple domains, both associated with and independent from the Internet – is also one of its biggest problems and hurdles for development. Implementors need to be able to dry-run features that piggy-back on BGP, and they need to be able to do so using real code points; yet those same implementations must also be legitimate for production use if integration testing is to be a useful exercise.
So not only has the Large BGP Community effort helped the here-and-now requirement of providing BGP Community attributes for newcomers to the Internet with 32-bit ASNs. It has also helped a greater understanding of how we manage the controlled extension of a protocol that has become the lynchpin of the modern global communications network, and whose resources require allocation with care.
And it has done so in a manner that restores some of the IETF’s original established mantra: rough consensus and running code.
References:
IETF RFC 1997 BGP Communities Attribute (1997), R. Chandra, P. Traina.
IETF RFC 4893: BGP Support for Four-octet AS Number Space (2007), Q. Vohra, E. Chen.
IETF Internet Draft: Large BGP Communities (2016), J. Heitz, J. Snijders, K. Patel, I. Bagdonas, A. Simpson, N. Hilliard.
NANOG 68 Dallas: Large BGP Communities (2016). G. Hankins, J. Snijders.
Images: Vladyslav Starozhylov.
Further reading:
Large BGP Communities web site
