Strategies for Managing Growing Networks

Building an extensible enterprise-class network requires the right tools and forethought. This article, the first of two parts, introduces the possibilities and issues of large-scale network design.

A s networks become ever more complex, it’s important to build networking infrastructure with scalability and reliability in mind. Best practices allow for networks to respond quickly to the demands of real-time business requirements with the least amount of manual intervention.

While small companies are able to work with simple configurations using static, point-to-point network links, that solution is wholly inadequate for sites with hundreds or thousands of servers and workstations. Dynamic routing eases the pain by putting intelligence into the network infrastructure, eliminating the need for human intervention when changes happen in the network topology. However, building such large networks takes some planning and forethought. This article, the first of two features on routing, introduces you to the possibilities and issues of large-scale network design. Next month, you’ll learn how to apply Linux routers in your network.

Designing large dynamic networks involves working with routing protocols, so a quick review is in order. Routing protocols are generally divided into two major classes: Interior Gateway Routing Protocols (IGPs) and Exterior Gateway Routing Protocols (EGPs).

IGPs, such as the Routing Information Protocol (RIP) and Open Shortest Path First (OSPF) protocol, provide routing services within an autonomous system (AS). An autonomous system is a group of networks under a single administrative domain, usually belonging to one organization.

EGPs, on the other hand, are used to interconnect different autonomous systems, with BGP-4 the predominant EGP being used on the Internet today. (For a detailed discussion of BGP, see “Demystifying BGP” in the May 2003 issue of Linux Magazine, available online at http://www.linux-mag.com/2003-05/bgp_01.html.)

All dynamic routing protocols are based on adaptive algorithms that calculate the most efficient path using a set of given measurements. Metrics vary from one routing protocol to another, some being more suitable for certain architectures than others.

The Routing Information Protocol, or RIP

RIP is one of the oldest and simplest routing protocols. It was originally designed to accommodate limited-size networks composed of not more than 250 routes, or networks having diameters not greater than fifteen hops.

In RIP, networks that are farther than fifteen hops are considered unreachable.

There are two versions of RIP: RIPv1 uses broadcast packets for its routing announcements, while RIPv2 allows IP multicast packets for its routing announcements. RIPv2 also supports variable length subnet masking (VLSM), which makes it easier to support non-contiguous networks.

RIP uses a cost metric that’s computed based on the distance, as measured in hops, to the given destination. In a scenario where a router has two or more paths to a given destination, a RIP router always makes decisions based on the cost metric and not on the path that has the greatest amount of bandwidth.

Such a scenario could mean a less than optimized path to a destination network.

A RIP router maintains a routing table that holds all of the routing information it needs, including the destination addresses and next hop addresses of its routes. Every thirty seconds, a RIP router sends an announcement to inform the neighboring RIP routers of the reachability of its routes.

The biggest advantage of RIP is that it is simple to configure and deploy. Its biggest disadvantage is that networks using RIP do not scale well when deployed in large environments.

Open Shortest Path First

OSPF was developed in response to the inability of RIP to serve large, heterogeneous internetworks. Unlike RIP, OSPF only sends updates when a topology change occurs, and instead of exchanging distances to destinations like RIP does, OSPF maintains a map of the entire network and constantly updates the map to reflect changes. The map is known as the link state database, and it holds all of the information necessary to compute the cost of each network link right after every topology change.

When a network change occurs, an adjacent OSPF router propagates the change across the entire network to ensure that each OSPF router has an accurate copy of the database at all times. OSPF then uses the Dijkstra algorithm to calculate the least-cost path to every known destination.

Instead of using hop counts as RIP does, OSPF bases its metric on the total bandwidth of a link. In the previous scenario where there were two paths to a given destination, OSPF chooses the path with the highest available bandwidth, thereby ensuring a much more optimized flow of traffic.

OSPF is hierarchical by nature. Networks running OSPF make use of autonomous systems numbers and areas to provide route filtering and summarization capabilities. An area is a logical grouping of OSPF routers that share routing updates.

The biggest advantage of OSPF is that it allows for faster convergence and is derived from a hierarchical design that facilitates network scalability. Its main disadvantage is that it’s more complex than RIP, and networks built with it must be carefully planned to accommodate future growth.

The Border Gateway Protocol

BGP-4 is the primary exterior gateway protocol employed on the Internet. BGP-4′s main purpose is to connect very large networks that are mainly autonomous systems. However, BGP-4 can also be used internally at large sites to route between domains. In this case, BGP is referred to as iBGP or Internal BGP. External BGP, or eBGP, is often used by sites to interconnect with ISPs and other autonomous systems.

Although BGP-4 routers can maintain a routing table that lists all reachable paths to a network, it need not record the location of every subnet within an organization. Instead, it uses address summarization and only communicates that which is defined and necessary, as illustrated in Figure One. BGP-4 routers operate by sending neighbor BGP routers their full routing information upon startup and incrementally advertising updates using TCP when changes occur.

Figure One: BGP records routing information only for autonomous systems.

Although static routing is commonly preferred to BGP-4 for WAN connectivity because of its simplicity, there are certain occasions when it’s advantageous to use the BGP-4 protocol. One of the more common situations is when an enterprise is connected to multiple Internet service providers (ISPs) or autonomous systems. Indeed, BGP-4 works best when an enterprise has more than one ISP and there is a need to do policy-routing on a link-by-link basis. Through the manipulation of the path attributes, it’s possible to achieve load-sharing/balancing and fail-over capabilities with multiple Internet connections. This scenario will be discussed in the later sections of the article.

Developing a Small Network

To illustrate the importance of routing in the management and scalability of a campus network, let’s develop a network solution for ABC, Incorporated. ABC is an outsourced call center that offers customer care services to its clients.

Starting with one hundred seats, the company wants to implement a network that’s easily expanded in the future. ABC’s decided to use the private network IP address range of for their internal LAN and use the IP address given to the company by their ISPs for Internet access. After thoroughly discussing the merits of a well-architected network, management wants to implement a network based on the following three requirements: ABC should be able to enforce secure access based on business function; there should be redundancy in the LAN; the WAN should incorporate load-balancing and fail-over capabilities.

In theory, ABC could build a network by simply plugging all of their computers into a few switches and hubs cascaded off of one another. However, that approach doesn’t support the segregation and redundancy needed to effectively manage future growth.

The best way to realize the requirements for ABC’s network is to build it based on a classic Cisco three-layer hierarchical model for network topologies, with core, distribution, and access layers. This three-layer hierarchical model separates the features of the network into distinct layers that allows for expansion without affecting existing systems. Figure Two shows an example of a collapsed core network based on the three-layer hierarchical model.

Figure Two: A collapsed core, three-layer network topology

In a collapsed core, both core and distribution layers are combined into one layer. A collapsed core architecture is often the recommended design for a small- and medium-sized campus network.

The core/distribution layer is where the core switching and routing takes place. Here, packets are fast-switched using dedicated ASIC chips, and routing definitions are created to allow end hosts to communicate with one another. Corporate servers are also directly connected to the core switches using Gigabit connections to provide maximum switching and forwarding capabilities.

The access layer is where the computers physically connect to. The devices on this level are normally made up of simple hubs and switches that provide end-user connectivity. Each access layer device connects to the backbone layer through the use of multiple Gigabit Ethernet uplinks. This redundancy is important in the event that a primary uplink goes down. When this happens, the routing definitions on the core/distribution layer automatically switch the packets to the secondary links to provide continuous network access.

Developing a network using the model described above is important for the following reasons.

* The model allows for the creation of Virtual LANs (VLANS) in the system. VLANS are used to logically partition networks into various subnets based on business or network requirements. Computers belonging to different VLANs are restricted from communicating with each other unless a routing path is explicitly added to bridge between VLANs.

* Routing segmentation allows the efficient use of IP addresses through the use of variable length subnet masking. As mentioned earlier, VLSM allows classful networks to be partitioned into smaller subnets. In the case of company ABC, the network address of can endlessly be divided into different subnets such as,, and so on, depending on the number of hosts needed for each VLAN. In a flat network, this logical addressing scheme is not possible, as the entire network shares the same IP address space.

* Finally, by introducing a routing or Layer 3 element, broadcast traffic is limited to each subnet. By default, routers do not pass broadcast traffic from one VLAN to another unless configured to do so. This is especially important in preventing excessive traffic from bringing down an entire network. If all hosts are connected to the same network, an ensuing broadcast storm could potentially bring down all the computers, causing a standstill.

Dynamic Routing Over the LAN

For small campuses, static routing is sufficient to propagate network information between routers. However, for bigger networks this becomes increasingly difficult, as more users and devices are added to the network.

Figure Three: Open Shortest Path First partitions a large network into smaller areas

As envisioned in the example above, ABC’s network is physically located within a single site. But what happens when ABC decides to expand and add new offices? That will require additional routers. If static routing is used between the routers, each router must be updated to pass the traffic correctly, and keeping tabs on such a network could become a huge task. To lessen the administrative burden of defining routes, a dynamic routing protocol such as OSPF can be used instead. Figure Three illustrates how OSPF can aid in partitioning a large network into smaller areas to help contain routing updates. In the figure, each major network, 10.X. 0.0/24, is contained within a single area only. The different areas need not send individual routes to the neighboring areas. Instead, the routers can send the summarized addresses to lessen the amount of traffic traversing each area. Network addresses can be summarized on the area border routers (routers belonging to more than one area) as new locations are added. In addition, new areas can be created to provide logical separations within the corporate network. Moreover, there is no need to explicitly define the routes on a per router basis. The OSPF process automatically facilitates the communication among all the routers within the network.

Dynamic Routing Over the Internet

Where dynamic routing is really required is for wide-area network connections over the Internet. Corporations seeking to connect to the Internet usually do so by subscribing to an ISP. A typical Internet connection is usually made using a subscribed lease line or Frame Relay service. When connecting to a single provider, static routing can be used in favor of dynamic routing for simplified WAN management. However, companies that have more than two providers within their network might find it more efficient to apply for their own autonomous system number and use a true exterior gateway protocol such as BGP-4.

In a multi-homed ISP scenario, the real challenge is to maximize the use of bandwidth. To achieve this, a process called load balancing is used. Load balancing allows a router to take advantage of multiple paths to a given destination. These paths are derived either statically or dynamically with protocols such as RIP, OSPF, and BGP-4. Most routing protocols support load balancing across parallel links that have equal cost.

When a BGP-4 router receives updates from multiple ASes that describe different paths to the same destination, it must choose the single best path for reaching that destination. Once chosen, BGP-4 propagates the best path to its neighbors. The decision is based on the value of attributes (such as next hop, administrative weights, local preference, the origin of the route, and path length) that the update contains. Because of the number of factors BGP-4 has to consider, the traffic might not take the same routes twice. This is illustrated in Figure Four.

Figure Four: ABC connects to two ISPs and maximizes bandwidth with load balancing

Consider a router that’s connected to the Internet through two ISPs; ISP A and ISP B. Each ISP link supports up to 1 Mbps of traffic giving a total aggregate bandwidth of 2 Mbps. Without proper configuration, the 2 Mbps bandwidth will not be fully utilized. However, using BGP-4, the traffic can be influenced to load-share across both lines by changing the BGP-4 path-attributes for both incoming and outgoing traffic. (The exact changes needed to achieve load-sharing goes beyond the scope of this article.)

Nevertheless, to do a complete load-sharing across multiple connections might not be possible since we cannot fully determine the path which the incoming traffic would take. In Figure Four, traffic sourced from AS 100 above might choose to go through ISP A while traffic from both AS 200 and 300 might choose to go through ISP B. Thus a 50-50 load-sharing split in this scenario is not possible.

One caveat of using multi-homed Internet connections is the possibility of a network being used transit network. A transit network is created when a local network is used by a foreign network as a bridge to reach other networks. In Figure Four, ISP A could use ABC’s connectivity to reach ISP B. Although this was not the intended purpose, such a scenario could potentially overload the WAN links of ABC with unintended external traffic. Careful policies should be created on the internet routers to avoid such scenarios from happening in the first place.

To enable routers within an enterprise to share the routes learned through different routing protocols, a process called route redistribution must take place. Route redistribution is the translation service that allows different routing protocols to interoperate and learn from each other. Ideally, all routers within an organization should run the same routing protocol. In reality, however, this is not always the case, as some routing protocols are more suitable for certain situations.

Figure Five: Route redistribution among subnets

For example, consider a network that’s running two routing protocols, RIP and OSPF, as shown in Figure Five. Each routing protocol computes the value according to its own algorithm and uses its own metric to calculate a best path. For illustration purposes, consider the propagation of routes in this scenario: RIP networks A and C needs to see each other’s routes. However they would first have to cross network B, which is currently running OSPF. To achieve this, a route redistribution has been configured on network A’s router to redistribute RIP into OSPF. The routes, along with their metrics, are converted from the RIP into OSPF, and are injected into the OSPF core. Upon reaching the boundary of Network C, the routes are converted back to their original RIP format. Because of route redistribution, both networks A and B can now see each other’s routes.

Using Linux as a Dynamic Router

Most of this article’s been concerned with the issues surrounding the design of large networks. So where does Linux come into the picture?

While Linux has good networking capabilities, its role in building large networks has thus far been as a small, special-purpose device such as a firewall, network address translator, gateway, or simple forwarding router. Low in cost, Linux can transform a small, stripped-down, multi-homed PC into a capable network device. This is the subject of several excellent open source projects, such as the Linux Router Project.

But with the right software, Linux can go beyond these simple functions and take on dynamic routing functions as well. The Zebra package (http://www.zebra.org) and its successor, Quagga (http://www.quagga.net), for example, enable Linux to handle the routing protocols explained in this article, as well as a few others. We’ll take this up next month, with an article explaining Zebra.

Equipped with multiple Ethernet cards and the right routing software, a Linux box can make a relatively decent core router, handling dynamic routing over a LAN. An organization might decide to use this configuration as a substitute for full-fledged routers while it goes through a period of growth and before its network architecture stabilizes.

How about as an edge router then? A Linux router would have to be equipped with the proper network cards to handle the uplink connections. These are less common than Ethernet cards, but are nonetheless available on the market. Cyclades, for example, sells several excellent communications boards that can be used for this purpose.

Staying Connected

Dynamic routing provides robustness and efficiency. With the right dynamic routing in place, a highly robust network can provide the enterprise with right levels of availability without having to manually replicate each change into every other router.

Moreover, network failures are immediately detected and are automatically taken care of, thereby minimizing downtime. A dynamic routing protocol also helps maximize the given network architecture by load-balancing using equal cost computations among the available given paths.

Both advantages are equally critical for enterprises who need to run their enterprises 24×7.

Rex Young is a network manager for C-Cube, a major call center in the Philippines. He has worked with large Cisco internetworking and VoIP implementations, and is a Certified Cisco Network Professional. He may be reached at youngrv@c-cubeservices.com. Dominique Cimafranca is a Linux IT Specialist for IBM Philippines. He has implemented Linux in all the eServer platforms from IBM. He may be reached at dgmc@sketches.kom.ph.

Comments are closed.