BT Addresses Rapid DNS Growth

BT Group is one of Europe’s oldest and most successful telecommunications companies, tracing its lineage from the first commercial telecommunications efforts in Europe. But the company isn’t stuck in the past; BT is bringing broadband connectivity into homes throughout the U.K.

But the 21st century presents new challenges as well as opportunities for this company. In early 2004, the company’s DNS servers were struggling under a sharp increase in query load. Although the subscriber base was growing, much of the traffic could be attributed to DNS queries generated by viruses and worms. Overloaded DNS queries put BT’s broadband and other service offerings at risk of slowdowns or outages. After exploring the alternatives, BT turned to Nominum’s Foundation Caching Name Server (CNS) to create a highly scalable DNS infrastructure that could handle dramatic increases in queries caused by both legitimate query growth and the effects of worms and viruses.

The Challenge: A Dramatic Increase in DNS Queries
Today, BT serves more than 20 million business and residential customers in the U.K. Broadband services are very popular; the company reached its target of 5 million broadband subscribers a year ahead of plan. BT’s profits were up 44 percent in the first quarter of calendar year 2005, and much of that growth is attributed to broadband services.

But subscriber growth alone couldn’t account for the increasing traffic on the company’s DNS servers. DNS queries doubled between May 2003 and February 2004, then grew threefold in the following three months.

Internet viruses and worms are the likely culprit. Whether or not they target DNS services, most viruses and worms generate DNS traffic as they attempt to propagate themselves. DNS overloads are a side effect of worm or virus attacks, magnifying their potential damage. If DNS servers cannot respond to legitimate requests, then network services essentially are unavailable for subscribers.

Exploring Alternatives for Expanding DNS Capacity
Faced with 1.2 billion DNS queries per day, BT had to act quickly to increase the capacity of its DNS infrastructure to ensure that customers had fast and reliable access to network services. The company’s BIND 9 DNS servers reached 100 percent utilization several times a day, offering little breathing room to handle unexpected peaks in usage.

The company attempted several alternatives for improving DNS throughput and performance.

Scaling up the DNS servers
By using faster hardware to run the DNS servers, BT achieved a small measure of throughput improvement. According to Jim Cavanagh, BT applications platform manager, using faster, more costly hardware “kept our head above water, but involved significant management and deployment effort.”

Scaling out by adding DNS servers
BT also tried adding DNS servers to stay ahead of demand. This is a common response to growing DNS traffic volumes. As with the scale-up alternative, this approach required considerable management and deployment effort, including reconfiguring DNS across many BT routers.

Load-balancing
BT used third-party switches to direct DNS traffic to the least-used servers. This load-balancing approach introduced another layer of complexity into the IP address infrastructure and failed to provide sufficient growth capacity at a reasonable cost.

After attempting these alternatives, it became clear that the most cost-effective way to improve DNS infrastructure capacity was to use more efficient DNS software.

After testing, BT deployed CNS in May 2004. The result was an immediate improvement in both the scalability and manageability of the DNS infrastructure.

Fewer servers
Today BT uses 10 DNS servers running CNS, fewer than required before. Consolidating DNS servers reduces the overall network management overhead.

Commodity servers
CNS runs on commodity servers rather than the higher-priced servers attempted previously to improve performance. If BT should add another CNS server, it won’t need to make a steep hardware investment to do so.

Significant performance headroom
The CNS-based infrastructure is handling 1.3 billion queries per day with ample spare capacity. Even at peak query loads (approximately 7,700 queries per second), the servers running CNS never exceed 17 percent CPU capacity. This performance “headroom” means that the BT servers are resilient to traffic spikes caused by widespread viruses or worms, and even to DoS attacks targeting DNS servers.

BT chose CNS for its superior performance compared with other DNS servers, saying, “The strategy has paid off, giving us a much greater total DNS capacity without a large investment in servers and management overhead.”

Ready for the Future
This is a critical period for communication service providers like BT that are counting on IP-based network services to drive profitability and growth. BT has sustained its success over the years by offering long-term value to customers and shareholders alike. Controlling costs is essential for offering competitive pricing. Subscribers expect the highest possible levels of performance and availability, and competitors are ready to take advantage of any problems a provider might have.

Using Nominum CNS, BT has created an efficient, highly scalable DNS infrastructure ready to handle new services and subscribers in a competitive and fast-moving market.

 
Jim Cavanagh is the IP Applications Program Manager for British Telecom Wholesale. He can be reached at jim.cavanagh@bt.com, or contact Nominum at info@nominum.com.

BT www.bt.com
Nominum www.nominum.com

Share this article: Email, Slashdot, Digg, Del.icio.us, Yahoo!MyWeb, Windows Live Favorites, Furl
Add this article feed to: RSS, My Yahoo, Newsgator, Bloglines