UC Berkeley

Welcoming Infoblox to the Ancestral Home of BIND

While Berkeley’s IP networking system met the university’s basic needs, its network management team found it was utilizing resources inefficiently. Multiple employees were required to maintain the legacy BIND system, which kept operational costs high and ran contrary to the university’s reputation as a technological innovator.

Compounding the challenge of an aging infrastructure was a network that is becoming increasingly complex as the number of connected devices climbs. With student enrollment expected to increase by 10,000 students by the 2018 – 2019 school year, the impact of service interruptions is increasing and has the potential to impact multi-national and interdisciplinary research projects. Additionally, the BIND legacy IPv4-capable network was unable to handle automated IPv6 addressing, requiring manual servicing of requests to meet the needs of a growing campus. Also, its current DNSSEC deployment required additional work hours for complex scripting when DNS changes were needed, potentially putting the entire network at risk in the event of errors.

Isaac Orr manages the Network Operations and Services Group responsible for the campus data network at the University of California, Berkeley. “It’s a large network,” he says. “We have around 60,000 wired ports, 4,500 access points, and 115,000 devices connected to the network. The group I manage ranges all the way from field installation technicians to more senior people involved in larger projects, such as deploying networking to new buildings on campus or developing new network services. That encompasses pretty much everything from DNS hosts to iPads and phones on the Wi-Fi network.”

Ironically, before Infoblox, the defacto standard used to manage core network services was BIND (“Berkeley Internet Name Domain”) which was developed and built by top experts at UC Berkeley. “IP networking has been big here since IP networking existed,” Orr shares. “Everything we had was actually custom built in house quite some time ago. The system was based on a PostScript database with PERL scripts.”

Functionality was not a problem; the legacy BIND system worked as it was supposed to, but IT no longer had the resources to maintain and continue its development. Since the last major work had been done on it in 2003, it had begun to lag behind in some of the things that Orr’s internal customers needed. “The BIND-based system was fairly inflexible in terms of making changes,” Orr says, “and having somebody to care for and feed the servers that did all those scripts was consuming two employee’s time—one of them a senior networking person.”

Looking for something more efficient that would lower total cost of ownership (TCO), Orr’s team selected Infoblox to manage its large and diverse network. “Infoblox has a very good reputation in the industry,” says Orr, “and we knew of other University of California campuses that had deployed Infoblox solutions. We also had people within our own group who had worked with it and were impressed.

“So we already had a pretty high opinion of the product, and when we started comparing it with other solutions in the marketplace, we concluded that it had the fullest feature set, and would allow us to do a lot of stuff.” The university purchased both physical and virtual Infoblox appliances running DNS, DHCP, and IP address management (DDI) on the centrally managed Infoblox Grid™ architecture.

Integration, Migration, and Simplification

By “allow us to do a lot of stuff,” Orr means, among other things, using APIs to integrate systems and automate tasks, using role-based access to allow the virtualization team to control its own zones, simplifying DNSSEC assignment, transitioning to IPv6, and flexibly combining virtual and physical appliances to leverage infrastructure already in place.

One of the first things Orr mentions is the APIs for integrating Infoblox DDI with solutions from other vendors, custom code built by the university, and with legacy BIND. As an example of what his team has done with these APIs, he cites a portal where campus users register the MAC addresses of their devices to their university-wide ID. If someone’s MAC address isn’t registered, that person doesn’t get DHCP or an IP address. And if a registered device is compromised, IT can block it from getting DHCP or an IP address. So the portal is both a way of providing access, and controlling it.

It also allows people to sign up for dynamic DNS. A lot of the back-end work involved in this is enabled by APIs. “This simplifies a lot of things for end users,” says Orr. “Having an API and an extensible database made it really simple to just uplift everything we were already using and drop it into Infoblox. It was next to nothing to create the application to talk to the database for our user portal.”

Infoblox Extensible Attributes is the key enabler. In the distributed environment typical of colleges and universities, individual departments have their own IT operations. For the central IT team, this meant that one tech might be the security contact for 300 devices. “We built our own web portal with its own concept of roles, since it handles other things than DNS/IPAM information. We extended its concept of roles into Infoblox via Extensible Attributes.”

Berkeley’s IT team also uses Infoblox’s Extensible Attribute fields to track contextual network data to help prioritize their operational actions to identify, prioritize, and remediate issues. “What we have is an Extensible Attribute in Infoblox that tells the security folks who is responsible for what network. Information related to that is stored in EAs on the network and host objects in Infoblox,” says Orr.

Another feature that has come in handy is Infoblox role-based access control. “Our virtualization team now has control over some of their own subnets and zones to handle address allocation and server naming,” Orr says. “It allows them to build automated workflows for provisioning in our private cloud infrastructure.”

The People Side of the Solutions

To help with the migration of data, Berkeley leveraged Infoblox Professional Services. “We wanted somebody who could actually work with us in terms of converting everything to Infoblox, who understood the product, and who could make recommendations about how it should be deployed and so on,” explains Orr. “Infoblox Professional Services was very reasonably priced, and we were very impressed. The person we worked with had a lot of experience in the educational environment, understood what we were trying to do, was very knowledgeable, and at the end of the day, really quite helpful.”

Infoblox Support has also helped out. “We worked with the support team to develop some enhancements to the solution,” says Orr. “It’s always nice when you’re working with a vendor and you say, ‘hey, we think it should really do this,’ and the vendor says, ‘you’re probably right,’ and they make it happen. To me, that’s the best sort of support experience you can have with a vendor.”

Taking the Complexity out of DNSSEC

Berkeley was an early adopter of DNSSEC, and for some time it has been signing all of Berkeley’s major zones. However, this required multiple scripts that had to be activated to re-sign all those zone files—which meant that making iterative changes to DNS was actually quite difficult. It worked, but it also created the possibility of errors that Orr says could break the whole system. “The worst-case scenario,” he says, “would be that the Berkeley name space would be unresolvable, and from a campus perspective, that would be pretty huge.” Infoblox features one-click, automated DNSSEC deployment. (See “A Best Practices Architecture for DNSSEC” by Cricket Liu.)

Taking the Lead in IPv6

Berkeley IT was well aware of the need to transition to the new IP address protocol before available IPv4 addresses were exhausted, but Orr points out that the legacy system didn’t understand IPv6 addresses at all. There was no way within the custom-built IPAM solution to allocate IPv6 address space, so they were doing it manually, and they had separate DNS servers that did the zones for IPv6.

“If somebody asked us for IPv6, we would provide it,” says Orr, “but it wasn’t a thing that we could just turn on everywhere and make available. With Infoblox we can. You have to remember that from our perspective, we look much more like a service provider than an enterprise IT department. So we have to allocate IP address space on campus just as a service provider does. Infoblox is perfect for that, and UC Berkeley is now a leader in terms of IPv6 instead of lagging behind.”

Virtualization Where It Makes Sense

Berkeley’s Infoblox implementation includes a mixture of physical and virtual appliances. “We already had virtualized infrastructure for the DNS BIND servers, so it made sense to deploy virtual appliances wherever we could,” says Orr. “But for our Grid Masters, based on scale, that wasn’t the right choice, and we have some remote Infoblox devices for redundancy that needed to be physical as well.” Managing this mixture of virtual and physical presents no problems, since the Infoblox Grid seamlessly manages the two in concert.

Economy, Speed, and Simplicity

When asked about the benefits Berkeley has gained, the first thing Orr mentions is reduction in TCO. “We’ve saved $75,000 per year on the senior labor we were using to help manage and maintain the DNS infrastructure.”

Time savings come next. “Five years ago when I started,” he says, “if you requested a new host on a subnet that was full, the process took at least two weeks to complete. Now, we’re down to three to five days. That’s a pretty big change in an organization of this size, and the Infoblox infrastructure was a big part of that.”

Then there’s the simplicity of operation. “Infoblox simplifies something that is really quite complex. At the same time, it gives us the flexibility to do the things we need to do. I would definitely recommend Infoblox to peers.”