Rabobank Nederland overcomes complexity and achieves reliable DNS thanks to Infoblox DNS and DHCP.
Part of the Rabobank Group, Rabobank Nederland develops policies and products across a wide range of areas including lending, payments, securities services, customer services, marketing, distribution, and personnel policy. It also operates its own banking business: Rabobank International. Rabobank International engages in international wholesale and retail banking activities, with a special focus on the food and agribusiness sectors.
As the manager of data-center networks at Rabobanks, Olav Roes is responsible for a network that spans the Rabobank data centers in Best and Boxtel. “The overall structure of the network is actually not that complex,” he says. “But when you look at it in more detail, you see that the layers consist of thousands of components which all have a function in the data center, and which need to be able to talk to each other.”
At the same time, the dependency on that complex network is enormous. Not only does the network enable the 35,000 employees to do their jobs, but it also forms the basis of the online services for the millions of Rabobank customers. “And this online presence of Rabobank must always be available via the various channels, without ever faltering,” says Roes. “If Internet banking can’t be accessed, for instance, that gets posted immediately on Twitter and NU.nl. That level of visibility really is enormous. And that is also where our great challenge lies: how do we guarantee that our online services are available to our customers 24x7?”
The Solution and Results
A first measure, which Rabobank implemented a little under ten years ago, is the sound design of network services such as Doman Name System (DNS) and Dynamic Host Configuration Protocol (DHCP). “Wanting to be permanently accessible also means that the customer who types www.rabobank.nl must end up on our pages,” says Roes. “Years ago many companies still had their DNS and DHCP running on a Linux or Windows machine somewhere in the data center. Gradually everything has started to be based around the acquisition of an IP address, and the Domain Name System has become increasingly important—to such an extent that you need a sound solution for it.”
Rabobank selected Infoblox appliances in 2004 for its internal as well as external DNS. “These are data-center-ready appliances,” says Roes, “with many added security measures to ensure, among other things, that a request is not turned to a different address.” In December 2012 Rabobank moved to a new Infoblox infrastructure, and 60 appliances were replaced in one evening without any downtime. “During this open-heart surgery not only the software but also the hardware was replaced,” says Roes.
Shortening lead times
A second step in guaranteeing online services was the provision of services through separate channels, to ensure that, for instance, an Internet banking failure doesn’t affect Rabobank services. By giving each service its own private access route rather than providing everything through one wide-access lane, failures can be kept as brief as possible because of the diminished likelihood that one failure will lead to new failures.
“We then were faced with the question of how to reduce a failure of an hour to a few milliseconds,” continues Roes. “In the Netherlands we usually think it decent enough if an incident is solved within an hour, but in fact that isn’t decent at all, because the impact of incidents is growing all the time.”
According to Roes, the answer to that is found in automation. “An engineer must be presented with everything: a system has already logged on, the logs have been searched, and ten possible causes have been found, of which three are most likely.” Better yet, of course, is the prevention of incidents and failures. “The challenge is to be proactive,” says Roes. “You have to act on the first signal by a component. If you are able to combine proactivity with a shortened lead time when there is an actual problem, you have found yourself a fortune.”
Rabobank used 2012 as a “year of reflection” to consider how to handle matters differently in order to arrive at more stable services. “We have technologies such as monitoring and automation tooling in house,” says Roes. “The question is to what extent they will serve us.”
A next step for Rabobank is to analyze the current setup of the data-center network. And on this level too, automation is important. For instance, virtualization has led to disconnection as a result of which the “thousands of components” Roes refers to can’t be physically identified. “Because of increased complexity, you also need more people to keep it all going,” says Roes. “That is not the way forward. We need to bring automation into our IT systems. And we should extract the logic that is now contained in the engineers’ minds—who must start at the same level each time when trouble shooting—and turn that into a logic that does the work for us.”
“We should extract the logic that is now contained in the engineers’ minds—and turn that into a logic that does the work for us.”
After a radical overhaul of the data center in Best, Rabobank started building a new data center in Boxtel in 2009. The sustainable group of buildings, which was opened in April 2010, consists of an office block and a computer center over two floors, and is supplied by four energy-provision buildings. The data centers in Best and Boxtel are connected according to the Twincenter concept, which enables data processing and storage in two locations. “Within this active-active concept,” says Roes. “one half is allowed to fail, as the other half is identical. We could run on only one of them.”