Summary: A Vision of Health for a Better World
With a more than 100-year tradition of providing care to the poor and vulnerable, Providence Health is a major healthcare system that spans seven states in the western United States. With core values built on Compassion, Dignity, Justice, Excellence and Integrity, their organization’s vision is Health for a Better World. Together, Providence’s 120,000 caregivers serve in 52 hospitals and 1,085 clinics, providing a comprehensive range of health and social services with a commitment to providing care and assisted living to patients in need—regardless of coverage or ability to pay. As Kris Acker, Senior Network Engineer at Providence puts it, this dedication to a vision of Health for a Better World “not only characterizes our approach to patient care, but it is also the way we treat our fellow caregivers. It isn’t simply a job, but more of a family and a way of life.”
In service of this mission, Providence requires powerful networking and data services that link and empower the numerous locations and affiliates the organization maintains. For well over a decade, Providence has relied on the Infoblox NIOS DDI platform to drive its essential DNS, DHCP and IPAM operations (collectively known as DDI), and deliver reliable, high-bandwidth connectivity to all its facilities, clinicians, administrators and patients. Providence administers all DDI related services through a single platform with Infoblox, making it far more manageable for the large organization’s IT operations team. As a result, Providence is able to provide network services not only for themselves, but also for third party facilities and services partners who utilize the same systems. However, as Providence expanded through multiple merger and acquisition activities (M&A) over the years, other non-Infoblox DDI technologies were brought into the organization, which over time began to present performance challenges.
Customer: Providence Health
Location: Renton, WA
Initiative: Consolidate all DDI operations onto Infoblox NIOS DDI, Manage network from a centralized data center, Migrate remaining Microsoft DDI operations onto Infoblox NIOS DDI
Outcomes: Drove down overall queries per second from around 90K QPS to just about 30K QPS, Reduced the dynamic registration process for clients from the 10- to - 15 minute range in the legacy environment to nearly immediately resolvable with Infoblox, Consolidated management of DNS and DHCP onto a unified system, with a single source of truth for IPAM
Solutions: NIOS DDI
The Challenge: Overlapping systems cause issues
The M&A push began in earnest in the early 2010s, with Providence dramatically increasing the size of the organization. Most notable were the acquisition of Washington-based Swedish Health Services in 2012, and a merger with St. Joseph Health of California in 2016. A happy coincidence of the latter was that St. Joseph and Providence were both running Infoblox NIOS DDI to operate core networking operations. Yet significant differences in the design of the networks, disparate versions of software deployed and varying ages of appliances and equipment posed real challenges. While the network merger and integration of the various Infoblox systems took time and effort, it progressed over the years and ultimately came to a happy resolution. Other technical challenges posed by Providence’s M&A activities, however, weren’t so easily overcome. Specifically, several entities that were running Microsoft Windows and Active Directory to manage DDI were brought into the organization.
“Over the past few years, the biggest challenge we faced was the split responsibilities of DNS and DHCP between Infoblox and Windows,” explained Acker. With Infoblox NIOS, IP Address Management (IPAM) is automated and integrated with DNS and DHCP functionality within the platform. Not so with Microsoft, which raised several serious issues. “Because we were using both platforms for DNS and DHCP services, there was no ‘single source of truth’ for IPAM. We had some IP address data in one platform, other data being spread across a series of Microsoft DHCP servers, and still more stored in spreadsheets. This dispersal of IPAM data resulted in IP overlap, inconsistent or incorrect DNS resolutions, as well as very slow response times as DNS was having to query from platform to platform depending on the data needed. We jokingly referred to this arrangement as ‘DNS Spaghetti,’ as our query path resembled a mess of lines going from platform to platform.”
The Situation: Mapping out a new centralized system
While the Microsoft / DNS Spaghetti situation was a well-recognized problem, Acker and his team had higher priorities that had to be addressed first. Specifically, they needed to move the organization from a distributed environment to one where Providence’s massive network could be managed from a central data center.
“Through all of the M&A activity over the years, our Infoblox systems were arranged as a distributed platform, with appliances spread across multiple ministries and small locations,” explained Acker. “Our research showed us that we could implement a much more resilient architecture if we transitioned away from the distributed design and adopted a more centralized framework. And because Infoblox’s DDI technologies are so advanced, and support standards and capabilities like AnyCast, DHCP Failover and Primary/Secondary DNS Configurations, we knew we could make the transition mostly with the team and the technologies we had in place.”
As the larger centralization effort progressed, Acker and team finally set their sights on resolving the Microsoft conflict in 2022. He explained, “Our Windows Servers were nearing end of support / end of life–that was the final straw leading us to consolidate onto Infoblox and deprecate our remaining DDI operations that were still running on Windows.”
The Solution: Consolidating all DDI operations onto Infoblox NIOS DDI
Acker and his team initiated a two-stage migration, first moving more than 600 DHCP Subnets from Microsoft DHCP onto Infoblox, followed by the migration of just over one million DNS records. “Our migration into Infoblox was an amazing experience,” recounted Acker. “As a critical healthcare provider we can’t have downtime, which meant all migrations had to be done in a live environment. Our engineering team was able to complete these migrations over a two-month period, and the process was nearly transparent to the end users, with very little impact to service offerings for our enterprise environment.”
Infoblox’s Network Insight and the Grid system’s native discovery capabilities proved highly useful in this instance, enabling network administrators to view, inventory and understand all of their network elements. “Network Insight was exactly what we needed to manage the discovery phase,” said Acker. “It gave us excellent visibility and control over our environment, with deep insight into all corners of the legacy networks. This higher level of manageability set the stage for us to clean things up wherever needed, and make fast, solid progress on our vision of having a single fundamental platform to manage all aspects of our critical DNS, DHCP and IPAM operations.
The Result: Much better control over the network environment
Acker and his team couldn’t be happier with the outcomes of the Windows migration, and the larger network centralization effort as well. “The benefits that have been observed have been astonishing,” said Acker “DNS queries are measurably quicker now, resulting in much faster application response times. Also, with all domains and IPAM operations now automated and centralized within Infoblox, we have eliminated cross domain or subdomain queries between different Domain Controllers and realized positive change in several areas:
- Drove down overall queries per second from around 90K QPS to just about 30K QPS.
- Reduced the dynamic registration process for clients from the 10-to-15 minute range in the legacy
environment to nearly immediately resolvable with Infoblox.
- Consolidated management of DNS and DHCP onto a unified system, with a single source of
truth for IPAM.
- Achieved much better control and understanding over the DDI environment.
One of their biggest benefits from switching to Infoblox’s NIOS platform has been the ability to standardize configurations across the enterprise. No longer does the team experience issues with local/regional configurations; the system is now standardized and a single configuration can be applied for every device, no matter the location. “With centralized management, we’ve been able to increase our insight into DNS views and what content we’re exposing to partners, so we just feel that we have better control over our environment.”
Looking forward, Acker and team are exploring ways to strengthen their ecosystem with a hybrid approach by tapping into Infoblox’s cloud and security offerings, and increasing NetOps to increase protection for users, devices and data assets. As part of this effort, Acker and team are looking to tap into Infoblox’s growing selection of cloud and security offerings to further automate networking operations and increase protection for users, devices and data assets through APIs. “From initial trials and tests, I believe the API for Infoblox is one of the most mature API’s we have worked with,” said Acker. “There are so many capabilities the API offers that are way beyond what you can do in the GUI.”
Additionally, the team has recently moved many services into cloud platforms and are actively working to automate many networking tasks both in cloud and on-prem. “We are running some proof-of-concept testing on the Cloud Integration and Automation licenses,” related Acker. “We expect that reporting and alerting will be taking a front seat along with automation. We have run through a trial period of Infoblox Reporting and are looking to purchase a permanent license in the months ahead.”