When it began its pre-emptive bi-annual migration in 2009 for business continuity during hurricane season, Woodforest National Bank quickly discovered the tools to replicate a 100 percent virtual environment were insufficient. “We needed two hardware-based solutions, from different vendors, to handle our virtualized servers and storage,” explains Richard Ferrara, CTO and EVP at The Woodlands, Texas, institution. “The process was cumbersome and script-based. There were various issues for which we had to develop workarounds.”
[Richard Ferrara was recognized in 2011 as an Elite 8 executive in part for leading the bank’s virtualization strategy: Woodforest National Bank CTO Richard Ferrara: Using Virtualization to Hurricane-Proof]
By 2011 Woodforest ($3.8 billion in total assets) was migrating its entire environment and had managed to reduce the duration from a weekend to an overnight. But, the tools remained inferior and, with all traffic travelling through a hardware box, they also created a single point of failure. Then, late in the year, Woodforest was approached by representatives of a new company, Zerto.
“Zerto’s founders were familiar with us because they had originally developed one of the two existing replication solutions we were using,” says Ferrara, who pilots an IT organization of about 130 people. “Their new Zerto solution combined the functionality we needed into one tool and it was more flexible. But, we were concerned it wasn’t mature enough.”
Woodforest eventually partnered with the startup in the late spring of 2012 to test the software solution and provide feedback. “Our team showed Zerto our entire process and the limitations of their solution as it applied to us,” says Andrew Wright, server and storage administrator at Woodforest. “They incorporated most of our suggestions throughout the next product iterations.”
By early 2013 Woodforest had successfully completed several large test migrations and began disengaging its systems from of the two existing tools. “The existing solutions operated within the data stream,” says Marcus Lohr, VP and network operations manager. “Our entire environment had to be moved.”
Then a serious disruption occurred. On a Saturday, with all 750 Woodforest branches open in 17 states, most storage area network (SAN) connectivity was lost, forcing branches to conduct limited business offline for six hours. “Essentially, one of the existing solutions blew up,” says Wright. “Richard [Ferrara] was away celebrating a wedding anniversary and Marcus [Lohr] was on vacation.”
“Yes, my wife was none too pleased that I worked remotely until 11:00pm that evening,” quips Ferrara. “Some of our branches operate 24/7 and of course ATM machines are always on. After that incident we turned off the existing solutions and finished integrating Zerto.”
Throughout the balance of the summer and fall Woodforest and Zerto hammered out outstanding issues in advance of the first complete migration from the hurricane-season site in Austin back to The Woodlands. “Each time Zerto supplied us with an update, we’d do a test migration,” Wright says.
When the first complete migration rolled around in early November 2013, Zerto replicated 204 virtual machines on 24 hosts and 72TB of data with no individual system unavailable for more than about 15 minutes. Transition required about five hours for the six-person migration team to complete with just one hiccup. “The incident took Zerto about an hour to locate and correct,” says Lohr. “We’ve been assured that bug will be fixed in the next release.”
Reduced Complexity, Improved Virtualization
Regardless, Woodforest considers Zerto a home run. “It provides us with granularity that’s very VMware-centric,” says Wright. “Removing the hardware required for the other two solutions has reduced complexity and improved virtualization.
“Not only can manage everything from a single pane of glass,” he continues, “we’ve eliminated the single point of failure -- if Zerto fails, our production environment will be unaffected.”
With data growth running at 20 to 30 percent annually, the Zerto solution is also more cost-effective, notes Lohr. “Zerto is licensed based on the number of virtual machines. The previous solution was by the terabyte.”
Overall, the new setup has proven strategically significant. “Anything we can do to improve our virtualization position has a positive business continuity impact by reducing the risks and costs of our bi-annual pre-emptive migration,” say Ferrara. “As a virtualization tool, Zerto definitely helps us meet our institution’s continuity-related goals."
Institution: Woodforest National Bank (The Woodlands, Texas).
Assets: $3.8 billion.
Business Challenge: Adopt a datacenter migration solution compatible with a fully virtualized banking IT environment.
Solution: Zerto (Boston); Business continuity solution.