Bank of the West Shaves Server Use With Terracotta

Continually expanding server resources without receiving a corresponding increase in application performance -- speed, efficiency, load and reliability -- was an ongoing frustration for Bank of the West (San Francisco; $66 billion in assets). The overriding problem facing our development services team was that many of our Web-based applications depended on only two Sun Microsystems (Santa Clara, Calif.) servers running the Solaris operating system. Twenty-four applications -- customer, partner and internal applications -- were deployed on these two servers. Reliability was critical, as hundreds of bank employees and thousands of external users accessed these applications daily.

The bank's Java (Sun Microsystems) application infrastructure included six Apache Tomcat (www.apache.org) open source instances, which create a dedicated Web server environment for Java code, on each physical server; two people managed the Tomcat environments. But running six Tomcat environments on each physical server strained system resources, creating bottlenecks and requiring IT staff to spend valuable time monitoring and trouble-shooting the servers. Continually adding Tomcat servers to solve reliability and scale constraints was an expensive proposition.

Bank engineers also faced database scale limitations: Employees ran daily business reports via a Web-based interface, but the database layer couldn't handle more than two concurrent users. Pulling data from the data warehouse could take as long as 30 minutes for some reports, and the users' browsers would often time-out before their reports were completed.

An enterprise broadcast system (EBS) compounded the technical challenges. This is used in lieu of e-mail to notify employees of emergencies, as many of Bank of the West's employees do not have e-mail access for security reasons. While the EBS ran on two additional servers, it shared those servers with various other applications and a database. Further, every desktop in the bank was connected to the servers at all times (enabling engineers to quickly alert bank staff to problems even if the e-mail system was down). The EBS handled hundreds of requests per second from every computer in the bank -- about 7,000 in all, each of which checked in approximately every 30 seconds.

We required a product that could help improve customer service; simplify internal development, monitoring and troubleshooting; eliminate costly database purchases; and generally improve overall performance of the bank's infrastructure applications. So in mid-2008 Bank of the West selected Java infrastructure software from San Francisco-based open source software company Terracotta.

The Terracotta software is designed to provide scalability for enterprise Java applications, enabling banks to reduce development and tuning effort while cutting ongoing operating costs. Until we deployed Terracotta, our applications required two Tomcat environments each to minimize the chance of downtime. Eventually the growing number of Tomcat servers became unmanageable. Now Terracotta provides seamless session replication between servers -- both physical servers and Tomcat environments -- so if a server fails, requests are immediately transferred to another server without interrupting user sessions. We can consolidate all of our Tomcat servers and even shut them down for maintenance without interruption. Additionally there's a lot less memory overhead since we don't need to run so many Java processes on each machine.

Terracotta's development and testing approach also enabled our developers to move legacy Web applications quickly to the bank's Java cluster. Terracotta's compatibility with many open source projects and enterprise Java application servers made the move virtually painless, and most applications required no changes to move to the new environment, which saved our engineers a lot of time and effort.

With the new system, running our daily internal reports became much easier. Response time and reliability are critical to the success of these reports, and we haven't had a single unplanned outage since first deploying Terracotta in December 2008.

Cache Rewards

Terracotta also is being used to offload database requests. Warehouse data can be cached indefinitely with Terracotta, which has really eased the load on the database. We can cache most of the data in Terracotta and use the database less frequently so that it responds more quickly when needed. Not only is our application response faster -- average response time is now less than five minutes -- but we also didn't have to spend $50,000 for a new database to handle the load on the data warehouse.

Bank developers have also used Terracotta to support our distributed alert system, EBS. Rather than buy a new standalone database, message queue or application server to support EBS, the bank uses Terracotta's distributed caching features to minimize demand on its existing database. With the caching solution, EBS handles thousands of requests with virtually no server load and now shares system resources with other applications without slowing them down. The bank is also able to shut down servers and restart them without losing the server cache. Average response time is less than 5 milliseconds. The server is answering client requests so fast that there are far fewer requests running at the same time, so the server doesn't have to work as hard as it would have without Terracotta.

Mike Johnson is a senior application engineer with Bank of the West. He's responsible for creating and maintaining applications, database development and front-end design.

Case Study Snapshot

Institution: Bank of the West (San Francisco)

Assets: $66 billion

Business Challenge: Deliver highly reliable Java applications for customers, partners and employees within existing server and database infrastructure

Solution: Terracotta's (San Francisco) v. 2.7 Java infrastructure management solution