ETS Manages Dramatic Growth on Next-Gen Storage and Recovery Platform
Client:Educational Testing Service (ETS)
- Tape backup approach unable to keep up with rapidly growing storage demands
- Need to complete backups within an eight-hour overnight backup window
- Incorporate all of ETS' systems and services into an end-to-end storage architecture
- Construct a dedicated 10 gigabit backup network
- Transform ETS’ storage estate with CSC Storage as a Service
- All backup data now replicated in real time and available for rapid recovery 24x7
- Backup times improved over 50 percent and a 320Gb database restored under 20 minutes
- A full recovery completed in eight hours
Founded in 1947, Educational Testing Service (ETS), a non-profit organization, is the world’s largest educational measurement and research organization. It administers more than 60 million achievement and college admissions tests each year — for individuals, educational institutions and government agencies at more than 9,000 locations in more than 180 countries.
CSC has provided comprehensive infrastructure outsourcing services to ETS since 2001. In 2011, ETS and CSC renewed a comprehensive contract that includes cybersecurity, virtualization, cloud services and CSC Storage as a Service.
Securely store growing amounts of data
The standardized testing business is experiencing a period of rapid growth. Government-mandated testing requirements at the elementary and high school levels are driving significant expansion within the U.S. Tremendous growth is also occurring across the world as emerging economies expand educational opportunities and ETS expands its offerings to meet the demand. With growth, however, comes the need to securely store rapidly-growing amounts of data.
Since ETS conducts multiple trend reports that compare new test results with past results, it keeps extensive data records that date back many years. Also, ETS has recently begun incorporating audio and video recording into tests, further adding to its data storage needs.
Due to the cyclical nature of its testing schedule, ETS requires the ability to rapidly scale storage capacity during peak periods. Some tests, such as the College Board’s Scholastic Aptitude Test (SAT), are administered six times every year, whereas other tests are administered only once a year. Each set of tests, and test results, produces a surge of new data that needs to be stored.
As ETS’ storage needs began to surge, the traditional storage and recovery processes — backing up data to magnetic tapes and shipping the tapes daily to underground storage vaults — were no longer sufficient. Typical backups, which had to be completed within an eight-hour window overnight, were taking 10 to 14 hours — sometimes running into the beginning of business hours, with the potential to disrupt ETS’ business operations.
As ETS expanded globally, operating across all time zones, testing data began coming in at all hours. A second shift was added to manage the global testing. But as new data flowed in during the night-time hours, the backup window shrank — requiring more data to be backed up in less time.
A greater challenge involved the potential for lost data in the event of an outage. Though most data was backed up daily, a full backup was conducted only once a week each Sunday. If a natural disaster caused a severe outage on a Friday, the potential existed to lose an entire week of data.
Data recovery posed a cause for worry. ETS is occasionally asked to provide specific records for fraud investigations, such as test-taking impersonation. Once the correct magnetic backup tapes are located, ETS had to restore the data — which could take some time. A single database would take approximately 14 hours to be restored. It could take up to three days to do a complete data recovery. And the organization had a limited number of tape drives to restore the data.
Security was also a big concern. There is no highly secure way to protect magnetic tapes fully. Motivated adversaries, who desire to compromise data, had no shortage of opportunities to gain access to the tapes.
Building an end-to-end storage architecture
To address the rapidly growing volume of data, CSC transformed ETS’ storage estate with CSC Storage as a Service, a disk-based storage and recovery utility service.
CSC first built an end-to-end storage architecture that incorporated all of the nonprofit’s systems and services. The ETS storage architecture was based on a standard CSC design that has been validated and refined across multiple CSC client implementations and that incorporates multiple CSC best practices.
Implementation of the new storage system required deep knowledge of technology from several different vendors, including Oracle, Symantic, Cisco and EMC — as well as an understanding of ETS’ IT landscape and the ability to seamlessly integrate the new technologies with ETS’ systems.
The implementation also required the project team to design and construct a dedicated 10 gigabit backup network. By separating all of its backup processes from the production network, the high-bandwidth infrastructure significantly improves performance and speeds backups.
CSC was then able to transform ETS’ storage estate with our disk-based, utility backup and recovery offerings, which include CSC Storage as a Service — Store, Recover and Archive. CSC Storage as a Service — Store is the utility-based service for storing and managing primary file and block-level data.
CSC Storage as a Service — Archive is a utility service that captures, categorizes, indexes and stores files and email. And CSC Storage as a Service — Recover is a disk-based utility data-protection service.
With CSC Storage as a Service, ETS pays only for the data storage used. These solutions not only dramatically speed up backup and recovery performance and improve security, but they cost less than magnetic backup tapes.
The entire implementation was guided by CSC Catalyst, CSC’s strategic methodology for delivering orderly business change. Catalyst provided a framework, language and project-management processes that allowed CSC and ETS staff members and data base administrators to collaborate closely and efficiently.
The end result was a seamless implementation that came in on budget and on time and successfully addressed all ETS storage challenges without disrupting business operations.
Improving operations and stability
By transforming its storage estate with CSC’s utility, disk-based storage, archive and recovery solutions, ETS is enjoying several benefits:
- Elimination of Backup Tapes – All ETS data is now replicated onto disk as it is produced and is available for rapid recovery 24 hours a day
- Improved Security – Rather than reside on magnetic tapes stored in underground storage facilities, all ETS data is stored using advanced encryption technology, within CSC data centers
- Ability to Add Storage without Server Downtime – In the past, ETS’ systems had to be taken offline to accommodate storage upgrades. This impacted business operations. ETS is now able to rapidly scale its storage needs during peak testing periods by simply requesting additional storage space as it is needed
- Flexible Storage Levels – ETS now has the flexibility to move its backed-up data among different storage-level tiers. The organization can place its business-critical data into the platinum storage tier, but move less business-critical data to lower-cost storage tiers — and move data among tiers at any time
- Improved Backup Times – CSC increased ETS’ storage bandwidth to 10 gigabits. Data recovery times have been improved by 50 percent or better across the storage estate
- Improved Recovery Times – ETS is also enjoying significantly improved recovery times. A 320 gigabyte database can be restored in fewer than 20 minutes. The entire ETS storage estate can be restored in fewer than eight hours.
"CSC has more than halved the time required to perform our back-ups," says Brad Peiffer, ETS director of IT. “Disk-based storage improves our operations and provides a far more stable environment. It also allows us, during peak Advanced Placement Testing time frames, to complete back-ups within an ever shrinking operational backup window."
Now, when ETS is asked to provide specific records to law enforcement agencies investigating testing fraud, ETS can locate the records in under 20 minutes.
"By eliminating the need to retrieve tapes from our offsite storage location and engage a service delivery team to restore them, we have greatly improved our operations," says Peiffer.ETS’ modernized storage estate has delivered the means to keep the nonprofit well ahead of its rapidly growing storage and recovery needs. This has freed ETS to pursue new business opportunities, enjoy reduced capital spending on storage, locate needed backup files much more quickly and enjoy improved business agility.