Better Connected: Moving Abt’s Survey Platform to the Cloud
The Challenge
Many organizations are struggling to take advantage of cloud computing without interrupting their core business and mission. Two years ago, Abt was facing that challenge. Our platform consisted of three different survey applications, all housed in a traditional, “brick and mortar” data center. This was difficult to maintain and limited the efficiency of critical capabilities. For multi-mode surveys (surveys conducted over phone, web, in-person, SMS, and through other distinct means), we had to use multiple tools that required inefficient manual interventions between systems and multiple telecom providers, which created a complex web for both system maintenance and reporting.
The process of migrating to a new platform wasn’t going to be much easier. We would have to reprogram surveys, and some of the projects had security approval from clients that were tied specifically to the then-current platform. Any changes we made to our survey and software tools had to also account for the larger ecosystem of tools that we used to pull samples, pay incentives, and manage the call center staff. (Figure 1 depicts both our old state, including three different applications and two different supporting corporate environments, and the proposed future state.) Fortunately we had experience transforming this type of infrastructure through the launch of our analytic computing environment (ACE), which supports all of our data science and statistical work, and had already been consolidated and migrated to the cloud. The following sections describe the plan for moving our survey activities: what we accomplished in each of two phases, and the results.
Overview: Consolidating for Greater Flexibility
In May 2018 we made the decision to migrate all aspects of our survey technology to the cloud using Amazon Web Services (AWS). Over two years, in one large coordinated effort, we moved out of our outmoded data center to take advantage of the scalability and high performance features of AWS. We consolidated our three survey applications into one modern, fully supported system and we are now able to conduct all surveys on the same platform. This includes not only phone, in-person, and web, but also interactive voice response (IVR), Audio Computer-Assisted Self-Interview (ACASI), and SMS. We also worked with a new vendor, Sytel, to upgrade our survey call center telephony, both on the networking side and also for our workstations. Sytel’s softphones, which replace physical phones with software, are now accessible to all of our call centers and contractor locations.
Through this process we also sought to ensure ongoing Federal Information Security Management Act (FISMA) Moderate security and Telephone Consumer Protection Act (TCPA) compliance. We tightly integrated our data collection environment with our primary analytic environment, which dovetailed with our overall analytic strategy. Best of all, we reduced costs by eliminating many of the legacy data center’s inefficiencies, eliminating the duplication of licensing costs, and reducing system administration tasks.
Phase One: Deploy Core System and Support One Fully Functional Project
During the first phase we built out the AWS architecture, deployed the Confirmit Horizons survey software and the Sytel telephony and deployed the surveys for the first big project to use the new platform. We also implemented all of the security controls and the data connectivity between the survey platform and ACE. Neither Confirmit nor Sytel had fully deployed client systems to AWS before, so we pioneered the effort and worked together to advance all of our technologies.
We went live in March 2019, becoming the first customers of Confirmit and Sytel to have fully functional multi-modal survey capability in the cloud, including web, phone, and in-person. A diagram of the data flow for the survey environment is shown in Figure 2.
Phase Two: Migrate all Projects, Supporting Systems and Data
While the goal of Phase One was to have one fully functioning project on the new system, in Phase Two we moved all our survey activity to the cloud and fully decommissioned our data center. Key streams of work for this phase included
- Moving all the active surveys to the new platform. We carefully planned this activity to allow as many projects as possible to end on the old system while starting new projects on the new one over the course of a year. Thus, we only had a handful of projects that had to migrate during live survey collection, which greatly minimized risk.
- Deploying a fully featured staging environment to test critical upgrades prior to deployment.
- Building new reports that run in real-time and can report across modes.
- Tightening security and obtaining a new “authorization to operate” on the new platform from a key Federal client
- “Lifting and shifting” the numerous supporting applications, including the incentive check system, the call center scheduling and monitoring system, the custom databases for maintaining project samples.
- Transferring systems to scrub out the data on “do not call” and mobile phone users, in order to ensure that we maintain TCPA compliance by never auto-dialing mobile phones.
- Implementing SMS tools for activities such as in-person appointment reminders, inbound calling, and soft transferring of calls to supervisors and external hotlines
- Fully decommissioning all servers in in the data center
Figure 3 depicts the current comprehensive survey ecosystem after Phase 2.
The Result
As of May 13, 2020, almost two years to the day from our meeting to launch this project, we turned off our last server in our data center. Abt now has fully migrated all of our survey work to the cloud, and integrated all of our survey projects into a single platform across all modes. We have streamlined our survey programming and implementation processes, consolidated our reporting, and fully connected all of our supporting services and analytics. Our survey collection, as complex and robust as it is, is now a part of our larger “Data Collection and Analytics Computing Platform” (DC-ACE). Taken together, DC-ACE allows us to seamlessly handle data across the entire data lifecycle, beginning with data collection, continuing with analytics, and culminating with dissemination and data visualization.
Epilogue: Reaping the Rewards During COVID-19
This new platform has served us very well in our response to COVID-19. Since our entire infrastructure—including telephony—was in the cloud, we were able to quickly deploy a solution for a “virtual call center.” Our interviewers can work at home using a virtual desktop with equally good performance as they did in the office while observing social distancing. This cloud-based platform gives us the scale and flexibility to serve our clients in even the most challenging circumstances.
