A Look Under the Hood at Abt’s Analytic Strategy: Part 2: The Platform and the Implementation

In the previous post we presented the challenges Abt had with multiple data and analytic environments and our vision of the future. In this post we lay out our key design decisions and implementation plan.

After determining that we wanted an integrated, yet multi-faceted platform offering the highest level of security compliance, the first design decision was to move everything to the cloud. The cloud provides us with the ability to scale up and down quickly and pay only for what we use. We chose Amazon Web Services (AWS) as our cloud provider because it provided a baseline infrastructure that met the Government’s FedRAMP security compliance requirements. AWS also provides the ability to create different types of servers for different products. For example, some analytic tools such as SAS require the ability to read and write to disks quickly. Stata and R require more memory. Thus we had the flexibility to match the server to the requirements of distinct analytic tools.

The overall cloud platform is called ACE3 since it is the third generation of our Analytic Computing Environment. Previously we had moved from desktop-based processing (ACE1) to an on-premise server (ACE2).

We devised a three-year strategy to turn our vision into reality.

In Year One, we concentrated on ensuring that we had all the data collection, analytic and dissemination tools on ACE3 that we use anywhere in the company. This included tools on any of our servers or on user computers. Since analytics is one of our core services we offer to clients, we invested in enterprise versions of each analytic tool.  And we implemented tools that made working on the platform easier for our staff. For example, we deployed a tool that scanned data copied out of the platform and intelligently identifies sensitive information. This enabled staff to cut and paste out text, code or non-sensitive data from the secure cloud environment without risking accidentally or intentionally moving sensitive data to locations where they don’t belong.

Once tools were in place, we migrated all of our data and our users out of ACE2 and decommissioned that platform, which cut costs. We also deployed our multi-modal survey data collection platform in AWS and linked it directly to the analytic environment. We moved data off of personal laptops.  We also conducted training on several of the tools that were newly available more broadly, including R, Python, ArcGIS and Tableau. We staged an “Expo” for our staff so they were aware of all the available products. These efforts helped advance the “People” prong of our strategy.

In Year Two, we are completing the migrations from the environments that our acquired companies manage. Most significantly, we are migrating all of our survey work from our legacy data center platform to the new cloud-based systems. This includes rebuilding a host of supporting tools that do cross-survey reporting, tracking of call center activity and linking to our financial systems to process incentive payments.  We are improving the overall architecture of the platform to improve balance of loads and the user experience.  We are also concentrating on tools for machine learning with an emphasis on natural language processing.

The original vision included a standardization of processes in addition to the platforms and people. So in Year Two, we are curating the best practices of Abt’s senior analytic and data programming staff. These include standardizing our approach to documentation, code review, reproducibility and data stewardship. We are both creating the processes and piloting them. Standard processes along with a single platform will facilitate easier collaboration among Abt staff and help provide an even greater assurance of consistent quality for our clients.

The plan for Year Three is to ensure training of our analytic staff on both the standard processes and the standard platform. We will determine the procedures for moving successful solutions from the Laboratory to the Factory. And we will develop the next long-term plan to ensure that our data and analytic capabilities continue to grow. That will enable our clients to benefit from all of the latest insights and developments of the ever-changing data and analytics landscape.