Building a test automation culture to reduce risks and accelerate feedback
A leading global professional services firm — with approximately 245,000 employees servicing over 150 countries — struggled with the testing process of its delivery cycle while building an enterprise auditing platform.
Recurring changes in business priorities slowed down the testing process, which impacted development and delivery goals. To overcome the time-consuming manual testing process, the firm’s team needed a smarter approach to understand what needed to be tested and how. Based on AgileThought’s success with phase one of the platform build, the service firm sought AgileThought’s help to automate testing and increase delivery speed, facilitate business agility and reduce risk
- Slower feedback from manual continuous testing as part of the build process
- Difficulty executing repeatable, accurate and meaningful manual regression tests during every deployment
- Lack of thorough regression testing across all previously delivered features due to shorter release cycles
- Inability to find high-severity defects earlier in the continuous deployment process
- Time-consuming test data preparation for pre-production testing
Creating a Test Automation Culture
AgileThought’s Software Development Engineering Test (SDET) team collaborated with the firm’s product owners, business analysts and test leads for a number of reasons: To understand the business context of stories being developed every sprint; to review which features in the application were stable for automation; and to triage what automated scripts needed to be developed. Establishing this collaborative culture helped provide insight on which tests could be automated, which not only increased coverage within the sprint increment, but also furthered support for the continuous testing process.
Reducing Risks for Mission-Critical Business Areas
Given the complexity and scale of the application, the SDET team focused on key areas of critical business risks and their impacts to the overall business processes. The AgileThought team collaborated with Product Owners and reviewed:
- Areas that would cause the business to lose money
- Areas that were error-prone and had the highest historical rate of failure
- Areas that often escaped the manual testing process
- High-risk, end-to-end business processes
The team developed a business-risk-driven automated regression suite of scripts based on this review, which reduced high-to-critical severity risks before deploying to a staging and production environment.
Accelerating Feedback Response Times to 15 Minutes with Automation Testing
After integrating automated build verification tests into the deployment pipeline — and running them multiple times a day — the development team received faster feedback on critical defects that blocked testing and development. These 10-15-minute build verification tests determined the stability of the deployed application by exposing integration issues early; it also reassured the team that changes introduced during development would not adversely affect features critical for supporting ongoing work. Now, with feedback response time down to 15 minutes, the business benefits from the continuous efficiency and repeatability of build verification tests every deployment. Development teams also improved feedback on defects and bugs found during deployments, allowing them to resolve defects faster and enhance quality before the end of the Sprint.
Reducing Regression Testing Times from Days to Hours
Historically, tests were written at the UI layer and took longer to write, longer to execute, and were dependent upon the consistency of the UI. On average, every UI test would take generally 1-3 hours to script, and 4-5 minutes to execute depending upon complexity. Instead of relying on UI automation scripts — which can be flaky and slow — AgileThought leveraged the service layer to test functionality closest to production code. Because tests at the service layer are faster, less brittle, easier to maintain, and extremely lightweight, the firm’s automated test scripting and execution can be performed twice as fast. Additionally, thorough regression across all previously delivered features — which historically took days — has been reduced to hours, and has covered about 60-80% of features deemed necessary for regression testing.
Scaled & Segmented Testing Across Pipeline
Segmenting and scaling tests — including individual automated test suites — across different stages of delivery pipeline greatly reduced test redundancy and execution time within the agile software development lifecycle. For example, build verification and comprehensive regression test suites would run in earlier stages of the deployment pipeline, while end-to-end test suites ran in later stages closer to production; this greatly reduced tests across the pipeline and drove purposeful testing at each stage of development and release.
Preparing Pre-Production Testing Data Twice as Fast
In preparation for pre-production testing, the user acceptance testers would spend countless hours before manual test verification — particularly in preparing data to test specific scenarios. To resolve this, the SDET team built a utility that would streamline data setup, leveraging the service layer automated scripts; now, manual data setup that would previously take hours to a day is accomplished more than twice as fast.