Tony Rehmer Mark Farr
On July 19, 2024, a routine software update from CloudStrike caused more than 8 million computers to crash, resulting in significant disruptions. Delta Airlines alone reported losses exceeding $500 million, and it took nearly 10 days to restore the affected machines. According to CloudStrike’s CEO, a full regression test was not conducted because the update was not believed to directly impact critical code.
This incident highlights a common yet costly mistake in both large and small companies—regression testing is an essential part of QA and is frequently undervalued and overlooked. The consequences can be severe, leading to financial losses, time delays, and loss of trust.
In the race to deliver products faster, companies often shortcut testing processes that could prevent these headaches. While the industry embraces methodologies like continuous development and Agile practices, it rarely adopts QA strategies capable of fully supporting these approaches. Most process improvements have focused on development efficiency—accelerating the delivery of new features. While this has driven innovation, it has left QA teams struggling to keep pace.
It’s time to rethink our approach to quality assurance. QA needs the support of a robust process that ensures quality without compromising speed, and we believe we’ve found a better way.
In this article, we introduce the concept of Continuous QA (CQA). CQA recognizes your existing QA teams have little time to focus on regression. Let alone create test cases and automation. Even their best efforts are poorly maintained and communicated. CQA changes this and allows your QA to focus on delivering healthy products. From new products and features to ensuring no new bugs were introduced. CQA combines specific functions and tasks along with process changes to deliver health back to our QA teams and systems.
The Need for Continuous QA
I’ve been delivering software for over 30 years , working from large corporations to small startups, and one challenge has always remained: determining how much regression is enough. We often put our QA teams in tough spots, and the phrase that always frustrated me, “Well, do you want quality, or do you want to release on time?” The answer, of course, is always “quality on time.” But like so many others, when the pressure was on, we often chose to release—and I have the ulcers to prove it.
As a CEO of a software company, we experienced this many times while delivering a critically important product to our customers. Despite our best efforts, the most frustrating moments for them and myself came when we introduced a new feature and unintentionally broke something that had been working perfectly. For most people, it’s hard to understand how fixing a flat tire at the front of the car could cause the taillights to stop working. If you’ve worked in software, you get it. If not, it’s a tough explanation—and honestly, no one wants to hear it.
What is Continuous QA?
CQA is a shift in mindset that acknowledges and supports the demands of a fast-paced, high-frequency delivery environment. While emphasizing QA coverage it includes Intelligent testcases, manual and automation regression tests As well as process that ensures it happens and it is maintained.
Its key component is a dedicated, separately resourced team known as the Regression Team. It is responsible for all QA assets such as test cases, coverage for automation, and manual tests, master test suites and KPIs. Building a scalable and reliable test suite that can be executed at any time and is a key part of your CI/CD process. As a product release approaches, the regression team prioritizes running both manual and automated regression tests finding and reporting defects. While the Agile/scrum team focuses on the new features the regression team focuses on the rest. After the release, the regression team shifts focus on creating and maintaining test cases and automation. Their goal is to produce a healthy QA system that is scalable, maintainable and, at any time validate product quality.
In CQA, test cases become the source of truth for your product. When created effectively, these test cases serve as the foundation of quality assurance. Test cases should be consistent and automatable. Initial test cases are created by the Agile team and become the Acceptance Tests for new features and products. Once the new features are released the regression team creates additional test cases to ensure proper coverage. When the product change occurs the test case is updated and in turn updates the automation. We like Gherkin test cases, which offer consistency, readability, and a structured approach that supports automation. To learn more about Gherkin, Click here.
With CQA’s divide and conquer approach. Your Agile/SCRUM team(s) are focused on delivering new products and features and working directly with your development teams on a daily basis. The regression team ensures the quality of the entire product(s).Proformance, load, stress, boundary are common here. Ever feel like your product is slowing down with each release. A simple timed set of automated test cases can verify performance issues quickly.
This team can work autonomously, allowing for flexibility in resource allocation but resources must be dedicated to regression. Regression teams are responsible for:
Test case coverage for full product
Automation that is maintainable and meets defined coverage requirements
The Master Test Suite, ensuring it integrates seamlessly with CI/CD processes
Automated and manual test regression testing
Peace of mind knowing your product is ready to deliver
CQA can be achieved with minimal investment.. Even starting with a single dedicated resource can make a difference. Since you are taking work off your agile/Scrum QA teams you can dedicate one of your existing resources to regression if they have the right skill set. Also taking the time to plan your CQA vision will save you time and regret. Knowing how to write efficient test cases that lend themselves to automation and choosing the right testing framework is crucial to maintaining a scaling of the regression team.
Children's Miracle Network Hospitals Journey to Continuous QA
Our journey began by admitting we had a persistent problem that wasn’t improving. Even with additional QA resources, the same challenges persisted—teams were too focused on new releases to adequately address regression testing, automation, or test case maintenance.
For our organization, Children’s Miracle Network Hospitals, managing QA for over 30 applications while remaining cost-sensitive as a nonprofit required a fresh approach. We partnered with Nebo Consultants, leveraging their expertise in QA processes, Gherkin test cases, and automation.
This partnership enabled us to maintain our current Agile/SCRUM development efforts while simultaneously building a sustainable test case library. It also allowed us to integrate automation into a Master Test Suite that can be executed anytime via a simple URL.
By outsourcing certain tasks to cost-effective offshore resources led by U.S.-based experts, we quickly realized the benefits of Continuous QA. Our 30 applications have been built over many years and against several common libraries. Tribal knowledge cannot keep up. Creating a master test suite has become vital and having Nebo it is becoming reality.
One thing that really helped us track progress of this team was our KPI’s. We established clear KPI with the critical ones being:
Test cases created per week.
Automations written per week.
Manual test cases executed per week.
We also made it a priority to integrate the master test suite into our CI/CD. This allowed my existing dev and QA teams to see the value.
Ancillary Benefits we have seen
Continuous QA offers several additional benefits:
User Manuals: We have used AI coupled with our Test cases to create accurate and comprehensive user manuals saving time and effort for both developers and end-users.
Shorten learning curve for new employees
Response and Recovery: A well-structured Continuous QA process can help organizations respond effectively to incidents and recover from outages with confidence in a comprehensive Master Test Suite and regression testing.
Additional testing: When time is limited for last minute sprint bug fixes prior to a push to production, the Regression Team can help test the application and specific fixes.
Other Success Stories
Continuous QA has delivered measurable results for Children's Miracle Network Hospitals as well as others. For example, Cloud Storage Security shared:
“Adding a team that is focused on regression and automation helped us maintain a high quality product. We were in a situation where our releases weren't always delivered on schedule, and our regression coverage was deficient. Adding Nebo Consultants has dramatically increased our quality and we have been able to consistently deliver a high quality product, on-time. They handle test case creation, automation, manual regression and upgrade testing which is critical for us. Their communication skills and understanding of our framework has greatly helped Cloud Storage Security. They are a critical part of our team.”
—- Aaron Gettings, Director of Engineering, Cloud Storage Security
Conclusion
Continuous QA is a game-changer in quality assurance. It's a cost-effective, time-efficient solution that empowers organizations to elevate product quality, mitigate risks, and deliver unparalleled value. Adopt Continuous QA today to transform your development process and stay ahead of the competition! For more information and details about implementing CQA in your organization view our Continuous QA Quick Start guide, Click here.
About the Authors:
With over 60 years of combined software development experience, we’re passionate about improving software quality. We’d love to hear from you—reach out and let us help or share your thoughts on Continuous QA!
For more information contact:
Tony Rehmer
SVP of Information Technology
Children’s Miracle Network Hospitals
Mark Farr
CEO
Nebo Consultants
