Faster test creation, greater test code efficiency, greater test code stability, catch more bugs earlier in the process.
I had the opportunity to speak with James Lamberti, CMO, and Raja Rao, Head of Growth Marketing at Applitools to discuss the current state of automated testing and what they learned when 288 Cypress.io, Selenium, and WebdriverIO quality engineers explored how visual AI augments code-based testing frameworks.
James’ premise is that today’s approach to automated testing is broken for several reasons:
Test creation is slow. Test authoring is time-consuming and this slows down releases and leads to poor quality tests.
Automated tests are brittle. Locators are required for browser interaction and assertions. The more thorough the test, the more costly it is to maintain.
Over time, more tests are added which means more locators and more flakiness.
Brittle tests running across multiple browser and viewport combinations further reduce stability which wastes time and money.
Automated testing is necessary for successful CI/CD environments, but constant false positives and slow tests ruin developer productivity.
Traditional automated testing frameworks are not built to detect modern app defects. As a result, major bugs still get into production.
Since I met James at DevOps World last August, Applitools has built out its platform to become a comprehensive end-to-end cross-browser UI testing solution addressing the problems with automated testing:
AI-powered computer vision replicates the human eye and brain.
Able to create tests more quickly with fewer brittle locations and labels.
SDKs augment all modern test frameworks, integrations with popular source control, CI, and defect tracking systems.
Execute tests in seconds versus minutes for faster builds and on-time delivery.
The ultrafast grid ensures visual precision across all browsers, screens, and viewports.
Open-ended, AI-powered assertions reduce code and maintenance while increasing test coverage.
To help users understand how to use visual AI, Applitools hosted a hackathon which engaged quality engineers to use visual AI with their code-based frameworks on five common use cases: 1) UI elements, 2) login functionality, 3) test table sorting, 4) displaying bar charts, and 5) testing dynamic content.
Results
5.8X faster test creation - testers are able to expand test coverage while simultaneously testing faster than ever before.
5.9X greater test code efficiency - testers can either achieve the same amount of test coverage using far less test code or dramatically expand test coverage using the same amount of test code as they do today.
3.8X greater test code stability - more stable code means tests break less often, allowing testers to expand coverage and spend more time managing quality.
45% more efficient at catching bugs early - The primary goal of the tester is to catch bugs early. With faster test builds that are more stable and easily maintained, more bugs get caught before escaping into production.
Applitools is currently running another hackathon through July 7, 2020 for cross-browser testing with different UIs and multi-version applications. I look forward to seeing what insights and benefits the testing engineers uncover this time.