Digital.ai Continuous Testing Reviews

Digital.ai Continuous Testing serves as my main architecture at a high level, sitting inside our CI/CD pipeline and acting as a test execution plus device lab and analytics layer. A central platform stores test scripts, reports, and device configurations. The lab has real devices connected via USB and cloud devices, and it integrates with our automation framework where we do not write tests inside Digital.ai, usually using Appium or Selenium.

Regarding the workflow, I can elaborate on what happens behind the scenes from code to production decision. A developer pushes code to Git, and a webhook triggers the CI pipeline. For example, when a feature login fix gets pushed, the pipeline starts. It uses Jenkins or GitLab CI to check out and build, then unit test, package, upload, and trigger tests. We can discuss build and artifact creation such as mobile APK or IPA, or in web deployment to the test environment, which is crucial for traceability. Then we upload it to Digital.ai Continuous Testing, where CI uploads the build via REST API or plugin. Internally, the app is stored with metadata, such as build number and branch commit ID. Test execution requests occur when the pipeline triggers the test run through Maven, REST API, or the Digital.ai Continuous Testing scheduler, including test suite name and devices list. Digital.ai Continuous Testing internally finds available devices, locks them for the run, and installs the app, managing busy devices with a queue system for scheduling.

Digital.ai Continuous Testing has positively impacted my organization with massive reductions in testing time. Prior to its introduction, our regression cycle took two to three days due to manual and limited automation. After integrating Digital.ai Continuous Testing, we can perform parallel execution across devices, completing the regression in two to three hours instead. This results in a 70 to 85% time reduction. We also experience faster release cycles; we previously released weekly or biweekly, but now daily on-demand releases are possible, allowing for continuous delivery.

Faster debugging, utilizing video recordings and device-level insights, reduces issue resolution time from four to five hours. Additionally, we have better device coverage, moving from testing on two or three devices to 15 to 20 devices in parallel, leading to fewer production issues. We analyze a typical improvement of 30 to 50% fewer production defects, thanks to real device testing and early feedback. Team productivity improves, as QA spends less time rerunning tests and debugging blindly while developers receive faster feedback loops, leading to a greater focus on feature development instead of firefighting. We also see cost optimization; despite the tool cost, we save on manual testing efforts and device maintenance, which lowers operational costs over time.

Overall, integrating Digital.ai Continuous Testing has reduced our regression testing time from around two days to approximately three hours and significantly improved debugging efficiency by allowing us to analyze video recordings and logs, reducing issue resolution time by more than 60%. Consequently, we have transitioned from weekly releases to nearly daily deployments while also reducing production defects.

The best features Digital.ai Continuous Testing offers are not just real devices but the combination of real device execution with rich debugging data or analytics, making it the most valuable. The reason is that in real-world projects, the biggest problem is not writing tests, it is debugging failures quickly and accurately. This feature solves that. With real device execution, not emulators, tests run on actual Android or iOS devices with different OS versions and real hardware conditions, catching issues emulators miss, such as touch delays or device-specific crashes. Full execution visibility gives video recordings, step-by-step screenshots, device logs, and network logs. For instance, if a test fails due to a login button not being clickable, without Digital.ai Continuous Testing, I would waste time guessing and rerunning locally. With Digital.ai Continuous Testing, I can watch the video to see the button covered by a pop-up, check the logs for API delays, and find the root cause in minutes. Digital.ai Continuous Testing also has smart failure analysis capabilities, identifying flaky tests or pattern-based failures, such as failures occurring only on Android 13 Samsung devices.

I appreciate parallel and scalable execution where I can run the same test across 10 to 50 devices and get results in minutes instead of hours. Combined with insights, it is fast and reliable, providing trustworthy feedback.

Digital.ai Continuous Testing could be better in certain areas, and I can share my experience-based view on what can be frustrating. One issue is device availability and queue delays during peak CI hours. Sometimes devices are busy, causing tests to queue and the pipeline to slow down unexpectedly, which is especially painful for large regression suites or tight release timelines. Improvements are needed in smarter auto-scaling of device pools and better priority-based scheduling. Additionally, execution speed variability occurs; the same test sometimes runs fast and sometimes slow, depending on device load and network latency, making results less predictable.

More stable execution environments and better performance isolation per session would help. Furthermore, debugging can still be indirect; even with logs or videos, I do not fully control the device as I would with local debugging, making it hard to pause and inspect live states or reproduce edge-case issues locally. More interactive debugging and improved local reproduction tools are necessary. Cost versus usage efficiency is another area of concern, as device cloud usage can be expensive and we sometimes have idle or inefficient tests that waste money. Improvements in usage analytics and cost optimization suggestions for smart test selection to run only impacted tests are areas where I believe Digital.ai Continuous Testing could improve.

I have been familiar with Digital.ai Continuous Testing for six months.

On a scale of 1 to 10, I would rate Digital.ai Continuous Testing 8 out of 10. The main reason for this score is that it is a strong enterprise tool but not perfect. Its strengths include the real device cloud, which is a major advantage for reliable testing on actual devices that catch real-world issues early, adding huge value. Parallel execution is another strong point, saving us significant time, as regression has gone from days to hours, providing a massive productivity gain. Excellent debugging support, including video logs and screenshots, significantly reduces debugging time. Seamless CI/CD integration works well with tools like Jenkins and enables true continuous testing, while enterprise-grade scalability handles large test suites and multiple environments effectively.

However, it is not a nine or ten due to some limitations. Device availability issues cause queues during peak usage, slowing down pipelines, and execution variability makes predictions difficult. Limited interactive debugging and dealing with flaky tests can also be painful, particularly given the instability exposed by the combination of parallel testing and real devices, which necessitates strong test design. If not optimized properly, the cost can escalate, making it essential to have a smart usage strategy. Overall, while Digital.ai Continuous Testing provides strong capabilities in real device testing, parallel execution, and debugging insights, which significantly improve testing speed and reliability, challenges such as device availability, execution variability, and limited interactive debugging prevent it from being a perfect solution.

My advice to someone considering Digital.ai Continuous Testing is to avoid the mistake of lifting and shifting tests without redesigning for the cloud. It is crucial not to simply take local Appium or Selenium tests and run them on the cloud, as this can lead to flakiness or slow execution. My recommendation is to focus on making tests more stable, employing explicit waits and retries, ensuring independence with no shared state, and keeping tests short and atomic. When using parallel execution, it is important to do it smartly rather than blindly. Running everything on every device could be counterproductive, so develop a device strategy where you run smoke tests on a few devices and execute regression tests on a wider range. For critical flows, run tests on all devices to save time and cost. Investing in test stability is essential; ignoring flaky tests would be detrimental, so always aim for a flaky rate of less than 2%. Utilize Digital.ai Continuous Testing's insights actively; merely checking pass or fail status will not suffice. Leverage failure trends, device-specific issues, logs, and videos to gain real value from Digital.ai Continuous Testing.

Additionally, build a smart CI/CD strategy with tools like Jenkins by following a recommended flow of performing PRs, smoke tests, and nightly full regression tests while ensuring that releases include critical and full sanity checks without rerunning everything each time. Optimize for cost to prevent idle devices or unnecessary tests from incurring unnecessary expenses; use tag tests such as @smoke, @regression, or @critical for running only what is necessary. Debug using video first rather than code; reviewing videos before diving into logs will save considerable time. I would rate Digital.ai Continuous Testing 8 out of 10.

The main use case for Digital.ai Continuous Testing has been automating test execution as part of the CI/CD pipeline, especially for ensuring builds are stable before the release. For example, I used it to run the automated test suite whenever a new build was deployed. It helped me execute tests across different environments and quickly identify any failures earlier in the pipeline. It made it easier to catch issues before they even reach production and also improved the overall release quality. Digital.ai Continuous Testing is mainly a tool to ensure continuous testing for faster feedback and better reliability in the deployment.

One of the key use cases for Digital.ai Continuous Testing is how it fits into the overall testing workflow, not just running tests. In my workflow, once a new build is triggered in the CI/CD pipeline, Digital.ai Continuous Testing automatically picks it up and runs the relevant test suite based on the stage, starting with a smoke test and then regression tests. After execution, it provides detailed reports and logs which help quickly identify where things failed. I have also used it to prioritize tests, so only critical ones run early for faster feedback, and the full suite runs later. This really helps speed up the release cycle. Another useful part is how it integrates with other tools, so the results can be shared within the team and issues can be tracked immediately. Overall, it acts as a continuous feedback loop to trigger tests, analyze, and improve, which makes the whole development and testing process more efficient.

What I really appreciate about Digital.ai Continuous Testing is how it fits into my workflow without requiring much manual intervention. The features I use the most are mainly around the automated test execution in the pipeline. Once a build is triggered, I rely on it to pick things up and run the tests automatically. That saves me a lot of manual effort. I also use the reporting and logs quite a bit. After a run, I usually go straight to the results to quickly see what failed and why instead of digging around manually. Another feature I find useful is the ability to run tests across different environments or devices, especially when I want to make sure everything is consistent before the release. Overall, I use it mostly for automation and quick feedback. It helps me catch issues early and keeps things moving without slowing down the workflow.

Digital.ai Continuous Testing has had a pretty positive impact on the organization, especially in terms of speed and reliability. In day-to-day work, it has reduced a lot of manual testing effort. Earlier, we had to run tests manually or trigger them separately, but with Digital.ai Continuous Testing in place, everything runs automatically as part of the pipeline. That saves time and made the process more consistent. It also improved how quickly we catch issues. Instead of finding bugs later, we started getting early feedback right after a build, which helps fix things faster and avoid bigger problems down the line. For me personally, it made my daily work smoother. I could focus more on analyzing results and improving the test cases rather than spending time on repetitive execution.

Digital.ai Continuous Testing is a solid tool, but there are a few things that can be frustrating at times. One thing I noticed is that the initial setup and configuration can feel complex, especially when integrating it fully into the pipeline. It takes some time to get everything running smoothly. Also, sometimes the test results and logs are not as easy to navigate, especially when trying to quickly debug an issue. A more simplified or clearer view would really help in day-to-day work. There are also moments where test execution can be slower than expected, particularly with larger test suites or multiple environments. I think it could also improve in terms of UI/UX. It can feel cluttered and a more intuitive interface would make things easier, especially for new users. It is definitely powerful, but making it more user-friendly and smoother to work with would definitely make a big difference.

I have been familiar with Digital.ai Continuous Testing for a while and have had exposure to it over the past year or so.

If someone is considering using Digital.ai Continuous Testing, my advice would be to start with a clear workflow in mind. From my experience, it works best when you use it as part of a CI/CD pipeline, not as a standalone tool. That is where you get value for automating tests, getting quick feedback, and improving the release quality. I would also suggest starting small, beginning with smoke tests or critical test cases first, then gradually expanding to full regression suites. That makes it easier to manage and avoid getting overwhelmed early on. Another thing is to spend time on proper setup and integration. It can feel complex initially, but once it is all set up, everything runs much more smoothly. This is important because the platform is designed to handle large-scale testing across devices and environments. Configuration matters a lot. Also, make good use of reporting and analytics features. They help you quickly understand failures and improve your tests over time. The tool provides detailed logs and insights that are also very useful for debugging and help in decision-making. I would say do not expect it to be a full plug-and-play solution. There is a learning curve and setup effort involved, but once you get past it, it becomes a strong part of your workflow. Overall, use it strategically, integrate it properly, and build gradually. That is when you will see the real benefits. I would rate my overall experience with Digital.ai Continuous Testing an 8 out of 10.

My main use case for Digital.ai Continuous Testing is to ensure application quality by running automated tests across different devices and browsers. In a typical project, I use it to validate new features and catch bugs early by integrating testing into the CI/CD pipeline. Whenever there is a new build, a test can be executed automatically on multiple real devices and environments. This helps ensure compatibility and stability before releases, reducing manual efforts and speeding up delivery.

Digital.ai Continuous Testing offers several strong features. Some of the best ones include real device cloud, cross-browser testing, test automation support, and seamless CI/CD integration. It also allows parallel test execution, which helps speed up the testing process significantly. The features I use the most are automated testing and real device testing.

Automated testing helps ensure that test cases run consistently with every build, while real device testing ensures that the application performs well across different devices and environments. Digital.ai Continuous Testing has had a very positive impact in terms of efficiency and quality. One of the biggest improvements is the reduction in testing time through parallel execution and automation. For example, instead of running tests sequentially on a few devices, I can execute them simultaneously across multiple real devices, which can reduce testing time significantly, sometimes by fifty to sixty percent. It also helps catch defects earlier in the development cycle through CI/CD integration, which reduces the cost of fixing bugs later. Additionally, it minimizes manual testing efforts, allowing the team to focus more on critical scenarios rather than repetitive tasks.

Digital.ai Continuous Testing is a strong platform overall, but there are a few areas where it could be improved. One challenge is that the initial setup and integration with CI/CD pipelines can sometimes be a bit complex, especially for teams new to automation. There can also be occasional delays when accessing specific real devices due to high demand. Additionally, debugging failed test cases in a distributed cloud environment can sometimes take extra effort compared to local testing. That said, these are relatively minor challenges, and the benefits in terms of scalability, speed, and coverage outweigh them.

I have been familiar with Digital.ai Continuous Testing for two years.

Using Digital.ai Continuous Testing helps improve test coverage and scalability. Instead of testing on a limited number of local devices, I can run tests in parallel across many real devices and environments. It also provides faster feedback to developers so issues can be fixed early in the development cycle. Additionally, it helps reduce manual testing efforts and ensures more consistent and reliable results.

My advice for someone considering Digital.ai Continuous Testing would be to first clearly understand their testing requirements and start with a well-defined automation strategy. It is important to begin with a small set of critical test cases and gradually scale up rather than trying to automate everything at once. Also, integrating it properly into CI and CD pipelines is key to getting the most value. I would also recommend investing some time in learning the platform and best practices, as that helps avoid initial setup challenges and ensures smoother adoption. I would rate this product an eight out of ten.

My main use case for Digital.ai Continuous Testing is enabling automated and continuous testing across web and mobile applications to improve release quality and speed up delivery cycles, as it helps with running automated test cases across multiple environments and devices, integrating with CI/CD pipelines so that testing happens continuously during development and deployment.

Digital.ai Continuous Testing offers several strong features, but the ones that stand out the most for me are automated test execution at scale, cross-browser and cross-device testing, and integration with CI/CD pipelines.

I have seen a clear positive ROI after implementing Digital.ai Continuous Testing, especially in terms of time saving, faster release cycle, and improved efficiency. Earlier, regression testing cycles used to take a significant amount of time due to manual execution and fragmented test environments. After implementing Digital.ai Continuous Testing, a large portion of these tests became automated and reusable, which reduced overall testing time substantially, often by 40 to 60 percent depending on the release cycle. I also saw a major improvement in execution efficiency because tests can now run in parallel across multiple devices and browsers instead of sequential manual execution.

Digital.ai Continuous Testing is a strong platform, but there are a few areas where it could be improved to make the experience even better. Especially when dealing with dynamic applications, while it integrates well with CI/CD tools, setting up and fine-tuning integration sometimes requires additional efforts compared to more plug-and-play solutions.

I have been familiar with Digital.ai Continuous Testing for around one year, and during this time, I have understood how it supports end-to-end testing workflows including test automation, cross-device testing, and continuous integration within CI/CD pipelines.

Digital.ai Continuous Testing is very strong in its core capabilities, especially automated testing at scale, CI/CD integration, and cross-browser and cross-device execution, significantly improving testing efficiency and helping accelerate release cycles. However, there is still room for improvement in areas like initial setup complexity, learning curve, and making the user experience more intuitive for new users.

I would recommend clearly defining your testing goals and CI/CD strategy before implementation, as Digital.ai Continuous Testing is a powerful platform, but it delivers the best value when there is a well-structured automation framework in place. I would also recommend starting with a pilot project first, focusing on a few critical applications, and then scaling gradually, which helps teams get comfortable with the tool and optimize workflows before a full enterprise rollout.

I rate Digital.ai Continuous Testing an 8.5 out of 10 overall.

I'm using Digital.ai Continuous Testing to create and test a mobile application. We're developing and testing a mobile app.

The most useful feature for me is Mobile Studio. It has a UI where I can click on elements, and it generates a script for me. Mobile Studio can generate code from testing steps. I'm using Python with it.

Another useful aspect is the subscription plan. It has many mobile devices available in the public cloud. This allows me to check different versions of Android and iOS devices. I can generate testing scripts and run them on all these devices.

The integration process was good, but I've faced some challenges. Every time they release a new version, I find bugs in the UI and features. Sometimes, buttons don't work well. When this happens, I submit a ticket to technical support, but they often have to fix it in the next version.

I have been working with the product for one and a half years. 

I'd rate the tool seven out of ten for stability. However, some devices in the cloud sometimes lose connection, which could be improved.

The setup experience is quite easy - I'd rate it eight out of ten. We use it on the public cloud. It only takes two to three hours to set up initially, but using all features might take two or three days. The solution is very scalable - I'd rate scalability nine out of ten. Our team consists of five people who use this tool. We are all QA engineers.

The tech support is good - I'd rate them eight out of ten. They have a very good attitude and work hard to resolve issues.

Positive

We haven't tested other solutions before, but we're now evaluating UiPath.

The solution helps us scale up our testing, which improves our application quality even if it doesn't directly save money.

The price is reasonable for our company, but I'm not the decision-maker.

I'd recommend this solution for beginners in automation testing because the UI is very friendly, and the integration process is good. Overall, I'd rate Digital.ai Continuous Testing eight out of ten.

I work for a government-based company and we are able to use Experitest to perform testing on our native app.

Experitest is one of the only companies to offer a real device on the cloud to perform testing. They also provide quality documentations that help you navigate and maximize the solution.

I would like for changes to be interchangeable between the cloud device and the real device.

I would also like to see more videos and descriptions that could make installation more efficient.

I have been using this solution for a few months.

I believe that it could be more stable. During times when something is not working, it is difficult to find the solution.

The solution is scalable but that depends on how much you pay. The more you pay, the more parallel executions you can have.

I am not able to work directly with Experitest, I communicate my technical problems with T-Systems who are able to help.

I worked with Tricentis Tosca awhile back.

The initial setup was a bit complicated and time-consuming. We had issues with going through proxies which took a lot of time as the company I work for has many security issues. 

We have two automators, myself included, and two manual users that work in Experitest regularly and maintain it.

I completed the initial setup by myself because there is a lack of support in my country. It took about one month in total to go through all of the documentation myself.

We make monthly payments. The cost is dependent on the number of devices we intend to support.

I would advise new customers to have a direct support person to help them with the initial setup, automation, and general support.

I would rate it a six out of ten.

We are mainly going to use it for accessibility testing on mobile devices. It is a cloud-based service with a desktop tool connected to it.

The most valuable part of Experitest is the number of real devices on which the test is run.

I have been automating tests for many years on many things but not on mobile devices. The amount of time that I have spent on just figuring out how to use Experitest and get it to work was quite long compared to what I have been doing before. I spent the first two weeks just getting it started. It would be good to have some video explanation of how to use it on your devices and get started. Their online documentation is quite good and extensive, but it would be quite good to have some end-to-end examples demonstrated.

I have been using this solution for a couple of weeks.

Its stability is okay.

We haven't scaled it. We're not that far yet. For our project, there are only two users, and they are administrators. 

I didn't interact with them.

I have built my own tools, and I have used UFT. Usually, the tools that you buy have some limitations in terms of flexibility. Therefore, we built our own tools. We decided to go for the Experitest mainly because it has quite a large number of devices that are available for testing.

Its initial setup is rather complex. Getting things installed was quite easy, but understanding how things should be connected so to make it work was difficult. It seems fairly easy when you first get into it, and you can also do some small testing, but when it comes to getting started and making things work, it is rather complicated. 

It was difficult to understand the concept of how it should work because when you develop automated tests, you need to have an easy way of trying out your automated script. You need to be able to run the automated test from the desktop on your own mobile devices. When you deploy it to Experitest, you can run it on several devices, which is a kind of starting step. These things are automated to a higher level. When you want to work with it and get it connected to your real devices, it starts getting complex.

I did the installation by myself.

It is quite fairly priced, but it really depends on your budget. It is somewhere in the mid-range of products. It is not free and it is not QGP that nearly costs a whole house. You pay for the number of users who require access to execute the tests.

I would recommend this solution in case you need to test on real devices and many different devices. We will keep using it just for the period of time of the project, which will be until May or June. This is a temporary project, and perhaps some of the tests will survive after the project ends.

I would rate Experitest a seven out of ten.