Try our new research platform with insights from 80,000+ expert users

SmartBear TestComplete vs TestRail comparison

 

Comparison Buyer's Guide

Executive Summary

Review summaries and opinions

We asked business professionals to review the solutions they use. Here are some excerpts of what they said:
 

ROI

Sentiment score
6.8
SmartBear TestComplete automation saves time, enhances client satisfaction, and boosts efficiency, with annual savings of approximately $10,000.
Sentiment score
3.3
TestRail improved productivity, collaboration, and accuracy in test management, providing valuable reporting, integrations, and an intuitive interface for users.
Once set up, only one person is needed to handle all tasks, reducing the requirement for multiple personnel.
QA Analyst at a tech services company with 501-1,000 employees
Time-wise, it saves about fifteen to twenty percent compared to Excel, and money-wise, it's around ten percent.
Test Architect at Zest Consulting
 

Customer Service

Sentiment score
6.9
SmartBear TestComplete's customer support is knowledgeable but inconsistent, with delays and unresolved issues needing faster escalation and responses.
Sentiment score
3.3
TestRail offers responsive support, but complex issues often require improved service, with users favoring self-service resources and community help.
Sometimes, you really need to speak to a person, and arranging such calls is not easy.
Test Architect at Zest Consulting
I personally escalate Leapwork issues with the Leapwork support tech team, and they are very quick to solve problems.
Automation Test Engineer Il at a computer software company with 11-50 employees
Just today we had a hiccup where it didn't want to integrate with Jira.
Release and Test Manager at a consultancy with 11-50 employees
 

Scalability Issues

Sentiment score
7.4
SmartBear TestComplete is scalable and adaptable, with flexible scripting, but may require licensing for wider deployment.
Sentiment score
7.0
TestRail offers robust scalability and stability, handling varied team sizes effectively, though user monitoring improvements are noted.
When I started automation testing in my current company, we only had 10-20 automated test cases integrated with TestRail, but we gradually increased to 100, 200, 1000, and 2000 cases.
Automation Test Engineer Il at a computer software company with 11-50 employees
It also doesn't have the analytics and dashboards that you need for large scale enterprise implementations, which is why it's not really scalable.
Release and Test Manager at a consultancy with 11-50 employees
I have never experienced any issues with its scalability.
Test Architect at Zest Consulting
 

Stability Issues

Sentiment score
6.9
SmartBear TestComplete generally stable, but users report crashes, memory leaks, and HTML5 testing delays in certain scenarios.
Sentiment score
5.9
TestRail is highly praised for its stability, with users experiencing rare, quickly resolved issues and consistently rating it highly.
It has never had any issues.
Test Architect at Zest Consulting
I would say seven or an eight because sometimes there are hiccups in the interface with Jira, for example.
Release and Test Manager at a consultancy with 11-50 employees
 

Room For Improvement

SmartBear TestComplete faces challenges in object recognition, integrations, licensing, performance, and support across browsers and mobile devices.
TestRail needs improvements in reporting, integration, user interface, defect management, API, technical support, pricing, and AI features.
While using SmartBear TestComplete, we are fine with the current capabilities, however, it would be beneficial to improve some performance aspects, especially the image comparison feature.
QA Analyst at a tech services company with 501-1,000 employees
If you want to use TestRail for enterprise-level waterfall projects with traditional reporting and a lot of analytics, it's not good enough because the analytics, dashboards, and reporting are not really there.
Release and Test Manager at a consultancy with 11-50 employees
I would appreciate AI features that help design test cases based on documented requirements.
Test Architect at Zest Consulting
 

Setup Cost

SmartBear TestComplete's pricing and licensing receive mixed reviews, seen as both reasonable and costly depending on usage and modules.
Enterprise users have mixed views on TestRail's pricing, appreciating discounts and scalability despite recurring costs and no perpetual license.
 

Valuable Features

SmartBear TestComplete excels in cross-platform automation, integration, and support for multiple languages, enhancing automated testing efficiency and maintenance.
TestRail offers intuitive test management, effective integrations, and user-friendly features for efficient, cost-effective test execution and organization.
The most valuable feature of SmartBear TestComplete for me is the image comparison functionality.
QA Analyst at a tech services company with 501-1,000 employees
The integration with Jira is good. I like that because it helps to show your test results and helps you to keep traceability between your user stories and your test results.
Release and Test Manager at a consultancy with 11-50 employees
It significantly saves effort in managing test execution and managing all test cases.
Test Architect at Zest Consulting
The reporting capabilities and the simplicity of it make it user-friendly, as new users can easily understand the tool.
Automation Test Engineer Il at a computer software company with 11-50 employees
 

Categories and Ranking

SmartBear TestComplete
Average Rating
7.6
Reviews Sentiment
6.8
Number of Reviews
76
Ranking in other categories
Functional Testing Tools (8th), Regression Testing Tools (5th), Test Automation Tools (7th)
TestRail
Average Rating
8.0
Reviews Sentiment
5.6
Number of Reviews
24
Ranking in other categories
Test Management Tools (2nd)
 

Mindshare comparison

While both are Quality Assurance solutions, they serve different purposes. SmartBear TestComplete is designed for Test Automation Tools and holds a mindshare of 5.8%, down 6.1% compared to last year.
TestRail, on the other hand, focuses on Test Management Tools, holds 7.9% mindshare, down 12.2% since last year.
Test Automation Tools Market Share Distribution
ProductMarket Share (%)
SmartBear TestComplete5.8%
Tricentis Tosca15.3%
OpenText Functional Testing7.1%
Other71.8%
Test Automation Tools
Test Management Tools Market Share Distribution
ProductMarket Share (%)
TestRail7.9%
OpenText Application Quality Management9.0%
Tricentis qTest8.7%
Other74.4%
Test Management Tools
 

Featured Reviews

Prakhar Goel - PeerSpot reviewer
Sr Test Lead at Emerson
Used for integration automation, user-based automation, and web automation
The solution's most valuable features are the drag-and-drop feature, keyword-driven approach, and reusability of the scripts. The solution has introduced a new feature that helps us identify objects we cannot normally identify. It gives you a fair idea of objects, resolving the object recognition issue. The solution can be used to perform different tests on different machines.
ChristineAnderssen - PeerSpot reviewer
Release and Test Manager at a consultancy with 11-50 employees
Provides quick test management for agile workflows but lacks customization for reporting and dashboards
The analytics and reporting capabilities in TestRail are not very extensive and tailorable. This doesn't make it very suitable for enterprise level programs. With OpenText, you have a lot of flexibility. I'm talking about the old version of ALM, the desktop client, which I used when it was still Micro Focus ALM. It has a tremendous ability to generate different types of reports. You can build different graphs, have Word document format, Excel format, or write your own Excel queries and incorporate that into your own reporting. TestRail has one type of format. If you want to generate a test summary report, there is basically only one format that you can use. You can't add your own components. You can choose which portions of your test results you want to include by selecting certain sections, but you don't have freedom to add and remove sections yourself. It's predefined and all that you can do is just take out the ones that you don't want. It is usable, but it's not tailorable, it's not configurable, it just gives you the bare minimum. It depends on what you use the tool for. If you want to use TestRail for enterprise-level waterfall projects with traditional reporting and a lot of analytics, it's not good enough because the analytics, dashboards, and reporting are not really there. However, if you use it for agile and you want quick and easy with very simple functionality, then it's very good because it's simple to use. For agile use cases, it's great and you don't need more than that. But if you are looking for an enterprise tool or waterfall project with large programs, even the enterprise version is not sufficient. There's no significant difference from the basic version. The only thing that you really get is the ability to have multiple project reporting, but the reporting itself is limited. There are no configurable dashboards. You can report across multiple projects, which is great, but the type of reports that you can run are limited to the current set of built-in reports. That's not good enough for enterprise level with multiple project and large programs. You want multiple reports across multiple projects and you want the tailorability and configurability of having dashboards and analytics, which TestRail is weak on for the enterprise level.
report
Use our free recommendation engine to learn which Test Automation Tools solutions are best for your needs.
881,082 professionals have used our research since 2012.
 

Top Industries

By visitors reading reviews
Manufacturing Company
15%
Computer Software Company
15%
Financial Services Firm
12%
Government
5%
Computer Software Company
12%
Manufacturing Company
10%
Financial Services Firm
9%
Educational Organization
7%
 

Company Size

By reviewers
Large Enterprise
Midsize Enterprise
Small Business
By reviewers
Company SizeCount
Small Business22
Midsize Enterprise20
Large Enterprise32
By reviewers
Company SizeCount
Small Business15
Midsize Enterprise3
Large Enterprise4
 

Questions from the Community

What do you like most about SmartBear TestComplete?
TestComplete has strong reporting capabilities. The reports they generate are really good.
What is your experience regarding pricing and costs for SmartBear TestComplete?
I am not involved in pricing or licensing; our management team handles these aspects.
What needs improvement with SmartBear TestComplete?
While using SmartBear TestComplete, we are fine with the current capabilities, however, it would be beneficial to improve some performance aspects, especially the image comparison feature. Occasion...
What is your experience regarding pricing and costs for TestRail by Gurock?
Pricing is reasonable for TestRail. It offers good value for money.
What needs improvement with TestRail by Gurock?
The analytics and reporting capabilities in TestRail are not very extensive and tailorable. This doesn't make it very suitable for enterprise level programs. With OpenText, you have a lot of flexib...
What is your primary use case for TestRail by Gurock?
I'm running an agile release train and we use TestRail for agile testing, mostly for manual testing, to keep track of test statuses. We link it directly to our user stories and execute our test cas...
 

Also Known As

No data available
TestRail by Gurock
 

Overview

 

Sample Customers

Cisco, J.P. Morgan, Boeing, McAfee, EMC, Intuit, and Thomson Reuters.
Apple, Microsoft, Boeing, Intel, NASA, Amazon, HP, Samsung
Find out what your peers are saying about SmartBear TestComplete vs. TestRail and other solutions. Updated: April 2023.
881,082 professionals have used our research since 2012.