OpenText Application Quality Management Reviews

ALM: You cannot just say one feature is most important. You get the most value
using all modules from Management to Defects. When you use the tool end-to-
end, you can pull efficient project reports (especially scorecards) from the
Dashboard. So everything is integrated and only then you can evaluate the tool
fairly. ALM is very flexible and each module can be used independently, but
when you do that you are only using the tool as storage, not as a test
management tool.

UFT: It became much more stable tool in terms of object recognition over the
years. It is easy to use as long as the user has basic software development
knowledge and understands that the software automation process is not just a
record/playback.

ALM: We currently successfully manage all testing projects due to ALM’s invaluable capabilities, which are listed below:

• Built on best practices with a flexible structure, organization, and documentation for all phases of the application testing process.
• Serves as a central repository for all testing assets and provides a clear foundation for the entire testing process.
• Establishes seamless integration and smooth information flow from one stage of the testing process to the next.
• Supports the analysis of test data and coverage statistics, to provide a clear picture of an application’s accuracy and quality at each point in its life-cycle.
• Supports communication and collaboration among distributed testing teams.
• Reduces time needed to create test execution summary reports.
• Reduces the time needed to write and execute manual tests with HPE Sprinter tool.
• Users can capture their actions automatically as steps in a formal test.

UFT: We save time executing smoke and regression tests. We also use UFT to create test data.

I would like to see better Reporting functionality especially more sophisticated graphs, for example Actual vs. Planned or high level progress graphs using indicators like traffic lights. I would like to see more sophisticated or flexible Dashboard views, such as editing and resizing. I use scorecards and pull them into the Project Reports using customized templates. Scorecards can only be refreshed from the Site Admin, which then test leads has to depend on the ALM Admin to refresh the reports if needed after the scheduled auto run. There should be ability to refresh scorecards (execute KPIs) from the project itself or give more frequent auto refresh option like even every 5 min. This is a really burden on the team.

I would like to see Requirements mapped to test steps so we can combine multiple requirements validation in to one test case but map the verification steps to the associated requirements, so if the step fails only fails one requirement not all. When we are operating in an Agile world we do not have time to write test cases to capture one-to-one coverage. I know ALM allows many-to-many mapping but we cannot get true requirement pass/fail status if we use many-to-many option. Test configuration option kind of on the right path, but can only be use for data driven test cases, I cannot add design steps. If we can add design steps to a subset of a main test using Test Configuration option, I think we may be able to differentiate individual requirement that was failed without failing all the requirements mapped to the main test case.

We have used this solution for 17 years.

I did not encounter any issues with stability.

I did not encounter any issues with scalability.

In terms of technical support, I usually get solutions to my issues. I did not have any issues to call technical support for a long time.

If you follow the instruction, the setup is straightforward. It definitely requires an experience user to do the installation and setups, especially for upgrades.

I always used ALM and UFT. However, I had training and evaluated IBM JAZZ tools.

We identified an object that was supposed to have a width of 30 characters, but instead had 100,000. No manual tester would have found it, forcing developers to take a second look at all objects which uncovered similar size issues.

While my experience tells me successful automation projects are at 70% coverage of manual test cases, we have been able to hit well into the 90% range of .Net automation with this tool.

So the first impression that hits me about HP UFT 14.0 (formerly QTP) is that it seems to be a whole lot faster! But that could be subjective, as I'm running it on a high end gaming system.

And my second impression was "Oh man, why does it still do THAT?"

Let's review the good stuff:

VBscript language -Easy to learn, surprisingly powerful and extendable.

What I will call the "PDM.DLL feature" provides a list view of any object property and methods at run-time from the code as well as the two other windows.

Built in Excel Datatables for Data-driven design

Revamped beautiful HTML results report with screen and movie capture

Terminal Emulator automation.

Modular design (through functions, ALM components and Flows)

Launch through Jenkins brings CI to the test automation development team.

Can leverage Windows API calls as well as custom AutoItScript for enhanced features.

The wide range of supported current and legacy web technologies, desktop apps, and WebServices testing is by far the most valuable feature.

Even in the case where technology is only partially supported, being able to customize out-of-the-box object methods is another time saver.

For example, we recently started to investigate automation of an AngularJS application. The problem was record/playback (UFT 12.54) did not work on it. However, the Object Spy correctly adds objects to the Object Repository. In addition Descriptive Programming worked from our custom framework. We had a basic login/navigate/verify Proof of Concept test operational with AngularJS Buttons, Links and Images quickly. Minor custom coding was required to override .Set methods of WebEdit objects, and more will be needed to support it's Angular WebTable objects. Totally doable for an experienced level team or user.

There are always new features and more support for new and legacy technology architectures with each release. But the bad news is a growing list of long-standing issues with the product rarely gets addressed. While I have a larger list of issues that make day to day work harder than it needs to be, these are the Top Five that I do wish would capture someone's attention in upcoming releases. All hit the tool's ROI pretty hard.

#1) Jump To Source - The Silent Code Killer: In older QTP versions a double-click on any function in the Toolbox window would take the developer to the function's source code, while a drag from the Toolbox would add it to the code window. Since 12.0 a double-click on a function in UFT's Toolbox window now ADDS the function (same as drag) to the Code window - to whatever random location the cursor happens to be at - even if it is off screen, and it will replace sections of code if it is highlighted. We are not sure what the intention was, but our Best Practice is to avoid the Toolbox window entirely to avoid the real danger of losing days of work and needless bug hunts.

Now Jump to Source is not all bad. A right-click on any function called from a Script takes us to the code source, which is great! But it only half works: in a Library, only for functions declared within the same library. Our advance designs have well over twelve libs so a whole lot of extra time is spent searching the entire project for a function's source on a daily basis.

Lastly, while we can add custom methods to object, a Jump To Source from these methods is long overdue. So again our only option is to search the entire project.

#2) Object Spy: It needs to have multiple instances so that you can compare multiple object properties side-by-side. It lacks a Refresh button, so that automation engineers can quickly identify the property changes of visible and invisible objects.

Or HP could skip to option #3...

#3) Add RegEx integer support for .Height or .Width object properties when retrieving object collections. If this were possible, our framework could return collections that contain only visible objects that have a .height property greater that zero. (Side Note: the .Visible property has not returned a False value for us in nearly five years - a recent developer decision, not a product issue) Eliminating the need to separate the non-visible objects from visible ones would decrease execution time dramatically. (Another side note: Our experiments to RegEx integer-based .Height properties found that we could get a collection of just invisible objects. Exactly the opposite of what we needed.)

#4) The shortcut to a treasure trove of sample code in the latest release 14.0 has been inexplicably removed. This impeeds new users from having an easy time learning the tool's advanced capability. In fact the only users daring enough to go find it now will be you who is reading this review.

#5) Forced Return to Script Code. This again is a no-brainer design flaw. Let's say we run a script and throw an error somewhere deep in our function library. Hey it happens. In prior QTP versions when the Stop button would be clicked the tool would leave you right there at the point where the error occurred to fix. Now in recent releases, UFT always takes us back to the main Script, far from that code area that needed immediate attention.

The product is surprisingly stable. For the flaws that I mention, if stability was an issue we would not have been using it for the better part of two decades.

The only time that the product gets unstable is when you try to do wicked advanced coding. For example, when you are trying to execute dynamic code strings with the Execute command that might not have been generated correctly. My years of experience tells me that if something is going flaky, then it's the developer's fault (me) and not the tool's fault.

Scalability is quick and easy implemented through a framework. Let's say we write a custom function

VerifyValue (oObj, sExpectedValue)

that can compare the expected value, string or integer, to an object's actual Value property. By adding the function as a .VerifyValue method to all WebEdit class object, the functionality is available to all current and future edit field objects regardless if they are in the repository or programmatic descriptions. And it is done with just a single line of code called RegisterUserFunc()

Now Let's say we now want to extend the verification to include value falls within an expected range? Add the code to the VerifyValue() function and all fields support the new functionality.

Scaling to new pages with new objects is not an issue either. The tool allows advanced users to design frameworks that can identify objects on the fly from "Plain English" descriptions ("OK LINK") without using the Object Repository. This may remind you of a Gherkin/Cucumber approach and the tool is certainly flexible enough to design just such a framework.

Orasi's technical support is an A+.

HPE's direct technical support is okay.

Having an issue getting UFT to work with your technology stack? Some versions of QTP used to include an oft overlooked document called the Product Availability Matrix (aka the PAM) telling users exactly which Technology version worked with HP's UFT tool versions. Unfortunately, due to a inexplicable "horse-and-cart" decision, HP has chosen to remove this document from UFT 14.0 install and provide access to it only AFTER users have purchased the product. So I have to buy it to learn how to make it work with my technology stack? Wait, what?

The install/uninstall of the updates are fast and easy. Many of the no-brainer configuration settings are set up correctly out-of-the-box. To be fair, we do have a 20-page document that walks through all the settings to check.

Sure, HP UFT is not free. But consider what you get for that cost: A stable product that is easy to use; the kitchen sink of technology stack support; decades of code (which in many cases actually is free); a version that is a stepping stone to an easier Selenium design; and a support base that is more that just the kindness of strangers.

Want to take UFT to another level for free? Add AutoItScript calls.

If Selenium is your thing then make it easy on yourself with LeanFT, aka UFT Pro. This gets you the easy object recognition of UFT, in Java, plus execution of concurrent test cases in multiple browsers which is a trick standard UFT does not do.

I am always downloading and evaluating other products. SmartBear TestComplete is the next closest option.

Take tool training by someone with years of experience.

HP, Orasi and RTTS all offer the level of training that gets you to the advanced state quickly.

And if you might be longing for the IDE toolset that Microsoft offers in Visual Studio, then look at Test Design Studio from Patterson Consulting to enhance your UFT toolbelt.

Defect Management: This feature allows us to track the defect status for our project, send the notification to the user via email and all the details about the defects can be maintained for the future reference as knowledge center.

Graphs and Dashboard: This is one of the top features, by which we can track the status of the project with ease to keep track of project management and executive reporting.

The live graphs can be exported via public URL's and can be integrated with SharePoint and others as required.

More collaborative, ease of work, and better documentation of all project activities.

Certainly on the UI part, it has to be improved to make it user-friendly and more presentable.

More than three years.

No issues with stability.

No issues with scalability.

Technical Support is great and always responsive to solve the issues.

Nope, the architecture is simple to implement and scale.

The price is a bit high.

Go ahead with this tool. It is for the project management and test execution.

Do consult a few of the other folks using this tool to understand the tricks.

Features that support documentation and test results from Requirements through Test Cases to Scripts to Execution results (pass or fail), then to Defects.

Life was made easy by shifting the MS Office documentation to the product. We deal with our test cases and execution, and the mapping is taken care by the product. You pass/fail a test case and product tells you what requirements/features of the product are good to go into production.

Auto-generation of automation scripts. Integration into the UTF (earlier QTP) has little more scope to improve.

I've used this solution for four years.

No.

No.

A nine out of 10.

No.

Straightforward.

Very expensive.

Some Open Source tools, but did not choose them as they lack support.

Have a skilled person to do the administration. You will love the ease of reporting that results in.

• Requirements Management
• Test Management
• Test Execution
• Defect Management
• Business Model
• Traceability
• Test configuration
• Test Parameters
• Integration with HPE UFT and HPE LoadRunner

All the options are designed so if you are using this tool for Test Management, then you must have them all enabled for efficient use.

It has basic and advanced facilities that allow you to arrange all of your test data in very well arranged, systematic manner.

Graphs can be further improved to manage more requirements at a time.

We have you used this solution for seven years.

We never encountered any issues with stability.

I did not encounter any issues with scalability. It depends on the number of licenses. If you have them all active at the time, it still responds the same way, which is fast.

I would give technical support a rating of 10/10.

We used Excel, but it was so difficult to manage the various Excel files and sheets just like maintaining a register book for your records. ALM allows you to handle them all with just one click.

The initial setup was straightforward as it it required little knowledge of the server to do a simple server-side installation.

If we have huge data and more number of users managing the data, then ALM is the best option for our organization. It also applies if your team is distributed, out-sourced or in-sourced.

We did not evaluate other options. I was aware of their earlier version which had a very good market and feedback, I didn't have to waste my time doing a PoC on any other product.

You must look to implement the minimum system requirements, Configuration, for the server for optimum and efficient use.

Customization is the most valuable feature. I don’t see other products with such a huge and detailed customization feature.

The Web UI and the Administration Page need to improve. These are not issues, but areas to improve.

Web UI: In some version (I don’t remember exactly the version number, unfortunately), Web UI was introduced with limited functionalities but it was cool in my opinion. Web UI has a lot of advantages and the main advantage is that the customer doesn’t have to install a big thick client on his machine. Web UI is faster, lightweight and easy to configure compared to the on-premise installation. But for some reason, this functionality (all Web UI) was removed in the next release.

Administration Page: It doesn’t have any issues. The main idea here is that it was written using old technologies and it would be great to improve/rewrite it using modern technologies. It doesn’t have an impact on the customer, it is more for developers who should support this area.

I have used this solution for five and a half years.

There were no stability issues.

There were no scalability issues.

Technical support is very good.

Previously, I have used Redmine and JIRA. Switching to HPE ALM was not my decision, I changed my job.

The setup was straightforward.

Just want to advise to use ALM SaaS in case you don’t want to use ALM all your life. It is cheaper and provides you with all the ALM features but for some period of time. Also, you don’t need to install it manually, the entire environment is already installed.

As an advise for developers who will develop such applications I would like to say - always try to support the area in an actual state, i.e., by using modern ideas/technologies if possible. Also, listen to the customer's needs, have flexible customization tools and do not forget about performance.

HP ALM is a good tool for a centralized and coordinated view of requirements, tests, defects, and iterations.

The main barriers of entry are cost and implementation, especially if an enterprise implementation is the best solution

Being able to have one place to review defects, testing progress, and defects was very useful.

Merging 40 different streams, just for defects, into one solution that had good search and reporting capabilities saved a significant amount of time in coordination, defect management, and by consequence, there was better control of the quality of delivered software.

The main barriers of entry are cost and implementation, especially if an enterprise implementation is the best solution

I have used this solution for eight years in a variety of versions and companies.

The biggest challenge was finding the appropriate resource balancing for the enterprise release. It is not very clear how that was going to be implemented due to documentation in 2010.

If there is a need for third-party integration, the documentation is not very good. We were able to integrate with FIT, but it took a very capable programmer to figure out how to do it. Again, this was in 2010. Hopefully, the documentation has improved.

I did not encounter any issues with stability.

I did not encounter any issues with scalability.

Customer service was adequate.

Technical support was adequate

We used a lot of home grown 'tools' and spreadsheets in one location and Lotus Notes in another.

In one instance, it was straightforward because anything was better than spreadsheets.

In the location that used Lotus Notes, there was a significant amount of resistance because of loss of control.

Neither instance was due to the tool, but it was due to cultural issues.

The implementation was done in-house.

At a Fortune 100 company, we achieved a reduction of 30% of defects in the first year and decreasing percentages the subsequent years.

The dollar figures were proprietary, but were significant even for an $11 billion dollar revenue company.

The main concern is that there is a significant dollar investment, so do good research to make sure the tool will meet your needs.

We evaluated IBM tools, as well and a couple of Open Source tools.

• Templates: Allows us to standardize fields, workflows throughout hundreds of HPE ALM projects.
• Customizable Events drive workflow: Saves a few seconds of time when we set default values, customizable dependent lists. When we translate this to hundreds of projects and hundreds of concurrent users, the productivity gains are immeasurable.
• Traceability: Traceability from Requirement -> Test Plan -> Test Lab -> Test Runs -> Defects.
• Rich sets of permission settings for different roles.
• Business Views: It is easier to pull reports for novice users.
• OTA, REST API: Utility for adding users and massaging data.

It has improved collaboration between our test teams.

• Client-side ActiveX with patch upgrades
• Support TDD/BDD
• Adding features like Kanban, project plans, resource utilization, and JIRA’s big picture.
• Traceability reporting: If HPE can generate a traceability matrix - Traceability from requirement -> Test Plan -> Test Lab -> Test Runs -> Defects.
• Defects aging report: A look at how long each defect is from the time it is created and how long it has remained at a particular status.
• Reports to build information: Currently, HPE ALM does not support the copy and paste of test instances.
  • 
    How can we duplicate a test set within a project?
  • How do we copy a test set from one project to another, and also copy the associated test plan along with it?
• Reports on automation:
  • How do we capture the number of automation test cases as some automation test cases have multiple test cases in each automated test?
    How do we calculate the ROI of automation?
    How do we determine which test cases should be automated, because sometimes the effort of automation does not have a good ROI?
    How do we calculate the ROI of HPE Sprinter if (1) Automated Script generation is used? (2) Mirroring is used?
    How do we know which testcase is created using Sprinter?
• Requirements coverage reports.
• Cumulative trending reports for test execution and defects outstanding.
• Auto-generation of test summary report.
• Inbuild best practices for fields, such as root cause category.
• Copy of test sets in the test lab.
• Upload of test execution results.
• Offline test defects to third parties and sync upon checking in.

We have been using the service for 18 years.

The solution is very stable, if you have the right person to manage it.

The solution is very scalable, if you have the right person to manage it.

The technical support has been deteriorating since the Mercury days. I would give technical support a rating of six out of 10.

We were looking at other solutions, such as JIRA, due to all the issues I have raised.

The setup was straightforward.

HPE ALM has been sold to Micro-Focus. I am not sure if Micro-Focus will be flexible.

If no flexibility is provided, you can easily move out in weeks, if you have the right resources.

We evaluated Zephyr, QASymphony, XQual, Perforce, and Rational Quality Manager.

For integration purposes, we evaluated Tasktop, Orasi, HP Synchronizer, and ConnectALL.

For automation purposes, we evaluated UFT, Selenium, and Robot.

Below is a checklist for others considering a test management solution:

• CIO sponsorship
• Number of projects
• Number of concurrent users
• Standardization needs
• Ease of control and management
• Access rights for individual roles
• Event driven workflow customization
• Extending beyond test management to TDD, BDD, test data management, test environment, and CI/CD tools.
• Focus on where you want to solve each problem and identify relevant tools for each of these.
• Availability of skilled resources
• Hiring the right resources to manage: There are easily millions of test management users, but it is not easy to find a true-bred expert.
• Keep in touch with what’s happening in the industry. You need to be focused and not swayed easily.
• Know your stuff.
• Support all your users and make life easier for them.
• Integration with automation tools, performance tools, security tools, and Jenkins/Bamboo/Team City.
• Define the test process that ties in with your test management too.l
• Form a training team to constantly train users.
• Open API for customization
• Export of info to MS Excel.
• Ease of migration.

Playback, test tracking, and defect management are the most valuable features.

Playback made it easy to create automated functional tests that could be reused.

Test tracking eliminated the need to track test executions on a separate application, such as Excel.

It provides ease of use, the reporting status and tracking of defects.

I no longer use the product as its price does not fit into my QA budget.

Its pricing does need to improve. As I recall, when I was working at my previous company, we paid over $100,000 a year plus maintenance. At that time, I could have purchased a RadView selection for that much and reduced the annual maintenance to around $15,000.

I have used this solution for about 10 years.

There were no stability issues.

There were no scalability issues.

Earlier, we used a home-grown, Lotus Notes-based system. Later on, I moved to a position where HPE QC was already being used.

The setup isn't too easy nor too difficult; I'd put it somewhere in the middle.

This is an excellent tool for larger companies. It has been my experience that it is not cost-effective in medium or small-scaled companies.

While at my previous employer, I evaluated other options and recommended moving to a more cost-effective tool. At that time, I recommended RadView.

Make sure it fits into your cost structure and consider the annual maintenance cost in your evaluation.

• Integration with other HPE products
• APIs
• Dashboards

Defects and Test management were earlier conducted with the help of Excel sheets. Now, they are tracked in the Quality Center leading to accountability, dashboards, and being tracked in a single place.

Licensing model: HPE has one of the most rigid, inflexible, and super expensive license models. It is an extremely heavy system application. The UI is very dated. Most applications these days have a light UI that can be accessed by pretty much any browser; QC still uses a UI which has a look almost the same for the past 20 years! I am guessing they are doing this to maintain the same look and feel so that they do not have to get their customers familiar with a new UI. When you compare this system's heavy UI with JIRA or TFS, the difference is evident!

I have been using this solution from the time it was part of Mercury Corp as a Test Director. That makes it around 17 years.

Past versions were a pain to use with frequent crashes. The current version also has its own set of problems with HPE deciding to do away with its HTML/Lite version leading to a lot of confusion.

While getting additional licenses is straightforward, HPE's licensing model makes life difficult with customers having to submit a "non-usage" agreement if they do not want active support for part of the licenses. For example, you have 100 licenses and decide to get an additional 50; later on you want to downscale to 50, you will need to sign a document that says that you will not use those licenses even though you OWN the licenses! We found this extremely irritating and impossible to explain to end customers who were (and still are) irate. Support should not have anything to do with usage!

Technical support is pretty good. However, their SLAs are based on locale, timing, etc.

We did not previously use a different solution.

HPE has one of the most complicated installations. Upgrades are a nightmare. Even HPE recommends doing a fresh installation and a cut-over.

HPEs licensing model is inflexible, rigid, and is not customer-centric.

We did not evaluate other solutions.

I would recommend going towards another solution unless you have an entire HPE shop. Other similar products offer more features, are lighter, and are very light on the pocket book, too. We are also moving away from this product, primarily due to licensing costs