Qualibrate Reviews

We're using it for testing, mostly regression testing.

Before Qualibrate, we did a release in about three months, and the regression testing would take us about three weeks with 15 people. Now, we do it in one night. The time savings are really important because we can now release things more quickly.

When it comes to problems that require changes, we can now do them in one week. Before, we would save all such changes and do them together over three months. The business is very pleased with the way we work now.

Qualibrate's low-code approach also enables users to get on board quickly and use it. We work with SAP and we know everything about SAP, but when we use Qualibrate, we can easily explain to our new people also how it works. Even business users can create automated test scripts without coding knowledge. Generally, getting them to do so would require about a month of training but now it is done in a day.

We're also able to do more frequent testing, without putting too much pressure on timelines and resources. In previous times, we didn't test security notes because there were too many. Now, every time we have a security note in the system, we do a test, and that's great. That has helped make our risk very low.

The most valuable feature is the way it works, the usability. From the first setup, the user interface is the most beautiful. It's very nice the way you can recover things and make it work.

We use the solution’s Test Planning & test Execution Scheduling features, and they are very important. They are easy to work with. We use SAP Solution Manager, and Qualibrate works with it, enabling us to manage all our tests, taking them from Solution Manager directly into Qualibrate. Because everything we do is in SAP Solution Manager, we are now able to do test automatization, combined with Qualibrate. All the defects are available in SAP Solution Manager and all the changes will be in Qualibrate, immediately.

From Qualibrate, we see whether tests are good or not. We can create a test-related incident in Solution Manager for those that are not good, and Test Suite in Solution Manager will notify the person who has to rebuild it. That works very nicely.

Not everything in SAP works well with Qualibrate. There is a development tool called xpath and you have to program it. We always thought it wouldn't be necessary to program it with Qualibrate, that everything could be solved by Qualibrate without programming, but you have to program some things. Using xpath is more complicated, and not easy for everybody. It would be helpful if there were a no-code solution for this.

We have been using Qualibrate for more than three years now. Our first version was version 2.0, and now we are moving to version 3.0.

It's very stable. We never have problems with it.

We haven't had any problems with scalability. Currently, we have more than 100 scripts and some are big scripts with 400 to 500 steps.

We have plans to increase our usage of Qualibrate because we are going to S/4HANA, and that's a big step for us. 

Qualibrate's technical support is great. They always work together with us. There are people from India who help us and there is no problem with support when there is a problem we need to resolve.

Positive

We did not have a previous solution. We did everything manually.

The initial setup of Qualibrate was not complex at all.

The group we bought it from, Fiori, was very willing to work with us. It was very good because there was very little waiting. I would call the manager and he would call me back and we would immediately have someone on the line who knows the business. That was really great. And they come onsite, making it easy, and explain the solution to you.

Our deployment took about two weeks.

We had 32 big scripts already and we wanted to put them in Qualibrate. The consultant gave us lessons and, in about two weeks, we were ready. It was easy. We had 15 people on the floor, and two of them were from the consultant, and they helped us with the scripts so that everybody understood everything. That was great.

We have seen ROI in our ability to release things a lot faster.

I compared the prices of the 15 solutions we looked at. Qualibrate was the most valuable because it could be integrated with SAP Solution Manager. Tosca had a big price, but it does everything, including defects. But because we already had defects taken care of with SAP Solution Manager, we just needed one module to work with and that was Qualibrate. It was easy to pick it and not so expensive.

We looked at Tosca and about 15 other solutions. Because we have a lot of things in SAP Solution Manager we tried an SAP solution, but that didn't work for us, so we looked further. Qualibrate was the best for Solution Manager. The big pro was the user interface. That was really the most important thing I saw.

If we didn't have SAP Solution Manager, I probably would have bought Tosca, but our other tools already covered a lot of its functionality.

When you are considering buying something, you have to see the user interface. If you buy something that is complex, people won't work with it. Qualibrate is so easy to work with.

I would probably rate the solution higher than eight out of 10 if we weren't using the on-premises version. With the cloud version, Qualibrate can look at things with you, but with the on-premises version they can't. That makes some things a little bit difficult, but because we work with defense material, it's not possible. The on-premises version is more difficult than the cloud version.

We are trying to convert a large majority of our regression testing from being manual to using this automated solution. We want to basically reduce the number of manhours that we have to spend three or four times a year to execute our regression test cycles. We want Qualibrate to offload those hours.

It is a web-based application, but my understanding is that it is hosted in the cloud on Qualibrate's server. So, I would say it is in the cloud.

It will alleviate some of the stress in terms of enabling us to do more frequent testing, and it will help us with our timelines. I certainly think that the potential is there, but we're not there yet because we're still kind of in the project rollout phase. So, we're not quite reaping the rewards or the dividends of deploying Qualibrate because of this upfront effort we're having to put into it, but myself, my manager, and my colleagues see the value. We do see the long-term value in terms of time savings and the stress it is going to alleviate for the team. That's the whole reason we wanted Qualibrate. It is just unfortunate that we're not far enough to have reaped the rewards just yet. There might also be some additional savings or benefits, but we haven't gone off to explore those in any sort of capacity since it is not really our focus.

It is the principle functionality that we're leveraging, which really can be defined as recordings and playbacks. So, you record the scripts that you want to execute and you also want to be able to playback. So, these are the features that we are largely leveraging. There are flows and scenarios, and they are the design aspects that fit within the playback and the recording solution. For me, they are the core of Qualibrate, and that's what we're using. 

There is a module that we would like to have. We would like Qualibrate to design a requirements module so that we can design our testing, our flows, and our scenarios based on our actual requirements. Right now, we're doing that, but we're having to do it outside of Qualibrate. For example, in Excel, we might have a list of 50, 60, or 70 different requirements and combinations of tests that need to be executed, and since that module doesn't exist in Qualibrate, we're doing it offline. We have already vocalized that wishlist to them, and they have acknowledged it, but I have no idea when they're going to get around to deploying something like that. It is probably number one on our list. 

We're using the testing schedule and execution a little bit. We're not really thrilled with the testing schedules just yet. They just recently enhanced the schedules module. So, I haven't got to play with those new enhancements just yet, but we have noted to the Qualibrate team that we were hoping for some expanded functionality in that space. I think they're working on that, and some of it might've been delivered recently, but there are still some things in their backlog that we're hoping they will deliver.

Their support has been quite good overall, but there have been some frustrations recently where we just feel like we're not always being heard. They are very inconsistent. In some cases, their support is very immediate, and then in other cases, there is absolute silence.

We just started this year. We got training in early February, so we have been using it basically for seven or eight months.

Its stability is pretty good. We had some issues early on, probably back in February or March. We raised some concerns with the Qualibrate team, and they deployed some development changes that created some improved stability. I still run into a stability type of issue once in a while, but it is 80% to 90% less than what I was running into six or seven months ago. So, it seems like it's a much more stable solution. There is still room for improvement, but I'm quite happy with it at this time.

It is a very scalable solution. We started out with one component, and we are already working on our second and our most important component. I don't see why Qualibrate couldn't be used basically for 90% to 95% of our testing. I do think there are going to be some test cases for which it just makes sense to leave them as manual because they may be highly complex, or there may be some nuances about it. So, it might make more sense that a person does those test cases on his or her keyboard and monitor, but we're going to scale it and really maximize it from a regression testing perspective. Our team is really looking forward to having all this completely done and be able to have the time savings. So, its scalability is great.

In terms of usage, I don't think we're going to use it on a day-to-day basis. This is going to be largely used for automated testing that's going to occur on a virtual machine. We want to basically schedule it and have these tests run hands-off. I and my two colleagues who are hands-on in these projects are basically administrators of the Qualibrate solution within the team. So, we're making the decisions and governing who's using it, etc. We do most of our regression testing on a quarterly basis or near quarterly basis, so there are three to four cycles a year. That will be the primary use case. We will probably run some of the core tests on a more frequent basis. I can see us doing it maybe weekly. I don't know if we'll ever do any sort of daily testing with it, and I'm not sure that we would ever do any sort of manual testing. So, I don't see any of my colleagues sitting at their desks and going, "Oh, you know what? I think I'll go run some tests in Qualibrate today." We're going to schedule these, we're going to offload these, and we're just going to check on what the outcomes are. If something goes wrong, a test fails or something else is amiss, then we'll address that. Otherwise, it will be kind of out of sight, out of mind as per our long-term plan.

I am going to give their support a seven out of 10 because I feel that the quality of the support is quite high, but we are a little bit frustrated, especially in recent weeks, with the timeliness of the support.

When we were a customer who was just starting out, they were always available for an email or for a quick call. They would be willing to jump on a call on short notice. They're probably assuming that we're much more self-sufficient now, which is largely true, but we've had three or four topics come up probably since the 1st of July. I know that my colleague has submitted three or four support cases, and he hasn't even got an acknowledgment that they were received by them.

The first thing you want to do with a customer is to acknowledge that we've got your ticket, and we're looking into it. We'll get back to you, even if it takes a while to get back. They're not even acknowledging his cases. I submitted one this past Friday, and I got a response the following Monday. So, it seems they are very inconsistent. In some cases, their support is very immediate, and then in other cases, there is absolute silence. 

This is the first functional testing tool that we've implemented. I have been harping on this to my management staff for years and years. We never even considered and evaluated any tools, but finally, management saw a need and said, "Yeah, okay, Steve, you're right." It wasn't really so much that use case that clinched the purchase or procurement of Qualibrate; it was the fact that we actually have a second use case on which we haven't really spent a lot of time, and one of our higher up managers wanted to use Qualibrate to generate data in our SAP systems.

Our solution sits on top of SAP or works within the SAP environments. So, we do a lot of demos and proof of concepts. The systems that SAP provides are actually quite old, and there is no more-recent data. We're talking data from 2013, '14, '15. So, this manager wanted to have one of my colleagues write up some scripts that would actually run in SAP and generate, for example, sales order data, purchase order data, and other business-like data that would look like more real meaningful data from a demo perspective and from the customer's perspective. Because that takes on a sales or pre-sales approach, he could go in and justify the expense. So, even though I've been screaming for it for 10 years and has basically been ignored, it was another use case that actually clinched it. My colleague is working on it, and he is having some success with that use case, but from a testing perspective, this is the first time we did any sort of evaluation or actual purchasing of an automated testing or an automated scripting solution, whatever you want to call Qualibrate.

I found it a bit more on the complex side. There is some flexibility within the solution, and that's fine. Flexibility can be useful. I like flexibility, but if you're brand new to the solution and you don't really know what you're doing, then it's hard to understand how to design what you want to do within that flexible solution. So, we had to scramble a little bit. We're now on our third phase of this project, and each time, I and the two colleagues who are heavily invested in this have learned what not to do versus what to do. We're improving each time, but we're having to figure that out kind of on our own with some guidance from the Qualibrate team. Most people learn by doing and by making mistakes, and that's where I feel like we're at. So, initially, we struggled with the design and how we wanted to design our testing within the Qualibrate solution. We struggled a little bit like that. We did reach out to the support team at Qualibrate. We got some insight from them, but I also feel like we were largely left on our own to kind of figure it out. That's not necessarily a problem, but some customers may not like that approach. They're going to want more direct answers.

We're still deploying it. The initial phase was three months. We then took a break, and then we went on for two months. We just started our third phase about two weeks ago. So, when we're reaching a milestone, we are putting that phase off, looking back at lessons learned, and then taking a small and kind of a mental break from it. We then come back to it a short time later. The phases are getting smaller, and that's also representative of the fact that we are getting better and more efficient. We have a better understanding of the solution and how our requirements stick within the Qualibrate solution. I actually expect this third phase to be as little as maybe six weeks. So, it started out initially as three months of learning the hard way and making mistakes. We've now whittled down to perhaps a month and a half for each of our phases. We still probably have another one or two phases to go. This will be something that would roll around probably through the new year. I hope to be done by January, and then we can effectively start using Qualibrate on a near-daily or perhaps a weekly basis. 

In terms of the implementation strategy, our software is fairly mature. It has been around for 20 years. We kind of felt like using Qualibrate to effectively transition all of our manual testing to automated testing. It was like an elephant. How do you eat an elephant? It is one bite at a time. So, we tried to look at our software from a component level, and we said, "We've got this really big component, and we have a couple of other smaller components." We then basically tried to choose one that didn't have a massive footprint. It was important, but it was not so large that we felt like we were going to try and drown getting it integrated into Qualibrate. So, that was phase one that we focused on. Once we got that done, we moved to phase two.

We had to break down phase two and phase three into two sets. So, our last phase and the current phase we're on will cover our most critical component within the solution. So, this will be a big milestone. When we're done with phase three, we'll be able to look back and say 90% of this component can now be tested through Qualibrate. Phase four, and probably phase five, is going to touch smaller and less important modules that ideally we'll be able to roll out and get done in a pretty compressed timeframe. So, we might be done sometime in December, but because of the upcoming holidays at that time of the year, it probably is going to roll over into January.

For the project that I'm closer to, we're not realizing the dividends, but we know that it's on the horizon. We're just not quite there yet, but we're confident that we're going to have it. So, we are looking forward to that day.

For the parallel use case, we are seeing some early returns. The reason is that it is a bit of a project. My colleague was able to stand it up in a fairly limited number of hours. It pretty quickly and easily started creating or generating data in the SAP demo systems that were already being used by our pre-sales team. So, we are seeing some benefit from that. 

We probably have 10 licenses, but I don't know what are the costs or anything like that.

We did not evaluate other options. The reason was that we had contacts on each company who already had some sort of relationship. I think it was a rather informal relationship, but there had been some dialogue and some interaction in terms of people shaking hands and knowing each other by name, probably in trade shows in Europe. So, they knew each other. Because we already had kind of that rapport, we just looked at Qualibrate.

They did a short PoC with us. We met with them over a period of three or four days, and it looked like a really good solution. It was young but mature enough to do what we needed to do, and we were sold. Perhaps, we should have done some due diligence. We would have done that if we hadn't had that relationship already there, but I think that's what kind of clinched it for us.

I would advise making sure that you understand the tool. You don't have to understand every aspect of it, but you need a pretty thorough understanding of its components, the playbacks, the designs, and the technical pieces that make up the solution. Our pain point was the very first phase. We were brand new to the solution. We went through some knowledge-sharing sessions with their support staff, but we really felt like we were struggling at first. Most of that was probably because we simply did not understand Qualibrate to the depth that we did in subsequent phases. So, it would be good if you could really spend a little bit of time upfront, maybe just from a sandbox perspective, and you test it, mess around with it, and get familiar with it. After that, you can start to decide and design how your solution and how your scripting should look within Qualibrate. This way, you'll avoid some of the pitfalls that we ran into. We found ourselves having to do rework over and over again, and that was simply because we learned a better way of doing it than we did initially. 

You can maybe test it on a smaller scale. We were so motivated to get started that we tried to run before we could crawl. We should have kind of spent a little bit more time doing that initial build-out testing. We should have made sure we were happy with the results and that we were taking the right approach. We did all this work and then went, "Oh, that's not really going to work." We then had to go back and do some redesigning. So, some upfront commitment would help there.

It is fairly advanced. It has only been around for three or four years, but it does seem like a pretty mature solution. Most of it is pretty user-friendly. There are some components that I have found to be a little bit more technical and a little bit more daunting to use, but overall, it is quite easy to use as far as this kind of scripting software goes. I'm quite happy with it. I'm not overly technical, and I can understand it fairly easily. That must make it pretty good.

We're not using it on a daily basis. We're still kind of in a project phase where we're still ramping up and getting all of our scripting and recording done. So, right now, everything is more in kind of a batched process or phased process. We do hope to get to the point of being able to use it for daily or near-daily testing, but we're still a little away from that. Since we've only had it in-house for seven or eight months, we're just not using it on that frequency just yet, but that is our long-term plan. There are a couple of components that we are not really using at this time. That's because they are still immature, and they have not been around very long as part of the Qualibrate solution. We'll get to those perhaps another day. We're focused on our primary use case right now.

We are quite happy with the flows and scenarios and those building blocks that really represent the core of the solution for scripting and playbacks. So, we're overall quite happy with this solution at this point.

Based on how it operates today, I would give it a nine out of 10.

We are using Qualibrate for our testing strategy, as well as for standard change requests.

We began working with Qualibrate for testing our on-premises SAP solution. After that, we started implementing the cloud application, SAP SuccessFactors, so we started using it to test that.

Regression testing is very important for this application, and we use Qualibrate for this, as well as to perform user acceptance testing.

Another activity that we use Qualibrate for is the automation of standard changes. These types of changes are pre-approved, and via Qualibrate, the configuration is done. The flow receives requests for changing an item in our ticketing system, ServiceNow, via our middleware system, which is connected to Qualibrate. In Qualibrate, existing recorded scripts will be started and then Transport in the SAP development system will be created automatically.

In the SAP landscape itself, the Transport will be moved automatically with all checks through the landscape. Of course, at the end when the testing is completed in the production environment, there will be a message sent back to the ticketing system via the middleware and the ticket will be closed.

I recently created a business case for Qualibrate and it has to do with regression testing. We have approximately 300 to 400 SAP end-users on-premises in the organization, across different business units. If you have to organize a regression test, it is because you have changed something and you want to know if it will have an impact on the processes in the other regions. In cases like this, you want to check to ensure that nothing was touched.

Normally you invite maybe 50 key users to test in the different functional domains. This will perhaps cost an average of four hours per person. Instead, you are running automated test scripts that have been pre-recorded. Even if the length of time per test remains at four hours, you are saving that much in terms of what it would cost to pay testers for the same work.

It is not possible to test everything, especially if you have interfaces with other external systems, but you do win a lot of the time. Also, it's not only for testing when your project is going live, or when you're bringing changes to an environment but also checking your daily operational support. It brings a really big benefit in terms of keeping your system stabilized. In the end, you have fewer disruptions for the business.

The low-code approach enables users to onboard quickly and then use it in their daily test creation, but I would always suggest centralizing it. The users have the scripts and you only need one or two people on a central level to keep control of it. 

People can make use of Qualibrate for training because that's also a benefit. If you have a new employee, you can show them in the system how one of the end-to-end processes looks. You can also explain the UFT in the same way.

The most valuable feature is that it's user-friendly. It is really easy to use and if you have your adapter downloaded, you can remotely explain to your key users, the testers, how to use it.

We have all of the biggest parts centralized, which means that we can run all of the scripts at the moment we bring changes into the UFT environment. They can easily be run in the background. 

You have the ability to reduce all of the recorded transactions so that you can build your own consolidated end-to-end process. If something in the end-to-end process changes then it is very simple to insert a new part into it.

We have just finished all of the scripts for SuccessFactors, which is an HR application. Every quarter, you have a new release, and we are dealing with scrambled data that needs to be reduced. Using Qualibrate makes it easy.

What I would really like to see is if you are running scripts in Qualibrate, and there is a defect, then you can have it automatically raise a defect in your own ticketing system. We do not use the defect system in Qualibrate, which is why this would be helpful. We won't want to have to use the ticketing system in Qualibrate in addition to the one we use now.

The reporting can be improved. For example, the reports can be better and show details like how many of your scripts have been executed, what the history is, which scripts have not been executed for more than a month, and other types of statistics.

It would be nice if when you add or change some specific script in your SAP system, it would automatically check to see what it implied with respect to Qualibrate. For example, it would link to the appropriate Qualibrate script if it needed to be changed. In the context of an end-to-end process, which has already been recorded, if next week I change something in the process then I would like to get an alert that we need to make the relevant change in the process.

My company has been working with Qualibrate for two years.

I worked with Qualibrate for another company approximately six years ago.

Stability-wise, so far so good. Last week, there were two times that the system was unavailable but that really does not happen very often. In case it does happen, the communication from their support desk is very fast. They inform you directly.

Scalability-wise, this solution is fine.

We don't have very many people who are working on it. There are two external, my test manager, and myself, so four in total. 

We have no issues with the technical support. When I have a problem and the system is not available, I want to know how soon I can get it back up and running. I do not run these kinds of systems every hour; even when your SAP is running 24/7, your testing does not. However, when you want to test something, it needs to be available, and support takes good care of this. When I have called they pick up the problem and they resolve it.

Approximately six years ago, I was with another company and I used several different test systems. I was involved in a pilot at the time and helped to define the requirements. One of the important ones was to keep it fair user-friendly, which Qualibrate is.

Over the period of a few years, I looked at test automation. At one point, in 2018, I compared the features in Qualibrate with what it had four years prior, and there had been new guidelines for things like data introduced during that time. A lot of improvements had been made to the product.

I have experience with HP QTP, but it is an expensive tool and I explained that I didn't need a Rolls Royce for testing. Rather, I just wanted a simple tool. I didn't want to have a very expensive tool where I only use 50% of the functionality when instead I can have one that's user-friendly.

Comparing Qualibrate with my other tool in terms of manual testing and the process of documentation, I would rate Qualibrate a nine and the other tool a five. The other tool was very technical and not user-friendly. Every time you need to change something, you have to contact technical support. It is also very complex when you want to explain it to your key users.

I have my own test manager who is responsible for the SAP on-premises solution, which includes testing it with Qualibrate. We had an external team help us with supporting the functionality of our cloud system, and we also asked them to take care of Qualibrate. Unfortunately, it took a really long time and the person responsible for it has now been replaced. Together, the new person, working with my test manager, has it under control.

If you have it under control, you just have to do the maintenance.

We have seen a return on our investment. There are a lot fewer people involved in our testing and the system is more stable. In addition, more people compliment you and less complain if the system is not available, which is a really big benefit.

We signed a three-year contract and the pricing is in line with our expectations.

If you look at the cost of your license and what is required, you will see that it is better to keep everything centralized and pay for perhaps five user licenses. This is more cost-effective than giving one to everybody in the business.

When I proposed to use Qualibrate, they were hesitant because it was not in the top four or five of Gartner. That necessitated some more evaluation which included a tool by IBM and at least one other one.

My advice is that if you know how an application is working then reduce it on a weekly or perhaps daily basis. If you are doing this then you will very quickly understand how the system is working.

The first time you record a regression test, I would advise doing it in a classroom session. In this scenario, people are doing the regression test manually and you ask them to execute the same script, but this time, start the Qualibrate recording function. People can make use of the backspace functionality and with it being recorded, it can be used the next time a regression test is needed.

I suggest that you not only look at the price but also the quality. This vendor listens to you and if you have requirements or need help, they are always available and not only by phone. If required, they will visit the customer from time to time. The application is user-friendly and there is documentation available. Overall, we are really happy with it.

I would rate this solution a nine out of ten.

With Qualibrate, the accent for us is on SAP, because that's our biggest application and a critical application for our company. Web was also an opportunity.

If you do a SAP regression test and you have to do it via manual testing, you need to organize a lot of people. The difference with automation testing is I can do it every day. What we have with SAP are the HR Support Packages, the service packages. We do them every half year, but if I want to check it, I can do it every day. That is a great advantage when using Qualibrate for automated testing.

Qualibrate has enabled our business users to create automated test scripts without requiring coding knowledge. That was a main part of the decision for the business, that it is easy to use. You don't have to be educated and have experience in Java or HTML. They have to know what kinds of flows they want. I have explained that they need to make their flows in building blocks. We are now planning to bring the Qualibrate solution to the business users.

Qualibrate enables us to do more frequent testing, without putting too much pressure on timelines and resources. I'm doing it every day and I'm showing management the differences between that and once a month or once every half a year. We now have continuous testing and that's what we also want to do with all the applications in the cloud. We need to be able to test development or when a change is planned for an application. We need automated testing in the continuous delivery pipeline.

The more frequent testing has affected the number of production incidents and software quality a lot. If you do testing every day, after a weekend of development changes in some formulas we use for the desktop application, running the scripts automatically means they see the changes in what they have deployed. That is a big advantage. I can immediately see, in the morning, what they have done over the weekend in deploying applications. I report things to the project leaders and the people responsible for functional maintenance of that application. They are surprised: "How do you know that so fast?" It's because by running Qualibrate we get the answer.

With Qualibrate, when we looked at the time and resources we are saving it was €1.5 million per year, mostly because of the move away from manual testing. Maybe that's a small amount if you compare it with my last employer, when I worked for ING Bank, but we are a small company. We only have 1,800 employees. Using Qualibrate we are saving a lot of time for the regular users because people on the business side are not testers. They have to take care of daily business as a priority. I am able to take work from them and that makes it easier for them to do their regular work.

Also, where it might have taken a day to create a test document, now, if you do work with a flow, it takes you four or five minutes and you have your automated test flow and your documentation for your process. That makes Qualibrate a fine application to work with.

What Qualibrate makes very easy to do is to record a process flow. Within five minutes you have a clear document produced by Qualibrate. Instead of using Word, and copying and pasting pictures into it from printscreens, within five minutes what you have was easy to make and it's easy for users to use.

We had an issue with SAP when using PDF forms. That was something that was not supported by Qualibrate, but we solved that issue by choosing another solution. That was the only wish we had with Qualibrate. All the other functionality provided is all the functionality we need.

We have been using Qualibrate for three years.

It's very stable. I'm working in Qualibrate almost every day and sometimes on weekends. I haven't experienced any issues. It's running smoothly.

Our use of the solution is still growing. We started using it on sub-solutions but nowadays we are getting a lot of requests from the business to do dev projects or help them with their applications. Instead of their manual testing they would like automated testing. It's easy for me, as an admin user, to create a new project and guide them with my knowledge in creating flows and scenarios. The number of projects and business partners requesting use of it is growing in our company.

We now have 301 flows using Qualibrate and in a year we did about a half million test steps.

We have five licenses. I use it as a test engineer. On the business side there are functional maintainers, and some of the business users, or the people around them, are guiding the process flow by first creating the test steps we have to record. So we now have 10 people directly involved with Qualibrate.

We have weekly contact with Qualibrate. If there are some wishes on our side, we can mention them there and they will be put into review for deployment. Our last request was that when you record a flow, that it would automatically create screenshots. Screenshots are much better than they were before for the documentation and for use as a user manual. That has improved now.

I'm very happy with the service level they give us, especially from Alan, Marina, and Martina. They are always available for me if I'm calling, or by email or by Teams. Their technical support is a nine out of 10. 

We previously used a manual testing system that I had adopted from ING Bank. Phase one was to collect the information and phase two was to create your manual testing.

We started using Qualibrate in 2017 with a pilot version because our organizations had a big need for automating our process flows. We would usually do it manually and part of it is still manual. But our organizations had a question: "Can we go for automated testing?"

I was the person who introduced Qualibrate as a choice in our company. The parent company is IOVIO, and I worked with IOVIO when I was at ING Bank.

The initial setup was very easy. It is a SaaS application. It was easy to install and easy to use. There was no problem installing it in our environment. Within an hour it was operational.

The only thing we needed to do to use Qualibrate was more due to our technical maintenance requirements. We weren't allowed to have local admin rights but because Qualibrate is using an updater. The updater has to be clear of admin rights because otherwise you can't install new features. That made for a little discussion, but happily that's over now. But there were no problems installing or setting up the whole thing. However, I have to mention that I'm experienced in test engineering, so that also enabled us to do it very quickly and easily.

We deployed it with help from Qualibrate consultants because it was our first time using Qualibrate. They were onsite and helped us to install it on each of the laptops, easily. We had three licenses to start.

The  consultants were great. Qualibrate is a young and dynamic team, and they were willing and able to help us every time I called.

Automated testing is not cheap. But other companies, for example, Panaya, required a minimum of 10 licenses. Qualibrate allowed us to start small, with three licenses, with a price that was competitive within the market.

We invited a lot of companies to do a presentation. In the end, our stakeholders were convinced that Qualibrate was the best solution for us. We looked at Tosca, Solution Manager from SAP, Panaya, and Testbirds. Because we are government organizations, and we need to be very transparent in choosing a company to provide us with automated testing, we had to invite several companies.

In the past I was a test engineer with Micro Focus LoadRunner and Solution Manager. Qualibrate is a very simple solution. You don't have to be a person who has skills in coding or creating or something like that. It makes for a very easy tool to use in Agile sprints. The difference with LoadRunner was that you had to be a coding specialist, otherwise you could not run the application. That makes a great difference. People without those skills can use Qualibrate as a tool for flow documentations, and it's easy to maintain while providing test automation as well.

While you don't need to have coding skills, if you have the skills it makes some things easier to use. There are some options within the data, in the advanced options, that you can use. There are building blocks created by Qualibrate and some code that you can use. For example, for a date field you use code they have already made for you. You can specify today plus seven days so that your script is always running smoothly.

I would advise using Qualibrate as a testing tool in a DevOps organization. It's a fine product. Other solutions are also okay. Tosca is also a great solution, but in the end, Qualibrate makes us better.

We don't use Qualibrate's Test Planning and test Execution Scheduling features yet, but we plan to do so this year. We are now growing from three to five licenses and management wants to know what the options are for test management and test planning so that we can schedule things for when the business wants to run the scripts. It can be every day or once a week. That's the way we plan to use Test Planning.

We don't use defects at the moment because we are now organizing API management to get JIRA onboard. Our process is still growing in production.

We are totally satisfied with what we now have in our possession with Qualibrate. We are very happy with the solution as it stands now.

Our most important use case was a carve out. We acquired an organization, so we needed to to carve out their SAP environment. 

We use it for regression testing. We set up an automated regression test set because we have to test everything to make sure that everything still works.

It is SaaS.

It allows us to change more quickly because we built that automated test set to do regression testing, which takes away the manual effort of regression testing. So, if we change things, then we don't have to worry about the time that it takes for that regression testing and can do that overnight. I think it has changed the way that our organization looks at the risk of changes in an SAP environment.

The solution has enabled business users to create automated test scripts without requiring coding knowledge. Initially, it doesn't speed it up significantly because the business users still need to do the first test. For example, you do your test, then you would typically hand that over to the business units through the testing. So, we still do that, but then they record it and it becomes part of the progression tests. From my end-to-end scenario testing, I don't have to do that much anymore. For regression testing, that has pretty much been all taken out of the equation.

Qualibrate has enabled us to do more frequent testing, without putting too much pressure on timelines and resources. Because of the way that we have the intervention regression test setup, we hardly have any production incidents anymore.

The widget's ease of use is the most valuable, which means it allows you or business people to record the automated test scripts. In most cases, it is really good because it is the business people who actually know how the system is being used. The simplicity of the design is valuable, where you can record your transactions, then create your automated scripts. You can automate it at the same time, and the automation features are cool.

It definitely has a low-code approach that has enabled users to get onboard quickly and use it in their daily test creation, running, and maintenance. If we change things in SAP, we still do the Unix test. Once that is signed off, we say, "Okay, it does what it should do, and we record it in Qualibrate." Then, we use it for end-to-end testing, so it becomes part of the regression test set.

What could be improved would be the intuitiveness of the reporting engine. It does have reporting, i.e., a dashboard, but it is preconfigured, predefined KPIs and datasets. That could be improved because the datasets don't have descriptions, so you really need to know what you're doing. Whereas, it would be great if it could have more descriptions and be easy to build your own KPIs.

I have been using it for two and a half years. The last time that I used it was about two months ago.

The stability is reasonable. We do have the odd incident, but it is pretty stable overall.

It is easy to scale. If we want new users, then we can do this on the fly.

We have about eight people at the moment using it. Most are business users or functional consultants in the IT organization.

Qualibrate is being used very frequently. At this moment, we don't have any plans for new systems.

The technical support is very good. If there are any issues, then they are always there to help out. They have good response times within a day, but typically within half a day. They are knowledgeable and able to help.

There is a lot of information online in the help files, but also the team themselves are always willing to answer questions and help out with explaining how things work. I think the overall support, not just technical support, is really good.

We were using Solution Manager for test management, but we weren't using any test automation.

I heard about Qualibrate from a guy that I know who works at SAP.

At that moment, we are starting to get an idea of test automation. We have been in contact with Qualibrate, and it is really good.

The initial setup was pretty straightforward and easy. We just had to make sure that a lot of scripting was enabled and the firewall was set up, then the Qualibrate team took care of it. It all worked.

The deployment took a couple of days. This was more because we had to organize it with the infrastructure teams. The communication with the team is what takes the time because the actual setup is really easy. We have to log a ticket and wait.

Someone from the infrastructure team and an SAP consultant needed to do about 10 minutes of work.

As many people involved in a change, those are typically who maintain it. It is not a huge maintenance effort. Because if you change something in your system, then you should change the test as well.

If you look at it from an overall testing perspective, the solution saves a lot of time and documentation. If you look at how individual change flows to the software development life cycle, it might be a 20 percent savings, but still significant. If you look at it from an overall perspective, where you look at the time that we typically would spend on testing, then it is probably a bit more like 60 to 70 percent time savings.

It has a more reliable time to market and delivery of software. You get significantly less defects.

Our first use case with Qualibrate couldn't have been done without Qualibrate because there was limited time for the project. We could only do it because we automated it. We were only able to create the end-to-end automated regression test set because we could leverage our business users. From an ROI perspective, we saved so much time in testing from an IT perspective that it definitely pays itself back.

Qualibrate is realistically priced. I can't compare it because I haven't looked at other tools, but I think it is good. What I like is you can simply add new users, if you want. It has a license model that comes with different types of users, which I think makes sense.

Since it is SaaS, the licensing is all based on subscriptions.

Just try it. The only advice would be to think differently about testing and how you organize testing. It comes with an easy setup to use that business users can do. 

When you look at test automation and the progression testing that is ready, testing is not really an issue anymore. You can just test everything all the time, because time is not an issue. It puts a different perspective on software development. However, that is not specifically for Qualibrate, but for test automation in general. With Qualibrate, it is just so easy.

Biggest lesson learnt: It can be easy to automate testing. I never expected that. Typically, automating was almost always very technical. Qualibrate made it easy to automate testing.

One of the things that we were really looking for is test planning, but that is a new feature that is now there. So, that is good.

Its Test Planning & Execution Scheduling features are definitely something that we want to look into using in the future.

I would rate it an eight (out of 10).