Worksoft Certify Reviews

We get user stories from the DevOps teams, in conjunction with a recording they make with another Worksoft tool. Then we will investigate if it has already been automated and, if not, to automate the process which has been delivered to us.

This is all set up in a Citrix environment. We have SAP being used at the moment and we still have the old SAP ECC up and running. I'm not sure which part of this is cloud-based, but the Certify solution is installed on Citrix.

Because of the fact that we started just six months ago and we have a small team. But slowly but surely we are gaining more audience; more people are starting to get interested. That should lead us to be able to start implementing it the way it should be done. We have done some regression testing and, when doing so, we found real issues. So it has proven itself to be useful during regression testing at least.

We have definitely seen savings in testing time. Scripts are executed five or even 10 times faster than any one of us could do by hand. While we don't do so at the moment, we are going to start executing them in a lights-out environment. We will run tests during the night and get more numbers, execute more tests. That should also help us save time. We have to get the experience and the numbers for this, but I think it will save us a lot of time.

The biggest feature is the fact that it's codeless. It takes away the problem of finding people with the correct programming language, since there are multiple such languages. It saves time in introducing people to the solution because they don't need programming knowledge, they just need to be able to think logically. This makes it vastly usable by more people who are not even acquainted with IT at all.

Also, the solution's web UI testing abilities for testing of modern applications is pretty awesome. Like with every product there are some parts which can be improved, but overall it's great.

It's very easy to use and to install. You have to know, as a user, what your exact application is on the test; you need to know which object recognition files you need to use.

You use the tool to do your automated testing. As far as I know at this moment, it can do a lot of stuff. It's usable in DevOps, so with regards to packaged and non-packaged software, it's good.

I use Capture from within Certify. I also have a stand-alone capture that I have up and running. If you look at the whole cycle, it takes the user a lot of time to create the records. During the capture, the responsiveness of the system is really slow. But after that, when you send it to Analyze, the documentation is really easy. Just click the button and choose the format. Automate is the same. You just create automation and choose a file name. Then, when you need it, you just download it into Certify and start using it. We've been doing it for some time now.

The Capture feature helps find the actual processes to test for and to create end-to-end testing. We ask the users, when we are making the recordings, not only to enter the proper data but also to provide us with comments or LiveTouch images of messages that need to be recorded. They know, "Okay, when I see this message then it's up and running." Because they take the end-to-end as a whole in the recording, we can use that as process knowledge as well. So the process is, in fact, being captured in the Analyze software.

In terms of the solution's ability to build tests and reuse them, I would rate it at eight out of 10. We record it on one environment. We make it completely environment-agnostic, data driven. Once recorded, we can reuse it on every single environment in the development cycle, which is awesome.

Performance on the web UI part, especially with some of the more comprehensive Fiori features, like the complex tables that are being used, could be improved. In those cases we have noticed a lot of execution-time increase with regards to the Certify solution.

I have been using Worksoft Certify for six months now. I started using it in September of 2019.

We've had no Certified-related stability problems. We have issues with things like Execution Manager and Analyze. I'm not sure if those problems are infrastructure-related or due to the Worksoft setup, but with regards to Certify it's stable. Sometimes there is a crash, but I think it's more related to the fact we're doing a lot of complex stuff in a Citrix environment with low resources.

The solution hasn't enabled us to scale up testing yet. We're on the way forward, but because of some issues in our own architecture we are not able to execute those tests. But I know how the setup is working, and I think you can scale up really easily; just add more machines, add more users, and have a go.

When I started within the program itself, no one else was using it. There were two users on the Railnova team. At this moment, about 10 or 12 people are using but within a couple of months we will be around 50 users in total.

I would rate Worksoft's technical support somewhere between a six or seven out of 10. In two-thirds of the cases we get a proper technical support member who has the knowledge to help us with our problem. But at other times we get someone who doesn't really know what he or she is doing or doesn't really understand the issue.

Another big part of the grade I gave is the fact that when you are in contact with the call center, a lot of times there really is a lot of background noise. With the accent, it's already really tough for me in fact to understand them, and with the background noise the problem gets bigger. But I've had also a lot of support from the Germans and from all over the world. Most of them are really capable.

It's an easy setup. There are some small configuration settings and then you can have a go. It's all up to the user to do the updates on the definition files. That's also easily available to us.

I started using the software without hearing from Worksoft. I only had to do some courses on the Worksoft University web page they provide. I didn't have any real, proper training, and I was up and running within two or three weeks. And within two months, I was able to provide enough support to get multiple teams within our company up and running with Worksoft. It's really straightforward.

I wasn't part of it, but I believe the initial setup and further configurations took two or three months in total.

Because of the fact that it's also able to do orchestration and because of the fact that our company is moving from the old SAP towards SAP Fiori — they wanted to have the main focus on Fiori for the UI part, in conjunction with the orchestration which Worksoft is able to do — at first it was only UI-driven. But we will expand into more and more Worksoft uses.

The company used a Worksoft consultant for the deployment but I don't know her name.

We, as a company, have good contact with one of the Dutch Worksoft managers and he introduced the integrator to us, as that manager is from the United States. The consultant came over for a week to give us some training on a number of things because we are not only using Certify.

They were really happy with her. A lot of questions were answered, a lot of issues were resolved. She was able to contact Worksoft support really fast. They had a blast while she was here.

We haven't seen ROI yet because we are in start-up mode with Certify. At this time we are only investing in the solution. Hopefully, we will be able to have some insights into ROI within a half-year from now.

The licensing is yearly.

My advice would be to think outside of the box. If you've chosen to work with Worksoft, you have to embrace the product as a whole. You will find, as with any other product, that some things that don't operate the way you want them to or would have expected them to. But if you teach yourself to view your problems from different angles using the software, then you will be able to come up with the most brilliant solutions. You can do much more with a codeless tool than you might think upfront.

The biggest lesson I have learned using comes back to the codeless part. I view myself as a smart guy, but I don't have the proper coding language knowledge. I was working for myself over the course eight years, before working here, and oftentimes the jobs were really cool, but most of the time I had to do Python and this and that. That was always a struggle because sometimes, when you've learned a language but you're not using it for a year or two years and you want to go back, you have to start remembering it. So I was turned down for those jobs. In this case, and we can show the world that it can be done codeless, if you have the proper tools.

When I was first introduced to Worksoft and they told me it was codeless, I was really skeptical. I said, "I don't see that happening," because I had been doing this for quite a while and was used to doing some coding. But the tool convinced me otherwise, which is really nice.

Overall, it's capable of being used in modern technology environments. I have been using it for six months now and I still have a lot of learning to do. And as a company, we need to start using more of the Certify features, not only scripting and rerunning those scripts.

Most of the people who are using it right now in our company already have some testing experience, but it's our goal to have business and IT people use the Capture feature as part of the process for DevOps.

We don't do test maintenance at the moment. We started out with test automation. We had to set up a base for the DevOps teams and then support them from that point onwards. So we are slowly moving into the maintenance part. Because we have split the data from the script itself — everything is data-driven — so it should be fairly easy for us to make the necessary changes. I think execution is faster when compared to human hand movements. But for changing or maintenance, I don't know.

The solution hasn't enabled us to find more defects at the moment, because we have been focusing on "happy path" testing. We need to get to the end-point of the end-to-end testing. But I believe, and I'm rather positive about this, that if defects are entered into the system, given that our regression test set covers a big percentage of the complete solution, it should be able to find defects really fast. Faster than we can.

The Certify users within our company are all in scripting. We're developers. And because we are in a scrum team, we don't have different roles in our team for test automation. A lot of things are being delivered by DevOps the teams, which you can view as functional consultants. As for the deployment and maintenance, a lot of it is outsourced to one of our partners. We do have functional and technical maintenance or support. I'm the technical guy and then we have two functional guys as well.

We have automated quite a number of test cases in last year.

We have a weekly release. A weekly release means every week that we have testing going on. The particular year where we started is called the asset management area. We could never run our Intuit testing, and we have a whole lot of Intuit testing. This one product helped us pass the testing with the Nighthawk testing, which is working on the Nighthawk manager. That's the one that we use. We can switch it on in the night and run the testing, then come back in the morning and see what has been completed. If there is any fail, we can even analyze it. We can use the evidence document to pass it on to development team to tell them, "This is where we failed."

It has improved on our defect management time. It has improved our test execution time. I don't need to manage these things, just sit somebody down to look at how the script runs. There are a lot of ways that it has helped us.

It is very easy to maintain. With scripts, I can change one line and in one step. Whatever I want, I can do. I don't need to be an expert to use it. Being a manager without a whole lot of technical knowledge, like an automation person, I can change Worksoft using what I learned during the training. That's what I like about it. Anybody can do automation.

I love the Capture 2.0 feature. When you are doing a normal manual testing, go ahead and switch on the Capture 2.0 feature, then capture everything and pass it on to your teams who can convert them quickly into test automation. With this feature, it is saving our automation creation time by about 60 to 70 percent. It is also helping our manual testing time in terms of catching all the evidence documents. 20 to 25 percent time is being saved because of this product and Capture 2.0 feature and what we are receiving with the good documentation.

I would expect more opportunities to automate Java.

I would like it to analyze what we are not using. 

I would like to see the impact analysis integrated with the performance testing tool. We have multiple tools doing multiple items. I would like to have one common tool. 

I'm also interested in load testing automation and whether we can create a script for it, then can we use the same script for my performance testing? 

Stability is quite good when compared to other things. I don't want to say it gets 100 percent rate in terms of stability, because I'm using this for about ten to ten and a half months right now. It has been close to a year, but I'm really amazed when comparing it to any other tool that I have used in the past.

I don't have the words to express its scalability besides awesome. The amount of changes that we have seen are tremendous.

We don't use them frequently, just when we have an issue. I would rate the technical support a seven out of ten. Most of the team is good and helpful. However, I would like them to evaluate the issues a little more sometimes before escalating them to engineering.

We were previously using Micro Focus UFT. The tool was good, and we did not have a lot of problems with it. The only problem was SAP changes a lot of things every time. The frequent changes were causing a lot of issues for us in terms of automation. We were able to automate many things, but the maintenance was a big problem for us. 

1. You needed to have a person who had the coding knowledge to do it.
2. The frequent changes made the scripts useless. Then, we would have to come back and redo a lot of things. 

This is where we were looking for a product where we could have minimum maintenance that anybody can automate. This is the concept why we came into Worksoft.

We used Worksoft team initially. That helped us through the setup and other things. They did a great job. We probably automated about 275 tests in less than five weeks, including the setup. 

I suggest people go head and use Worksoft, along with their services, when you buy the tool. They will help you to onboard it quickly and set it up for you. They will do lot of the automations. They will help you with lot of these practices, then you can take it over from there.

We have seen ROI.

By using automation, it reduced about 75 percent of the time when compared to any other tool. The changes for Worksoft used the same script as the script automating UFT. Manually, running our tool takes about 4 hours, but with Worksoft, we were able to do it in less than 30 minutes. Whereas, the same thing that you had to do would take you almost 55 minutes to an hour. There is now a 50 percent savings in terms of other automation tools and an 85 percent savings in terms of manual to automation.

We have seen more than 40 to 50 percent reduction, in terms of all around time, where we were doing five days a week for a major maintenance testing of our first cycle. We have reduced it down to three days now. So, that is a 40 percent savings which we are seeing. We are not completely automated because we are still doing sampling. We have just automated a good 40 percent of our things. When we go to 80 to 100 percent, it will reduce 60 to 80 percent of our time, which is what we are looking for.

We looked at Panaya, but Panaya is not a desk automation tool. We are using Panaya for some of our impact analysis.

We did try with Selenium and many other vendors. A lot of other tools have a small test director that we tried. The ease in using this tool is very good because our business team can go in and use it. So, anybody can record for me and capture. Then, we have a very small team of automation testers who can convert the information immediately into a reusable component, parameterize it, and do the records sets. In that way, with a very small set of test automated guys, we can do much more.

I recommend this solution already to my colleagues worldwide.

We run this on seven different multiple applications. It starts from SAP, goes to the UI, comes back to SAP for violations, and then goes to mainframe for validation. Then, we use Java Web as a Java. After that, there is another HDM which we try to validate. Also, we are trying to validate a third-party application using it, because we have used a lot of their components trying to do a mock type of filing import/export option with the tool. 

We have used this solution for web UI testing, as we are on SAP Web UI 5.0 right now. We use this very heavily right now in our asset management area. It is very easy to use. The Capture 2.0 together with it is helping us, because we are now able to recognize some objects through Capture 2.0. We also have LiveTouch. This is another advantage where you can use this to capture multiple items at a time.

We use Worksoft Certify to test our SAP System. We have a global instance of SAP, which we started implementing in 2012, and we are still in the process of implementing. We have rolled out SAP to about 80 percent of our manufacturing and distribution. Right now, the remaining projects are a small distribution center and sales offices. We have ongoing projects, and three times a year, we release a new version of SAP. We rolled out SAP to a new geography, and we also added new features for our business users. Thus, as part of those projects, we use Certify to do regression testing of our existing business processes, and we also use it in the project to test new functionality.

When we are rolling out in a new country, we do a configuration for that new country. We use the automation test to test the business processes and prices of that new country. It is sort of semi-automated. Our business analysts generate sales orders from the new country, and we will run them through the shipping orders to cash, the shipping steps, and the concrete steps. Then, we get a set of documents to review. The business analysts review those documents to make sure the order is processed correctly. So, it's not fully automated, but it does help cut down on testing a lot when we roll out to new countries. 

For regression testing, that is fully automated. We have tests where the software checks the results and either returns a pass or fail. These are run as a regression suite anytime we push a change to production.

We do use it for the end-to-end testing of packaged applications, primarily SAP. We do have some plugin applications that we use it to test, which are part of the business process. We use Salesforce for CRM, and we have a custom built eBusiness application. While we don't do extensive testing of those applications using Certify, when the business processes touch one of those applications, we do cover those application with another certified test.

It has cut down on the amount of low level, grunt work that business analysts have to do and can free them up to do more critical thinking. Before we had test automation, we were running tests and relied on people, which was very time consuming. A business process test might have 100 to 150 steps across different applications, and we don't have a single person who has expertise in all those applications. When executing a manual test, we have to balance the test between different people to do their steps. In a typical project, we might have 100 to 150 of these types of tests running. The coordination of the testing process where you have to have different people available at different times is very time consuming and inefficient. What automation has done is cut that cycle dramatically because automation does not have to worry about having to find the right order management or warehouse person to do their steps. The automation just runs through, then the business analyst can review the results afterwards. Therefore, it has been more efficient, cutting our testing part down by almost two-thirds to 75 percent.

Our automation tests are more robust than our manual tests. We found our test lab would grow over time because we didn't have a lot of discipline within the team for manual testing to have a master test which could be used repeatedly and revise as necessary. So, they were creating a new test for every specific little thing that they wanted to test. They were  setting up these manual tests where they had ten to 15 tests which tested the same thing, but not quite. Therefore, it became a bear to manage. Whereas, with automation, because it is more controlled, we have a core set of about 125 automation tests entered into our library. That's in our change control. Therefore, we know exactly what the state of our tests are. 

If there is a new business process or new wrinkle in a business process, we didn't have a defined process, so now we are updating automation tests. The quality of the data that we're getting out of test from automation is much higher than we received out of manual testing. If we know the automation suite is parsing, then the application is working properly. With automation, we have more confidence that if the test is parsing than the application under test is working correctly.

It is fairly straightforward. We have some deep expertise after using it for five years. We have some people who know it now very well. 

This type of marginalization of the code inside Worksoft Certify has been very valuable to us. The ability to capture documentation. We are a technology company and are regulated, so we have pretty stringent requirements. We use Certify to capture screenshots and evidence during testing. We can capture every screenshot during the business process including a document and hand it off to the auditors. It makes defending an audit very simple. We can, if they ask for evidence, produce a document that shows the business process of every step and the screenshots showing all the pertinent data, which has been pretty useful as well. This is the report feature with Certify. When you run a test, you can either have it generate documentation or run it in the background. Most of the time when we were running regression tests, we just run them in the background. 

We don't generate documentation, but we could turn Capture 2.0 on, where it creates a screenshot. As we're starting the test, it creates a screenshot of the application it is testing every step of the way and produces a word document or a pdf at the end that you can then hand off to auditors and show them the actual flow of the process that you're testing. However, we do not use this feature.

One feature that we have been asking for has been to treat tests as code and store the source code for tests in a configuration management tool. Right now, for version control of testing, it's all internally within the tool. If we have a test of a business process and want to revive that test, our methodology now is purely manual work. We go into the tool, create a copy of the existing test, and call the next one: v2. Now, we have two of them and the only way you can tell them apart is by its naming convention.

This is not an efficient way in terms of how modern applications do version control. If this was code, we could plug it into a tool like Git or GitHub to manage of our versioning and branching. The reason why we want to do this is that the application which we are testing branches. When we branch the code, we put a bunch of new functionality on the new version while our production version stays unchanged. Then, at the end, we merge the two together.

From an automation testing perspective, we have to run tests on both. Then, we have two current versions of our test. So, it's a bit hard to manage in the tool right now because you can only have this manual approach where we are tracking it via the name convention. Whereas, a modern way of doing it would be to have our application plug it into a version management tool, like GitHub, where we would store the code and could just pull in the version of the test that was applicable to the version of the software that we were testing.

This is something we have been asking for for a while now. I understand that it's in the pipeline, and it may be in their latest version (version 11). This is something that we will be looking into this quarter.

The challenge that we face everyday for test automation are more internal (people issues). We need change management and getting people to accept automation instead of the technical limitations of the tool. The tool does what we need it to do from an SAP testing perspective.

We have no stability issues. 

Maintenance-wise, we have one system administrator who is not full-time, since it has been pretty stable. We don't change much compared to other applications. This application is pretty hands-off.

We should be upgrading to the latest version in next couple of months.

At our peak usage, we have had seven people working on it, and have had no issues with that. Now, with our current work load, we only have three people. We only run our test suite. It was one of my goals on this project that we had the infrastructure setup, so we could always run our entire test suite overnight. As we built out our library, this meant expanding our infrastructure. Right now, we have 100 to 150 integration tests, and some of them can take ten to 20 minutes to run. A single instance of Certify can only run one at a time. Thus, we have had to think about how we set up our infrastructure in such a way that we can run the entire suite of 150 tests in six hours.

The way that we have done this is to split it up amongst servers. Therefore, we still have extra servers for execution. We have four servers now and run the tests in batches of about six queued up at a time. In this way, we can run our suite of 150 in parallel across four machines and get it done in about six hours. Right now, we do this manually. We do the manual breaking up and monitoring. I know Worksoft has some tools which automate this. This is something that is on our radar to look at as we grow. However, right now, we just manually manage the process.

We have three test developers using it. These are the people actually building tests. In terms of consumers of the test automation, we have probably 35 to 40 business analyst.

The technical support is pretty responsive. We haven't had many issues with it. When we were doing an investigation into doing web testing, we ran into some roadblocks. The team at Worksoft was very responsive. At the end of the day, it came back to technical limitations of a tool. I have been pretty impressed with how responsive the team. They were always able to answer our questions to the extent that the tool was able to do what we needed to do.

It was all manual. For convenience, we used Micro Focus ALM for tracking our manual tests. We still use that as our central hub for our test documentation. We weren't using any test automation tools in IT. Within the organization, we have R&D groups that develop software for various systems and medical devices. Those teams are running tests and code. They are in automation test suites, and I was part of one of those teams before joining IT. However, in IT, before we started using Certify, we weren't using any test automation.

Manual testing was costing us a huge amount of money. We did a double rollout of SAP. We split it over three deployments:

• With deployment one, it was just one division in North America. We had over a 120 people doing manual testing for a period of about sixteen weeks. Add up the cost of that. 
• As we moved into deployment two, we were going to have to test new functionality and also regression test what we'd already booked. If you took the amount of testing that we'd done in deployment one, even if we weren't going to redo all of that, we're going to have to do 50 percent of that. It was going to be a huge manual effort and a sunk cost. We'd put all that money into manual testing and wouldn't have an asset. It would be money that we are basically suspending with no reuseability.

It was a pretty easy decision to convince the team to move to automation because it would be an asset that we could reuse again. Over the last five years, we've shown that we've had a positive ROI on it. The initial upfront cost in terms of licenses, plus all the money that we spent developing tests, has proven it's worth. Now, we can do a regression test suite in ten days as opposed to sixteen weeks.

The setup was very straightforward. We did a proof of concept with Worksoft. They came in and had an engineer onsite. We set them up on a server and pointed them at our test SAP system. They built a couple of prototype tests for us. When it came to implementation, we had an existing prototype that we looked back on. I have a systems administrator on my team, and he was able to pick it up pretty quickly. 

The documentation was good. We did the install on our production system, copying over our prototype tests. We used that as our starting point for building out our library. We also sent out a couple of guys for training.

We were up and running with a functional system within a couple of weeks. The challenge, at that point, came down to training our business analysts on how to use the tool. This took longer than getting the system up and running, which was pretty straightforward.

We did the deployment ourselves. It took less than a week. Internally, we had one system administrator do the bulk of the work.

We ran the deployment on Windows Server. We have two machines: a database server and an application server. Our test developers can logon via Windows Remote Desktop to access those machines. They built all their tests out on that system. Architecture-wise, it is hosted all behind our firewall, but it is all server-based. No one is building tests on their local desktops. It's all server-based, and we can share some of our scripts amongst our team members.

My primary team is offshore. They are in India and Bangalore. Therefore, all of the test development is done there. However, we can access the central test library seamlessly, and  the test strategy for setting up and standing up servers and installing the software was pretty straightforward.

Our ROI is primarily a reduction in testing time. The testing, when we were doing it manually, was 30 to 40 percent of the project's cost. This was a $450 million USD deployment of SAP, and testing is 30 to 40 percent of that cost. We spent probably about a million and a half in test automation, but managed to reduce our testing times from weeks to days. There is a clear cut return.

If we write a new test that's 80 percent the same as an existing test, it is pretty straightforward to reuse the steps from existing tests for our new tests and build upon them. We found that there has been increasing ROI automation as we built up our library. When we write new tests now very seldom is a new test build from scratch. It is normally a variation of something that we already have, so we can turn those around pretty quickly within a couple of days to two weeks.

We ended up buying too many licenses. They were very good at selling it to us, and probably oversold it a little. We bought 45 licenses and have never used more than twenty. However, they gave us a pretty significant discount on the bigger license, so it made sense for us to buy enough that we wouldn't have to go back and ask for more. 

At that time, we had budget to do that. The licensing is pretty straightforward. We have considered using them to do robotic process automation and may still do that. Initially, we were worried that our license might preclude us from using the tool for something other than testing, but when we checked into that, there is no limitation. 

We could use Certify to do robotic process automation, which is basically running a process on your correction system instead of your test system. Therefore, we may do that in the future. 

We also looked at HPE UFT (the HPE automated testing tools) and SAP TAO (SAP's own internal test automation). The reason we pretty quickly went with Worksoft was primarily the responsiveness of the team. The evaluation happened between deployment one and deployment two.

When implementing SAP, we had IBM as our system integrator. We went to both SAP and HPE asked them to show us what they could do for test automation. We also looked around and found Certify as a third candidate. The response from the Worksoft team was far higher than the other two. IBM wasn't able to produce sufficient expertise to demonstrate the SAP test automation tool and same with HPE. I also didn't have a good response from them. We felt, " If this is the level of support that we were getting during the sales cycle, how will it be after the sale has occurred and we have to go to them for support?" Whereas, Worksoft was very responsive. They sent people onsite. They did a proof of concept using our system and data. There was a pretty clear cut night and day difference in teams and companies involved. I didn't get a chance to evaluate the technology of the SAP or HPE solution because their sales teams weren't responsive.

We have a dedicated team of what we call test developers who are specialists in this application. While I don't use the application myself, but they're pretty productive with it. We have a team using Certify for SAP Test Automation and a team using Selenium for web application development. The SAP test development is more efficient than the web test development. For a similar sized test development project where they have to test and develop five automated tests of a certain method, we can turn them around in SAP faster than we can turn them around in Selenium.

Now, it might be Selenium has a higher learning curve than Certify. Or, it is easier for test developers to get good at developing test units using Certify. Selenium is far more technical. Of the two tools that we use, Worksoft is more user-friendly than Selenium.

The technical instrumentation was pretty straightforward. The tool does what we need it to do. The primary challenges that we have had with test automation have been change management, getting the old, greater IT organization to accept automation as a substitute for manual testing. Culturally, within our organization, we put a lot of pressure on our business analysts to thoroughly test the application, and if they have never used automation before, there is a fear factor there saying, "I'm responsible. Then, I want to see it with my own two eyes." 

I recommend expanding, training, and coaching people that automation is just as good, if not better, than manual testing in terms of finding bugs and proving that the system is working correctly. It is far faster, and you will get a lot of your life back. That has been the biggest challenge for us: Telling that story and expanding the use of automation throughout our organization. Now, automation is pretty mainstream and accepted, but that was the biggest challenge for us. It certainly wasn't technical challenges.

We don't use Capture 2.0. We found it easier because we have a large pool of business analysts who are not certified users. Our process for capturing the business process which needs to be automated, therefore we use Zoom Recorders. It is like a WebEx tool. It has a screen sharing device and a record feature with audio. We find the audio is quite beneficial. When we capture the business process, we will have people record in Zoom, annotating with their voice (doing a voice over of what they're doing). Then, we handed it off to the test engineers to build up the automation. We look at Capture some time ago and felt it wasn't as efficient. Capture 2.0 is the newest version, and we haven't really looked at it in-depth. We will certainly reconsider it, but right now, we are not using Capture 2.0 to do business processing.

We use web UI testing to a smaller extent as part of the SAP business process. For a business process which incorporate Salesforce, a field service engineer might order a spare part. This is a post process that spans both Salesforce and SAP. For the first half of the processes, we use Certify. We did attempt to use an in-depth testing of web applications sometime ago. At that point, we felt there were some technical limitations. The project was to use Certify to do comprehensive testing of our Salesforce application. However, we found when we did a deep dive that there were some aspects of Salesforce and proprietary screens which Certify already struggle with. At that point, we decided to switch to Selenium which is the industry standard for web testing. Now, we do most of our tests on Salesforce in Selenium. While Certify has become a lot more capable with web testing since then and the newer versions are better at it, at the time we investigated it, we felt that Certify probably wasn't up to scratch as a web testing application.

Going forward, we will look at Certify again as a web testing application tool since it is more efficient than Selenium. We are finding that it's costing us more to develop a test for a web application than it does to develop a test for a SAP based application. We want to take a look at them again as a solution because it might help increase our efficiency as most our applications from this point forward will probably be web applications. So, there's a lot of work to do in that arena.

With our eBusiness and Salesforce suite, we are not even close to full test automation coverage. We still have a lot of work to do. So, it's worth us looking at Certify again. We're expanding into big data and big data analytics. There are a whole slew of terms around that with regard to testing. E.g., how do you verify that your data's accurate? We are just dipping our toes into it, as we haven't done any model testing yet. That is something that we have to look into. There are a lot of areas where we could use it.

In the last couple of years, we have become an established and accepted part of the SAP testing in the organization. We are a fairly conservative group. Now that we've done the SAP testing, we need to start looking at different horizons of mobile, big data, and web testing where we still have a lot of work to do in terms of building up our automation.

We are in the retail business, so our primary use case is automated regression testing on our supply chain. It helps us oversee our internal economy, including the warehouse, HR, and the stores. 

We prefer Worksoft over other platforms because it's a low-code solution

We can't get the process intelligence module to work properly. We can't get the impact comment that analyzes the incoming development code to run, either. We've also had bugs in the CTM and execution manager in the past year. It took technical support a long time to resolve this issue. We escalated it so that the vice president of the company was included as well. 

I have used Certify for a few years, but the company has been working with it for five years. 

I rate Worksoft support five out of 10. They have some general issues that they're they're not testing, but when we open a ticket, they give us a hot-patch fix. It seems like they're not testing their products enough. On the other hand, Worksoft support has been responsive to our requests. They've addressed the whole list of issues, so we have no more open tickets. Once they escalate something, they're efficient. 

Neutral

The vendor installed it for us, but there were still many issues. We've reported over a hundred bugs and defects. In that time, we could run stuff, but sometimes it has been standing still. 

Worksoft is reasonably priced. 

I rate Worksoft Certify six out of 10. Before implementing the solution, you need to think about how you will use it because it has so many modules. You should consider how you will use those and do a proof of concept. Worksoft hyped up the process intelligence and impact modules for two years, and they still aren't working for us. Before you buy it, make sure it works in your environment. In the future, I hope they have a more stable product with more internal testing and quality control. 

Our primary use case for Worksoft Certify is the automation of test cases. Performance-wise Worksoft Certify is very good. 

The most valuable feature of Worksoft Certify is that it has a codeless environment, so you do not have to be a programmer to use it. 

Worksoft Certify supports multiple interfaces and applications like SAP, Web, or Silverlight Java, and Mainframe. It is easily integrated. 

I have used the tool for a long time and much has improved over the years. Many new features have been added and the tool improves on a daily basis. The UI is much better now and it looks much nicer as well. 

Worksoft Certify's tech support's response time could be improved. 

I have been using Worksoft Certify for nine years. I use the product on a daily basis. 

Worksoft Certify is easy to scale. We have somewhere between 60 and 80 people using it at the moment. 

We do have technical support from Worksoft Certify. When we do run into issues, we first see if we can sort them out in-house. If we are not able to, we reach out to them and they look into the issue and sort it out. 

Positive

I previously used Selenium. Comparing these two products, I can say that you need much more IT knowledge to use Selenium. Worksoft Certify is quite easy to debug. 

The initial setup was easy. On a scale of one to five, with one being very complex and five being very easy, I would give the Worksoft Certify deployment process a five. It also does not need very much maintenance at all. 

We deployed Worksoft Certify in-house. The deployment did not take much time at all. It took just a few hours. 

Worksoft Certify is a bit costly. It is a good product so the cost is a bit high. But I'm not sure about how much we pay for it exactly. 

The advice I would give to someone looking to implement Worksoft Certify is this: contact the provider and request what you need to be installed. Once the solution is installed, you can open up the user manuals and start learning it. It is easy to learn and implement a project in Worksoft Certify.

On a scale from one to ten, with one being the worst and ten the best, I would rate this product a ten overall. 

The primary use is developing automation on SAP at a medical device company moving from Micro Focus UFT scripts.

We were able to increase the percentage of automation from between 40% and 50% to 80% within a year.

The most valuable feature is the ability to automate quickly and to maintain and update scripts.

We would like this to be able to be used outside of SAP applications, as it would be good for other types of products.

I have been using Worksoft Certify for three years.

We expanded licenses after a year.

Prior to this product, we used Micro Focus UFT.

We evaluated an SAP escalator from SAP in UFT.

My advice for anybody who is implementing this product is to get consulting and on-site training.

The reporting is good.

It is very user-friendly.

Certify integrates with other tools and it works very well with other machine testing applications.

If we can import process from other automation tools like UiPath, Selenium, or Automation Anywhere, then it would be more helpful.

I have been using Worksoft Certify for more than seven years.

This is a stable product and we used it on a daily basis.

This is a scalable solution.

We have created tickets in the past and the support is absolutely fine. Their responses are very fast.

We did not regularly use a similar solution prior to Worksoft Certify, although I have worked with others here and there.

The installation is simple, taking less than an hour to complete.

Our internal IT team performed the installation and there is no maintenance required.

I have been comparing other tools, such as Micro Focus UFT One, to see whether they are feasible for our purpose.

Overall, this is a very good tool and I recommend it.

I would rate this solution a ten out of ten.

We use it to do end-to-end testing for the business. After development has occurred and once we're into verifying that no regression has been broken, it's at that stage of testing that we deploy it.

We are a subsidiary of a larger company and we are focused on rolling it out, at the moment, to our larger company. With the tool's simplicity of use, where we are able to have a code review occurring, in that sense it will be useful in being able to roll it out to the greater company. We will be able to give it to the people who are experts in their areas, rather than trying to pass off test cases to one centralized location. It will be centralized automation and we'll just have one central COE.

Automation using the solution has saved testing time. I couldn't give you a number of hours or days because we're still in the beginning stages of trying to roll it out globally. We haven't been able to use the product and reuse automation. The whole point of automation is that the upfront cost to automate something is heavier and then, as you reuse it, it reduces the testing cycle. We're still investing in the earlier stages where perhaps we have spent equal parts right now, but we intend to see a reduction as we capture more and more.

Certify has also enabled us to find more defects. While I'm focused solely on automating and testing, so I don't have access to the defect count number, I know we have found defects, which tells me that we are finding defects that wouldn't have been found otherwise, or defects that wouldn't have been found as quickly.

• The dataset.
• The reusability.

It does allow for good reusability. When it's designed properly and utilized properly, we can put things in a way that allows for reusability, meaning a lot of reuse of VA01, if they're very similar flows, to keep it simple. And if we do have problems with a more complex flow, we'll make another version of VA01 that targets edge cases.

In terms of web UI testing, we've done very limited Fiori testing, but we have done Salesforce and a few others. Our experience is that when we get that stuff applied properly and working properly, it works very well. They're usually built well and if we do have problems with them we can get Worksoft to fix them. A lot of the times, if we're running on something that doesn't have an XF definition for it, by understanding how it's building objects, we are able to easily map objects fairly well and quickly.

The solution's ability to automate testing for packaged applications like SAP and Salesforce is related to when they do have that XF definition, but I do think it works very well. That's especially true for the SAP integration. That interface is very solid and objects are just about always discovered properly.

Since they updated the Capture feature to a more "Snagit" look and feel, it has become our primary tool. We've moved off of the old LiveTouch functionality. We will use it occasionally, but with Capture being built-in, it's easier for users to be trained on one tool. That tool has enough capability to be able to do both verifying the properties and recording the playback. It works well for us.

Looking at it as a product fully packaged, I would like to see more documentation or ease of use of the documentation. Sometimes documentation does exist but we have to search three different sites to find the proper way to do things or track down the technical document that explains certain fields. 

That, in turn, relates to the ease of use and how objects interact with each other. The application could lend itself to be simpler.

Another area that I would like to see improved is how the permissions are applied. If you're applying permissions groups to a user, one of the options is to delete the group entirely and lose the entire permission group, rather than just deleting the permission from the user, which seems a little silly. In my opinion, that whole module of permissions is very confusing and lends itself to common errors. We have to rebuild permissions occasionally.

The functionality is all there. I just think the way it's packaged can be confusing. We are successful and we can get things working the way they're intended to in Worksoft. It's just that sometimes finding how to do that, or where it is described, can be difficult.

I've been using Worksoft Certify for about the last year-and-a-half.

It's a pretty stable application. When it works, it works well, and it seems to work consistently. And when it doesn't work, it does not work — if that makes sense. When we see it functioning, we've got everything just right, it frequently seems to function solidly. And then, when we seem to have problems, it seems to not function at all, meaning tests will not run, or we cannot get a script to work in this or that particular way at all. But we've been able to work through all of our non-functioning issues through their support.

The solution will enable us to scale up our testing. With our focus being more on regression testing, it increases the testing of existing functionality first, and then we'll bring in that new functionality.

We are planning on rolling this out to more people, multiples of the number we have using it today. We think that it should be scalable but we haven't done it yet on that scale so we don't know for sure. But we do feel it will be scalable and that it will scale well.

Our extent of usage is pretty narrow at the moment. Approximately 10 people are using it right now and they are mainly automation engineers. There are a few directors using it to understand what the product is. People who we would consider to be "automation champions," who will help champion the product at our global headquarters, are being trained on it right now. They're not actually going to use the application, they're just going to understand it so they can help champion it and bring it on, full-scale, with user acceptance. 

Our main users in the future will be those information business analysts who know their respective products very well, the ones who are making the changes in targeted areas and who can easily reach out. They will be able to quickly test and record whatever they need to record for testing. We're looking at anywhere between 20 and 50 additional users within the next year, depending on how well user-acceptance goes, and expansion will continue from there.

I'll start with our positive experience. We always end up with some kind of resolution whenever we do submit something through support. 

There have been times though where their support has been very slow or difficult, where we end up with a level-one support for what feels like much longer than we should have a level-one for the issues we have. These are high-end issues that mean we can't function. That's been a frustration point for us. We've had to meet with Worksoft to talk about the support that we're getting.

As we start to build better in-house knowledge of some of the caveats of Worksoft though, that support has been needed less. That has made things a little better for us and that's why we focus heavily on training and having supporting documents on what we're doing.

I have used Winshuttle as well as DataLoad, which is an open-source and much more simplistic. Winshuttle is used more for something like an RPA function.

Certify has a much deeper bench in terms of what it can actually do. Winshuttle is only functional, to my knowledge, with SAP applications because it's built on the scripting portion of SAP. Its focus isn't for testing, so it's not a good tool for testing. But it is more simplistic in the sense that it looks like a spreadsheet and the result is provided in the last column of what the status bar gave you. It is really designed for one Pcode at a time, in my opinion. Whereas with Certify, you can run a larger-scale test or function or even a larger-scale RPA function, compared to what Winshuttle can support. The complexity involved in that is much harder. It's something of Catch-22, but Certify does enable you to do much more.

I can't speak to about the installation process, as we have a different person who manages installation. As far as setting up users goes, it's fairly simple within the application, once it's installed and functioning on the servers.

We started out with one model of being centralized and we're rotating to a decentralized model of sharing this out with more users and increasing usage. It's almost like we're in a second deployment of the product, and using more of the tools.

We're rolling it out to the specialists in each business area, on the information systems side. These are the people who are producing changes and who understand the changes and updates quite well. We'll have them write the scripts themselves, with our support as the center-of-excellence team. The idea is that they will be submitting the scripts that they've written back to us for code approval and then promotion to gold, to be able to be run regularly, as a script that's been validated. It should work well and be successful for them. We'll give them help with training, etc., in the Worksoft product itself. We're trying to focus on somebody becoming an application expert, for each application we're testing, and to be an application expert for the automation product, allowing them to function well enough within the Worksoft application.

The person who is responsible for installation is also responsible for maintenance of the solution. Like me, he is an automation engineer, but we have different focuses.

My advice would be to develop a very good training program to go with it. Also, understand how to build a good structure to allow for success and to limit exposure where people are editing things that they shouldn't be editing. You should also partner or work with other businesses that have used the solution successful. Build up industry contacts who can help you understand where they're going and where they're having problems, as well, with the model they're implementing.

The biggest lesson I have learned from using Certify is that you can design it to be way more complex than you need to, and you need to be very careful, when you're designing the solution, to design it in a very simplistic manner. It's almost like code in that it enables you to do things that are very complex, but you need to be very cognizant that you shouldn't always do the most complex flow, and that you shouldn't overly design logic out of any one script. They should be relatively simple.

Regarding ease of use, once you understand how to use it you can use it very effectively. But at times it's difficult to understand what the application is doing, what you are actually editing, within the application. So at times, when it comes to certain objects, you might not realize you're editing another object, in a way, unless you've used the application and understand how it actually builds together. It is simple once you know what you're doing, once you understand how all the objects work together, but leading up to that it can be more complex. We overcome that with training, reference documents, and a lot of training documents. We did an intro training with our team just yesterday. We're rolling out more globally, so we're training and trying to have a center of excellence team that can help out with these concepts. For example, they can help design better training to understand, "Hey, when you're editing here, you're doing this." We're trying to do more targeted training to the things we do with our standards inside of Worksoft.

As far as the Capture documentation goes, for us, it's almost too detailed. We've actually implemented a custom solution for documenting, because we need something that's simple, almost like what users would experience for test cases for manual testing. We also designed our own solution for that, in part, because we utilize a lot of Selenium-style code and we need to be able to record results that are occurring in that application. We'll call Selenium and Worksoft and we need to have a consolidated results report. We don't utilize, and, just to be clear, we've never purchased, BPP (Business Process Procedure) so I don't know any of that functionality. But with our unique set up, it did not make sense to utilize those reports. The reporting that is built into Worksoft is good for development cycles, developing scripts, but we don't use it for result-reporting, in the sense of whether the test passed or failed. We've narrowed it down into a custom application.

While it does allow for good reusability, even if best practices are followed, at times it's hard to identify if you have the same components or processes being built. That can be hard to recognize. For example, there will be duplicate login scripts. The application doesn't seem to lend itself to being easy to manage for duplication of processes. We are trying to put workflows in place on our team to help identify duplication and to reduce it. We do intend to use Analyze as a way to help catch duplicate workflows.

We are working towards use of the solution for RPA testing, but our primary charter is to industrialize our testing cycle, and then we can move into something like that.