Runscope [EOL] Reviews

We run all our production-level test cases on it. We use it to run against APIs to make sure those API endpoints are still there, that nothing has happened to them, that they haven't stopped running. This is a way of checking every five minutes. We use it as a monitoring tool.

It helps make sure that the program running the production server hasn't crashed without anybody knowing. Runscope makes sure that it runs properly and, if it does crash, something happens. It sends an email. Then, with Slack, we can notify somebody that something is not right.

It uses JavaScript which is easy.

I've noticed, once or twice, when there were some updates pushed by Runscope, at times it didn't refresh the pages. I was running it, there were multiple tests running at the same time, and it didn't update the webpage I was looking at. So I had to refresh it manually. Occasionally, it still happens.

Another thing I have found is that the built-in editor for the scripts can be improved a lot. Instead of making such a basic editor, maybe include a little more so it is easier to look at things and debug.

Finally, other than the Runscope documentation, I haven't seen too much elsewhere. I haven't seen a user community.

In general, it's pretty stable. Once or twice we have had some issues but that's it.

It is scalable, within the scope of what we're using it for.

I have not had an opportunity to contact technical support yet.

Although I wasn't with the company when they chose Runscope, one of the main things I was told was that they chose this product is because it's cloud-based. The company has different data centers. You can run it off the cloud and against different data centers. That's one of the plus points for most people for this product, as opposed to some other solutions.

I had a local setup done. It was pretty easy and straightforward.

I set it up in the development environment so that certain tests run there. I also did quite a bit of the production environment setup. That was more like creating an environment, not installing anything, such as a Runscope agent.

It was pretty straightforward. All the documentation is there. You just can't search the documentation.

It's a pretty good product, as long as the price is okay for you.

We use Runscope to test our API endpoints for our product, both on-demand and via a schedule.

Not only does Runscope allow us to keep our tests organized and available to everyone on our team, but it also alerts us when tests fail, via integration with Slack, so that we can address bugs that have been introduced.

• We love the fact that we can have our API tests run on a schedule as often as we need.
• We also take advantage of being able to set up different environment settings so that we can use the same test in our production, integration, and QA environments, easily.
• The ability to string together a number of API tests for a test suite is very important to us as well.

Navigation can be a little tricky when changing run environments and switching between test run results and editing the tests. It would be nice if you could switch the environment for all tests in a Bucket to run, rather than having to change each test environment.

I would also like to see easier integration and display of test results so that the data can be shared via a dashboard within our company. (Some integration is possible, provided you use one of the supported platforms).

We have rarely seen any issues and have never seen an outage with Runscope's servers. Just make sure the agent is run on the proper hardware.

We are not limited in the number of tests that we run, nor the frequency, provided that we stay within our contracted rate. It is easy to manage that information.

We have had good success with tech support, especially when we were getting started. They are knowledgable and will help you all the way through your process until you understand the solution to your problem.

We have used Postman quite a bit, but Runscope makes it easier to group tests together in a logical way. In addition, being able to run tests on a schedule and make them available to everyone on a team is great.

Easy setup, easy to make your first tests. There is a bit of a learning curve when you are trying to decide how to group tests and use variables, but nothing that their tech team isn't able to help you through.

Pricing is very reasonable and is easy to pay when you are supporting a team of engineers.

We did not evaluate other options, other than continuing to use Postman for our API testing.

Tests can be brittle, so make sure you include checks to validate data and variables that you are using.

We use it for doing smoke tests in production and testing in QA and in team environments.

It speeds up production deployment.

Pre-request Scripts and Post-response Scripts are very handy. Instead of adding a bulk of assertions to those associated test cases one by one, I can easily just write scripts, then copy and paste them into my test cases. This really saves me a lot of time on debugging and revising.

The documentation of Runscope needs to be improved. For instance, there is very much a lack of examples in the documentation. It is a bit difficult for beginners and intermediate users to get started quickly or to go deeper when creating test cases.

All my test cases failed when Runscope did a security patch a few months ago. That was the first time I found the stability wasn't great. Otherwise, it has been doing fine.

I have experienced no issues with scalability.

I have not had to contact tech support yet.

We had a previous solution, but Runscope has a better user interface and is easy to set up, so that is why we switched.

The initial setup is pretty straightforward.

I am not aware of the pricing and licensing as these topics are not part of my job.

Ask for more examples in the documentation.

We use it for API regression tests and API production monitoring.

It has allowed us to go from no API tests to scheduled, automated tests integrated with PagerDuty in a few months. I was able to do this without having to hire a QA Developer. Existing manual QA testers were able to learn simple JavaScript and Runscope UI. Regression tests were executed in minutes instead of days.

It is a cloud-based environment for building and executing tests, which allows me to focus on building, running, and reviewing test results. I do not have to waste time managing development environments on local machines. Being cloud-based, I can check the status on the test results from anywhere on any device. I am not constrained by a physical machine where the tests are running.

File upload is a big part of the products that we test. The lack of file upload in Runscope requires us to still use UI (Selenium) automated tests for these scenarios. Having Runscope support file uploads is the biggest improvement that we would benefit from. I would be willing to pay more if file upload was supported.

Stability has been good. It is not an issue for us.

We have experienced issues with scalability so far. A few months back, I needed to do some load testing which was not what Runscope was designed to do. I was able to hack together a solution in Runscope which provided the data that we needed. This is a complement to its flexibility.

I have only contacted tech support a couple of times. The experiences were positive.

Before Runscope, I only used UI (Selenium) testing solutions.

The complexity of the setup was due more to the complexity of the API calls in the system that I was testing, not the complexity of Runscope. To make it easier to onboard new customers, CA may want to have example tests with a couple API calls demonstrating assertions and variables.

The pricing plans that we have used so far have been selected purely on volume needed per month.

We evaluated Assertible and Postman.

• Prioritize tests.
• Use environment variables.
• Budget time to experiment with features before going full speed on test creation.

Initial method testing and regression testing.

We have been able to set up regression tests that cover everything we need to monitor. We then set them to run automatically and forget about them. We are notified whether they pass/fail, and have been able to catch issues quickly when changes were made to our methods.

• Being able to add tests as sub-tests in a parent test for regression testing
• Notifications for tests that pass/fail
• Being able to customize expected results with assertions
• Being able to set tests to run daily
• The intuitive user interface that makes it easy to set up everything without a programming background

We would like to be able to select the actual time a test will run automatically. It appears that you can set up the automatic schedule, but the test runs based on the last time it ran and not a time you select. Since we want these tests to run in the middle of the night, it would be nice to be able to select that.

Also, it would be nice to be able to generate API documentation.

No issues with stability.

No issues with scalability.

Technical support is fantastic.

We used Postman. Runscope makes it much easier for those without a programming background to accomplish API testing.

I found setup to be pretty straightforward and anything that gave me trouble was solved by looking at the Runscope documentation.

Pricing is fair.

It's really easy to use and the documentation is extremely helpful if you get stuck on something. The product is flexible and allows you to set up variables and assertions in order to customize tests for your needs.

API monitoring for IoT data pipeline in the cloud.

This solution has improved our responsiveness to issues in both production and non-production environments.

Through integration with Slack, our development and DevOps teams are quickly aware of issues related to new changes deployed to non-production environments, which tightens our feedback loop during testing. Through integration with PagerDuty, our DevOps resources are able to reduce the adverse impact of a given production issue.

Runscope has also helped to expose subtle network latency or routing issues with components managed by our cloud provider.

API monitoring where specific tests are designed to emulate production data-flows through our pipelines. Tests are per-step configurable with script capabilities and support for token-based authentication using variables. Configurable test frequency and integration with both Slack and PagerDuty for alerting our DevOps team are also important. General use of the dashboard by multiple team members is also valuable.

1. Ability to configure Runscope behavior when a given test has its schedule modified. That is, in a non-production environment, we often need to pause a test (either by removing it from the schedule or adjusting its frequency) and this results in the automatic kick-off of this test in all other environments. We would like the ability to simply pause a test for a specific shared environment without causing additional tests to be started.
2. We definitely make use of historical test results, especially when tracking more difficult issues such as timeouts. Per our test frequency (15 minutes) we often need to go back further than the last 100 test runs. We are aware that results further back in time are available through the Runscope API, however, it would be great if the dashboard allowed easy access to a larger amount of historical data.

There have been some Runscope outages in the last year. Though not many, Runscope availability is of critical importance in our production context. This was the primary reason for the "recommend this solution" score of eight rather than nine or 10.

No issues with scalability.

Tech support is responsive through website chat (this is the only communication channel we have used).

Setup was fairly straightforward.

Having the different plans organized by team size, buckets, and requests per billing period works for our needs. Larger companies may need to take a close look at these elements.

Some alternatives were evaluated prior to choosing Runscope, however, I do not have access to that information at this time.

Create a DevOps team story to evaluate Runscope, which includes setup for all tests/buckets which will be used in your production environment. Exercise all use cases and outcomes, including integration with your desired tools (e.g. Slack, PagerDuty, etc.), and tune the available configurations as needed.

We are using it for API end-to-end testing.

It helps us verify that production deployments do not break the features our customers use.

We have the ability to populate environment variables using scripts. This allows us to keep the configuration mostly dynamic and avoid a lot of manual updates when something changes. 

Schedules are a great way to have repetitive checks against the environment.

We would like to have an improved ability to share a set of tests across different environments (e.g., dev, stage, and prod). The shared environments feature is great in practice, but in reality, if you have two or three of them using the same set of tests, it gets complicated to figure out which environment is failing. 

Overall, the user interface (UI) requires a lot of clicking around. 

Waiting for asynchronous results requires a lot of "Pause" steps, which is quite difficult to get right. It would be great to have some type of "Waiting for Results" step, which would include success/failure conditions, then eventual timeout for retries.

There have been no stability issues.

There have been no scalability issues.

We have found working with the technical support to be great.

We did not previously use another solution.

The initial setup was straightforward, but it included a lot of repetitive manual work.

Not applicable.

We evaluated AppNeta, New Relic Synthetics, and custom implementations.

Create a script to populate tests from a config file. Store it in a source control and load/update it through the API. Doing things through the UI wastes a lot of time.

Our primary use is for REST API automation.

Runscope allows our QA team to spend less time on our API smoke test during releases.

Environment initial variables: These allow our team to run tests against multiple environments on the fly. 

We utilize modular tests based on environment variables. These have saved QA a lot of time when setting up our back-end automation.

One area that I seem to forget adjusting when creating new tests is setting the default environment within test settings. If you forget to set the default environment during test creation, you have to go through every test and make sure that it is set to the environment that you prefer if using shared environments. If there was an option to force all tests within a bucket into a specific environment with one click, this would save a lot of time.

Stability has been fantastic.

Not so far.

Technical support has been amazing. I have reached out with chat many times, and I have received help immediately every time.

We did not previously use another solution.

The initial setup was straightforward. Now that I have more experience with REST API automation, anything that I ran into during setup was due to my lack of knowledge at the time.

Runscope pricing is unbeatable.

I did not evaluate another solution. I had a lot of recommendations that Runscope was the tool to use.

Runscope is the best option for REST API automation.

We use it to monitor the APIs. We always have them scheduled there.

As we have them scheduled, we use the notifications. We know the exact moment it is breaking something that might impact the customer system, that might result in a feature not working. We know "in advance" and we can fix it before the customer notices. It helps us to be more proactive.

We have it all monitored. Even when it's a problem on our side, the server is down, or there is a lack of memory, the first thing to break is the API, because the calls can't be processed. So we know exactly at that moment because we receive notifications, and we can take action. It doesn't only notify us of the type problem, but it also links it to our servers.

The schedule is definitely the most valuable. Also, being able to check the JSON attribute values is helpful. We always check the attributes. Finally, the notifications are helpful as well.

One thing that can be improved is the logging. Sometimes, when we have an error, we don't know how to find more information about the problem that has happened. Sometimes, it just returns something like "that gateway." More information would be a way to improve it.

Also, currently we can only schedule to the nearest five minutes, one hour, two hours, etc. If we could set up the exact time that we want it to run, that would be good because sometimes we have one service that crosses another service. It's rare that this happens, but sometimes it does happen because the service loads are loading at the same time. What our tests were doing is creation, addition, and deletion. Before we deleted, this one service crossed the results of another one. And then we deleted the thing and broke another service. It was just a coincidence of the same millisecond, so one thing that could solve this would be if we could schedule the exact time that we want, or to schedule the order of service execution. That would be a good improvement.

No issues with stability.

No issues with scalability. It's fine.

Tech support has been pretty good. We had a problem. We accidentally deleted a bucket. We were trying to create a script and deleted a bucket. I needed to ask support to use the backup to get things up. Maybe a good feature would be to show this backup inside the tool, so end-users can just do backups by themselves.

We did use Postman but not for scheduling; just to do a quick check, we ran it from there. I don't know too many tools in the market, so it's hard to make a comparison. I think Postman does notifications and schedules stuff, but I don't know the price, so I don't know if it would be similar or totally different.

The setup was very nice. It was fine.

I think it's a good price. Once I asked my boss to approve it, he approved it immediately.

I would rate Runscope an eight out of 10 because it's still missing flexible scheduling, the exact time or the ability to sort the services the way we want them to run.

Our primary use case for this solution is API tests.

Since we are working with Runscope, we have a tool which is shareable between all the team members in a uniform standard. Now, each of us can easily run all our tests in a couple of minutes before deployment and all the team can see the results. We can receive an email and slack notifications if tests are failing/passing.

• Ability to schedule and compare the test suite results over time periods. 
• It has a relatively simple user experience. 
• Ability to copy tests suite between buckets. 
• Ability to add JS libraries. 
• Email and slack notifications. 
• Ability to run a large amount of tests simultaneously. 
• Revision option (to track changes). 
• Shared environment for tests.

• Ability to create code which runs before and after API tests. 
• The area of "initial script" is relatively small and incomplete related to code editors. 
• There are no regular features and validations for editor. 
• If triggering test bucket via trigger API, there is no option to see if the tests which were triggered passed or failed.

During the using period, we experienced several downtimes.

There are not many options for scalability.

The support is very good. They answering quickly and effectively.

We were using editors to create scripts and run them locally. This was unstable, not shareable, or a uniform solution. We were not able to run big amount of tests simultaneously and receive notifications.

The initial setup was pretty straightforward. The work is in the cloud, you have to set integration options and libraries, if needed, then start to work.

The price for the product is very reasonable for a team of more than four to five members.

We did evaluate other solutions before choosing Runscope.

It is a relatively reliable tool for running big amounts of API tests, which you can schedule to run periodically, as the product is very good.