StresStimulus Reviews

Following Microsoft’s Visual Studio Web & Performance testing tools being deprecated, StresStimulus had been adopted to continue performance testing work of an enterprise scale application with web-based user interfaces for customer interaction & back-office processing. 

StresStimulus is being used to validate the non-functional requirement commitments made to clients, ensuring the product satisfies the volumes of concurrent users specified in contractual documentation and helping to identify system performance bottlenecks that can then be investigated and better optimized by development teams.

StresStimulus is being used as an alternative to Microsoft’s Visual Studio Web & Performance testing tools used previously, as the Microsoft tools have been deprecated. StresStimulus has been beneficial as it is a product in active development with a responsive support team, whereas, in contrast, the Microsoft tools are no longer supported or developed. 

For small simple test cases and scenarios, StresStimulus offers similar functionality to the Visual Studio Web & Performance testing tools. However, for larger, more complex tasks, we’re finding it necessary to reduce the scope and scale of aspects of test scenarios that were previously possible with Visual Studio – either capturing fewer PerfMon metrics, reducing the monitoring sample rate, or reducing the concurrent usage profiles & test durations.

Technical support is great. StresStimulus’ support team have been excellent. An example (for which this review is being provided in the interests of reciprocity) was their response to our request for enhancements to the way tests were organized and executed. 

We required a way to start a collection of tests running which emulate behaviour related to contractual Non-Functional performance requirements of the system, then in addition to that, subsequently also ramp-up another collection of tests to stress the system and identify the point at which bottlenecks, errors & unacceptable levels of performance degradation occur. 

A Webex call with their technical staff was arranged immediately to discuss the issues & requirements. A solution was proposed to introduce a concept of ‘Tiers’ to enable sets of tests to be ramped up in a pre-defined order. The functionality was developed within the space of a week and delivered to us as a bespoke build prior to it being merged into the next version of StresStimulus made available to the public shortly thereafter.

Source control integration and versioning could be better. Currently, StresStimulus does not natively support any form of change control management.

Result analysis (snap-out tabs for side-by-side comparisons) need improvement. Multiple sets of analysis results can be opened in the user interface at the same time, and comparison reports can be generated to compare some aspects of pairs of test runs. However, the interface that displays the multiple sets of results only allows a single result set to be viewed at any one time & uses tabs to switch between the individual result sets. It would be useful to have the ability to view multiple tabs at once, either side by side in the interface or by snapping tabs out of the interface into their own windows to be viewed together with others.

Graphing (collect PerfMon data as part of the tests and then have the ability to create different graphs with that data after the test completes to help with analysis) could be better. Graphs and their contents have to be defined before tests are run. Often, as a result of running tests and viewing the results collected, it is necessary to have graphs showing/comparing different metrics collections than those specified before the run. Other tools collect metrics that are not tied to pre-defined graphs & allow the metrics to be used to construct graphs as required by subsequent analysis & investigation.

Nested tests/test case groups/test solution files (ability to build a hierarchy of test cases/test case groups/solution files) are required. Within a StresStimulus, tests are organized in just two levels, test cases, and test case groups. As is possible with other tools, it would be helpful to have a hierarchy with multiple levels of nested test cases/test case groups (or even the solution config files themselves). It would make it easier to manage multiple sets, combinations of tests, load profiles & test mixes if a high-level solution file (or test case group) could run a collection of lower-level ones, which in turn end up executing multiple different base-level test case groups and individual test cases.

I've used the solution for five months.

The solution is stable with frequent version updates. 

The most effective way we’ve found so far is to create and manage several sets of different load profiles (number of VUs, ramp-up, test length, etc) and test mixes (which tests are being exercised & how the VUs should be distributed over those tests) is through multiple separate .sslnconfig test solution files. The number of these .sslnconfig files & related folders quickly add up, making the management and maintenance of large numbers of test cases/scenarios/test suites a challenge.

Analyzing results from test runs is proving to be challenging. For example, test runs that take 3 hours and capture PerfMon metrics from 14 ‘System Under Test’ and three ‘Test Rig’ machines – we’ve seen it take two hours to load test result data for analysis after the test completes, as well as experiencing repeated timeouts, errors & analysis reports indicating the tests were aborted.

StresStimulus does not integrate with any source control systems natively – versioning has to be managed by hand with external source control tools making it hard to work as a team on a single performance test project.

StresStimulus currently doesn’t effectively support multiple users working on the duplicate solution files which are being managed using a source control system. .sslnconfig test solution files currently use absolute file paths to locate resources in the solution, these paths differ between multiple users working with .sslnconfig files. This causes merge conflicts & test errors every time different users commit/get-latest the versions to/from source control.

The product has excellent technical support where queries are responded to and actioned quickly. 

Positive

StresStimulus is being used as an alternative to Microsoft’s Visual Studio Web & Performance testing tools used previously, as the Microsoft tools have been deprecated.

The initial setup was simple.

We did an in-house implementation.

Pricing plans are dependent on number of licences and virtual users required for testing. 

We evaluated Loadrunner and Apache JMeter.

We use Stress Stimulus to run performance tests and scalability tests both for the web and APIs. The tests are run on local PCs, on-prem VMs, and cloud VMs. Our application is mainly web-based and changes very often, so we need a solution where recording and running tests need to happen in a short period of time. For this, StresStimulus perfectly fits in. 

Since our tests are fast-paced, getting stuck with problems needs attention immediately. The support team does an excellent job of responding within a day.

We have small-scale and large-scale tests running frequently. Thanks to the flexibility of StresStimulus, we were able to get the license customized to fit exactly our needs. 

Most of the time, the graphs coming out of the tool are very well done and little work is required on our side to plot any new graphs. 

Installation and upgrading on VMs are almost seamless, hence not having to rely on infrastructure teams for installation on new machines or upgrades, enabling us to move much faster.

As with any performance testing tool, correlation is something that takes time and needs to be done carefully. For the most part, StresStimulus does really well with the autocorrelation of all the basic values. However, sometimes it overdoes it, and we have to be careful to remove those.

The solution offers a simple UI. Compared to other tools, there is not a whole lot to StresStimulus UI. This makes it easy to use, be it while scripting, executing, or having to train someone new to the team.

The ability to run the same script across different domains in just a single step is great.

StresStimulus simulates browser HTTP calls. I would love to see where it can also simulate a real browser to measure visually complete time. For example, to be something similar to TruClient.

We run tests on SQL statements and stored procedures directly, even before integrating with UI. I wish there were a way to connect to my application DB and run SQL queries using a JDBC connection.

In a world where the applications are changing every day, I hope StresStimulus comes up with an option to compare HTTP requests before and after deployment and tell me what requests have changed. This would help avoid rescripting every time, and this would be a huge deal.

I've been using the solution for 3.5 years.

We have always used StresStimulus though we have explored other tools.

They are very transparent in terms of price. Installation, as mentioned previously, is very easy. The testers themselves could do it.

We considered Micro Focus Load Runner. It's just too pricy. JMeter just seemed very raw.

We use StresStimulus for all our performance test requirements which include performance regression tests, performance test automation, reporting of performance results, and performance testing new features. Performance test environments include dedicated on-prem test labs as well as testing against cloud instances or hybrid apps.

We run StresStimulus in our CI/CD pipelines on all our release branches which cover changes, updates, performance improvements, and new features with automated release triggers.

Designed to be a modern testing tool, StresStimulus made it easy for us to upgrade from an older deprecated testing tool and adapt to Agile DevOps testing principles by shifting left with performance testing. 

The ability to automatically create and send out test result reports increased test coverage momentum. Detailed reporting and dashboarding increased results visibility and tracking.

Compatibility with other testing script types means we can use existing test collateral in the form of Postman and Selenium scripts as performance test scenarios, increasing code coverage with less rework required.

The product is a very user-friendly and easy-to-use tool. There was no steep learning curve to get going with performance tests. It has very accurate automatic dynamic parameter detection and correlation. There's a wide selection of user browser types and network emulation.

The ability to import and detect dynamic parameters missed in Visual Studio web test scripts makes it easy to increase web test accuracy.

Automating the test results reports offers less human intervention so more attention can be given to the actual results itself than manually creating results.

Automatic test result comparisons against multiple tests make it very easy to compare base test results to post-change results.

Automatic emailing of test results to the relevant role players keeps all interested parties in the loop.

Integrates with chaos engineering tools like Gremlin.

Can use Postman API tests to run performance tests.

Additional features like the recent ability to import and run Postman and Selenium test scripts as performance tests have already been introduced with every new major release. 

StresStimulus has a dedicated blog where users of the tool can log new feature requests or enhancements to existing features. The goal of StresStimulus is to ease the implementation of performance test automation so expect more features around expanding compatibility with CI/CD tools and modern Web technologies. 

It's working on a better solution to automatically manipulate user licenses in automated pipelines.

I've used the solution for more than two years.

The solution is stable with good after-sales support.

The solution is very scalable. Load agents can be deployed via on-prem, Azure, or AWS VMs or a combination of on-prem and cloud agents.

After-sales customer support is excellent as is the user blog for learning and logging new feature requests. 

Premium Support offers training at no extra cost.

Positive

We used Visual Studio Enterprise load testing features which have been deprecated. Therefore, we had to find a new modern load testing tool going forward.

The initial setup is a simple run-alone install that does everything. 

Testing out-of-the-box is a simple matter of importing existing scripts on the controller or creating new scripts by recording the user actions. Using test agents is as easy as running the installer on the load agent and pointing the controller to the test agent.

The in-house performance testing team did the evaluation and implementation.

The accurate web test scrips are dependable and repeatable. There is better code coverage. We're shifting left with performance test automation. The tool is here to stay as a long-term investment.

The setup cost is very competitive and easy. Pricing/licensing options are very flexible to adapt to test requirements/needs.

Yes, we evaluated Neoload, Neotys, Webload, JMeter, and a few other offerings.

The criteria we looked for was a solution that provided ease of use, compatibility with existing test scripts, support, and pricing structure.

If you are in the market for a modern web load testing tool that is easy to setup and use, competitively priced, offers good support and is frequently updated with new features you need to give StresStimulus a long hard look.