Tricentis NeoLoad Reviews

Less manual scripting and ease of setting up monitoring are very valuable features of the tool.

Loadrunner has extensive list of analysis features available whereas NeoLoad had limited when I worked.

It helped us deliver the solution for various customers within budget and efficiently.

Analysis can be further improved. I have been using LoadRunner analysis and it has been a very powerful tool.

I have used it for six months.

I have not encountered any deployment issues.

I have not encountered any stability issues.

I have not encountered any scalability issues.

Technical support's turnaround time for issue response is too good.

I have worked on various different performance testing tools, and I found NeoLoad very much within budget without compromising the quality of the product.

Initial setup was straightforward.

A vendor team implemented it.

More protocols can be added, and I would love to see if any profiler is included in it.

In my opinion, correlation of dynamic data is the most important advantage of this tool. Normally, it is difficult to record performance scripts, but with this tool it is much easier.

Since this tool is lowering the risk of monthly releases to live environment, it reduces the stress on people. In addition, performance problems that we could not solve for a long time have been solved by the tool with Dynatrace integration.

It would be good to make some updates on the reporting side. For example, correlating the performance of server, throughput and hit count in a specific time interval would be a nice functionality.

We use the tool full-time on our test environment and we have been releasing it to our live environment for 2 years.

Support side of the tool is very strong. We have seen fast feedback and answers to other questions we faced in the past about licensing and other general problems.

I have used many performance testing tools. This tool goes one step further with easy integration with code analysis tools, server monitoring, the success of correlation studio and the advantages on the mobile side.

Licence cost is very attractive compared to other vendor tools and also there are many license alternatives. Licence costs are good enough compared to the benefit of the tool to the company.

The ease of recording scripts and the correlation process, the force-on monitoring, success on the mobile side, and easy integration with code analysis tools are the reasons I have rated this product so high.

Ease of use. Technology support, and robust features.

We are able to use NeoLoad to quickly expand performance testing to all critical business applications and address some key performance issues.

Team Server - Neotys Team Server - This currently stores project data on the local drive. I suggest having Neotys Team Server Data stored on an actual server.

Since 2014.

No issues. Simple deployment process.

None.

Excellent.

Excellent.

Yes, HP LoadRunner. We switched due various reasons including cost, ease of use and robust features.

Yes, the initial setup process is really straightforward.

In house.

The ability to quickly record and correlate a script very quickly with a lower learning curve. A complex LoadRunner script could take days to create, but the same script in NeoLoad could be created within hours. We had contractors coming in, and within months they were able to script and start contributing to the team very quickly due to the simplicity of this software.

NeoLoad has the ability to compare a test report to a previous run. We can easily compare to see if this new code base is performing better or on par with the previous code base.

We were able to create more scripts using NeoLoad within a short time frame. We increased the number of applications we support/test due do the quick scripting time. Now we have more time to do analysis and engineering work which is very important to us.

The ability to show transactions per second during the test run. Currently, we have to eyeball the TPS using the graph. NeoLoad 5.2 will have TPS reading after the test is completed but not during the test run.

16 months

None.

None.

None.

9/10

We used LoadRunner before we switched over to NeoLoad. LoadRunner is a powerful tool, but the learning curve is very high. It would not be feasible for our team because we have a lot of newcomers and a lot of applications to support. NeoLoad costs a fraction of what Load Runner costs.

Setup is very straightforward. Neotys also provides guidance and support, setting up the team servers and licenses.

We did the installation in-house, with vendor support when needed.

Identify what kind of testing your company needs (mobile, database, web, and so on) and find out if NeoLoad supports it.

All-in-one product (design, runtime, results), easy scripting and variable correlation, and load generators deployed all around the world to simulate global traffic.

It is easier to do load tests using NeoLoad.

The only improvement that I wanted is support for load tests in agile mode. Neoload 5.2 makes this possible by updating the old versions of scripts with data from the new HTTP requests.

I have 4 years of experience.

The only issue that I had was with network virtualization on load generators installed on Windows Server 2008 R2. The link is sometimes cut between the controller and the load generators.

10/10. Support is available, they handle issues very quickly, and they are very skillful.

I used HPE LoadRunner. I think that NeoLoad is more user-friendly.

NeoLoad is easy to deploy using the install wizard.

Well, first I got trained by a vendor team. After that, I implemented more projects using NeoLoad.

My advice is that there should be a minimum of 4GB of memory on the servers on which you are installing NeoLoad.

Neotys provides a free evaluation licence to use NeoLoad with up to 50 virtual users and all protocols included.

The product licence remains cheaper than HP LoadRunner, for example.

The way it interacts with applications and how the dynamic values are handled.

LoadRunner was not able to record a .NET application, we had many issues with correlations, requests were not capturing properly, it was time consuming and there were many other issues. When we tried the same application in NeoLoad, it worked like a charm without any hassle.

It needs improvements when handling binary values.

I am using it for more than 2 years.

None

None

None

First I tried HP LoadRunner, but it didn't work.

It was straightforward and easy to setup.

Vendor team

The market always pays for the one who comes up with something new and that is easy to customize.

Keep enhancing it and adding new features.

Wasted time in low-end work (scripting) is greatly reduced with NeoLoad. It enables my team to record and modify test scripts very quickly. This reduces the effort for testing and enables the team to do more high-value work like analysis of results. We use external data analytics tools (Tableau) for result analysis. The export raw data feature in NeoLoad provides the extension with Tableau.

Performance testing is done by testing frameworks in which NeoLoad is the driver of load. The ability to extend the tool is crucial for organisations. Our test framework is composed out of NeoLoad as a centralised component surrounded by Tableau for analytics and AppDynamics for deep-dive diagnostics. This load test framework enables the team to more quickly test, find and resolve issues and system bottlenecks.

The weakest feature of most of the load test tools is result analysis. It is important we can customise graphs to visualise results in such a way that stakeholders can understand the outcome of testing. To be able to visualise results better, we extended NeoLoad with Tableau Software.

With Tableau, you can centralize performance related data in a database. Data can come from any tool, from log files or RUM tools and system utilities. Tableau provides you with the ability to connect to any data source, analyse patters and find bottlenecks more easily.  NeoLoad has made huge improvements over the last few years and finally has the ability to view the raw data (and not only aggregated results). This is fundamental for any type of performance engineering.  Aggregated results hide bottlenecks. 

Customer support is very good.

The price of the tool is competitive.

The features introduced in the recently released new version of NeoLoad fit really well into the DevOps way of working, further reducing the time for rescripting. This enables rapid delivery and lean ways of working.

The two standout features for us are:

• Team Server Collaboration - as is the case with many MNCs like ours, teams/team members are spread geographically at various locations. The Team Server module is a boon for such teams as multiple users can work on the same project simultaneously and check-in the changes. It also helps manage and allocate virtual users (of shared license) for execution of multiple projects at same time.
• Results & Analysis - NeoLoad provides comprehensive results for any load test. The results can be drilled-down to minute levels (from scenario level all the way to individual request level) of any transaction which makes root cause analysis very effective. NeoLoad can also consolidate results from different sources and the results are graphical (one can build custom graphs for any transaction) and colourful for easier interpretation.

Other advantages are

1. GUI driven recording makes it easier to record and prepare test scenarios without need for extensive coding knowledge thereby enabling tool users to ramp up faster
2. Supports a ton of protocols plus additional plugins such as Jenkins, AppDynamics, etc.
3. Custom scripts and frameworks can be built
4. Recording and simulating mobile applications is simple yet realistic

NeoLoad is a robust and versatile tool. With plugins for CI tools like Jenkins, NeoLoad is configured to automatically trigger small load tests on modules each time there is a build update. This has enabled us to identify performance issues for our clients at a very early stage. It has undoubtedly an impact during larger scale enterprise performance tests when all the modules are integrated and tested as cost to fix performance issues increases with later stages of deployment.

We have raised a few support tickets for certain feature sets/improvement areas.
These include permanent fixes for a few stability issues discussed in the following question and implementation of more robust solutions for media streaming protocols, TCP/IP and websocket protocols.

Also, there are a few changes in the latest version, 5.2, which have brought in many improvements but have removed certain features. For example, v5.2 has done away with the 90th percentile reporting for response times, which was present in earlier versions. While this is a small deviation, it still is important for organizations using it.

The tool can hang and take a long time to open projects if the size is large and if many graphs and result templates are stored. Certain protocols although supported by NeoLoad need to have custom solutions built (e.g. web socket connections) while solutions for certain other protocols such as media streaming protocols like HDS, HLS, etc. are still in the nascent stage.

We have used NeoLoad since late 2009.

There is no specific maximum project size defined (actual NeoLoad project size). This size is dependent on number of user journeys in the project, number of scenarios defined, number of graph templates stored, and number of results stored and so on. The bigger the project, the slower it opens in the tool (as expected) but many times can be excruciatingly slow. The other drawback is that the graphs and results may not get saved correctly. With the latest versions of NeoLoad, this issue has been significantly fixed but it can still occur at an undesirable moment as the optimum size of a project can only be determined by trial-and-error. Our experience has taught us to keep multiple backups of same project under two categories for such exigencies – the first category would contain backup of the project with previous results and graphs while the second one would have only the project (VUs and scenarios) but with previous execution results and graph templates removed.

The other issue we faced was with respect to results generation. If we need to abort a performance test that is in progress, all one has to do is click the stop button. However, if the stop button is clicked twice inadvertently or clicked twice because it still showed “consolidating results” message for eons after the first click, the results ultimately generated would get corrupted. Either the results would not show for all virtual users executed or would not get generated at all. The last time we checked, Neotys did not have a concrete solution to this problem even after analysing the execution logs and troubleshooting it. Their “solution” was to check our stop policy for the scenarios executed (to make sure all virtual users reached their ‘end’ state to generate results) and also to check if any antivirus or firewall was impeding result generation (an antivirus or firewall can prevent NeoLoad from writing results to your folder). From our side, our stop policies were fine and there was no antivirus/firewall to block NeoLoad so the solutions were not that useful. As a workaround, we would click just once to abort an execution run and wait for the results to be generated even if it took a while.

The customer service and technical support is excellent if you are a paid customer. The turnaround time and SLA for a priority one - Urgent - issue/ticket is almost immediate. For less priority tickets it is quick (usually a couple of hours) and the representatives even help build a custom solution for your project if need be. However, if you are using the trial version, you would have to depend on NeoLoad forums for answers. On the flip side, Neotys documentation available on their side is detailed and exhaustive which should answer most questions when you are evaluating the product.

We have used a host of other solutions such as LoadRunner, JMeter, WebLOAD, SOASTA, etc. While the choice of the solution depends on various factors such as open source vs commercial tool, protocols used, number of virtual users etc. NeoLoad comes up trumps on most factors which were needed for our clients namely:

1. Cost
2. Ease of deployment and scalability to test for high user loads (in the range of 100,000s users for large enterprise tests)
3. Ability to monitor performance at various levels of architecture
4. Graphical representation of results
5. Detailed comparison can be seen in the sheet attached

Initial setup is a breeze. The installation is a complete package without having to switch between standalone modules for recording scenarios, execution and analysis which is not the case in say, LoadRunner or WebLOAD which has different modules for each (like VuGen, Controller & Analysis).

Neotys also offers their own cloud infrastructure to leverage upon for tests with larger user loads which can be configured to each project needs. However, if you are setting up your own cloud infrastructure on say, Amazon AWS, you would have to be equipped with enough knowledge to set up the images (AMIs) and configure the necessary IPs & locations for the test.

When we started out in 2009-2010, we did use the vendor team to guide us through and setup the necessary infrastructure. Over the years, we have built a strong in-house team and have setup our own infrastructure and cloud instances.

Depending on the technical expertise of your team, you can choose to implement in-house or avail the support of the vendor team. While the basic installation is straightforward, the vendor team could help you choose the right hardware systems/configuration for localized load generation or help you configure Neotys cloud for IP simulation or handling larger user loads.

Neotys offers various licensing plans to suit every business. The free version comes with 50 users with all protocols included for small load tests. The paid licensing plan depends on number of virtual users needed (to simulate the load), duration, the protocols under test, number of concurrent applications being tested, cloud usage etc. For example, the licence can be availed even for a day, a month, or a year depending on the need. In that sense, the licence cost can range from $1500 to >$20000. A general rule of thumb is the longer the licence period and the greater number of virtual users results in a lower price over the long term.

Leverage the free version of performance tools to its maximum capability to evaluate and cross check against the performance testing needs for your business to make an informed decision on buying the tool. The pros outweigh the cons and so NeoLoad is highly recommended over its competitors.