The ability to test mobile applications, support cloud based testing, real-time monitoring during a test, advanced auto-correlation feature, infrastructure monitoring, detailed results reporting etc. It also gives the ability to manipulate the script by inserting JavaScript. It's one of the features that very few other tools have.
First and foremost, it took out the painful resource hours out that went in to manually correlate each and every action of the script while using LoadRunner. NeoLoad does not require much of scripting efforts, and correlation and parameterization are fairly easy.
We would like the product to be able to handle large project suits with large number of test scripts. Currently, we are not able to upload/publish huge projects back and forth from the team server to the NeoLoad controller. There is no support for VT3270 terminal emulation to support mainframe-hosted applications. There is no summary graph that shows the distribution of all of the response times on a single graph. We currently export the data as a coma separated variable (CSV) file and import into Excel to generate this summary graph.
4 years.
A small number of times, when load testing huge projects with huge number (~10,000) of vUsers, we have seen that it completely froze. We had to recover from the previously saved version of the project.
Very good, the vendor has been very accommodating with the ebb and flow of our business needs. Neotys is a partner.
Technical Support:Good. (They decommissioned the support for 4.2 though)
LoadRunner. Cost effective and with almost same features that are required for our project.
Straightforward and the technical support folks will be available to help.
We utilized a vendor team. They were fairly knowledgeable and knew how to quickly troubleshoot the issues or contact Neotys.
A great deal of improvement in application performance happened because we could pin point the bottlenecks. The real time performance monitoring allowed to find out the troublesome queries and tune them. Increased resource efficiency.
Low in cost when compared to tools like HP LoadRunner. Talking about licensing, they offer a cool feature that enables us to split the Vuser bundle across multiple machines. In other words, we can have multiple controllers running multiple load tests at the same time.
JMeter. Figured that it would not be suitable tool for our project and requirements.
NeoLoad is the performance testing solution for Web and Mobile applications that can also be used to run the tests from cloud by having load generator’s from various locations. If there is a strong performance team that can analyze and correlate the test results with the raw data, then NeoLoad would be the right solution.
It gives us the ability to quickly script a load test scenario and to develop complex testing scenarios. If you ever want to know more about your application, just recording a session can be invaluable.
Outside of the obvious benefits of load testing, the data we get back and are able to provide at the executive level provide a level comfort in the given application at peak periods, and drives direction for future performance-oriented projects and goals.
They have done well with patching and enhancements in each version. More of a “community” of shared experiences, help, and collaboration would be a good enhancement to their site.
We've been using it since 2011.
No issues encountered.
At times we have had issues with load generator agent stability, but tuning the heap and documented network hacks greatly reduced this.
No, it's very easy and quick to scale.
7/10
Technical Support:7/10
No, not previously but we have demonstrated or performed proof of concepts of other products.
The infrastructure setup was very straightforward, and getting started with our first test was relatively easy with online videos and resources.
We did it in-house.
Difficult to really estimate, however we do know that by load testing we have found countless issues over the years that would have affected our e-commerce revenue had these issues made it to our production environment.
We evaluated other products of similar nature as well other team member’s prior experiences with similar products.
We evaluated other products of similar nature as well other team member’s prior experiences with similar products.
Start with some small, simple scenarios and expand complexity from there. Your project will take shape quickly, and something that seems overwhelming and complex will all of a sudden will not be so hard. Use the online resources and videos.
The six phases of an IT project:
This article has been put together as part of an evaluation of the performance test tools NeoLoad and LoadRunner. I consulted a variety of sources, including user groups, discussions with colleagues, telephone chats with HP (the vendors of LoadRunner) and Neotys (the vendors of NeoLoad), and of course the Oracle – Google. The opinions in this article are my own and do not represent the views of any particular company, the software vendors or any organisation (and I’ve assumed that readers have some knowledge of web application performance testing).
The versions discussed in this article as of April 2014 are:
LoadRunner 11.52
NeoLoad 4.2.2
I hope this evaluation and comparison will assist you with your own evaluation of LoadRunner and NeoLoad.
Introduction
LoadRunner and NeoLoad are considered the top two best performance testing tools on the market. Comparing the two tools is just like comparing a Mercedes Benz and a BMW: both are high performance, perfectly engineered machines that also have an associated prestige.
Performance testing is a critical component of the software testing process. It determines the actual operational boundaries that will simulate the real world use of an application. Performance testing is load testing, stress testing and scalability testing.
Why are we here?
The philosophy behind performance testing is we don’t want any of our IT systems to crash – it’s terribly embarrassing and can be very damaging, never mind expensive, when a crash happens.
A load test determines how a system behaves under a particular workload. The main objective is to see how various components react to a gradual increased ramp up of workload. The usual outcome of load testing is throughput, response times, CPU load and memory usage.
Stress testing determines the breaking point or the unacceptable performance point in a system. It reveals the maximum service level the system service can sustain.
Scalability testing evaluates the effects of adding additional hardware, virtual or physical, in order to distribute work among system components.
The following evaluation considerations have been taken into account for LoadRunner and NeoLoad:
LoadRunner is written in C++, and NeoLoad is written in Java. LoadRunner scripts after recording can be manipulated and debugged using C. NeoLoad is a scriptless tool – instead, it involves dragging and dropping functions into a script. Also IF statements and loops can be dragged and dropped. Even exception handlers are controlled the same way within NeoLoad. There are some good and some cumbersome qualities attached to both – the NeoLoad feature is less powerful than the LoadRunner vugen feature, albeit LoadRunner takes more hours to script record, parameterise and debug.
Recording scenarios
If you’ve ever used a proxy debugger like fiddler, you’ll see what’s happening behind the scenes when surfing the internet. There are send requests and responses from your computer’s browser to the website’s web server. Recording load testing scenarios works just like a proxy: when the record button is pressed, it actives a proxy that acts like a ‘man in the middle’ capturing http requests and responses from the system’s web servers.
LoadRunner architecture
LoadRunner, which has a trial version, consists of three components: the virtual user generator, also known as vugen (script recorder and editor), the controller where all the vugen scripts are controlled into a scenario, and the Analysis, the results component. These enable the user to keep scripting in vugen while a scenario is running in the controller component, which usually happens because the load tester is hired at the end of the System Development Life Cycle (SDLC) and is under a huge amount of pressure.
NeoLoad architecture
NeoLoad consists of one component encompassing all-in-one software, which means that if a load scenario is running you cannot create and edit scripts.
Recording web with LoadRunner
Nowadays Chrome and Firefox work perfectly as recording browsers in LoadRunner’s vugen. Once upon a time, just Internet Explorer worked. Other browsers made vugen crash when the recording button was pressed, so LoadRunner consultants were limited to the browser people use to download other browsers.
If you’d like to get the time it takes for a user to login to the system under test, press a simple ‘Start Transaction’ button during recording – you can name the transaction appropriately, log the user in, then end the transaction when the logged in page has downloaded.
The transactions look like this for a simple login script:
Recording web with NeoLoad
NeoLoad records scripts in a ‘tree-view’ like structure. A wizard pops up after recording to automatically correlate the recorded script for you. Start and end transactions are not like those in LoadRunner – they are called ‘containers’ in NeoLoad and there is no end to the container as the ‘end’ of the container is actually the beginning of the new one, like this:
Correlating
Correlating is extracting dynamic values and placing each dynamic value into a variable from GET requests and substituting it into the POST response, usually in the request after the GET. This is the most time-consuming task in load testing – just ask any performance test analyst.
Correlating with LoadRunner
Design Studio is the tool LoadRunner’s vugen uses to try to correlate dynamic values in a newly recorded script.
‘No correlation parameters found’: I glanced at the recorded script below and found many dynamic parameters.
So why didn't the automatic correlation utility work for LoadRunner? This is a good question considering some of the other protocols that LoadRunner uses correlate really well. Like me, many of my colleagues have become used to manual correlation when using LoadRunner for http/web.
In fairness to LoadRunner, it can find one or two dynamic parameters, depending on the size of the recorded script. It looks like this:
It works (sometimes).
Manually correlating with LoadRunner involves placing this statement to grab a parameter from every dynamic GET request, which you’ll have to then place before a POST request if required.
Correlating with NeoLoad
Correlating is definitely where NeoLoad has an edge over LoadRunner. NeoLoad takes hours of painstaking manual correlation work out of the equation, and that saves bucket loads of cash in consultant fees and maintains deadlines. Any IT manager will know how much a LoadRunner contractor costs daily. If that figure is going to be potentially halved (or otherwise cut down by the speed NeoLoad can do the job), then maybe the next word that enters the manager’s head is ‘bonus’ or even ‘gratitude’, for finishing the project on time and within allocated budget parameters. Gratitude goes a long way.
There is some manual correlation in NeoLoad that involves a not-so-very-cumbersome two clicks of a mouse.
NeoLoad is so intuitive that if the same extracted value exists in subsequent requests within the scenario, this pops up (music to my ears):
And if you’d like to replace all, then:
The music keeps on playing – click, click, click ... and done.
Playing back debugging/recorded script with LoadRunner
Debugging is instantly more informative in LoadRunner than in NeoLoad as you can see the script executing and outputting to the log (circled below). It’s easy to watch variables and check for errors in real-time script compilation and execution.
Playing back/debugging recorded script with NeoLoad
One quick and easy way to check and debug a script in NeoLoad is to use the virtual user checking option by right clicking on your script. This runs the script and shows errors in requests that need to be addressed.
In NeoLoad, a more involved debugging analysis (after checking using one method) is to use JavaScript. Do this by dragging the JavaScript function into a script and coding the appropriate write to a text file. All variables can then be sent to a text file and used as a watch list.
NeoLoad will also check the JavaScript syntax for you.
Parameterising
Parameterising is adding a text or csv file to a load testing script. For example, when a script is recorded, there is a hard-coded user id and password in the script. Creating a file of five user ids and passwords and thus adding to the script while replacing the hard coded values is parameterising. You can control this, for example, using the five user names – the script can randomly pick the user id and password or it can pick them one line at a time incrementally.
Parameterising with LoadRunner
I’ve always found parameterising with LoadRunner straightforward, as long as you’re aware that creating a new parameter creates a new .dat file as well. It’s easy to have a user name and password in the same file (like above) instead of in two separate files. You can also edit the text file in Notepad, which makes pasting large amounts of data relatively easy.
Variables in LoadRunner
Where variables need to be declared as well as the variable type, like char or int etc., and values need to be added to them during the script execution, then do the following:
Parameterising with NeoLoad
Adding datasets to NeoLoad isn’t too dissimilar to LoadRunner. Also you can import spreadsheets into the variable manager or just create datasets manually.
Variables in NeoLoad
NeoLoad has an edge over LoadRunner, as none of the above coding is necessary. Variable types are added using the variable manager as in the screenshot below:
NeoLoad has an added SQL variables feature that queries the backend database for test data.
Randomising user input/actions
A very important aspect of a realistic load test is to have as many virtual users doing as many different things as possible to the system under test. The more random actions the better, such as if there are 10 links for two vusers to click, then there should be a one-in-forty-five chance that the two users click the same link (if you don’t know what I mean, check out permutations on Google).
Randomising with LoadRunner
Picking a random row from a result set you’ve just recorded with LoadRunner involves doing something like the code below. First declare the variables, then using the web_reg_save_param_ex() built in LoadRunner function, grab the number of rows, place the number of rows into a random number function between 1 and 10 (if there are 10 rows), and finally get the script to click on any of the rows – this should be different for each iteration in your script, depending on the test you’d like to conduct.
Randomising with NeoLoad
Randomising with NeoLoad is very easy, but to randomise you have to record every request and place them in a container. To give you an example, imagine you are filling out an online form to register for an online shopping site (see the screen below). Just before you submit the form you have the option of subscribing to the store’s newsletter, and also entering into a draw if a radio button is checked. Now that’s four combinations:
In NeoLoad you have to record this randomising scenario four times and place each into a container. It’s then possible to control the randomise settings for a realistic and valid load test.
The fun part – performance testing
After all the hard work is done – the correlation, the parameterisation and getting admin access to every single pre-production server in the system – it’s time for the real fun. Every performance tester loves this part because it’s the breaking stuff part: Lego for grown-ups.
Other forms of load testing include what hackers do – distributed denial of service (DDOS) attacks. Well, that’s more like a stress test.
IP spoofing can be implemented with both LoadRunner and NeoLoad – this means that an even more realistic performance test can take place. You don’t want a router in the network to reroute the web requests to a honeypot if there’s a rule in place that redirects heavy traffic away from the system under test.
Load Testing using the LoadRunner controller
All servers in the system under test can of course be added into the load scenario, so that real-time monitoring of server resources can be directly correlated to the number of vusers being ramped up, to a peak hour load test. So for the layman, let’s see what maxes out first if 1,000 users login to your website all at once, or all in the space of five minutes – things will start to break depending on a lot of factors, too many to discuss in this article.
Server statistic graphs can be added to the load scenario, and all the gathered stats as well as transaction time taken can be collated in LoadRunner’s Analysis component. A performance Test Summary Report (TSR) uses a template that can be modified to your liking and then presented to your IT Test Manager. Make sure you let him or her know that you typed the report from scratch – work is all about perception after all.
The client controller component of LoadRunner has all the bells and whistles needed for a realistic load test. There’s only one drawback: there’s a limit on the number of performance counters that can be added to a scenario. If there is a requirement for more performance counters you have to install and configure monitoring software called SiteScope. All servers in the system have to then be added to SiteScope after SiteScope and the SiteScope license server are installed.
Load testing using NeoLoad
The ‘runtime’ component of NeoLoad and the LoadRunner controller are on a par in terms of the capabilities they have for you to add the appropriate graphs.
LoadRunner vs NeoLoad – the cost
Here’s what every IT manager wants to know: Which tool is cheaper? Is it value for money? Now this is a tricky one: How much does LoadRunner cost?
Performance testers have always known that when it comes to price, LoadRunner is not a BMW or Mercedes – it’s a Ferrari. I wasn’t very successful in my endeavours to obtain a quote for LoadRunner. I contacted HP in London, and they referred me one of their local partners here in Australia. My enquiry was pretty simple: I wanted a quote for a 5,000 user Web vusers license, but HP’s partners said via email that they needed more information.
Using Google to ascertain the cost of a LoadRunner license had mixed results. I came across an old article mentioning that a 5,000 vuser perpetual web license with support for three years was about a million dollars. That article was written in 2005.
When I contacted Neotys in France, they were a little more helpful with a licensing quote for NeoLand, and there was a lot of room for negotiation as well, including email support, monitoring for Oracle and a 5,000 vuser license that never expires.
Based on my very ball-park figures, there’s an approximate cost correlation between NeoLoad and LoadRunner. Imagine that LoadRunner costs $100:
LoadRunner: $100
NeoLoad: $10
NeoLoad is about 10% of the cost of LoadRunner, and Neoload includes technical support as well as perpetual licensing.
There are also several product packages associated with both HP and Neotys that can be negotiated as per your allocated budget, so keep that in mind. All you have to do is pick up the phone and talk to them.
So there you go – my take on the top two best performance tools on the planet. Taking into account industry experience, time frames scripting, hours spent completing the job (usually after working hours) and downright value for money, the winner has to go to NeoLoad. It just works.
If your company has spent its money on LoadRunner and highly paid consultants developing test scenarios for your application, then the money has not gone to waste. If your company wants a cheaper product that does the same job as LoadRunner, then evaluate NeoLoad and ask an experienced performance tester’s opinion on the tool. The downside to this option is finding experienced NeoLoad consultants.
LoadRunner is the preferred tool for large organisations, such as banks and financial institutions. Right now, NeoLoad is the preferred performance testing tool for universities and smaller organisations.
Some final questions
My employer has been using LoadRunner for performance testing for many years. How expensive would it be to hire people to convert these scripts to NeoLoad?
Weighted Rankings 1 – 5 (5 is excellent)
Can LoadRunner scripts be migrated to NeoLoad?
Yes, and vice versa. It doesn’t take as much time or resources as you think. Companies don’t need to hire specialists to re-code LoadRunner scripts into NeoLoad scenarios.
The real plus of Neoload IMO is the script-ability. This makes it much easier to use than say HP Loadrunner. The tool has some serious work in the maturity department though. On the other hand, if you find a an issue and report it, its treated very seriously and its usually solved in a timely manner.