Apache JMeter Reviews

We use Apache  JMeter for API testing. Our latest task combines test cases with our CI/CD pipeline for deployment to Kubernetes. Tests are developed locally and automated for the build using standard Unit and Integration testing techniques. Once deployment is successful to Kubernetes, we develop the JMeter tests for automation in promotion scenarios. Thereafter, once a deployment is successful, we can automate promotion to the next level based on test success.

Test automation moves our organization close to rapidly deploying products. Unit and Integration testing is easy to automate, and most organizations perform these as part of their day-to-day operations. However, end-to-end testing, smoke testing, load testing, and performance testing are much harder to automate. Apache JMeter has aided in that challenge.

While there is a User Interface, the scripting ability is highly beneficial and is easy to use. Tests can be added to a CI/CD Pipeline for integration with testing and deployment scenarios once finalized and operational.

This is a difficult question to answer. On one side, JMeter is very flexible and allows for a high amount of customization. On the other, some tasks are common enough that it merits simplifying the process.

Authentication for API testing could use improvement. Currently, it is a multi-step process to call, extract, and utilize a bearer token securely for API calls. This process is becoming a common enough task that a "wizard" for creating and consuming popular authentication models is merited.

I have been using this solution for about six years.

Apache JMeter is stable, and I personally have not encountered any issues. Depending on the size of test runs, one might need to adjust their JAVA settings to align with the test requirements.

Its scalability works. It is a typically Java run. Therefore, it is limited only by what you can do in Java in terms of scalability.

Developers write tests, verify tests, and maintain tests using version control. They identify and tag each to ensure they are appropriately labeled for test purposes (E.g., unit testing, integration testing, performance testing, and the like). Unit and Integration test coverage is normally high. However, we require testing from outside of the system, and JMeter allows us to create tests automating this process.

Apache JMeter utilizes community support. It is well-documented and has an active community. As far as I know, there is not a "pay-for-support" option.

I have used Postman in collaboration with other developers. However, I prefer Jmeter only out of personal familiarity and not for any technical deficiencies of Postman.

Apache JMeter setup is easy. However, there is a medium-to-heavy learning curve for developing tests and getting started using it for practical uses. Depending on its intended uses, there could be a significant configuration task for a given set of tests.

Apache JMeter is under Apache License, Version 2.0 licensing. Understanding licensing requirements is important for the implementation of any tool.

Understand the use case. Choosing the correct tool for any task is always a challenge. Jmeter offers a significant amount of flexibility and will work for a lot of solutions. Jmeter requires a commitment to learning for optimal operation; without that investment, tests may not yield the appropriate outcomes.

Public Cloud

Amazon Web Services (AWS)

I am using this solution for a banking use case. We have an older version of this solution. We are planning to upgrade to a newer version.

A lot of things are valuable. It is free. It has a lot of features, such as report generation and integration with CI/CD, which makes it very competitive with the other paid solutions available in the market. It is a good solution.

They can improve it a little bit in terms of distribution load testing. We struggled with it during the distribution. 

In terms of reporting, runtime monitoring is not currently included, and it should be included. They can also improve it on the reporting side in terms of the comparison of the reports. 

They can also focus more on integration with CI/CD. Currently, people are using their own customized tools. It would be nice if Apache can provide some standard tools and procedures for integration with CI/CD tools like DPR. There are some tools, but it would be nice if official standard tools and procedures are available.

I have been using this solution for almost three months.

It is stable. There is good support, and fixes are also available immediately.

We have not scaled it, but it is scalable. We plan to increase its usage. 

I didn't have to call Apache because if anything comes up, we are able to find a solution in the community.

Its initial setup was straightforward.

We are using the free version, and if required, we can easily switch to the other version.

I would definitely recommend this solution if it suits a use case. It is good, and it also looks promising. We plan to keep using it in the future.

I would rate Apache JMeter an eight out of ten.

On-premises

It is specifically used for performance systems. It is used for identifying the areas where we need to improve the application bottlenecks and for load testing. We are using its latest version.

It has helped us to build robust application cater to the learning domain and identify bottleneck prior go live. It helped us refine our deployment strategy and capacity planning.

It is open source as well as relatively extendable. It allows us to extend and add additional functionality and features. Its deployment is also very easy.

It should start supporting the presentation layer. It currently provides performance testing specifically at the application and API level. It can be extended to the presentation layer, which includes mainly Angular and React frameworks. 

It should also be easy to use and easy to train people.

I have been using this solution for more than ten years.

It is stable.

It is scalable. It allows us to extend and add additional functionality and features.

We have around 10 to 15 people who use this solution.

It is open source, so I don't think any support is available.

Load Runner, replaced with JMeter due to lower ROI

Its deployment is easy. It didn't take much time. It took less than 15 minutes.

We deployed it on our own.

1. Scalable Product and solutions

2. Plug and Play with CICD process

3. Reduction in licence cost 

No Licensing cost for JMeter

Yes.

I would recommend this solution. We plan to keep using this solution.

I would rate Apache JMeter a seven out of ten.

Private Cloud

We're primarily using the solution for its performance. It's on our data center.

The tool has been a great help to our organization, which is focused more on quality testing as opposed to performance testing. It's provided a lot of insights. With the digital transformation that is currently happening, the need for that shift was rather critical.

The performance of the solution is excellent.

They have designed the product so that it is very easy to configure. 

You can basically do anything you like with the product. It's not very restrictive.

We like the fact that the technology is open-source.

The user interface could be improved. If they had better UI, it might make it easier to use.

You really need a technical team in order to really utilize the product.

The scalability could be better, or the process of scaling itself could be a bit more clear.

We've used the solution for a couple of years at this point.

The stability is quite good. There haven't been any issues with bugs or glitches. It doesn't crash of freeze. It's very reliable.

Nobody ever has used any tool to its full capacity, including this one. Half of the time it's how you set up and how your environment is. We can easily scale on cloud. So far, we're worried about its ability to scale. Our setup is basically a hybrid cloud where we can have private data centers and we can know exactly where each is located. However, the scaling is a big concern for me. I'm looking at BlazeMeter as well, and, although I've only really read about it, it looks like it scales quite well. With scaling, I'm sure I'll find more complexities, especially if I look into kiosk testing. I'll want to explore further test cases.

Currently, those that are on the product are mostly testing communities approaching it from a performance aspect. Every project is different and therefore the number of users goes up and down.

There isn't really a need for tech support with JMeter. There is a vibrant community, which is a good way for a lot of users or developers to go on and post a question and get an answer. The team prefers it this way. It's a positive aspect of JMeter.

I have some experience LoadRunner, however, it is expensive for what it offers. This product is open-source which makes it affordable.

The initial setup is pretty straightforward. My team was very hands-on. They didn't struggle with the UI at all and it wasn't too complex for them. That said, if a person with lesser skill levels, maybe someone who has a lower skillset, may struggle a bit with the implementation. It's geared slightly towards more technical users.

The product is an open-source solution.

I understand the Apache also has something called BlazeMeter, which is something I recently looked into.

I'm not quite sure which version of the solution our team is using right now. It's likely the latest version. We did the upgrade on our tools in 2019. It's open-source, and therefore everybody picks up the latest one.

As an open-source tool, there's a good community surrounding it. If you have automation frameworks or a DevOps pipeline, you can connect that and easily configure everything to streamline processes. Due to the fact that it's open-source, you don't have to wait to get the approvals and the budgets in order. That part alone can take months to sometimes years. If your company has the time, they should review the tools. If it seems to fit your organization, I'd recommend trying it out. 

Overall, I would rate the solution seven out of ten. It's a good open-source tool that you can configure easily, and it's very competitive in the market in terms of usability

Hybrid Cloud

When I was last using JMeter, we were simulating 200 concurrent users and evaluating performance based on transaction times. We were defining SLAs based on the results.

Essentially, we created load scenarios and testing different ones using different workload models.

The recording and playback functionality is helpful.

The reporting is not very good.

When we run with multiple users, it takes a lot of memory.

With respect to the recording and playback functionality, the auto-correlation parameterization is not easy and should be improved.

I have been using Apache JMeter for about four years.

There are issues with stability when running with multiple users because it consumes a lot of memory.

Scalability is fine, although it is important to remember that JMeter doesn't run on its own. It needs to work with load-generations such as BlazeMeter. LoadRunner is the same in that you need a cloud-based infrastructure to run it.

There is no official support. There is a forum where you can ask questions and they respond to you, but the technical support that we have with LoadRunner or NeoLoad is not available.

I have used many similar solutions in the past such as New Relic, AppDynamics, NeoLoad, and Micro Focus LoadRunner Enterprise.

JMeter is not as good as LoadRunner or NeoLoad, and it isn't as easy to use, but it's okay because there is no cost. LoadRunner is too expensive, in my opinion. NeoLoad is cheaper, although not significantly.

From what I have seen, many companies are adopting JMeter because it's free. Especially in Canada, using JMeter seems to be the new trend. Some companies are choosing NeoLoad over LoadRunner because it is easier, faster, and cheaper. Whatever they need to do can be completed quicker. The main problem with NeoLoad is that obtaining resources is harder.

Given all of the choices, my preference would be to implement NeoLoad.

The initial setup is straightforward. I would not say that it is complex and if you already have the file downloaded then it will only take about half an hour to deploy.

I took care of the deployment myself.

I was using the free version of the software.

My advice for anybody who is considering JMeter is to just install it and try it. Creating scripts is a different process when you compare it to LoadRunner or Neoload. There is different terminology compare to these two products, so if somebody has not used JMeter then it may seem difficult at first.

I would rate this solution a six out of ten.

On-premises

Our use cases are for applications such as E-commerce sites where there are multiple concurrent users. In some cases, there is also API load testing where a high number API calls are required for load testing. 

Apache J meter has proved to be the tool of choice for Performance testing with various features and also it comes without any cost.

A valuable feature is that when there's a high number of TPS I can achieve more transactions per second given the hyper-limitations. If the TPS is more than 5,000 TPS, the CPUs get very high (for Load runner)  and JMeter handles that well. I'd also say that the scripting flexibility like co-relation using regular expression is good, it's in the central control. Those would be the two major valuable features. 

We have some scenarios for diameter load testing where TPS requirements are very high, 30K or 40K TPS. In the telco area, this is for simulating mobile usage. However, diameter load testing can be difficult in J Meter. The only way to imitate Diameter requests and process the responses of these requests is to implement them in the code of JSR223 samplers. JMeter generally provides synchronous calls. It's something that could maybe be improved in the future, because for achieving that very high kind of TPS, more than 30K, 40K requires a asynchronous solution. It's not a common thing, it's really very specific to the telco domain and a very few projects. 

I've been using this solution for about 18 months. 

We haven't encountered any major issues with the stability.

Scalability is good. 

Support is more open-sourced. There is a lot of material available online so we haven't had to contact Apache technical support and we have our own team in the company who maintain the product.

We sometimes use LoadRunner for various purposes but currently, we use JMeter 80% of the time because LoadRunner's CPU consumption becomes very high for TPS more than 5000. Also, LoadRunner is not free so we have to pay for the license. We're using it for very few projects now. JMeter can easily achieve a high TPS but LoadRunner has a lot of high-resource requirements. 

LoadRunner's advantage is that the output report is very comprehensive. JMeter reports are only good for the people who understand the detailed logs. Out-of-box, LoadRunner has nice reports and ease of execution. In JMeter, it's also simple but requires some online training or knowledge on the loader and the setup. 

Initial setup is quite straightforward, it takes about two hours, no more, although each project is different. It depends on the scripting that goes into the environment. In terms of the product alone, the deployment doesn't take long. If you include the hardware, it can take about a week because the hardware calls for a lot of couplers and storage units and that takes time.

For the load test which requires high transactions per second to be achieved, in the range of 15K transactions per second, this is a very good tool to use and the biggest advantage is that it is free. It has a big user group, so we can find solutions easily if there are any challenges. It's really a very good open-source tool for any organization to look at in terms of performance tests.

I would rate this solution an eight out of 10. 

Public Cloud

Amazon Web Services (AWS)

We primarily use the solution to manage our capabilities. Wherever the business goes, we need to work within the expected benchmark. In terms of the user bases we deal with, most of these are products from different locations with different types of functional flow-out. There are environmental integrations also running in the background, which a user is basically accessing while going through a particular workflow. 

We'll also handle the reporting section of our data management responsibilities utilizing this system.

The reports and analysis tools are very good. They are the solution's most valuable features.

The GUI could be improved. When we go into GUI mode, there are occasions where it will not sync with our expectations. There are crashes that happen that will stop the solution from performing. It seems we get minor glitches when we go into GUI mode.  

The data client architecture that we have isn't so great. If we are to consume the data, it won't clear because there is tech running on different agents. 

When I need to pull the reports from different agents, it's not user-friendly. The reporting can be difficult to handle. It's hard to increase it if you are working on a client's architecture.

It's not easy to get the data from one place or to do customizations. 

There are other solutions that allow users to model their load and structure with them. You can't do that on JMeter. 

On other solutions, like Silk Performer, you can do network packeting, which you can't do on JMeter. They should add this to the solution as a capability in the future.

The support management needs improvement. Support is coming from consultants; you will not be able to get on-premise support from all of their agents in one place. On Silk Performer, for example,  they have the capability where you can basically have a summarized report from different agents.

Personally I've been using the solution for the last 10 years. However, in the program we are in, I've only been using this tool for six months.

If you want stability, you will have trouble, because the system is not uniform. You have to do a lot of modeling. 

If you're talking about mobile platforms, JMeter will not allow you to capture the performance of applications on your mobile platform. It's an area where the solution is lacking. It doesn't have the capability to allow developers to check the performance on an AP app.

You can scale the solution, but you will run into issues if you want to run, say, 1,000 users.  You will have to configure multiple agents and then distribute your load. It's quite difficult to manage everything on a single agent.

The initial setup is straightforward. We found it quite easy. For us, it wasn't complex at all. The solution is up and running quite quickly and doesn't take long at all to set up and deploy.

We've looked at Silk Impact and Load Balancer in the past. Each of these solutions offers a few different functionalities that aren't necessarily available on JMeter. For example, on both, there are more network features available that aren't included in JMeter and you can model your load and structure with them as well.

If companies have mobile development, they should use JMeter. If they want to integrate it with the PA pipeline, JMeter will work well. However, there are other platforms and solutions as well.

Other options include Silk Performer and Load Impact. These two solutions have other capabilities that JMeter doesn't have. You can basically model your load and structure with them. You can't do that on this solution. On Silk Performer, you can also do network packeting, which you can't do on JMeter.

Then there are some network features available in Load Impact as well as Silk Performer that aren't in JMeter. 

I'd rate the solution eight out of ten. The solution has certain limitations, but it's still a good product. 

Public Cloud

Microsoft Azure

The tool helps us in the capacity assessment planning for our services. It helps us to make sure that the APIs will be able to afford the traffic that we expect to get from these services in the production environment. 

The product helps me get the expected performance from applications or servers and reduces costs. It also enhances the performance of the services and helped them reach their ultimate capacity. 

The tool needs to have a better Graphical User Interface. Many of the solution's features are difficult to understand due to the complex user interface and user experience. The product needs to add plugins. It should also work on the integration with external partners like IDE and API gateways. 

I have been using the solution for more than three to four years. 

The tool is stable. 

The product is scalable. My company has around four to five users for the solution. 

The product's setup is not complex. The tool's installation was very smooth.  It doesn't have any deployment process. You just need to implement a set of packages to get started with the product. 

I would rate the product an eight out of ten.