What is our primary use case?
We use the solution for enterprise performance testing of various technologies including web services, APIs, and web GUIs.
We deployed the solution to increase our performance testing footprint, which we needed to upscale for the maturity of our operation.
We have six on-prem load generators on our network, and the rest of our deployment is in the cloud. It's a very simple architectural design.
How has it helped my organization?
BlazeMeter opened up performance testing for us. Our old solution was a client-based performance testing tool, and for staff to access it, they needed to remotely connect to a Windows VM and book time with that controller. Now our tool is web-based, and we onboarded 12 to 14 teams to BlazeMeter, which would not have happened before. Our CoE team was the go-to for performance testing, but the solution has opened up the practice to the whole enterprise, making teams more self-sufficient, and that's the most significant benefit. Performance testing is no longer segregated to one team.
What is most valuable?
Using cloud-based load generators is highly valuable to us, as we can test from outside our network and increase load generation without having to upscale our hardware as much. The cloud load generator is there when we need it and is the feature we leverage the most.
We have a very high opinion of the range of test tools the solution provides, it has a great deal of potential and we are just scratching the surface of it currently. As our maturity and skillset with the product increase, we'll be able to leverage that more. For example, we don't really use mock services yet. We know how to, but we're still set in some of our ways.
BlazeMeter being cloud-based and open-source is vital; it was one of our top priorities when choosing a solution. Much like the rest of the world, we're moving away from the old paradigm of the Windows days where we would bring up a server, get Windows licenses, an operating system, and maintain it all. With BlazeMeter, most of that is done for us, and we don't have to worry about infrastructure. We have on-prem load generators for teams needing to run load tests from within our network, and we need to maintain that capacity. However, we don't have to host anything outside of the load generators in the network, so the maintenance effort and cost are much less than they would be as a legacy system.
The solution does bridge Agile and CoE teams. It's a shift-left tool, and testing comes in much earlier than in the past. BlazeMeter is a valuable asset in this regard.
The tool helped us to implement shift-left testing. Many of our teams with the required skillset can include performance testing as part of their build runs. This may not be high-level testing; internally, we refer to it as early performance testing. It allows teams to confirm the software is functioning correctly early, which was not the case before. We would wait until a certain point in the SDLC before running a performance check, and now we're able to implement that much earlier in the process.
We currently don't have any stats on changes in our test cycle times, but there is no doubt in my mind that BlazeMeter improved our software quality.
We have not faced challenges in getting multiple teams to adopt BlazeMeter. We onboarded around 50 users in three quarters, which is incredible considering we had two performance testers before implementing the solution. Our only challenge is skill sets, our staff wants to adopt the tool and understand its importance, but they may not have the resources or skillset to do so. Those with the necessary skillset are onboarded as soon as their project is greenlighted.
What needs improvement?
Our biggest challenge is the skill set required to operate the solution because we used to have a centralized performance testing team. Now we've opened it up to other teams; some needed to onboard new resources. The solution is simple and user-friendly, but we still need the right staff to use it.
We encountered some minor bugs, and I would like to have the ability to add load generators to workspaces without having to use APIs. We can't do that now, so we're beholden to the APIs.
For how long have I used the solution?
We have been using the solution for about nine months.
What do I think about the stability of the solution?
The solution is very stable. We had a few issues with users getting 404 errors recently, but that's the first time we have encountered any problems in three quarters.
What do I think about the scalability of the solution?
The scalability is incredible. We could scale it to as big or small as we want, with our license being the sole limitation. The resources are in Docker containers in Docker images. We could scale within a few minutes if necessary.
How are customer service and support?
The technical support is excellent. When we had hiccups during deployment, they responded quickly with effective solutions for us.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
We used other tools and switched because they weren't as user-friendly. BlazeMeter offered us the ability to increase our performance testing footprint without requiring a high level of performance testing expertise from our QA staff. Additionally, our old solutions were client-based, and BlazeMeter is cloud-based, providing all the advantages that come with that.
How was the initial setup?
The deployment is very straightforward. That was one of our criteria, as we didn't want a complex new enterprise solution rollout. There were a few bumps during deployment, but most of that was on our side. BlazeMeter is relatively simple compared to other enterprise solutions we implemented.
Less than ten staff were involved in the deployment. We used Linux Enterprise to house the six on-premise load generators, and there were a couple of employees responsible for Docker, our solutions architect, and myself as the admin.
What was our ROI?
I don't have a concrete figure, but I can say once we sunset our old solution, that will save us a significant amount of money on infrastructure, licensing, and maintenance. I also think there is an ROI associated purely with the increased quality of our software, thanks to BlazeMeter.
What's my experience with pricing, setup cost, and licensing?
The product isn't cheap, but it isn't the most expensive on the market. During our proof of concept, we discovered that you get what you pay for; we found a cheaper solution we tested to be full of bugs. Therefore, we are willing to pay the higher price tag for the quality BlazeMeter offers.
Which other solutions did I evaluate?
We carried out a proof of concept of four tools, which included BlazeMeter. It's more stable and mature, with well-documented APIs. BlazeMeter University was a significant consideration for us due to our requirements; it helped us roll out the solution to multiple teams. It seemed like APIs for the other solutions were an afterthought.
What other advice do I have?
I would rate the solution an eight out of ten.
The solution enables the creation of test data for performance and functional testing, but our use is focused on performance testing. We don't particularly use functional testing, but we are currently talking about using test data management for functional testing. We have our in-house automation framework, so the ability to create both functional and performance test data isn't a high priority for us.
We don't use BlazeMeter's ability to build test data on-the-fly, not because we aren't aware of it, but because we are still at the early stages with the solution. Until fairly recently, just one other person and I were in charge of performance testing for the entire company, so having self-sufficient teams is an immense change for us as an organization.
I would say it's critical to have the appropriate skillsets among the staff as we could deploy just about any solution in an enterprise. Still, it won't be used to its total capacity without the proper skills. BlazeMeter showed us how little performance testing we were doing before and how vital increasing that footprint is. We've onboarded 50 users; that's 50 users who were not engaged less than a year ago and can all carry-out performance testing.
This solution can work very well for enterprise companies with a more advanced skill pool to draw from. For beginners in this area, specific skills such as JMeter scripting are required to use the application. It's easier to use than most solutions but requires a particular skill set to deploy and operate successfully. A good solutions architect and QA leads are essential in evaluating any product.
Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.