What is our primary use case?
Our DevOps and our development team used Catchpoint exclusively for synthetic testing of API and URL endpoints. Our DevOps team is like a composite team for the solution users. Their job was to operate the tool and build the test, and they were the primary folks using it daily.
The developers used Catchpoint for pre-production testing to ensure that the tests ran and gave them the needed data. They made improvements to the systems that were being monitored. We had around 100 users on the whole platform. The third group was my team, the platform owners. We were in there running daily reports.
When we started, it was pretty light because we were trying to evolve our thinking, and then it grew. The contract was renegotiated when I was leaving, and we were increasing the test count even higher than we had, indicating a higher level of interest in what we could get out of it.
Catchpoint is a natively built cloud. There are also ways to deploy an agent internally for tests that sit behind firewalls and internal systems, but the platform is definitely SaaS.
How has it helped my organization?
Integration is essential. We can tell what's going on in our infrastructure and all the other events in our environments simultaneously. Our hand-built, in-house integration solution had a lot of overhead. at the same time, Catchpoint was essentially ready out of the box. We only need to configure a connector. That was a huge win.
Also, our outside-in tests from our previous provider would trigger an alert. Someone would look at that test failure and say, "Well, the site's up. What's wrong here?" We couldn't eliminate ourselves as the culprit. We couldn't tell if it was being routed through an entirely different AS across the internet because someone between us and the test site has a provider issue.
Catchpoint helped us establish that something is in a provider network, so we could tell our customers to check their internet provider because the traffic is not getting to us. You need to be gentle when you tell them that, but the fact that we could do it was crucial.
What is most valuable?
Catchpoint's UI is well designed, which was a major selling point for us. It's easy to get the visibility you want. The essential piece for us was the external integration with other tools. We needed to get alerts and test result data into other systems we utilized to roll up that information about how our enterprise was working.
Lastly, we appreciate the visibility Catchpoint gives us into the provider networks we were traversing when our test ran from an endpoint in a data center to our internal applications.
What needs improvement?
There's still too much manual involvement in getting customized test configurations out there. It's good, but it still takes a lot of effort. In other words, it's when you need to configure it to collect a specific variable and that kind of thing.
The other issue is the cost. The more data you collect, the more expensive it becomes. You sell your organization by saying we can get this feature set, but then you have to walk that back because we'll need more money to run every test.
This is something hard to get out in your initial scoping. You provide Catchpoint with a series of tests and get a cost estimate, not realizing all the data you might have to collect long term. That was a big deal for us because we partly switched on the promise of saving money.
For how long have I used the solution?
I no longer Catchpoint, but I used it for approximately a year and a half.
What do I think about the stability of the solution?
Catchpoint is incredibly reliable. I rate the stability 10 out of 10.
What do I think about the scalability of the solution?
SaaS products are built for scalability, so you never see the infrastructure behind them. The only limitation on scalability is the on-prem element, but honestly, you could CI/CD pipeline that and remove the scalability question there.
We never got around to it when I was there, but it's all on the customer to do that. If you're looking at it from the perspective of the core product, it's entirely scalable. I would rate Catchpoint 10 out of 10 for scalability.
How are customer service and support?
I love Catchpoint support. I'd rate them 10 out of 10. They're incredibly easy to work with, and you don't need to go through layers of bureaucracy to get to people who can answer your questions. I'm connected to their CEO on LinkedIn, and I communicate with him occasionally to let him know how things are going. We never had a problem getting the answers we needed, including post-sales.
Which solution did I use previously and why did I switch?
Before Catchpoint, we had a custom-scripted integration solution. We switched because it was cheaper, integration was more straightforward, and the UI was better. We were trying to create a UX that hadn't existed before because we understood the need to evolve. We felt we could lower the cost, and Catchpoint would allow us to see into provider networks.
Ironically, we ended up disabling a lot of that functionality because it became too expensive. Ultimately, we only selected basic tests to stay under our budget. But in the beginning, it was a reason we switched.
It wasn't the only reason, but we had a significant visibility issue with China. We didn't have good node coverage in China with our previous provider. The third reason relates to integration. We needed to seamlessly put products together and have them tell our end-to-end story.
One of our goals was to transition away from managing by exception to actually utilizing the tool. That's why UI became so important. It was the first time we could get teams to use it proactively. It wasn't just, "Oh, there's an alert. Let me go into the console. Okay, yeah. I know what that is. Let's go fix it."
How was the initial setup?
I rate Catchpoint a solid eight out of 10 for ease of deployment. We migrated somewhere around 2,000 tests in less than a month. While it took us a few weeks, we migrated a ton of stuff in that 30 days.
What about the implementation team?
I don't know if it is standard per se, but we had a project manager assigned to our migration. You tell them the tool you're moving from. We didn't pay any extra for this other than what was already in the contract.
We told them, "Here are the types of tests we have, and this is what we need them to be." We provided them with technical guidance for questions about the tests because some things don't convert directly.
There were times when they would reach out to us and say, "Okay. What about this group of tests? What were you trying to do here?" They helped us a lot with the conversion. We had to do very little in-house.
What was our ROI?
Abandoning the old solution involved switching tools and transitioning from our custom integration. I look at that holistically. Without either one of those being optimized, there would be problems.
We could deploy a test around 25 to 30 percent faster with Catchpoint than we could before them. Their test-writing language for complex cases needed work, but it was still better than we had. I can't even put a number on the ability to detect whether we were looking at a provider issue or our problem. We couldn't automate it. :
We could eventually triage and troubleshoot it, but we couldn't see it in real-time. That was a considerable return on investment. The ability to definitively tell a customer that the issue is on their end is invaluable because it helps us talk them down off a cliff. The capacity to identify those issues regionally, nationally, and globally was an unbelievable return on our investment.
What's my experience with pricing, setup cost, and licensing?
The pricing is based on consumption and works on a point scale. For example, let's say I want to look at www.google.com, and I'm going to test it to see if it's there. It will bring back all this data that tells me how long it took to connect and how long it took to get the first byte. It will list all the resources on the page, showing that they all work and there are no broken links.
It brings that data back. That test has an assigned point value depending on what you decide to extract from that test. If all I do is check to see whether it's available, it might be one point. I don't know the exact point values off-hand. This is just an example.
If I decide to add performance checking and all those time metrics I just mentioned, that might be 1.5. It'll slap on an extra 0.5 because you're pulling back more data and taxing their systems further. You can add screenshots. What did it see at this point? What they call snippets in some tools. That typically adds another point or more in some cases because that is very intensive.
Using google.com as an example, if I have the test login for me, that's a step in the test. It also gets a point value because each action can have all the same data recorded. A multi-step or transaction-based test could cost you seven or eight points every time it runs. You buy an allotment of points up front. And as you consume those points, your balance goes down until you run out.
You buy an allotment of points for a year on a contract with the expectation that you have planned your tests and what you expect to do with them. They will give you a rounded estimate, and you negotiate the price of your contract based on the points you buy. They are tiered, so you will get better deals as you put more testing into the system. Smaller contracts tend to be more expensive per point than larger contracts where you get volume discounts.
The price is reasonable, but we didn't save as much as we'd hoped. When we pitted them against Dynatrace, our Dynatrace sales guy was ready to negotiate. It's like buying a used car. We didn't save a ton, but we felt like the feature set we got kind of made up for that.
Value-wise, ThousandEyes would've saved us almost 40 percent, but there were a couple of hangups with that product that my leadership was unwilling to overlook. I thought they were minor, especially considering we're always trying to save money. However, the leadership of the DevOps team decided it wasn't worth saving that much money."
Ironically, ThousandEyes is now priced higher than Catchpoint. They have tried to regain our business, but now they're more expensive. There's an ebb and flow. If they're trying to get new business, they will probably treat you better than they do if you're an ongoing customer.
I give Catchpoint three out of five for affordability because they're transparent, there's no haggling, and they're willing to undercut anybody else in the industry. I don't think there's anything special about their pricing model. I don't believe that they're particularly insanely fair. It's more of a you-get-what-you-pay-for type thing.
It's not easy to figure out the point scale, but there are built-in tools to quickly simulate a test and determine what it will cost. I think they are highly transparent. I don't remember there being a single hidden fee. There is a cost if you attempt on-prem testing, but no mysteries there.
Which other solutions did I evaluate?
Catchpoint is one of several that can perform this same function. I generally consider the raw capabilities of all these tools to be somewhat commodified at this point. They're relatively mature, and I don't see much difference.
You have an endpoint somewhere and configure a test. It runs from the endpoint and tells you if your stuff is available. That's generally what these tools do. Most importantly, we don't need people sitting in front of this tool all the time. They don't have to sit there and watch it.
We evaluated other options like ThousandEyes, and our existing product was Dynatrace, which was rebranded as Gomez. There was one more. We took the three and did a bake-off.
What other advice do I have?
I give Catchpoint eight out of 10. It's excellent software, but when I start thinking of quality software, it's not the first thing that comes to mind. Depending on your cost model, I would've recommended ThousandEyes before they increased their prices. I don't think it's overwhelmingly good, but it's a solid eight.
My advice to Catchpoint users is to know your tests. You will struggle when they're trying to estimate your point usage. Ultimately, we had to disable several features because we had poorly calculated that. Plan for increased adoption because you can buy more points ad hoc, but they're not as cheap as buying them up front. At the same time, don't overbuy because anything unused is wasted.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Other