What is our primary use case?
We had two use cases. In the beginning, log centralization was the main thing, and this was the most frequent use of Graylog, but we also tried to use it for analytics. Graylog was maintained by the data lake team, and we were looking for tools that were suitable for analytics. We felt that Graylog looks real-time. It had some graphs and dashboards. So, we had an idea to use it for analytics.
What is most valuable?
What I like about Graylog is that it's real-time and you have access to the raw data. So, you ingest it, and you have access to every message and every data item you ingest. You can then build analytics on top of that. You can look at the raw data, and you can do some volumetric estimations, such as how big traffic you have, how many messages of data of a type you have, etc.
What needs improvement?
We stopped using it for analytics because of its price, and at the moment, we are using it mostly for log centralization. If you use it with high traffic for analytical purposes, as well as for the logs, the infrastructure costs are unbelievable.
Graylog is a great product backed by Elasticsearch as the storage and query engine. It is just an interface on top of Elasticsearch and some Elasticsearch management. The indexes that are kept in Elasticsearch are managed by Graylog software. Elasticsearch is a decent product, but it's very infrastructure-heavy. It requires lots of resources, and if you make a mistake with provisioning, you are likely to not get a cluster back. We had a couple of outages like that, and we hated that. So, we ended up over-provisioning resources just to avoid such situations from happening. If you have a whole team trying to fix the Graylog instance for two days, that's a bit too much. That may be my Norwegian take on it, but the engineering resources are expensive. It's better to just provision the infrastructure.
Overall, the product is great, and the features are just fine, but the infrastructure cost is what is killing it. The infrastructure cost is the main issue. I like the rest. If the infrastructure costs could be lower, it would be fantastic. I'm not sure if they can improve the infrastructure cost with the way Elasticsearch is. If they keep using Elasticsearch, maybe there are some opportunities there, or they can support other backends with cheaper storage. They could have a different backend to replace Elasticsearch or do some tweaks to Elasticsearch to reduce the costs. There could be partial parsing of logs or parsing on demand so that when you write data through Graylog to Elasticsearch, it doesn't need to crunch in every detail requiring that much CPU.
For how long have I used the solution?
I was a part of the team that was managing a Graylog instance for five years. We were both the maintainers and users of the platform. The last time I checked the Graylog interface was half a year ago. We switched to Loggly for day-to-day activities. Not all the teams in our company did that, but we did.
What do I think about the stability of the solution?
It's stable if you do not overload it. If you go over the boundary that it can handle with the existing infrastructure, it becomes a nightmare. Otherwise, it just runs smoothly.
What do I think about the scalability of the solution?
It's pretty scalable. We didn't hit any ceiling when scaling it up. You throw hardware at it, and it just performs fine. The only issue is the cost. It's not dependent on how big an instance you have. The problem is that it costs a fortune.
We have about 300 users, but it's not about the users. It's about the traffic we have. Our traffic is pretty big. We have thousands of messages going per second through the Graylog instance. It's not that we have many users making queries concurrently, but still, to have the data ready for querying, Graylog needs to crunch it, process it, and write it to Elasticsearch, and that's what consumes resources most of the time.
How are customer service and support?
We didn't have any issues that we needed to fix in Graylog. It just worked for us. We do use other open-source products and commercial products, and the commercial product support is not always fantastic. For Graylog, we could solve all the issues by searching over the internet and on Stack Overflow. With other products, such as Metabase, which is an open-source business intelligence tool that we use, because half of our company is made of software developers, they can write a patch. We did that a couple of times, but with Graylog specifically, we haven't had a need to introduce any patches. It was just tweaking the config.
Which solution did I use previously and why did I switch?
We switched to Loggly because of the infrastructure costs of Graylog. Loggly is an all in the cloud commercial offering. Even though Graylog is free and doesn't require any maintenance and we just pay for the infrastructure, surprisingly, Loggly costs less than Graylog. So, we save money with Loggly. That was a big surprise to me.
Graylog is very stable, but Loggly is less stable. Maybe they're trying to cut costs. We have had a couple of outages, and quite often, we had indexing delays. When the data is not available right away, we have to wait for that. With Graylog, with our over-provisioning, we never had these issues, but we use Loggly because it's more efficient money-wise for our volume.
One drawback of Loggly is that they have just a few sites where they can store data. Previously, the site was only in the United States. We couldn't choose anything else. For us, it wasn't a problem because there was some agreement between the European Union or European Economic Area and the United States on data processing, but then this agreement got canceled. So, Loggly introduced one European site. They do have something over here, but we still lack support on the locality of the data because our customers are in Asia, and we want services to be placed closer to them. We want more sites. If Loggly, for example, could be deployed on any AWS instance, such as on Amazon Cloud or Google Cloud, which do have data centers in, for example, Thailand or the Asia Pacific, that would be beneficial. Loggly still doesn't have that. They are developing something, but that's an advantage of Graylog. It can be placed anywhere you have the infrastructure.
There is one particular mode with which we could not agree with Loggly, but there is some progress there. We have high traffic, but we don't want to store the data for long. Loggly suggests 90 days by default. We don't need 90 days because we need to troubleshoot situations that happened today and yesterday. We only need a couple of weeks of data, but we need to process a lot of traffic. Loggly wasn't ready for this type of load. I do understand why Loggly does that. It's not the storage that is most expensive; it's the CPU resources that you need to put into the indexing process when you ingest logs into the system. So, Graylog is more flexible because you still can tweak it to your particular load. You can say that you need just two weeks of high-traffic data, but you would need the infrastructure built specifically for this use case. With Loggly, we spent a year negotiating this mode. We came close, but it's still not ideal.
The other competitors, which we haven't had in production, such as Humio, are promising lower prices. It seems like the next generation of log processing. Graylog is based on Elasticsearch, and it seems that Loggly is also based on Elasticsearch or at least some mutated version of it. Humio seems to be based on something else. They don't have Elasticsearch. So, they don't have this burden of maintenance.
How was the initial setup?
We have top-level engineers, and we didn't have any problems at all, but any random guy could also set it up. It involved the magic of regular Linux commands. It was pretty easy. I would rate it a five out of five in terms of the ease of the setup. It's great.
What's my experience with pricing, setup cost, and licensing?
It's open source and free. They have a paid version, but we never looked into that because we never needed the features of the paid version.
If you have a small amount of traffic or you are a small company, Graylog is just fantastic because it's open source, and it's free. You can run a Graylog instance on pretty modest hardware, but when it comes to large volumes, as we have, it becomes too expensive.
What other advice do I have?
I'm pretty happy with the features of the product, but I'm not happy with the infrastructure costs. Feature-wise or from the end user perspective, Graylog is just great.
For small enterprises, it's a good start because they tend to use cheaper products, at least until they grow. Graylog is a good fit there because you can pick a very cheap cloud provider and then just install it there. It is pretty cheap. For big companies that are focused on reliability and availability, Graylog either requires over-provisioning or will cost a lot, which is not ideal. There are better solutions out there in the market, but one important point is that Graylog can be placed in your local data center. Some companies are very suspicious of clouds or have some restrictions from authorities or as per their policies and business model. There are countries, for example, Pakistan, where the network is poor, and if you use the closest data center of any cloud provider, that will most likely be Thailand. For these types of setups, Graylog is pretty much the only choice.
Before Humio, I would have rated it a 10 out of 10. It's a great product, but because of its cost, I would rate it a 9 out of 10.
Disclosure: My company does not have a business relationship with this vendor other than being a customer.