Apache Kafka Reviews

We are using Apache Kafka to extract data from a Portuguese data source, utilizing an open-source project for data capture. The connector for this project is linked to both Kafka and Confluence platforms. We then transform the extracted data and store it in Elasticsearch.

We appreciate the ability to persistently and quickly write data, as well as the flexibility to customize it for multiple customers. Additionally, we like the ability to retain data within Apache Kafka and use features, such as time travel to access past customer data. The connection with other systems, such as Apache Kafka and IBM DB2. 

Apache Kafka can improve by adding a feature out of the box which allows it to deliver only one message.

I have used Apache Kafka within the last 12 months.

Apache Kafka is a stable solution.

The scalability of Apache Kafka is good. It can process many requests simultaneously.

We have approximately 600 people using this solution in my organization.

I have not contacted the support from Apache Kafka.

The initial setup is relatively easy as I am using Docker and the files provided by Confluent. However, setting up Apache Kafka in a production environment is not as straightforward. I prefer to use solutions, such as Confluence that already have everything preconfigured. As a developer, creating an environment for it is not a problem for me, but I think it can be challenging for those responsible for the production environment. There have been issues with data loss and other problems in the past. Configuring it for production is not easy.

My deployment was very quick because I am using it locally. We have someone else that does the cloud deployment.

I did our local implementation and we have someone else that does the cloud deployment.

The price of Apache Kafka is good.

I rate the price of Apache Kafka an eight out of ten.

I don't see any major issues with using Apache Kafka. Many companies use it and it's a good solution. My advice would be to use it as a software-as-a-service rather than setting up your own cluster. This way, you can benefit from a preconfigured and maintained platform. It's better to opt for a software-as-a-service solution.

I rate Apache Kafka an eight out of ten.

Our primary use cases allow software developers, and application developers, the option to not have to code in their own logic for the retry mechanism. A lot of software, and applications, have this feature of retry built in some way or the other, but they all have some kind of a pre-alpha version of Kafka, up to a certain extent.

So it is a good backbone for microservices. So basically you want to write microservices, which you can shut down and bring it up whenever you want. You want to be able to shut it down to actually replace it with a newer version and bring it up. The bottom line is you can kill the microservice and bring it back up and do all the things that you want to do with it. But whenever it comes back up, it should pick up and run from where it had left off. That is what everybody tries to do. And in order to build such a system, they have to write several logical pieces of code, and most of that code has already been built for in Kafka so that you don't have to do it yourself.

One of the most valuable features I have found is Kafka Connect.

Basically, the bootup time, if you have large messages, sometimes takes up more time than I would really like it to. So that is the area that Kafka can actually improve upon. But that is okay, the way we get around it is to make sure that Kafka has started up first and warmed up before anything else starts up. I would also like to see monitoring service tools.

I have been using Apache Kafka for the past three years.

The stability is good as long as you have a short retention period.

Confluent is the cloud version of Apache Kafka and it is scalable.

We do the implementation in-house.

If you are managing your own implementation the return on investment is pretty good. What you need is good developers.

Apache Kafka has open-source pricing.

Apache Kafka is a good choice, so I would recommend people not have a real-time application if they do not have to. It is better to have a very fast batch operation than a real-time operation. I would rate Apache Kafka a nine on a scale of one to ten.

My primary use case for Apache Kafka is replacing ETL and doing data transformations.

Kafka allows you to handle huge amounts of data and classify it into different categories. If you have huge amounts of data, Kafka is a very good solution for data classification. When you need to route it in different directions, you have to take a look at the messages that you get, interfile them, and then send them to the correct place. Kafka is a good product to use in the backend.

The feature I find most valuable is the classification feature. Kafka enables you to tag content with a category.

Kafka contains two components. The component that does the synchronization between the rest of the components, that's an older version of the software and it causes all kinds of configuration problems. The Confluent, which is the company that sells a commercial version of Kafka is getting away from that component precisely because of that. Kafka is a nightmare to administer.

In the next release, I would like to see that one troublesome component that causes configuration issues removed.

I have been using Apache Kafka for a couple of years.

The stability of this solution depends on whether it is properly configured. Having said that, Kafka is incredibly complex to configure, set up, administer, and maintain.

My opinion is that Apache Kafka is a scalable solution. In our organization, there are hundreds of thousands of users using Kafka.

The initial setup was extremely complex. In our case, it took a team of 12 two months to deploy.

These systems were installed by somebody else, not me.

I would advise others to schedule a month or two to just set it up and have it up and running.

There are other options. For example, Databricks is a Kafka alternative. We decided to go with Kafka because one of our clients already chose Kafka.

While evaluating, we found out Databricks is more expensive, for the level of activity that Kafka handles (in this case, millions of requests per day). Databricks could do it, but it would be overly expensive.

I would rate Apache Kafka's pricing a seven out of ten, with one being cheap and 10 being very expensive.

Since it has become so popular, large enterprises especially want to do it. For smaller enterprises, Kafka would probably be too expensive because they would have to hire people to maintain it.

I would rate the Apache Kafka solution a seven out of ten.

Apache Kafka can be deployed on the cloud and on-premise.

We use Apache Kafka internally to build a service on a cluster. Additionally, we use the intermediate persistence layer for events. There are many teams who leverage it as a message queue and further their microservice connections.

Apache Kafka has helped out the organization because we leverage it for all our eCommerce real-time analytics use cases.

The most valuable feature of Apache Kafka is Kafka Connect.

Apache Kafka could improve data loss and compatibility with Spark.

I have been using Apache Kafka for approximately five years.

Apache Kafka is stable.

The scalability of Apache Kafka could improve.

We have approximately 10,000 users using this solution.

The support from Apache Kafka could improve. Their engineers at times do not know what the solutions can do.

We previously used IBM MQ, Tipco, and AMQ.

The initial setup of Apache Kafka was complex. We were able to simplify it by doing registry-based integration of the services.

Apache Kafka has given a substantial return on investment.

The number of people required for maintenance depends on the team. They need a centralized team to offer Apache Kafka and services. Each team does have knowledge of Kafka.

This solution has a lot of features and there is no other solution on the market that has similar advanced features. It is a very good solution.

I rate Apache Kafka an eight out of ten.

We use Apache Kafka to process messages, specifically payment type messages, and incorporate the data from those messages into our analytics and reporting. It utilizes data from additional sources in real-time for our analytics and reporting purposes.

The real-time nature and the ability to use multiple offsets are the most beneficial features of Apache Kafka for our data streaming needs. This allows us to replay the same messages using different offsets. Although I haven't set up Kafka's scalability and fault tolerance myself, I know it can be configured with redundancy and fallback options. We primarily consume the messages using different clients, so the setup for fault tolerance and redundancy is transparent to us.

It's quite flexible and comparable to other solutions like ActiveMQ in terms of features and guarantees, especially with offsets for message handling. While ActiveMQ may be preferred in some use cases requiring guaranteed message delivery, Kafka's offset management provides similar functionality. Overall, I would recommend Kafka for real-time data streaming without hesitation.

The main challenge we faced while integrating Apache Kafka with other tools was setting up SSL and securing connections. Managing certificate changes and ensuring all clients connect smoothly, especially outside Kubernetes environments, posed ongoing challenges. Once initially set up, maintaining and sharing these security configurations became more manageable, but ensuring compatibility across different environments remained a continuous effort.

I have been using Apache Kafka for the last five years.

I would rate the scalability nine out of ten.

The technical support, typical for open-source solutions, is also responsive and helpful.

Positive

We switched to Kafka from paid solutions like IBM's MQ due to cost considerations, finding Kafka's multiple offsets and popularity advantageous.

As for pricing, Kafka is open-source, so it's free to install and use.

We use Kafka for Elastic Stack and Kafka SCRAM login.

I have many users of Apache Kafka. It's like a subject to study in enterprises. However, we have not decided if the systems should generalize Apache Kafka for every application and every IT system.

We use Kafka for mapping and ThoughtSpot data from one IT system source to the destination. We also prefer it to exchange data from our internal IT systems.

Kafka is a new method we opted to apply to our need for data exchange. Also, we use the solution's integration capabilities.

Irovement-wise, I would like the solution to have more integration capabilities. Also, the solution's setup, which is currently complex, should be made easier.

I have experience with Apache Kafka.

It is a stable solution.

A lot of my experience indicates that Apache Kafka is scalable. We can have ten or even fifty hundred users on the solution. So, it's possible because we are a big enterprise.

I have experience with Apache Kafka's technical support.

The solution's initial setup process was complex. The deployment process took three or four years.

Right now, I can't deliver the planning process required for deployment.

For deployment and maintenance, we have a manager and an operational person. However, I can't give an exact count of the people required for deployment and maintenance.

To be able to recommend Kafka to others, especially considering every context, we will have to set a benchmark and compare Kafka with other tools.

I rate the overall solution a seven out of ten.

We have a camera monitoring security system, in which we post messages onto the queue, which involves various steps in processing the message, like checking for the number of clients, running it against the police data, etc. So Apache Kafka is a security application with many types of consumers. We set up a workflow system with different sites, which works well.

As a software developer, I have found Apache Kafka's support to be the most valuable. The support team sends available information regarding the library and how to use the plugins. The solution is easy to integrate with any of our systems. We have other alternatives, but this is the one that seems to be the most popular database support.

The solution should be easier to manage. It needs to improve its visualization feature in the next release.

I have been using it for three years.

It is a stable solution. We never faced any issues. I rate it a ten out of ten.

It is a scalable solution. We set up a category with different consumers balancing things, which works as I thought.

I did not contact the technical support as it was not required.

We used Linksys for visualization along with Confluence, but there needed to be more value. For us, Apache Kafka is the best solution based on the support and third-party systems as it builds our subsystems around because we have a lot of development teams.

The initial setup was straightforward because I've got a lot of experience in this field. But even for a junior person, it would be fine. There are so many resources, and it's very well documented as they are a premium service provider. So it makes the setup just easier.

The deployment takes a few days.

We set up a free cluster for this service because we use a lot of data. We use ZooKeeper to secure different products for instruction with the cluster. But, it was easy as it is a popular product, and much information is available. It can download data, like fifty gigs per day. We can effectively handle it all as well. I never developed any issues.

It's a premium product, so it is not price-effective for us.

Apache Kafka is an out-of-the-box, reliable solution. For people in the fiber business, we need a reliable solution, and this solution is hundred percent reliable. If it is set up correctly, it hardly has any issues due to the more extensive user base; even if there are issues, it is sorted by the community. I rate it nine out of ten.

I primarily use Kafka in the investment banking sector to update prices and inform clients of updates.

Kafka's most valuable feature is its user-friendliness.

There are some latency problems with Kafka.

I've been using Kafka for more than three years.

I would give Kafka a rating of seven out of ten.

Apache Kafka is used for more than only a messaging bus but also served as a database to store information. It functioned as a streamer, similar to ETL, to manipulate and transform events before migrating them to other systems for use. The database could also act as a cache. Apache Kafka is used as a database broker, streamer, and source of truth for multiple systems due to its ability to maintain events for at least 10 days. It provided both synchronous and asynchronous communication, making it a complex system that would be easier to understand through diagrams or sketches.

We use reactive frameworks.

From my experience with Apache Kafka, one of the most notable advantages is its ability to maintain a comprehensive record of historical data that includes every update, alteration, and version of information, unlike a conventional relational database. This feature allows for seamless tracking and analysis of the progression and transformation of the data over time, enabling users to easily review and analyze the history of the information.

The solution has the capability for various systems to effortlessly interact with one another without prior knowledge of their existence, current operational status, or specific configurations. By utilizing service buses and dynamic integration, data can be distributed across networks and retrieved in a way that is most suitable for each system's requirements. In addition, Apache Kafka allows for the modification of data to provide diverse clients, consumers, or observers with unique and varying data. The replication of data can produce multiple versions, and this data can be adjusted to fit various needs. With the use of probes, one can alter the behavior of the transformation process, thereby changing the way in which data is transformed and the output produced. Overall, working with Apache Kafka has brought about an array of benefits, enabling seamless system interactions and allowing for the customization and modification of data to meet individual requirements.

There are numerous possibilities that can be explored. While it may be challenging to fully comprehend the potential advantages, one key aspect is the ability to establish a proper sequence of events rather than simply dealing with a jumbled group of occurrences. These events possess their own timestamps, even if they were not initially provided with one, and are arranged in a chronological order that allows for a clear understanding of the progression of the events.

There have been some challenges with monitoring Apache Kafka, as there are currently only a few production-grade solutions available, which are all under enterprise license and therefore not easily accessible. The speaker has not had access to any of these solutions and has instead relied on tools, such as Dynatrace, which do not provide sufficient insight into the Apache Kafka system. While there are other tools available, they do not offer the same level of real-time data as enterprise solutions.

One additional area that I think could benefit from improvement is the deployment process on OpenShift. This particular deployment is quite challenging and requires the activation of certain security measures as well as integration with other systems. It's not a straightforward process and typically requires engineers who are highly skilled and have extensive experience with Apache Kafka to carry out these tasks. Therefore, I believe that there is a need for progress in this area, and some tools that can provide information, assistance, and help make the whole process easier would be greatly appreciated.

I have been using Apache Kafka for approximately four years.

The solution is stable if you have set it up correctly.

Apache Kafka is a scalable solution.

I have not escalated any questions to technical support because Apache Kafka is an open-source system. However, Confluent and other companies sell support and enterprise solutions to make it more convenient and streamline the work. They offer tools, such as a monitoring tool with a visual interface, which provides a lot of information and buttons to press for correction or change without touching the code. Each of those buttons hypothetically could have helped the situation, but it is unclear what they do exactly, it is best to call the data center and ask. If you buy their service, you have access to all the enterprise comforts.

Setting up Apache Kafka is, is not an easy task, especially when trying to containerize it and make it controllable. This is because Apache Kafka has its own distributed mechanism for staying alive, checking readiness, replicating, and scaling. Ensuring that it complies with Kubernetes or OpenShift Orchestrator requires careful attention, as there is a risk of two masters attempting to perform the same task and ultimately undoing each other's work.

In comparison to Kubernetes, OpenShift is a highly skilled and advanced implementation infrastructure that automatically manages and orchestrates all the steps required for an application setup. It operates at a higher level of abstraction and eliminates the need for manual operations that are required with Kubernetes. While Kubernetes can run an application with some pipeline and configuration, OpenShift takes care of everything from finding the required images to creating ports and connecting databases. Although manual changes can be made, it's not necessary as OpenShift offers a much more course-grained management approach.

One skillful DevOps engineer can implement the solution.

Apache Kafka is an open-source solution.

The maintenance of Apache Kafka is crucial due to the complexity of the system with numerous microservices and systems communicating through Apache Kafka, requiring proper integration and configuration to prevent overloading and ensure a healthy cluster. The task is not easy and requires knowledge of the various adjustable parameters, as misadjusting even one of them can greatly slow down the cluster. For example, if the consumer group changes frequently, the messages must be regrouped and reassigned, causing significant delays. Therefore, configuring Apache Kafka correctly is essential to avoid high latency issues.

I would strongly suggest others give Apache Kafka a chance and explore the various advantages that it can offer, especially since it should not be perceived as a message bus or broker but rather an enterprise bus designed for data manipulation. It has the ability to transform data, store and reject it, and even maintain different versions of the same data simultaneously. Moreover, it operates on a pull mechanism rather than a push mechanism, which takes away the risk of losing data and places the responsibility for data loss on the consumer. On the other hand, it also ensures that the data is always available within the specified window and allows for easy replication of the past, which is extremely helpful in situations such as those involving a hacked bank database. With Apache Kafka, you can efficiently go back in time, obtain the required status and events, and make changes accordingly, without the need to go through each transaction separately. Thus, using this solution can make data management much more efficient and convenient.

I rate Apache Kafka an eight out of ten.

In order to improve its user-friendliness, engineer-friendliness, and DevOps-friendliness, the system must undertake various tasks, such as enhancing the overall operation and configuration, ensuring seamless integration with other systems, and adapting to security layers in a more comprehensive and generic manner. This will require significant efforts to make the system more functional, secure, and efficient.

I use Kafka to send network packets from different sources to my cluster. We have around 10 users at my company.

Kafka is scalable. It can manage a high volume of data from many sources.

The interface has room for improvement, and there is a steep learning curve for Hadoop integration. It was a struggle learning to send from Hadoop to Kafka. In future releases, I'd like to see improvements in ETL functionality and Hadoop integration. 

I have used Kafka for around six months.

I rate Apache Kafka seven out of 10 for stability. 

I rate Kafka eight out of 10 for scalability. 

I rate Apache support six out of 10. It was hard to find the information I needed. 

Neutral

Before Kafka, I sent feeds directly to Hadoop.

I initially found Kafka difficult to set up, so I would rate it about five out of 10 for ease of setup. After I learned more about the platform, I would rate it eight out of 10. It is deployed on-premises over a cluster of three or four PCs. You can deploy Kafka in a few hours with one person. 

Kafka is open source. 

I rate Apache Kafka eight out of 10. I would recommend it to others.