What is our primary use case?
My main use case for Kpow for Apache Kafka is navigating and inspecting and checking out the message flow in the different applications that our system supports. Our team currently builds an application which consumes messages and also sends out messages. It has an inbound and also has an outbound. We also talk to external teams such as Salesforce and Cerner, which listen to our messages. In the current stage, imagine a chain of applications wherein one listens and sends the message out, consuming an input and giving back an output. Checking that use case is mainly what we are doing with the different applications we support.
A specific example of how I use Kpow for Apache Kafka is in my company, particularly in the group domain. We use Kpow for Apache Kafka to identify the topic. Based on the topic, we identify which domain the message belongs to. We query and filter out the messages by selecting the cluster, whether it is test, dev, or pre-prod. Then we select the topic and filter by the message unique reference or message ID and select the time window. We analyze results to validate the message headers and the JSON payload, and inspect the structure. We also use it for our data loads to check for the lags on a topic and the rate of reading the messages. In troubleshooting and debugging, if one application in the group domain had the group transformer send out messages to the group inbound app, but we see that the group inbound app did not receive the message, common use cases such as missing data, incorrect data, or delayed data are helpful to debug and filter with Kpow for Apache Kafka.
I find myself using Kpow for Apache Kafka very frequently. Just before deploying our application or deploying a feature, we usually do a round of testing, which usually involves an end-to-end test of the complete life cycle of a message. I use it on a weekly basis and in our company, it is split across five or six environments including dev, test, test batch, training, pre-prod, and prod. Performing data loads and debugging is pretty common, and I use it on a weekly basis.
I want to add two things about my main use case. One is a fun fact. I recently got to know what Kpow stands for; it stands for Kafka power, which aligns with the web UI tool that is really powerful and user-friendly and helps in monitoring and observability, and even fast debugging in a cross-team setting. The second thing I would mention is that we sometimes use Kpow for Apache Kafka to validate the schemas. On the menu, there is a section for schemas where we can inspect the message structure or the field structure and validate the payloads if it is the field level data structure. That is really helpful as well.
What is most valuable?
I believe that the best features Kpow for Apache Kafka offers include the monitoring and debugging aspect, which is a very handy feature, as well as object management or the life cycle of an object or message, starting from managing the topics, the partitions, the groups, consumer groups, and the schemas, all of which can be configured in Kpow for Apache Kafka. While browsing or inspecting the messages, it offers the KQuery, which can be really handy; you can just filter out based on writing the query, and then it does its magic.
The KQuery feature helps me in my daily work by allowing us to filter a life cycle of a message using an identifier called the message unique reference, which is our primary key. In all the applications, that particular ID is similar, so we know if the message went through a certain app, whether the next one received and processed it, and subsequently if it passed on till the end with a happy path. Adding querying based on the timestamps can be really helpful in which we can filter it down.
Kpow for Apache Kafka has positively impacted my organization and has been very beneficial. Having it across different environments has helped our team manage the project really well, especially working in a cross-functional team that has a dependency with Salesforce and Cerner teams, and also as we work on modernizing a legacy app. Supporting different teams and having a tool that acts as a great UI for Kafka messages is wonderful.
What needs improvement?
To improve Kpow for Apache Kafka, I believe that even though the UI is really user-friendly, it can be made more intuitive. Sometimes I find it a bit laggy or it does not update itself properly, which could be due to the load or the number of messages being processed. There is definitely scope for improvement in terms of the loading or UI aspect. Nonetheless, the features it offers are really great compared to similar tools such as Kafdrop and AKHQ and Conductor.
For how long have I used the solution?
I have been using Kpow for Apache Kafka for more than a year now, as part of my current project at VHI.
What other advice do I have?
Kpow for Apache Kafka has certainly increased collaboration and reduced time in our team. For example, there are instances where we have to say that a certain message hits a particular topic, which in a usual path would go through the application when it runs or something is triggered. Kpow for Apache Kafka also has a feature where we can trigger messages or send out a message based on configuring it on the fly by updating certain configs. I can tweak a message that it received and send it again to debug and verify how the app processes it. That has been really handy, saving us time and helping find issues that could be fatal in a production environment, adding a security layer as well.
Regarding Kpow for Apache Kafka's AI capabilities, I think it is a nice tool in terms of security and authorization. I can only access it via certain roles that I need to be granted, and it is only through an in-house portal that I can access this tool. The data or the messages it supports are secured end to end, so I believe the security aspect is brilliant.
I have not explored Kpow for Apache Kafka's AI capabilities much regarding its accuracy and reliability of output.
My advice for others looking into using Kpow for Apache Kafka would be to get familiarized with the UI. On the menu, there are certain features it supports including topics, partitions, schemas, groups, and clusters. Selecting the clusters was tricky for me based on the different environment selector, whether it is dev, test, or prod. Starting off with browsing the different topics, listing and searching and viewing the partitions and offsets, then inspecting the messages would be helpful. You can filter by different options and view the message and headers, then view the lags to identify any downstream issues.
I would say that Kpow for Apache Kafka is a really powerful tool, as the name suggests, which offers fast visibility, and the debugging it provides makes it structured and organized. It can be handy for supporting different environments. I give Kpow for Apache Kafka a rating of 9 out of 10.
Which deployment model are you using for this solution?
Private Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)