I'm not sure exactly what benefit we have because we are using multiple AWS tools. We have AWS DMS, which is also the same as Amazon MSK, and we have Fivetran, which is a third-party website providing similar features to Amazon MSK. Since we are working in the financial domain, we want to ensure everything meets security requirements, which is why we chose Amazon MSK. We need to create connectors in Amazon MSK, but there are no default connectors in AWS for that purpose. We need to create custom plugins, which requires writing Java code and uploading it to the S3 bucket. From the S3 bucket, we need to create a custom plugin, making it a lengthy process. In comparison, Fivetran has all plugins readily available. When creating a source for Snowflake and DynamoDB in Fivetran, there's no need to create custom plugins. Everything is available, requiring only the selection of source and target, and configuration of URLs and IP addresses. With Amazon MSK, we must create custom plugins and upload zip files to the S3 bucket, making it a long and hectic process.
The cost of using Amazon MSK is high, which is a significant disadvantage, as the increase in cloud costs by 50% to 60% does not justify the savings. There were no other notable issues.
Horizontal scale-out is actually not easy, making it an area where improvements are required. There is a limitation on the size of the message that you can put and it needs to be increased.
Senior Software Engineer at a recreational facilities/services company with 1,001-5,000 employees
Real User
Top 10
2024-06-12T19:48:00Z
Jun 12, 2024
In my opinion, there are areas in Amazon MSK that could be improved, particularly in terms of configuration. Initially setting it up and getting it connected was quite challenging. The naming conventions for policies were updated by AWS, and some were undocumented, leading to confusion with outdated materials. It took us weeks of trial and error before discovering new methods through hidden tutorials and official documentation.
Integration Solution Architect at a consultancy with 11-50 employees
Real User
Top 5
2024-05-24T18:34:00Z
May 24, 2024
From AWS, I would consider more MSK schema validation is needed. It is easy to integrate if you have an application, but on-topic integration is more complex. You can do it with EvenBridge very easily, but not with MSK. One of the reasons why we prefer Kafka is because the support is a little bit difficult to manage with Amazon MSK. You have Azure and AWS there, but if you use a connector from the market, the support model is marketplace-based, which means you have to go to the developer of this connector.
Senior Data Engineer at a computer software company with 201-500 employees
Real User
Top 5
2024-05-09T20:45:00Z
May 9, 2024
We don't have many use cases involving ingesting large amounts of data and scaling up and down. We have a clear understanding of our data volume, which remains relatively constant throughout the week. While we're aware of other features Amazon MSK offers, we feel confident in our current setup. If our requirements change significantly in the future, we'll reassess our needs and consider adopting Amazon MSK.
The product's schema support needs enhancement. It will help enhance integration with many kinds of languages of programming languages, especially for environments using languages like .NET. They should build schema access as efficiently as developed by Confluent Cloud.
Using Exabeam has provided one valuable aspect, which is the availability of separate agents for Windows and Linux. However, there is a difference when it comes to Linux. On Linux, we need to perform certain command line operations, whereas, for Windows, we have the convenience of an MSI package. One thing I personally feel is that it would be beneficial to have command line installation for Windows as well. Additionally, we have to install the package one by one on each server. It would be really helpful if Amazon MSK could provide a single installation that covers all the servers. If there are multiple servers, I would otherwise need to perform individual installations on each one. All of my servers are either VDI or virtual machines, so I have to log in to each virtual machine, copy the necessary files, and then initiate the installation. In future releases, I would like to see if there a probability of having a one-click installation that can be applied to all servers.
There's a bit of operational overhead, but it's not a transmission. It's something you can live with because the setting out of the Kafka cluster is less managed compared to Kinesis. Once it's set up, everything goes smoothly without going through that operational overhead again. So maybe that could be some disadvantage, but it's not a transmission related to what you're getting from Kafka. Operational overhead for setting up the Kafka cluster compared to the Kinesis data stream is easier. For Kinesis, you have to do a few clicks and your data stream is ready if I need to do some config to set up the transfer. The solution does not auto-scale. The pricing can be improved.
Team Lead at a tech services company with 10,001+ employees
Real User
2021-03-16T03:58:00Z
Mar 16, 2021
MSK should be easier to integrate with other solutions. It should be more flexible, integration-wise. MSK has a private network that's an out-of-the-box feature. For our case, we needed something public. We can't handle private-only. So, in order to make the data public, we had to do some extra work. It became too risky for us. That's the main reason we stopped using it.
Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that enables you to build and run applications that use Apache Kafka to process streaming data. Amazon MSK provides the control-plane operations, such as those for creating, updating, and deleting clusters.
I'm not sure exactly what benefit we have because we are using multiple AWS tools. We have AWS DMS, which is also the same as Amazon MSK, and we have Fivetran, which is a third-party website providing similar features to Amazon MSK. Since we are working in the financial domain, we want to ensure everything meets security requirements, which is why we chose Amazon MSK. We need to create connectors in Amazon MSK, but there are no default connectors in AWS for that purpose. We need to create custom plugins, which requires writing Java code and uploading it to the S3 bucket. From the S3 bucket, we need to create a custom plugin, making it a lengthy process. In comparison, Fivetran has all plugins readily available. When creating a source for Snowflake and DynamoDB in Fivetran, there's no need to create custom plugins. Everything is available, requiring only the selection of source and target, and configuration of URLs and IP addresses. With Amazon MSK, we must create custom plugins and upload zip files to the S3 bucket, making it a long and hectic process.
The cost of using Amazon MSK is high, which is a significant disadvantage, as the increase in cloud costs by 50% to 60% does not justify the savings. There were no other notable issues.
Horizontal scale-out is actually not easy, making it an area where improvements are required. There is a limitation on the size of the message that you can put and it needs to be increased.
In my opinion, there are areas in Amazon MSK that could be improved, particularly in terms of configuration. Initially setting it up and getting it connected was quite challenging. The naming conventions for policies were updated by AWS, and some were undocumented, leading to confusion with outdated materials. It took us weeks of trial and error before discovering new methods through hidden tutorials and official documentation.
From AWS, I would consider more MSK schema validation is needed. It is easy to integrate if you have an application, but on-topic integration is more complex. You can do it with EvenBridge very easily, but not with MSK. One of the reasons why we prefer Kafka is because the support is a little bit difficult to manage with Amazon MSK. You have Azure and AWS there, but if you use a connector from the market, the support model is marketplace-based, which means you have to go to the developer of this connector.
We don't have many use cases involving ingesting large amounts of data and scaling up and down. We have a clear understanding of our data volume, which remains relatively constant throughout the week. While we're aware of other features Amazon MSK offers, we feel confident in our current setup. If our requirements change significantly in the future, we'll reassess our needs and consider adopting Amazon MSK.
The product's schema support needs enhancement. It will help enhance integration with many kinds of languages of programming languages, especially for environments using languages like .NET. They should build schema access as efficiently as developed by Confluent Cloud.
The configuration seems a little complex and the documentation on the product is not available.
Using Exabeam has provided one valuable aspect, which is the availability of separate agents for Windows and Linux. However, there is a difference when it comes to Linux. On Linux, we need to perform certain command line operations, whereas, for Windows, we have the convenience of an MSI package. One thing I personally feel is that it would be beneficial to have command line installation for Windows as well. Additionally, we have to install the package one by one on each server. It would be really helpful if Amazon MSK could provide a single installation that covers all the servers. If there are multiple servers, I would otherwise need to perform individual installations on each one. All of my servers are either VDI or virtual machines, so I have to log in to each virtual machine, copy the necessary files, and then initiate the installation. In future releases, I would like to see if there a probability of having a one-click installation that can be applied to all servers.
There's a bit of operational overhead, but it's not a transmission. It's something you can live with because the setting out of the Kafka cluster is less managed compared to Kinesis. Once it's set up, everything goes smoothly without going through that operational overhead again. So maybe that could be some disadvantage, but it's not a transmission related to what you're getting from Kafka. Operational overhead for setting up the Kafka cluster compared to the Kinesis data stream is easier. For Kinesis, you have to do a few clicks and your data stream is ready if I need to do some config to set up the transfer. The solution does not auto-scale. The pricing can be improved.
Amazon MSK could improve on the features they offer. They are still lagging behind Confluence.
MSK should be easier to integrate with other solutions. It should be more flexible, integration-wise. MSK has a private network that's an out-of-the-box feature. For our case, we needed something public. We can't handle private-only. So, in order to make the data public, we had to do some extra work. It became too risky for us. That's the main reason we stopped using it.