StreamSets Reviews

My primary use case with StreamSets is to integrate large data sets from multiple sources into a destination. We also use it as a platform to ingest data and deliver data for database analytics.

One major benefit that we have realized with StreamSets is that we are now able to run pipelines that scale horizontally, instead of using a static service to host the service. This has improved efficiency and reduced our workload by around 85 percent. Initially, we started out with around 40 users. Now, there are 100 users. We have definitely scaled up, in terms of usage, with StreamSets.

The fact that it is a single centralized platform saves us a lot of time. It's very intuitive and very effective, saving us a lot of resources with its built-in capabilities. No manual intervention is needed, and nobody needs to oversee it. It's an "all-in-one" deal for us. We are able to save 15 to 18 hours per week. Tasks that required three people can be done with StreamSets itself.

And with its ability to integrate large data sets, we are now able to pull thousands of records instantly, thereby reducing the need to do some complex coding for this asset. That has also been a very big plus for us.

We also use it to connect our Apache Kafka with data lakes and, as a result, this connection has gotten much more efficient and quicker for us. The overall efficiency has also drastically improved for us with this. Connecting these enterprise systems using StreamSets is pretty easy. The StreamSets platform is very straightforward. There is no major coding required, so any non-technical person can also do it.

Without the need for any complex coding at all, we are able to pull records. The records are vast and very large and pulling them usually requires coding, but the fact that there is literally no coding required is a very big plus for us. Once you start to code, there is a lot of time involved and a lot of QA involved, but all of that is eliminated here.

And it has definitely helped us break down data silos. With our large amount of data, we have different data formats, and as a result, there are data silos that are present by default. With StreamSets, we were able to completely eliminate that because StreamSets has become a centralized system for us to accommodate everything. We have been able to get a single, centralized view of all our data.

We have a lot of different data formats, and transforming them manually without any tool or system is a cumbersome and frustrating process. We use StreamSets to do that. It has made that process much more elegant and efficient for us.

For me, the most valuable features in StreamSets have to be the Data Collector and Control Hub, but especially the Data Collector. That feature is very elegant and seamlessly works with numerous source systems. 

Also, the intuitive canvas for designing all the streams in the pipeline, along with the simplicity of the entire product are very big pluses for me. The software is very simple and straightforward. That is something that is needed right now. 

Apart from that, the user interface of StreamSets is very good. It's very user-friendly and very appealing. Moving data into modern analytics platforms is a very straightforward procedure. There is no difficulty involved in it.

In addition, the ETL capabilities of StreamSets are also very useful for us. We are able to extract and transform data from multiple data sources into a single, consistent data store that is loaded into our target system.

In terms of the product, I don't think there is any room for improvement because it is very good. One small area of improvement that is very much needed is on the knowledge base side. Sometimes, it is not very clear how to set up a certain process or a certain node for a person who's using the platform for the first time.

Some visual explanation or some visually appealing knowledge-based content would be very good. That is something that I could have done with, once I started using it, because I found it very difficult.

I have been using StreamSets for about a year.

It is definitely a stable product. In fact, it is one of the top products in the market in that particular category. We have not faced any stability issues so far, in terms of server speed, latency, or deployment.

It's a scalable product. In our company, the platform is used across seven teams in our organization.

A couple of more teams are evaluating StreamSets in our organization. They're running things and asking for some feedback from our side as well. There are plans to expand our use of it.

I have been in contact with their technical support and I would rate them very highly. They're very knowledgeable and patient. That is something that I like very much. For a very new user, it's not very easy to understand and we contact the support team over email.

We do have a relationship manager as well, who acts as the central point of contact for us. They're very prompt, knowledgeable, and friendly.

Positive

This was one of the first products we used.

Within about three months we were able to see benefits from the system. We saw a lot of time being saved, and about a 30 percent increase in our overall efficiency.

Apart from reducing our workload and improving our efficiency, we saw a 12 percent increase in our revenue last year after we implemented StreamSets. I know people will definitely see a return investment on their money from it.

From what I hear from my team, I believe it's moderately priced because they're happy with the pricing.

Server update maintenance is required, but that is minimal. Any product would require that type of maintenance. I don't think we are investing a lot of time and money in maintenance. The maintenance is just another cost for us. We have only two guys working on the maintenance part of the software.

It's a very intuitive product, modern, and very user-friendly in terms of the UI. Almost all our requirements have been met by StreamSets and we don't have any complaints so far.

I would recommend starting to use it as soon as possible. No tool is perfect. You have to choose the best of the lot. I certainly believe StreamSets is at the top of the ladder when it comes to similar software.

My biggest lesson from using StreamSets is that data integration can be done much more easily now. I only knew that after starting to use StreamSets. When it comes to data integration from multiple sources, and having multiple destinations, people always assume it's a time-consuming, cumbersome project. But once we started using StreamSets, all those assumptions were broken. It's very straightforward and elegant software.

I was working on an integration project where I was using the StreamSets platform. I was looking at both their data collector and their transformer. The idea was to integrate it with AWS SageMaker Canvas. Both of them are what they call no-code options. StreamSets is for data pipelining, managing your data flow, and transforming your data. SageMaker is AWS, and Canvas is basically their no-code option for machine learning.

I was trying to connect it to a data object repository. For AWS, that's a specific managed service called S3. I wasn't trying to run it with a data warehouse.

It's still in the trial stage. I don't get a 30-day trial period or anything like that. I just got to write about what's involved and then see if that's something that justifies the use case for going ahead and purchasing the license for it.

It enables you to build data pipelines without knowing how to code. It abstracts away the need for Spark or anything like that. This ability is highly important because it reduces development time.

It saves time because you don't have to write code. 

It saves money by not having to hire people with specialized skills. You don't need Spark or anything like that for doing the same thing.

It helps to scale your data operations. You can get to the execution engine and provision bigger machines or bigger clusters. You can scale out to however much data you need to scale out to.

The most valuable would be the GUI platform that I saw. I first saw it at a special session that StreamSets provided towards the end of the summer. I saw the way you set it up and how you have different processes going on with your data. The design experience seemed to be pretty straightforward to me in terms of how you drag and drop these nodes and connect them with arrows.

The execution engine could be improved. When I was at their session, they were using some obscure platform to run. There is a controller, which controls what happens on that, but you should be able to easily do this at any of the cloud services, such as Google Cloud. You shouldn't have any issues in terms of how to run it with their online development platform or design platform, basically their execution engine. There are issues with that.

It can break down data silos within the organization. One person can do the whole thing with StreamSets and SageMaker Canvas, but it hasn't yet had any effect on our operations or business because it's one of those situations where you can either get a demo from them or you basically have to go to one of these sessions and they give you temporary credentials and try to work with your use case. Personally, I would change their model a bit and give a two-week trial license for a cloud platform at the very least. You can then try to get something to work or call up their technical department and say, "Look, I've been evaluating this thing for the last few days. I don't know exactly how to resolve this issue."

I started using it in June of this year. 

The whole issue of the execution engine needs to be better resolved. If you pick a cloud, why isn't it working with this cloud? Or what do I need to do to get it to work with one specific cloud service if it can be deployed across multiple clouds?

It seems pretty highly scalable to me. That's not going to be an issue. Just the administration of it could be an issue.

It's currently being used in a dev department for machine learning. It's being used by the business analyst team.

I haven't contacted their support.

AWS has native solutions. There are AWS Data Wrangler and others that come bundled with their services, like AWS Glue. We haven't yet switched to StreamSets. It's still in the evaluation stage, but the no-code and the drag-and-drop option with a GUI are some of the things that seem to resonate with people. 

I was involved in its setup. I was the one who basically had to try to get it to run with whatever process or custom processor I developed. 

It was complex to set up. I had to go to the sessions. On a couple of occasions, I was doing it directly from the cloud platform, and apparently, that wasn't the way to do it. You have to go through their universal designer platform first. 

In terms of maintenance, once you're deployed from the cloud, that's all handled for you. It's managed for you directly from the cloud service. So, you don't have to worry about that. They maintain their design platform.

I didn't use any consultant.

I didn't get into that with the StreamSets representative. It seems to be pay-as-you-go, but I don't know exactly how they do it.

Alteryx is another option. It's a similar tool, and it looks almost the same as StreamSets. Alteryx is something that's available for any cloud. It doesn't matter which cloud. You go on the various clouds, and you look and see what they have.

To those evaluating this solution, I would advise looking into how it integrates with the cloud service that they're going to try it with. Does it naturally integrate better with AWS or Azure? It's one of those situations.

I used StreamSets' ability to move data into a modern analytics platform. That's what the AWS SageMaker Canvas is. It's like predictive analytics. In terms of ease of moving data into this analytics platform, doing the design on the StreamSets platform is one thing, but having the execution engine and getting that provision is a totally different ball game. Basically, that's where its limitation comes in.

Overall, I would rate it a seven out of ten. The issue that was never resolved for me was if you're running a compute or execution engine on AWS versus Azure versus GCP, how does that integration work because that has got nothing to do with StreamSets? That is outside of StreamSets. You're now dealing with the cloud service, and there's a good reason for that.

The project which I work on is developed in StreamSets and I lead the team. I'm the team leader and the Solution Architect. I also train my juniors and my team.

For the last year and a half, I’ve been using this tool and this tool is very effective for data processing from source to destination. This tool is very effective and I developed many integrations in this tool.

The solution is really effective.

It's very effective in project delivery. This month, at the end of June, I will deploy all the integrations which I developed in StreamSets to production remit. The business users and customers are happy with the data flow optimizer from the SoPlex cloud. It all looks good.

Not many challenges are there in terms of learning new technologies and using new tools. We will try and do an R&D analysis more often.

Everything is in place and it comes as a package. They install everything. The package includes Python, Ruby, and others. I just need to configure the correct details in the pipeline and go ahead with my work.

The ease of the design experience when implementing batch streaming and ETL pipelines is very good. The streaming is very good. Currently, I'm using Data Collector and it’s effective. If I'm going to use less streaming, like in Java core, I need to follow up on different processes to deploy the core and connect the database. There are not so many cores that I need to write.

In StreamSets, everything is in one place. If you want to connect a database and configure it, it is easy. If you want to connect to HTTP, it’s simple. If I'm going to do the same with my other tools, I don’t need many configurations or installations. StreamSets' ability to connect enterprise data stores such as OLTP databases and Hadoop, or messaging systems such as Kafka is good. I also send data to both the database and Kafka as well.

You will get all the drives that you will need to install with the database. If you use other databases, you're going to need a JDBC, which is not difficult to use.

I'm sending data to different CDP URL databases, cloud areas, and Azure areas.

StreamSets' built-in data drift resilience plays a part in our ETL operations. We have some processors in StreamSets, and it will tell us what data has been changed and how data needs to be sent.

It's an easy tool. If you're going to use it as a customer, then it should take a long time to process data. I'm not sure if in the future, it will take some time to process the billions of records that I'm working on. We generally process billions of records on a daily basis. I will need to see when I work on this new project with Snowflake. We might need to process billions of records, and that will happen from the source. We’ll see how long it needs to take and how this system is handling it. At that point, I’ll be able to say how effectivly StreamSets processes it.

The Data Collector saves time. However, there are some issues with the DPL.

StreamSets helped us break down data silos within our organizations.

One advantage is that everything happens in one place, if you want to develop or create something, you can get those details from StreamSets. The portal, however, takes time. However, they are focusing on this.

StreamSets' reusable assets have helped to reduce workload by 32% to 40%.

StreamSets helped us to scale our data operations.

If you get a request to process data for other processing tools, it might take a long time, like two to three hours. With this, I can do it within half an hour, 20 or 30 minutes. It’s more effective. I have everything in one place and I can configure everything. It saves me time as it is so central. 

If you use JDBC Lookup, for example, it generally takes a long time to process data.

StreamSets enables us to build data pipelines without knowing how to code. You can do it, however, you need to know data flow. Without knowing anything, it's a bit difficult for new people. You need some technical skills if you are to create a data pipeline. When procuring the data pipeline, for example, you need the original processor and destination. If you don't know where you're going to read the data, where to send the data, and if you have to send the data, you have to configure it. If the destination you're looking for is some particular message permit or data permit, then you should write your own code there. You need some knowledge of coding as StreamSets does not provide any coding.

StreamSets data drift resilience has not exactly reduced the time it takes for us to fix data drift breakages. A lot of improvements are required from StreamSets. I'm not sure how they're planning to make it happen. There are some issues in the case of data processing, and other scenarios.

If the data processing in StreamSets takes a long time as compared to the previous solution, then we will reconsider why we use StreamSets.

I've been using this StreamSets for the last two years.

In terms of stability, there have been one or two issues. Good people work on the solutions when we have issues. However, sometimes we don't get a good solution. 

As a user, I expect a lot more and that the solution will come quicker as compared to keeping projects on hold or keeping them for a long time. If they do not have any solution, then we can plan accordingly how to use the other processors. They just need to let us know quickly. 

The scalability is good.

We do plan to increase usage. 

In terms of technical support, they generally do a detailed analysis from their end. They always try to give a proper solution. However, sometimes, they won't get to any proper solution. They'll come back and look into it and sometimes it takes time. If they can speed up the process a little bit that would be ideal. We are always sitting on the edge. If we don't get a proper response from them, then it will be very difficult for us to answer to higher management. 

Neutral

This is my first solution of this kind. Previously, I was working in open source systems, with scripting, et cetera. This is the first time I've worked in the data area. I've got full support. As a new data user, I'm still getting used to it.

The setup is straightforward, it's not complex and it is simple. 

We treat it like a pipeline. We are not writing code and putting things in. In the case of a pipeline, you can export it and input it, or you can make it a pipeline. It can be auto-deployed into a respective environment. That's what we did.

We have different destinations we need to send to. We aren't using a single destination. In that sense, we do have multiple computations. We set up, send the data and do the deployments. 

There is occasional maintenance needed. Sometimes, if something goes wrong, we'll have to correct the data. We just check here and there for the most part.

We did not need an integrator or consultant to assist with the setup. 

As a team, we do the deployment. We won't send it to others, whatever we develop, we will test and deploy. We already have the system in place and it is really helpful for the deployment of the solution.

I haven't seen an ROI. 

It's not exactly saving us money as it's a new tool. If I'm going to hire someone new, I will not hire based on the StreamSets tool or some specific tools, and I might save money right away. However, I'm spending time on my side. StreamSets is not being used by many horizons. In some places in Europe, fewer companies are using StreamSets. People should get to know StreamSets and they should get some expertise in the area, the way AWS and Azure do. I’m spending a lot more time and therefore I’m not saving money. That said, I’m also not losing money.

Higher management handled the licensing. However, I can't say how much it costs. I'm more on the user side.

I did not evaluate other options. 

I have not yet used StreamSets' Transformer for Snowflake functionality. I created one POC, not with Snowflake, however, I'm going to use Snowflake in my next project.

I'd rate the solution seven out of ten. They are doing a good job. Using this solution I can feel the data and see the user flows. 

If you are going to withdraw on-premise, and you're just copying the data to a table, you're not going to see how much data has been copied. With this, I'm seeing how much data has been transferred, and where the processor is. It gives a clear picture with metric details and notifications. That's the reason I used this tool for the last two years. 

We typically use it to transport our Oracle raw datasets up to Microsoft Azure, and then into SQL databases there.

It is really easy to set up and the interface is easy to use.

We found it pretty easy to transform data.

The online documentation is pretty good.

We've seen a couple of cases where it appears to have a memory leak or a similar problem. It grows for a bit and then we'd have to restart the container, maybe once a month when it gets high.

We have been using StreamSets for about one year. We may have been experimenting with it slightly before that time.

Other than the memory issue that we occasionally see, the stability has been really good.

We haven't seen a problem with scaling it.

I haven't had to deal with technical support. We would first check the online documentation or web documentation, and usually found what we needed. We haven't had to call them.

Prior to using StreamSets, we were using Microsoft CDC (Change Data Capture). It was a fairly old product and there were lots of workaround and lots of issues that we had with it. We were looking for something more user-friendly. It was pretty stable, so that was not an issue. 

This product was a lot easier to use than the one we had before it. It took us half an hour and we were set up and running it, the first time.

We are running the community version right now, which can be used free of charge. We were debating whether to move it to the commercial version, but we haven't had the need to, just yet.

It performs very well. The main use is to extract information from some of our Kafka topics and put it in our internal systems, flat files, and integration with Java.

It facilitates the consumption of the data in batch mode to the system where it is required. We don't do a lot of transformations or joining or forking of the information. It's more point-to-point connectivity that we implement over StreamSets.

The ease of configuration for pipes is amazing. It has a lot of connectors. Mainly, we can do everything with the data in the pipe. I really like the graphical interface too. It's pretty nice.

I would like to see it integrate with other kinds of platforms, other than Java. We're going to have a lot of applications using .NET and other languages or frameworks. StreamSets is very helpful for the old Java platform but it's hard to integrate with the other platforms and frameworks.

StreamSets works great for batch processing but we are looking for something that is more real-time. We need latency in numbers below milliseconds.

It's pretty stable. StreamSets has been up and running up for months without any intervention in terms of the operations team. It's great.

I don't know if they can implement some kind of high-availability. I really don't go deep into that kind of configuration because, with only one node and running as stably as it is, we have no problem with that. But for critical operations, I'd like to know if I can facilitate some kind of high-availability, in case one of the nodes go down.

It's pretty scalable.

I don't use support. I mainly use the community or web searches; self-learning.

The initial setup is pretty straightforward.

If you are looking for something to do batch processing in Java, this is the right solution. We did the exploration when we were trying to implement a batch processing system and decided that StreamSets is the best for that. If you're looking for real-time, you may want to look at another system or the next version of this one.

Because of the kind of system that we need to implement with this kind of solution, the most important factors I look at when selecting a vendor are things like latency and real-time processing.

I would rate it at nine out of 10. What would make it a 10 would be, as I said, I'd like to have more integration with other kinds of languages or frameworks and also more real-time processing, not batch.