Google Cloud Dataflow Reviews

The primary use case for Google Cloud Dataflow is when a brand has a lot of data and wants to store it in their warehouse. They can use BigQuery to store their data or use big data solutions to store large volumes of data that arrive on a daily basis. Once they have structured data in the platform, they can create a unified persona of the user. This is one use case, and once they have a single persona of the user, it enables us to create numerous use cases for Google Cloud Dataflow, depending on the brand that will be used.

Google Cloud has many streams and products. In Google Cloud, everything is translated in the backend, so we do not have to use services such as Apache Beam. When you want to use Google Cloud Functions, you write the code, and the backend talks to all the libraries or Apache, so we do not need to be concerned about those. We just need to use our functions that translate and have many tools and services readily available.

Google Cloud Dataflow has made it very easy for detailed monitoring and logging features for pipeline performance assessment. For example, if I am using Google Cloud Functions, I can easily see what changes I have done and trace it properly. I can see what is happening with this script, how many users are affected, whether the script is working, what is failing, and how we can rectify issues with proper monitoring.

I feel there could be something that they can introduce, such as when we have data in the tables, a feature that creates a unique persona of the user automatically, so we do not have to do that manually. If this feature is enabled and we have all the sources and data available, it will be much easier for us to maintain everything in one place, rather than sending it again from GCP to another platform just to get the user.

I have been working with Google Cloud for almost three years, and we are using the tool and engaging with it.

I do not see any issues right now or any improvements needed because they are doing their work well.

Compared to other support systems, such as those in Braze, Tealium, Google, and others like Adobe, Google Cloud takes more time because it is a bigger company. We do get a response, but the responsiveness that we get from other brands is quicker.

The initial setup involves basic implementation, but as we move to the next stages with Google Cloud Dataflow, it becomes complex, including more sources and channels.

Since we are an agency, we also work with other agencies for Google Cloud Dataflow, so there are dedicated people allocated to each project. It depends on how the agency contract is based and how many people will work on it. From my team, we have basically five to six people only on GCP, but there are other teams working in other areas as well.

I am still a learner with Google Cloud, and I think they could introduce more features.

With BigQuery, you can either directly go with the authentication method, which involves authenticating your account and calling the BigQuery tables you want to ingest. However, using Google Cloud Dataflow with Cloud Storage can be tricky due to unstructured data.

I do not know much about the session windowing feature in Google Cloud Dataflow, but for each service, such as Cloud Functions, they have specific session windows.

Google Cloud tech stack is cost-effective for small businesses with less data, offering a good deal if properly structured. I would rate this solution an eight out of ten.

Our primary use case for Google Cloud Dataflow is processing both batch and streaming data. This involves integrating Dataflow with Pub/Sub for message ingestion and BigQuery for data warehousing and analysis. This entire pipeline is crucial to our data analytics workflows, and the accessibility of the processed data in BigQuery is vital for various departments across the company, enabling data-driven decision-making at all levels.

Google Cloud Dataflow has significantly improved data democratization across our organization. Dataflow processes data, making it readily available in BigQuery. Team members without specialized data engineering backgrounds can now leverage their existing SQL skills to directly access and analyze this BigQuery data. This accessibility has transformed how different departments operate - from data scientists conducting advanced analytics to finance teams running business reports and production teams optimizing operations. By removing technical barriers to data access, we've accelerated our analysis cycles and enabled faster, data-driven decision-making across all levels of the organization.

As a data engineer, I find several features of Google Cloud Dataflow particularly valuable. The ability to test solutions locally using Direct Runner is crucial for development, allowing me to validate pipelines without incurring the costs of full Dataflow jobs.

The unified programming model for both batch and streaming processing is exceptional - requiring only minor code adjustments to optimize for either mode. This flexibility extends to language support, with robust implementations in both Java and Python, allowing teams to leverage their existing expertise.

The platform's comprehensive monitoring capabilities are another standout feature. The intuitive interface, Grafana integration, and extensive service connectivity make troubleshooting and performance tracking highly efficient. Furthermore, seamless integration with Google Cloud Composer (managed Airflow) enables sophisticated orchestration of data pipelines.

Apache Beam represents a powerful data processing solution that deserves wider recognition in the broader tech community. This technology offers remarkable capabilities for handling data at scale, yet its full potential remains untapped. Through increased advocacy and community engagement, we can help more organizations discover and implement this robust tool, enabling them to benefit from its advanced streaming and batch processing capabilities.

I have been working with Google Cloud Dataflow since November 2022.

Dataflow has been incredibly stable and performed flawlessly in my experience. I haven't encountered any problems, and it's clearly benefiting from Google's robust infrastructure.

Google Cloud Dataflow offers dynamic resource optimization through intelligent autoscaling. By configuring worker machine specifications, users can fine-tune performance and optimize cost efficiency to match their specific processing needs.

Google's proactive updates and the absence of any need for technical support demonstrate excellent service.

Positive

I haven't used any comparable tools previously. AWS Glue and Azure Data Factory are potential alternatives, but I haven't had the opportunity to work with them.

The initial setup was remarkably straightforward. It only involved writing Python code to create a pipeline using either Direct Runner or Dataflow Runner from the terminal. No special requirements or complex configurations were needed.

This project was implemented entirely by our team. We leveraged Ansible for automation to streamline the process.

Google Cloud Dataflow costs are primarily driven by compute resources (worker type and count) and data volume. However, other factors like pipeline complexity also contribute significantly to the total cost.

I haven't worked with other solutions.

Don't let initial challenges discourage you - persistence is key when learning Apache Beam and Dataflow. There are excellent resources available to help you get started, particularly Beam College (https://beamcollege.dev/), which offers structured learning paths and hands-on exercises. The Beam Summit (https://beamsummit.org/) is another valuable opportunity to connect with the community, share experiences, and deepen your knowledge through presentations and workshops.

I highly recommend exploring existing projects as learning examples. The technology's straightforward design and robust integration capabilities make it accessible for developers at all skill levels. Once you grasp the fundamentals, you'll appreciate why I would rate this solution a perfect 10/10.

Positive

Positive

It is used for exporting data, such as customer clicks, customer interactions with emails, and link tracking. The Google Analytics streaming data is used to establish customer behavioral patterns.

The solution is beneficial because it helps to identify potential customers who are interested in our products.

We primarily use Azure Databricks and other related Azure cloud-based components.

We work with various Azure services they provide, including S3 bucket, Lambda functions, and other related tools.

Power BI is another tool that we regularly work with.

It can be improved in several ways.

The system could function in an automated fashion and provide suggestions based on past transactions to achieve better scalability.

Implementing AI-based suggestions for higher scalability would be beneficial.

I have been using this solution for the last three to four years.

The scalability is good. It can be rated as seven out of ten.

Integration capability is a feature that Databricks provides, and we are using Databricks for that purpose.

At this time, I do not have any negative features to report.

Those features are available and working as expected.

That information is confidential as it is part of the company.

On a scale of 1-10, I rate this solution as 8.

We use the solution for data streaming analytics.

The solution is suitable for any environment. We can customize it as per specific requirements.

The solution's setup process could be more accessible.

We have been using the solution for more than eight months.

It is a stable solution.

It is a scalable solution. We have 10-15 users in our organization. We plan to scale it out across all the departments.

We have a dedicated manager in our organization who handles technical issues.  Thus, we reach out to him in case of queries.

The solution's setup process is a learning curve for us. It is a bit tricky overall. We receive vendor assistance in terms of infrastructure and expertise to build it. The time taken for deployment depends on the status of the environment, security and governance setups, etc. It takes a few hours to days to complete. Also, it is self-maintained, saving us the cost of it.

We currently work with two architects from the vendor's team. Also, we have a few engineers, report developers, and model designers from our team to implement the solution.

The solution is cost-effective.

I rate the solution an eight out of ten.

I use the solution in my company for data transmission and data storage.

One of the good features of the product is the overall capacity that it provides to its users. Though the speed of the product is good, the main feature of the product that I like is its capacity.

The authentication part of the product is an area of concern where improvements are required. For some common users, the solution's authentication part is difficult to use.

The scalability of the product is an area of concern where improvements are required.

In the future, the product should be made available at a cheaper rate.

I have been using Google Cloud Dataflow for two or three years. I am an end user of the solution.

The product offers normal stability. Stability-wise, I rate the solution a six or seven out of ten.

It is a scalable solution.

Scalability-wise, I rate the solution a six out of ten.

In our organization, we need around 100 end users of the tool.

I haven't used any other products in the past.

The product's installation process is easy, but in my company, since there are some security issues, the tool should be installed by our IT admin.

The product mainly saves a lot of time, like 70 percent for our organization.

The tool's maintenance part is somewhat easy.

I rate the tool a seven out of ten.

We use Google Cloud Dataflow mainly for batch pipelines, like migrating workload from on-premise data movement to BigQuery or Storage Bucket.

The most valuable features of Google Cloud Dataflow are scalability and connectivity.

Google Cloud Dataflow should include a little cost optimization.

I have been using Google Cloud Dataflow for two and a half to three years.

I rate Google Cloud Dataflow an eight out of ten for stability.

Around 10 to 15 developers use the solution in our organization.

I rate Google Cloud Dataflow an eight out of ten for scalability.

The technical support team’s response time is slow.

Neutral

I rate the solution’s initial setup a seven or eight out of ten.

Google Cloud Dataflow was deployed within 15 to 30 minutes. A group of two to three people is more than enough to deploy the solution.

On a scale from one to ten, where one is cheap, and ten is expensive, I rate the solution's pricing a seven to eight out of ten.

Google Cloud Dataflow is an extended or managed version of the open-source Apache Beam service. Google Cloud Dataflow is a cloud-based solution.

I would recommend Google Cloud Dataflow to other users for batch pipelines.

Overall, I rate Google Cloud Dataflow an eight out of ten.

we are using Google Cloud Dataflow for retailers and eCommerce.

The most valuable features of Google Cloud Dataflow are the integration, it's very simple if you have the complete stack, which we are using. It is overall very easy to use, user-friendly friendly, and cost-effective if you know how to use it. The solution is very flexible for programmers, if you know how to do scripts or program in Python or any other language, it's extremely easy to use.

Google Cloud Data Flow can improve by having full simple integration with Kafka topics. It's not that complicated, but it could improve a bit. The UI is easy to use but the experience could be better. There are other tools available that do a better job.

I have been using Google Cloud Dataflow for approximately one year.

Google Cloud Dataflow has been around for a while and it is stable.

Google Cloud Dataflow is scalable because it is on the cloud. If we were hosting it we might have some troubles.

We have approximately five people in my organization using the solution. We plan to increase usage but not the engineers that use it.

The technical support is very hard to reach.

The initial setup of Google Cloud Dataflow is simple. The deployment tool is approximately 10 minutes.

I rate the complexity of the initial setup a one out of five.

We did the implementation of Google Cloud Data Flow ourselves. We have five people in our engineering team and one of us that is free does the maintenance of the solution when required. Whoever builds the ETLs and the flow has to build the observability part and monitoring.

We're a startup and we've only recently been building the data architecture and it is too early to tell.

The price of the solution depends on many factors, such as how they pay for tools in the company and its size.

I rate the price of Google Cloud Data Flow a two out of five.

We evaluated other options before choosing Google Cloud Dataflow, such as Confluent. We chose Google Cloud Dataflow because we went towards an all-GCP cloud initiative in the companies. It was more of a position as the stack we were going to use rather than having multiple different components of other third-party companies. 

There are other cool solutions, such as Databricks, but those are for different things. Google Cloud Data Flow is mostly for transformation in the cloud, in streams. There are other nice solutions out there on the market but choosing Google Cloud Data Flow made sense because we have everything integrated into GCP.

I rate Google Cloud Dataflow an eight out of ten.

Our primary use case for the solution is running batch jobs. It is mainly used for running computations on large batches of data. So in a case where you have big data, you need to know the analytics on the data, process the data, and present it. Google Cloud Dataflow gives you the scale and processing engine to run expensive computations on your data, quite similar to big data processing engines.

Migrating our batch processing jobs to Google Cloud Dataflow led to a reduction in cost by 70%.

The service is built on an open-source framework - Apache Beam, which has a lot of documentation. Additionally, the service is relatively cheap compared to other batch-processing engines, and the SDKs are well-supported. Hence, these are three significant features that are valuable.

Currently, not all error logs are available to users and this could make debugging failed jobs very difficult.  

The startup time of Dataflow jobs could also be reduced, and some features available in Java SDK can be included in the Python SDK.

We have been using the solution for approximately one year.

The solution is stable.

The solution is scalable and scales very well, regardless of how much data we process. Currently, four members of the team are using the solution.

The initial setup assumes much background knowledge about how the Google Cloud platform works, so new developers will come to Google Cloud Dataflow and find the solution challenging. Therefore, I rate the initial setup process as medium.

The solution is cheap compared to other similar products and emphasizes many features compared to many other streaming engines.

I rate the solution a six out of ten. The solution is good, but notifications for error messages could be more detailed, and deployment times could be reduced.