Try our new research platform with insights from 80,000+ expert users

Apache Spark vs Google Cloud Dataflow comparison

 

Comparison Buyer's Guide

Executive Summary

Review summaries and opinions

We asked business professionals to review the solutions they use. Here are some excerpts of what they said:
 

ROI

Sentiment score
5.6
Apache Spark provides up to 50% cost savings, boosting efficiency and reducing expenses significantly in machine learning analytics.
Sentiment score
5.6
Google Cloud Dataflow was appreciated for cost savings and time efficiency, though some considered its impact not fully assessable yet.
 

Customer Service

Sentiment score
6.0
Apache Spark offers vibrant community support and resources, with commercial support available through vendors like Cloudera and Hadoop.
Sentiment score
6.6
Google Cloud Dataflow support varies, with users praising technical resolution but highlighting inconsistent response times and accessibility.
I would rate the technical support of Apache Spark an eight because when we had questions, we found solutions, and it was straightforward.
Consultant, Chief Engineer, Teamleiter at infoteam Software AG
I have received support via newsgroups or guidance on specific discussions, which is what I would expect in an open-source situation.
Data Architect at Devtech
The fact that no interaction is needed shows their great support since I don't face issues.
Data Engineer at Accenture
Google's support team is good at resolving issues, especially with large data.
Senior Data Engineer at Accruent
Whenever we have issues, we can consult with Google.
Senior Software Engineer at Dun & Bradstreet
 

Scalability Issues

Sentiment score
7.4
Apache Spark's scalability and versatility enable efficient large-scale data processing, making it a reliable choice for diverse teams.
Sentiment score
7.3
Google Cloud Dataflow excels in scalability and efficiency, making it ideal for real-time data processing and dynamic needs.
Google Cloud Dataflow has auto-scaling capabilities, allowing me to add different machine types based on pace and requirements.
Data Engineer at Accenture
Google Cloud Dataflow can handle large data processing for real-time streaming workloads as they grow, making it a good fit for our business.
Senior Data Engineer at Accruent
As a team lead, I'm responsible for handling five to six applications, but Google Cloud Dataflow seems to handle our use case effectively.
Senior Software Engineer at Dun & Bradstreet
 

Stability Issues

Sentiment score
7.4
Apache Spark is praised for its robust stability and reliability, with high user ratings despite minor configuration challenges.
Sentiment score
8.3
Google Cloud Dataflow is stable, reliably handles tasks, and benefits from automatic scaling, with minor issues on complex tasks.
MapReduce needs to perform numerous disk input and output operations, while Apache Spark can use memory to store and process data.
Data Engineer at a tech company with 10,001+ employees
Without a doubt, we have had some crashes because each situation is different, and while the prototype in my environment is stable, we do not know everything at other customer sites.
Data Architect at Devtech
I have not encountered any issues with the performance of Dataflow, as it is stable and backed by Google services.
Data Engineer at Accenture
The job we built has not failed once over six to seven months.
Senior Software Engineer at Dun & Bradstreet
The automatic scaling feature helps maintain stability.
Senior Data Engineer at Accruent
 

Room For Improvement

Apache Spark needs improvements in real-time querying, user-friendliness, logging, large dataset handling, and expanded programming language support.
Google Cloud Dataflow needs better Kafka integration, improved error logs, reduced startup time, and enhanced Python SDK features.
Various tools like Informatica, TIBCO, or Talend offer specific aspects, licensing can be costly;
Data Architect at Devtech
I find that there really lacks the technical depth to do any recommendations for future updates of Apache Spark.
Consultant, Chief Engineer, Teamleiter at infoteam Software AG
Outside of Google Cloud Platform, it is problematic for others to use it and may require promotion as an actual technology.
Data Engineer at Accenture
I would like to see improvements in consistency and flexibility for schema design for NoSQL data stored in wide columns.
Senior Data Engineer at Accruent
Dealing with a huge volume of data causes failure due to array size.
Senior Software Engineer at Dun & Bradstreet
 

Setup Cost

Apache Spark is cost-effective but can incur high infrastructure costs, especially in cloud setups like Databricks, with setup time variability.
Google Cloud Dataflow is praised for cost-effectiveness and scalability, offering competitive pricing influenced by pipeline complexity and company size.
It is part of a package received from Google, and they are not charging us too high.
Senior Software Engineer at Dun & Bradstreet
 

Valuable Features

Apache Spark provides scalable, in-memory data processing with flexible support for distributed computing, streaming, and machine learning integration.
Google Cloud Dataflow offers seamless integration, multi-language support, scalability, and serverless data handling for efficient batch and streaming processes.
The most important part is that everything can be connected, and the data exchange across overseas connections is fast and reliable.
Consultant, Chief Engineer, Teamleiter at infoteam Software AG
Apache Spark is the solution, and within it, you have PySpark, which is the API for Apache Spark to write and run Python code.
Data Engineer at a tech company with 10,001+ employees
The solution is beneficial in that it provides a base-level long-held understanding of the framework that is not variant day by day, which is very helpful in my prototyping activity as an architect trying to assess Apache Spark, Great Expectations, and Vault-based solutions versus those proposed by clients like TIBCO or Informatica.
Data Architect at Devtech
It supports multiple programming languages such as Java and Python, enabling flexibility without the need to learn something new.
Data Engineer at Accenture
The integration within Google Cloud Platform is very good.
Senior Software Engineer at Dun & Bradstreet
Google Cloud Dataflow's features for event stream processing allow us to gain various insights like detecting real-time alerts.
Senior Data Engineer at Accruent
 

Categories and Ranking

Apache Spark
Average Rating
8.4
Reviews Sentiment
6.9
Number of Reviews
69
Ranking in other categories
Hadoop (1st), Compute Service (5th), Java Frameworks (2nd)
Google Cloud Dataflow
Average Rating
8.0
Reviews Sentiment
7.1
Number of Reviews
14
Ranking in other categories
Streaming Analytics (13th)
 

Mindshare comparison

Apache Spark and Google Cloud Dataflow aren’t in the same category and serve different purposes. Apache Spark is designed for Hadoop and holds a mindshare of 13.3%, down 18.6% compared to last year.
Google Cloud Dataflow, on the other hand, focuses on Streaming Analytics, holds 3.9% mindshare, down 7.4% since last year.
Hadoop Mindshare Distribution
ProductMindshare (%)
Apache Spark13.3%
Cloudera Distribution for Hadoop14.1%
HPE Data Fabric13.5%
Other59.1%
Hadoop
Streaming Analytics Mindshare Distribution
ProductMindshare (%)
Google Cloud Dataflow3.9%
Apache Flink10.9%
Databricks9.0%
Other76.2%
Streaming Analytics
 

Featured Reviews

Devindra Weerasooriya - PeerSpot reviewer
Data Architect at Devtech
Provides a consistent framework for building data integration and access solutions with reliable performance
The in-memory computation feature is certainly helpful for my processing tasks. It is helpful because while using structures that could be held in memory rather than stored during the period of computation, I go for the in-memory option, though there are limitations related to holding it in memory that need to be addressed, but I have a preference for in-memory computation. The solution is beneficial in that it provides a base-level long-held understanding of the framework that is not variant day by day, which is very helpful in my prototyping activity as an architect trying to assess Apache Spark, Great Expectations, and Vault-based solutions versus those proposed by clients like TIBCO or Informatica.
PR
Senior Data Engineer at Accruent
Enables real-time insights and efficient data preparation for machine learning
Google Cloud Dataflow's features for event stream processing allow us to gain various insights like detecting real-time alerts. For integration, we use Dataflow to extract data from different sources like APIs and flat files. We then perform data cleansing, including deduplications, schema standardizations, and filtering of invalid records. We also use it for preparing data for machine learning models, transforming data, and accelerating models.
report
Use our free recommendation engine to learn which Hadoop solutions are best for your needs.
884,873 professionals have used our research since 2012.
 

Top Industries

By visitors reading reviews
Financial Services Firm
23%
Manufacturing Company
8%
Computer Software Company
7%
Comms Service Provider
6%
Financial Services Firm
17%
Manufacturing Company
13%
Retailer
11%
Computer Software Company
7%
 

Company Size

By reviewers
Large Enterprise
Midsize Enterprise
Small Business
By reviewers
Company SizeCount
Small Business28
Midsize Enterprise16
Large Enterprise32
By reviewers
Company SizeCount
Small Business3
Midsize Enterprise2
Large Enterprise10
 

Questions from the Community

What do you like most about Apache Spark?
We use Spark to process data from different data sources.
What is your experience regarding pricing and costs for Apache Spark?
Apache Spark is open-source, so it doesn't incur any charges.
What needs improvement with Apache Spark?
I find that there really lacks the technical depth to do any recommendations for future updates of Apache Spark. I used it for two years for our prototype work and testing things, but because I had...
What is your experience regarding pricing and costs for Google Cloud Dataflow?
Pricing is normal. It is part of a package received from Google, and they are not charging us too high.
What needs improvement with Google Cloud Dataflow?
It can be improved in several ways. The system could function in an automated fashion and provide suggestions based on past transactions to achieve better scalability. Implementing AI-based suggest...
What is your primary use case for Google Cloud Dataflow?
It is used for exporting data, such as customer clicks, customer interactions with emails, and link tracking. The Google Analytics streaming data is used to establish customer behavioral patterns.
 

Also Known As

No data available
Google Dataflow
 

Overview

 

Sample Customers

NASA JPL, UC Berkeley AMPLab, Amazon, eBay, Yahoo!, UC Santa Cruz, TripAdvisor, Taboola, Agile Lab, Art.com, Baidu, Alibaba Taobao, EURECOM, Hitachi Solutions
Absolutdata, Backflip Studios, Bluecore, Claritics, Crystalloids, Energyworx, GenieConnect, Leanplum, Nomanini, Redbus, Streak, TabTale
Find out what your peers are saying about Apache, Cloudera, Amazon Web Services (AWS) and others in Hadoop. Updated: March 2026.
884,873 professionals have used our research since 2012.