Apache Spark Reviews

I am not just an end customer of Apache Spark; I use it for the solutions that I produce, primarily data integration solutions and data access solutions based on Apache Spark, but it may depend on the situation to align with other tools present in various customer locations, such as Informatica.

Essentially, my main reason for using Apache Spark is data integration, and the two major use cases are data access and data integration.

I do use Apache Spark for event analysis.

Apache Spark, specifically PySpark and the tools available there, have been quite helpful in my event analysis work.

The in-memory computation feature is certainly helpful for my processing tasks.

It is helpful because while using structures that could be held in memory rather than stored during the period of computation, I go for the in-memory option, though there are limitations related to holding it in memory that need to be addressed, but I have a preference for in-memory computation.

The solution is beneficial in that it provides a base-level long-held understanding of the framework that is not variant day by day, which is very helpful in my prototyping activity as an architect trying to assess Apache Spark, Great Expectations, and Vault-based solutions versus those proposed by clients like TIBCO or Informatica.

Areas for improvement are obviously ease of use considerations, though there are limitations in doing that, so while various tools like Informatica, TIBCO, or Talend offer specific aspects, licensing can be costly; I prefer to work this way, which does not imply being anti-tooling, but since your focus was on my technology, these will continue to be my technologies.

I have been using Apache Spark for approximately four years.

Without a doubt, we have had some crashes because each situation is different, and while the prototype in my environment is stable, we do not know everything at other customer sites, and we have not identified a way of assessing the sanity of the whole environment, so there have been some crashes.

As long as one knows how to scale the environment, scalability is not a problem at all and is not difficult to manage.

I have had experience with technical support from Apache, mainly through newsgroups, which have been reasonably forthcoming when required.

I think it would be farcical to compare technical support from Apache Spark to what I would receive under an Informatica license, but I have received support via newsgroups or guidance on specific discussions, which is what I would expect in an open-source situation.

Neutral

I think things are becoming easier for installation and deployment, and I have cleaned up the process over a number of years to minimize pain, although the process of installing a commercial tool might be easier.

API management is not my interest at the moment, so I do not remember reading information about API management products and Enterprise Service Bus, ESB in the past, and I am not using any solutions like that.

I am working much more in data science and data engineering.

I can discuss my experience with data engineering tools, and I am using some now.

I work primarily with open-source tools, and that is the way I work instead of big data solutions like Informatica.

The open-source big data stacks I use include Apache Spark, Hive, and tools such as Vault for protection and Great Expectations for data quality, which are my primary tools along with the PostgreSQL database for database support.

I have been using Apache Spark for quite a long time.

Real-time data analytics is an area of interest for me, but I have not had to do that in most of what I have done, although it is increasingly becoming an area of interest that I am looking seriously at Apache possibilities for instead of going down the Kafka path, using the Apache streaming API to see if it fits my use cases, but I do not know it very well.

Very often in many of my experiments, the data set has had to be partitioned, and there have been issues in handling very large data sets, with most of my work done using Python machine learning libraries, requiring chunking, and speed of prediction has been an issue of concern in some experiments where we have had to shut down processes due to CPU requirements, then restart with different Apache configurations, and resourcing support is a major determinant if I were to name a constraint in terms of running machine learning experiments.

Apache Spark is basically free compared to other similar products like Informatica and Talend.

Sometimes, we have to use cloud resources due to insufficient on-premise resources for certain types of computing, so it is hybrid.

Our builds have all been based on the Azure Apache marketplace.

While I have worked with Hive and not really on Talend, I confirm I indeed work mainly with Apache Spark and Hive.

In the past, Hive has been a kind of a default part of the process, but that is not the case anymore in recent times.

I have given this review a rating of 9 out of 10.

I attempted to use Apache Spark in one of our customer projects, but after the initial test, our customer moved to another technology and another database system. I do not have any final remarks on what would be better or what is not as good because we only tried it for the customer and made a running technical prototype. After we presented it to our customer, he moved to another system, which was preferred by one of his consultants. I do not know exactly what was the reason to move away from Apache Spark or the underlying database system, but it was simply a decision driven by the customer.

The best features in Apache Spark that I appreciate are the fast database access, the data transformation, and the data exchange.

I see that very good integration with other platforms, including interfaces that can connect to other vendors and technologies, and integration of the MCP protocol of one of AA systems, would be an interesting direction for me personally and as a company to integrate the technology in our customer projects. The most important part is that everything can be connected, and the data exchange across overseas connections is fast and reliable.

I find that there really lacks the technical depth to do any recommendations for future updates of Apache Spark. I used it for two years for our prototype work and testing things, but because I had no final project with a release and running at the customer side or other side, I cannot say what I would expect if I wanted to use it in a real project.

Regarding the current licensing cost, I would say it is in the medium range. However, because I do not have a licensed project for our customer, I do not know if it would be too high for our customers if they have to buy the license for themselves. For me, compared to other things, the licensing was acceptable.

I have been working with Apache Spark for two years.

I would rate the technical support of Apache Spark an eight because when we had questions, we found solutions, and it was straightforward.

Positive

For me, the implementation of Apache Spark was straightforward. The deployment required mixed time; I cannot give any specific duration. It was step by step, over approximately a week, because I have other things to do and I have to do it in parallel with other topics I have to maintain. It is hard to give a duration.

I did it all by myself.

As for real-time data analytics with Apache Spark, I do not know if I would use inbuilt analytics or other options. Most of our customers have external AI systems polling the data from different data sources and doing their analytics in their preferred tool. I do not know if I would use integrated systems in Apache Spark itself, because we are working with customers who have a huge toolchain and have dedicated tools for nearly every task, and they are choosing their task for AI, predictive maintenance, and all the different features. I gave this review a rating of eight.

I don't use a big data solution such as a Data Lake. I have used Apache Spark code on a lot of big data. We don't have a Data Lake and these new technologies, but we have used Apache Spark code to get data from big data, CRM, and Siebel. Siebel is the application that exists in Teleco branches in Egypt, and the customers communicate with it. We have a lot of data from the finance team, CRM, Siebel, and other sources. We consolidate all of this data and perform transformations.

With Apache Spark, we can perform various transformations. For instance, when a customer calls their mother and consumes many minutes, we should consolidate all of them and calculate the net minutes. For the monthly invoice, we should determine how much the customer should pay for ADSL, phones, and family phones. We take all of this information, transform it, and use it to generate the invoice.

We enhance the data processing by using Apache Spark SQL. I haven't used Apache Spark machine learning, but I've used Apache Spark SQL because we have data in HDFS tables. We take the Apache Spark code, get the data, and then get the aggregate. For example, we can request aggregated data about a customer's consumption since last week. You can use Apache Spark code or Python code itself, and if you know SQL, you can type SQL code within the same script and output it to any table or Excel file. It's durable and easy.

When a customer has an issue with their phone number and can't call or access the internet, they visit a branch and speak with an agent. We need to take action based on the data, so we need real-time data processing to get aggregated data for this customer from the last week or month. All data solutions serve the customer and business needs, which is what I appreciate about data solutions.

I can improve the organization's functions by taking less time to make decisions. To make the right decision, you need the right data, and a solution can provide this by hiring talent and employees who can consolidate data from different sources and organize it. Not all solutions can make this data fast enough to be used, except for solutions such as Apache Spark Structured Streaming. To make the right decision, you should have both accurate and fast data.

Apache Spark itself is similar to the Python programming language. Python is a language with many libraries for mathematics and machine learning. Apache Spark is the solution, and within it, you have PySpark, which is the API for Apache Spark to write and run Python code. Within it, there are many APIs, including SQL APIs, allowing you to write SQL code within a Python function in Apache Spark. You can also use Apache Spark Structured Streaming and machine learning APIs.

Regarding Apache Spark, I have only used Apache Spark Structured Streaming, not the machine learning components. I am uncertain about specific improvements needed today. However, after five years, there will be many new cloud providers, connectors, and solutions. Every year, there should be some enhancement to remain competitive in the market. As new solutions emerge frequently, the basic improvement would be to have integration with these solutions.

I have used Apache Spark for three years. However, I have only used Apache Spark Structured Streaming, this specific library in Apache Spark, for one year.

I don't have any issues with Apache Spark that are better handled in Hadoop. I have experienced the opposite. I have used Hadoop, MapReduce, and Apache Spark, and I see many problems in MapReduce that Apache Spark resolves. Every new solution comes to solve problems from previous solutions. Apache Spark resolves many problems in the MapReduce solution and Hadoop, such as the inability to run effective Python or machine learning algorithms. MapReduce needs to perform numerous disk input and output operations, while Apache Spark can use memory to store and process data.

I have used it daily for one year without issues. I also have colleagues with five years of experience who haven't encountered any problems.

I have used the Hadoop ecosystem, Hadoop HDFS, Impala and Hive. Hive is similar to SQL but within the Hadoop ecosystem. It queries data that HDFS has stored in folders. I have used Python, Apache Spark, and Apache Spark Structured Streaming from Apache Spark. I have also used MapReduce, which is a processing engine similar to Apache Spark. To use MapReduce, you need to be familiar with the Java programming language. Since not everyone knows Java, Apache Spark is considered better because users can use Python and SQL within it.

I prefer to use Apache Spark as an open-source solution if you already have experienced data engineers who use Apache Spark. If you don't have this expertise, you need to hire additional employees. However, if you have a Databricks solution that includes Apache Spark, you have customer service and may need to hire fewer employees.

I am unable to suggest specific improvements at this time. This review has a rating of 10 out of 10.

Neutral

We use it for real-time and near-real-time data processing. We use it for ETL purposes as well as for implementing the full transformation pipelines.

There is no other platform that can challenge its features. Apart from the restrictions that come with its in-memory implementation.

Apart from the restrictions that come with its in-memory implementation. It has been improved significantly up to version 3.0, which is currently in use. 

Once I get those insights, I can let you know if the restrictions have been overcome. For example, there is an issue with heap memory getting full in version 1.6. There are other improvements in 3.0, so I will check those. 

In future releases, I would like to reduce the cost.

We have been using this solution for 11 to 12 years. Now, it is deployed on the cloud or premises. Previously, it was on-premises when the version was below 1.6.

After version 1.6, it will be on the cloud. I have used it in all the major cloud providers: AWS, GCP, and Azure.

It is a stable solution, but when it comes to patch updates and different reports being updated, it can be a headache. 

When an application is built on top of certain reports with specific versions, and then the version changes, it can lead to a lot of things needing to be adjusted. This is something that definitely needs improvement.

I contacted customer service and support six or seven years ago when Spark was still on version 1.6. 

We were struggling with memory limitations and the need for a lift and shift mechanism in a hybrid cloud mode. I contacted one or two people at that time.

Neutral

It is quite expensive. In fact, it accounts for almost 50% of the cost of our entire project.

If I propose using Spark for a project, one of the first questions I get from management is about the cost of Databricks Spark on the cloud platform we're using, whether it's Azure, GCP, or AWS. If we could reduce the collection, system conversion, and transformation network costs by even just 2% to 3%, it would be a significant benefit for us.

If your use case involves real-time applications frequently changing columns or data frames, then Spark is a fantastic option for you. 

However, if you have a batch process and don't have a structural data analysis, I would suggest avoiding it. The high cost of cloud infrastructure combined with Apache Spark can be a significant burden in such scenarios.

Overall, I would rate the solution a nine out of ten. 

I have some hands-on experience with Spark. I have one year of experience that should be considered as one year working with Spark, which is six months to one year. We use it for faster processing, especially compute. 

Spark is used for transformations from large volumes of data, and it is usefully distributed. We receive data from various sources and need to transform it. The data is enormous, in terabytes, and often from specific databases. We perform transformations, aggregations, and deduplication. 

We meet business requirements by computing data, minimizing it, aggregating it, or performing other operations. We typically write to Hive downstream.

Spark is faster and distributed. Previously, everything relied on MapReduce, which was slower. With Spark, multiple computations and transformations hold in memory for faster processing. 

Real-time communication is possible, connecting with platforms like Kafka for real-time data import and compute. We implemented Spark and NiFi for integration. Spark replaced other costly products, reducing costs by thirty-eight percent.

The Spark solution could improve in scheduling tasks and managing dependencies. Spark alone cannot handle sequential tasks, requiring environments like Airflow scheduler or scripts. For instance, one task should trigger another based on completion, however, Spark can't manage these dependent loads. We focus on specific compute tasks that we can deliver.

I have six months to one year of experience working with Spark.

Spark is stable, however, efficient use is necessary for running jobs seamlessly.

Spark is scalable.

I didn't work on any AI build projects for Spark, however, it supports external AI capabilities.

The initial setup is complex. Logging methods require configuration, and it depends on matching with the cluster. Communicating within the node and setting up external logging supported by Spark are challenging.

On the compute side, I worked on Snowflake as well.

I recommend Spark for working with large-scale big data. It is crucial to have skilled technicians. Overall product rating: seven out of ten.

The primary use case for Apache Spark is to process data in memory, using big data, and distributing the engine to process said data. It is used for various tasks such as running the association rules algorithm in ML Spark ML, running XGBoost in parallel using the Spark engine, and preparing data for online machine learning using Spark Streaming mode.

The most significant cost savings come from the operational side because Spark is very typical in operations. There are many experts available in the market to operate Spark, making it easier to find the right personnel. It is quite mature, which reduces operation costs.

The most significant advantage of Spark 3.0 is its support for DataFrame UDF Pandas UDF features. This allows running Pandas code distributed by using the Spark engine, which is a crucial feature. The integration with Pandas syntax in distributed mode, along with the user-defined functions in PySpark, is particularly valuable.

The main concern is the overhead of Java when distributed processing is not necessary. In such cases, operations can often be done on one node, making Spark's distributed mode unnecessary. Consequently, alternatives like Doc DB are more preferable. Additionally, performance in some cases is slower, making alternatives two to five times faster.

I have more than ten years of experience using Spark, starting from when it was first introduced.

Spark is very stable for our needs. It offers amazing stability.

Scalability depends on how infrastructure is organized. Better balance and network considerations are necessary. However, Spark is very stable when scaled appropriately.

Customer support for Apache Spark is very good. There is a lot of documentation and forums available, making it easier to find solutions. The Databricks team also does a lot to support Spark.

Positive

The initial setup of Spark can take about a week, assuming the right infrastructure is already in place.

A few technicians are typically required for installation and configuration. SRE engineers or operational guys handle the setup, as they need to understand the details about installation and configuration. Maintenance usually requires just an SRE engineer or operational guy.

The main benefit in terms of ROI comes from the operation side. Spark’s operational costs are lower due to the availability of experts and its maturity. However, performance costs might be higher due to the need for more memory and infrastructure.

Compared to other solutions like Doc DB, Spark is more costly due to the need for extensive infrastructure. It requires significant investment in infrastructure, which can be expensive. While cloud solutions like Databricks can simplify the process, they may also be less cost-efficient.

I'd rate the solution eight out of ten. 

Most of my use cases involve data processing. For example, someone tried to run sentiment analysis on Databricks using Apache Spark. They had to handle data from many countries and languages, which presented some challenges. Besides that, I primarily use Apache Spark for data processing tasks. I work with mobile phone datasets, around one terabyte in size. This involves extracting and analyzing data before building any models.

Apache Spark is known for its ease of use. Compared to other available data processing frameworks, it is user-friendly. While many choices now exist, Spark remains easy to use, particularly with Python. You can utilize familiar programming styles similar to Pandas in Python, including object-oriented programming.

Another advantage is its portability. I can prototype and perform some initial tasks on my laptop using Spark without needing to be on Databricks or any cloud platform. I can transfer it to Databricks or other platforms, such as AWS. This flexibility allows me to improve processing even on my laptop. For instance, if I'm processing large amounts of data and find my laptop becoming slow, I can quickly switch to Spark. It handles small and large datasets efficiently, making it a versatile tool for various data processing needs.

Apache Spark lacks geospatial data.

I have started using Apache Spark in 2015.

Apache Spark is open-source software. The documentation is pretty decent. I have used the online resources and mailing list but have never received a positive response.

The initial setup is very easy. It's proper to start. Setting up Apache Spark has become more accessible, even when creating a new environment. It is available in the Python package ecosystem, so you can install it using pip. However, you still need to set up the Java environment for it to work properly.

Apache Spark is different. I can use Apache Spark on my computer. Even though I don't have a super high-performance computer, it has nine cores and 40 GB of memory, sufficient to run Spark on my laptop without much setup.

We also have a cluster of much larger computers with hundreds of cores where we run Spark. Managing Spark on this cluster requires more setup. Initially, we used HDP, but it has since changed to something else. Setting up Spark on a cluster takes some time because Spark doesn't necessarily run independently in a cluster environment. You need a cluster management platform or framework like Hadoop or Kubernetes to manage and run Spark efficiently.

So, while setting up Spark on my laptop is straightforward, setting it up on a custom cluster involves more effort and configuration.

I rate the initial setup an eight or nine out of ten, where one is difficult, and ten is easy.

Apache Spark is my go-to solution for processing large-scale datasets. I would recommend it 100%. One of the main reasons is its ease of use. You can start using it on your laptop without any extra infrastructure, and then you can take that same code and run it anywhere else, including on the cloud. You're not locked in by any vendor, which is a significant advantage. 

Overall, I rate the solution a nine out of ten as a big data processing engine.

Our main use cases for Spark are Apache Spark SQL and sometimes Spark Streaming to process streaming data.

Like most solutions, we got data from SAP or Azure Data Warehouse. Suppose they were using Azure Cloud technology. So, the data comes from there, relational or sometimes semi-structured data like JSON files and all. 

So, we process the data with Spark, writing this code with PySpark, actually Python, which Spark allows, to create the data forms and all and load it into the Tableau format, basically. 

So, we try to load it into some database, like SQL Server or any other database. From there, the business data scientists or analysts pick up the data. So, any sort of different sources, basically, like e-commerce sites. 

So, previously, we used mostly structured data, which was stored in SAP, mainframe Oracle, or any other system provided in structured formats like CSV. 

Now, when we're tackling sentiment analysis using NLP technologies, we deal with unstructured data—customer chats, feedback on promotions or demos, and even media like images, audio, and video files. For processing such data, we rely on PySpark.

Beneath the surface, Spark functions as a compute engine with in-memory processing capabilities, enhancing performance through features like broadcasting and caching. It's become a crucial tool, widely adopted by 90% of companies for a decade or more.

Before Spark, there was MapReduce, but it was much slower. Even running the same query a second time would be time-consuming due to the I/O operations with disk storage. Spark was introduced to address these issues, offering processing speeds a hundred times faster than MapReduce, an initiative that saw contributions from Adobe Systems among others.

So, in response to the evolving needs of the industry, Spark has proven to be the solution, efficiently handling the processing requirements we face today.

Spark supports real-time data processing through Spark Streaming. It allows for batch processing of data. If you have immediate data, like chat information, that needs to be processed in real-time, Spark Streaming is used. 

For data that can be evaluated later, batch processing with Apache Spark is suitable. Mostly, batch processing is utilized in our organization, but for streaming data processing, tools like Kafka are often integrated.

In-memory processing in Spark greatly enhances performance, making it a hundred times faster than the previous MapReduce methods. This improvement is achieved through optimization techniques like caching, broadcasting, and partitioning, which help in optimizing queries for faster processing.

There could be enhancements in optimization techniques, as there are some limitations in this area that could be addressed to further refine Spark's performance.

I've used it for four years. 

In the community forums, I asked questions a while back when I was new. However, the responses came from other users in the community, not the official Apache Spark organization. So, I am not sure about the proficiency. 

Since it's open-source, most questions happen in the community. For enterprise support, I imagine the response speed would be different. 

I have also used Hadoop.

The main reason for choosing Apache Spark was for big data solutions. Hadoop was introduced earlier, and most organizations were using Hadoop or cloud data platforms. 

Then, Apache Spark came into the picture, and it was much faster. It's kind of taking the place of Hadoop. Organizations using Hadoop are now primarily focusing on Apache Spark for support.

So, for big data computing tasks, what you do with Hadoop is like a top-level layer. Spark is another layer on top of that. Organizations using Hadoop technologies and big data technologies in general have adopted Spark. 

There aren't really other comparable tools for big data computing tasks. But, resource managers like Kubernetes and YARN are used with Spark. YARN was used in Hadoop big data technology, but now Kubernetes is more commonly used for resource management.

Resource allocation and optimization in the computing tasks are different for on-premise systems. 

In cloud environments, resource allocation is already handled by the cloud provider, so you don't need to worry about it. 

On-prem, if you're using Hadoop with Spark, resource allocation might be handled by Kubernetes or YARN. These tools provide feedback to the Spark driver about available resources, and the driver allocates tasks to worker nodes based on that information.

Overall, I would rate the solution a nine out of ten. 

I would recommend this tool to someone considering it for scalable data processing.

Nowadays, Apache Spark is on the market, and most organizations are using it. There are people with more experience and knowledge than me, and they're confident about this tool. 

That's why it's become a solution for organizations. It's not a one-man decision but rather a group or community effort.

The only issue I faced with the tool was that I used to choose the compute device to support parallel processing, and it has to be more like scaling up horizontally. The tool should be more scalable, not in terms of increasing the CPU or something, but more in the area of units. If two units are not enough, the third or fourth unit should be able to come into the picture.

From my perspective, the only thing that needs improvement is the interface, as it was not easily understandable. Sometimes, I get an error saying that it is an RDD-related error, and it becomes difficult to understand where it went wrong. When I deal with datasets using a library called Pandas in Python, I can actually apply functions on each column and get a transformation from the column. When I try to do the same thing with Apache Spark, it is okay and works, but it is not straightforward; I need to deal with it a little differently, and even after trying to do that differently, the problem I face there is, sometimes it will throw an error saying that it is looping back to the same, but I was not getting that kind of errors in Pandas.

In future updates, the tool should be made more user-friendly. I want to take fifty parallel processes rather than one, and I want to pick some particular columns to be split by partition, so if the tool is user-friendly and offers clarity and flexibility, then that will be good.

I have been using Apache Spark for four years.

Stability-wise, I rate the solution a nine out of ten. The only issues with the tool revolve around user interaction and user flexibility.

It is a scalable solution. Scalability-wise, I rate the solution an eight out of ten.

Around five people in my company use the tool.

The solution's technical support is helpful, but I faced some problems which were more of a generic issue. If I face any problems which are non- generic issues, I get help from the tool's team. For the generic issues, I get answers mainly from the forums where the problem was already resolved. When it comes to some unknown problem or specific problem with my work, then the support takes time. I rate the technical support a seven out of ten.

Neutral

I only work with Apache Spark.

The product's initial setup phase was easy.

I managed the product's installation phase, both locally and on the cloud.

The solution is deployed on the on-premises version.

The solution can be deployed in two to three hours.

Apache Spark has helped save 50 percent of the operational costs. Time was reduced with the use of the tool, but the computing part increased. Overall, I can see that the tool's use has led to a 50 percent reduction in costs.

I did not pay anything when using the tool on cloud services, but I had to pay on the compute side. The tool is not expensive compared with the benefits it offers. I rate the price as an eight out of ten.

Previously, I was more of a Python full-stack developer, and I was happy dealing with PySpark libraries, which gave me an edge in continuing the work with Apache.

Speaking about Apache Spark's use in our company's data processing workflows, I would say that when we deal with large datasets of data, if we don't use Spark, then when we try to use a data frame consisting of one year of data, it used to take me 45 minutes to an hour. Moreover, sometimes I used to get the memory out of space errors, but such issues were avoided the moment I started using Apache Spark, as I was able to get the whole processing done in less than five minutes, and there were no memory issues.

For big data processing, the tool's parallel processing and time are areas that have been helpful. When I try to apply a function, I can directly data write one code. Basically, I used Apache Spark to forecast multiple units at the same time, and if not with Apache Spark, I would be doing that one by one, which is more of a serial processing process that used to take me around five hours. At the moment, we use Apache Spark in parallel processing, where computing happens parallelly, and all these computations are cut down by at least 90 percent. It helps me significantly to reduce the time needed for operations.

The tool's real-time processing is an area that I have not tried to use much. When it comes to real-time processing of my data, I use Kafka.

I am handling data governance using Databricks Unity Catalog.

When I try to apply an ML model, I am unable to get that model done on a table partitioned by a particular column; it makes me get the job done in a reduced number of partitions. If I go with five partitions, I am able to get at least three to four times the benefits in a lesser amount of time.

Regular maintenance exists, but it is not like I have to sit week by week and upgrade a patch or something like that. The maintenance is done mostly in about six months to a year.

I take care of the tool's maintenance.

I recommend the tool to others.

I rate the tool an eight out of ten.