Apache Spark Reviews

We use the product for extensive data analysis. It helps us analyze a huge amount of data and transfer it to data scientists in our organization.

The product’s most valuable feature is the SQL tool. It enables us to create a database and publish it. It is a useful feature for us.

At the initial stage, the product provides no container logs to check the activity. It remains inactive for a long time without giving us any information. The containers could start quickly, similar to that of Jupyter Notebook.

We have been using Apache Spark for eight months to one year.

It is a stable product. I rate its stability an eight out of ten.

We have 45 Apache Spark users. I rate its scalability a nine out of ten.

The complexity of the initial setup depends on the kind of environment an organization is working with. It requires one executive for deployment. I rate the process an eight out of ten.

The product is expensive, considering the setup. However, from a standalone perspective, it is inexpensive.

I advise others to analyze data and understand your business requirements before purchasing the product. I rate it an eight out of ten.

We use Apache Spark for storage and processing.

The most valuable feature of Apache Spark is its flexibility.

Apache Spark's GUI and scalability could be improved.

I have been using Apache Spark for four to five years.

Around 15 data scientists are using Apache Spark in our organization.

Apache Spark's initial setup is slightly complex compared to other other solutions. Data scientists could install our previous tools with minimal supervision, whereas Apache Spark requires some IT support. Apache Spark's installation is a time-consuming process because it requires ensuring that all the ports have been accessed properly following certain guidelines.

While installing Apache Spark, I must look at the documentation and be very specific about the configuration settings. Only then I'll be able to install it.

Apache Spark is an expensive solution.

I would recommend Apache Spark to other users.

Overall, I rate Apache Spark an eight out of ten.

Our customers configure their software applications, and I use Apache to check them. We use it for data processing.

The data processing framework is good. The product is very useful.

The solution must improve its performance.

I have been using the solution for four to five years.

The tool is stable. I rate the stability more than nine out of ten.

We have a small business. Around four people in my organization use the solution.

The deployment was easy.

The solution was deployed with the help of third-party consultants.

Overall, I rate the product more than eight out of ten.

Predominantly, I use Spark for data analysis on top of datasets containing tens of millions of records.

I have an example. We had a single-threaded application that used to run for about four to five hours, but with Spark, it got reduced to under one hour.

The distribution of tasks, like the seamless map-reduce functionality, is quite impressive. For the user, it appears as simple single-line data manipulations, but behind the scenes, the executor pool intelligently distributes the map and reduce functions.

The visualization could be improved.

I have been working with Apache Spark for only a few months, not too long.

I haven't faced any stability issues. It has been stable in my experience.

When it comes to the scalability of Spark, it's primarily a processing engine, not a database engine. I haven't tested it extensively with large record sizes.

In my organization, quite a few people are using Spark. In my smaller team, there are only two users.

In terms of maintenance, when the load hits around 95%, we need to prioritize scripts and analysis within the team. 

We coordinate and prioritize based on the available resources. If there were self-service tools or better hand-holding for such situations, it would make things easier.

Currently, we extensively use pandas and Polaris. We are leveraging Docker and Kubernetes as a framework, along with AWS Batch for distribution. This is the closest substitute we have for Spark Distribution.

Both Docker and Kubernetes are more general-purpose solutions. If someone is already using Kubernetes and it's provided as a service, it can be used for special-purpose utilization, similar to Docker and Kubernetes.

In such cases, users may need to write the parallelization logic themselves, but it's relatively easy to onboard and start with a distributed load. Spark, on the other hand, is primarily used for special-purpose utilization. Users typically choose Spark when they have data-intensive tasks.

Another significant issue with Spark is its syntactics. For instance, if we have libraries like Panda or Polaris, we can run them single-threaded on a single core, or we can distribute them leveraging Kubernetes.

We don't need to rewrite that code base for Spark. However, if we are writing code specifically for Spark Executors, it will not be amenable to running it locally.

I would recommend understanding the use case better. Only if it fits your use case, then go for it. But it is a great tool.

Overall, I would rate Apache Spark an eight out of ten. 

We use Spark for machine learning applications, clustering, and segmentation of customers.

Apache provides a lot of good documentation compared to other solutions. 

The migration of data between different versions could be improved. 

I've been using this solution for four years. 

The solution is stable. 

The solution is scalable. 

If you pay for customer support then you get a quick and efficient response, otherwise the community support offers good help. 

The initial setup has been simplified over the past few years and is now relatively straightforward. 

Licensing costs depend on where you source the solution. 

This is a good solution for big data use cases and I rate it eight out of 10. 

Apache Spark is a programming language similar to Java or Python. In my most recent deployment, we used Apache Spark to build engineering pipelines to move data from sources into the data lake.

One of Apache Spark's most valuable features is that it supports in-memory processing, the execution of jobs compared to traditional tools is very fast.

Apache Spark could improve the connectors that it supports. There are a lot of open-source databases in the market. For example, cloud databases, such as Redshift, Snowflake, and Synapse. Apache Spark should have connectors present to connect to these databases. There are a lot of workarounds required to connect to those databases, but it should have inbuilt connectors.

I have been using Apache Spark for approximately five years.

Apache Spark is stable.

I have found Apache Spark to be scalable.

Apache Spark is open-source, there is no team that will give you dedicated support, but you can post your queries on the community forums, and usually, you will receive a good response. Since it's open-source, you depend on freelance developers to respond to you, you cannot put a time limit there, but the response, on average, is pretty good.

If Apache Spark is in the cloud, setting it up will require only minutes. If it's on Amazon, GCP, or Microsoft cloud, it'll take minutes to set everything up. However, if you are using the on-premise version, then it might take some time to set up the environment.

I rate Apache Spark an eight out of ten.

We are using it for big data. We are using a small part of it, which is related to using data.

It is useful for handling large amounts of data. It is very useful for scientific purposes.

There are some difficulties that we are working on. It is useful for scientific purposes, but for commercial use of big data, it gives some trouble.

They should improve the stability of the product. We use Spark Executors and Spark Drivers to link to our own environment, and they are not the most stable products. Its scalability is also an issue.

We are building our own queries on Spark, and it can be improved in terms of query handling.

In my company, it has been used for several years, but I have been using it for seven months.

It is not scalable. Scalability is one of the issues.

It is open source from my point of view. So, there is no support.

I would advise not using it if you don't have experienced users inside your organization. If you have to figure it all out on your own, then you shouldn't start with it.

Overall, I would rate it a six out of 10. For a commercial use case, it is a six out of 10. For scientific purposes, it is an eight out of 10.

The solution can be deployed on the cloud or on-premise.

We are using Apache Spark, for large volume interactive data analysis.

MechBot is an enterprise, one-click installation, trusted data excellence platform. Underneath, I am using Apache Spark, Kafka, Hadoop HDFS, and Elasticsearch.

Apache Spark can do large volume interactive data analysis.

Apache Spark is very difficult to use. It would require a data engineer. It is not available for every engineer today because they need to understand the different concepts of Spark, which is very, very difficult and it is not easy to learn.

I have been using Apache Spark for approximately 11 years.

The solution is stable.

Apache Spark is scalable. However, it needs enormous technical skills to make it scalable. It is not a simple task.

We have approximately 20 people using this solution.

If you want to distribute Apache Spark in a certain way, it is simple. Not every engineer can do it. You need DevOps specialized skills on Spark is what is required.

If we are going to deploy the solution in a one-layer laptop installation, it is very straightforward, but this is not what someone is going to deploy in the production site.

Since we are using the Apache Spark version, not the data bricks version, it is an Apache license version, the support and resolution of the bug are actually late or delayed. The Apache license is free.

We are well versed in Spark, the version, the internal structure of Spark, and we know what exactly Spark is doing. 

The solution cannot be easier. Everything cannot be made simpler because it involves core data, computer science, pro-engineering, and not many people are actually aware of it.

I rate Apache Spark a six out of ten.