Databricks Reviews

I worked with Databricks pretty recently. The particular design processes involved in Databricks were also a part of that specific design/architectural process.

We have used the solution for the overall data foundation ecosystem for processing and storage on a Delta format. We have also seen use cases where we were trying to establish advanced analytics models and data sharing where we leverage the Delta Sharing capabilities from Databricks.

A very valuable feature is the data processing, and the solution is specifically good at using the Spark ecosystem.

There are some aspects of Databricks, like generative AI, where they are positioning things like DALL-E. They're a little bit late to the game, but I think there are some things that they are working on. Generative AI is catching up in areas like data governance and enterprise flavor. Hence, these are places where Databricks has to be faster, and even though they are fast, I'm not sure how they'll catch up and get adopted because there are strong players in the market.

Databricks is coming up with a few good things in terms of integration. But I have to put one point forward that covers multiple aspects, which is the ease of use for the end user while operating this particular tool. For example, a tool like ADS gives you a GUI-based development, which is good for the end user who does development or maintenance. Looking at the complexities of data integration, a GUI might not be easy, but Databricks should embrace something on the graphical user development front because it is currently notebook-driven. Also, in terms of accessing the data for the end user, Databricks has an SQL interface, similar to earlier tools like SQL Management Studio. Since people are mostly comfortable with SSMS already or not, Databricks can build integration to known tools for data access, and that also helps, apart from what they're doing. I would like to see improvements with respect to user enablement, which is a good part of enterprise strategy. I would like to see their integration with a broader ecosystem of products. If you have to do data governance in tools like Microsoft Purview, it's manual and difficult. Now, I'm unsure if that momentum must be from Databricks or Microsoft. But it would be good if Databricks had some open interfaces to share metadata, which could be viewed in tools enabling data governance like Collibra, Purview, or Informatica. The improvement has to do with user and metadata integration for tools.

I've worked with Databricks for over five or six years, but it's been on and off.

The solution is scalable. In this particular ecosystem, there is no one else who can catch up with Databricks for now.

Databricks' customer support is very good. They have a lot of ways in which they interact with vendors and service partners across the globe. They have periodic touch-up sessions with vendors, where their engineers answer your questions.

The implementation is not challenging because the solution integrates well with the platforms on which they are established, whether it's Azure, AWS, or GCP. The solution is not difficult to set up, but you'd probably need a technical user to operate it.

It's the same story with maintenance, where you'd need a technically proficient person with programming knowledge to maintain it.

Databricks integrates many enterprise processes because data processing and AIML are a small part of a larger ecosystem. Databricks has been a part of other platforms, and they are trying to establish their platform, which is a good direction.

Most of the capabilities of the underlying platform can be leveraged there. But the setup isn't difficult if the database lacks some capability, you can't find it in the database, or you're not comfortable with a certain feature in the database. It integrates well with the underlying platform. For example, with scheduling, let's say you are uncomfortable with workflow management. You can utilize integrations with EDA for any other tool and probably perform scheduling. Even if what you're trying to do is not easy, it is enabled with integration. Either they build a required feature in their tool later on, like a GUI, or you perform integrations to make the features possible.

We did evaluate licensing costs, but it had more to do with the Azure ecosystem pricing since whatever we are doing has more to do with Azure Databricks. Many optimizations are recommended, but we haven't exercised those for now. But considering that the processing is a bit more efficient, the overall price won't be much different from what it could be for any other similar component or technology. We haven't had specific discussions with Databricks' folks on pricing.

My advice to users who would like to start working with Databricks is that it is a good solution to work with for data integration and machine learning. Databricks is maturing for other use cases, so there are two points to be considered. One is that you need to evaluate how they will mature, which will be on a case-to-case basis. Second, how will it align with the overall platform story? There will be many overlapping aspects over there as Databricks expands its capabilities. In that case, it must be considered that if those capabilities overlap, how will the underlying platform vendors handle it? How would that interplay happen if many of Databricks' new capabilities align with Microsoft Fabric? That has to be very carefully considered. Otherwise, if you utilize those new capabilities, there might be a discontinuity where you cannot use Databricks because the platform does not support that.

If I specifically talk about Spark-based processing transformations, the data integration story, and advanced stability, I would rate Databricks around eight out of ten. However, with respect to new capabilities like cataloging, data governance, and security integration, I rate Databricks around five because it has to establish these features. And since Databricks integrates with platforms, we must see the interplay with the platforms' capabilities.

I overall rate Databricks a seven out of ten.

It's mainly used for data science, data analytics, visualization, and industrial analytics.

Specifically for data science and data analytics purposes, it can handle large amounts of data in less time. I can compare it with Teradata. If a job takes five hours with Teradata databases, Databricks can complete it in around three to three and a half hours.

So that's why it's quite convenient to use for data science, for training machine learning models. By using more computing power, you can make it even faster.

There is room for improvement in visualization.

I used it for two years. I worked with the latest update. 

I would rate the stability a nine out of ten. I didn't face performance drops.

I would rate the scalability an eight out of ten.

Databrick's support is great. If we need any support, they are very quick with it. And they genuinely want you to use Databricks. So, whatever we ask them, they come up with multiple solutions to problem statements. That's really good.

Overall, the customer service and support are very good.

Positive

I personally prefer using Databricks. However, we also considered using Snowflake, but the pricing was different. It's  price per query.

So, as per your storage, a data scientist or a data analytics team needs to query again and again, which does not suit a data-heavy organization.

It's a good return on investment for Databricks from a delivery perspective. Delivered multiple dashboards. So, it's quite a good return on investment. And being a small organization, everyone can use Databricks, and cost-wise, it's also good for small organizations.

If the company is a startup, Databricks might be suitable. If a big company needs a lot of storage, Teradata might be best for them. It depends on the situation.

Overall, I would rate the solution a eight out of ten. I would definitely recommend this solution for small organizations. 

Our primary use case is in our project; we are dealing with Duo Special Data, where we need a lot of computing resources. Here, the traditional warehouse cannot handle the amount of data we are using, and this is where Databricks comes into the picture. 

The processing capacity is tremendous in the database. We are dealing with Azure as storage, so we have not faced any challenges. And also the connectors to different data sources. Moreover, it is not a language-dependent tool. Therefore, development also takes place faster. It is one of the best features of Databricks.

There is room for improvement in the documentation of processes and how it works. I was trying to get one of the certifications, so I saw an area of improvement there. 

I have been using Databricks for eight to nine months.

It is a stable product for us. We didn't see any challenges. 

There are around 30 to 35 users in our organization. 

The initial setup was easy because the third-party team made the clusters for us. 

A third-party team enabled the cluster to make the setup easy for us. 

I would advise using it based on the use case because it easily handles big data. It is your go-to tool if you are dealing with massive data. 

Overall, I would rate the solution a nine out of ten. The tool performs well in various use cases, availability of documentation online, and compatibility with big data systems like GCP, Azure, or AWS.

Our company uses the solution's Spark module for big data analytics as a  processing engine.  

We do not use the module as a streaming engine. The historic perception is that Spark is for batches, machine learning, analytics, and big data processing but not for streaming and that is exactly how we use it. 

The solution's features are fantastic and include interactive clusters that perform at top speed when compared to other solutions.

The ATC monitoring experience and the maturity of the APIs are very good. 

CI/CD needs additional leverage and support. Community forums are helpful for gaining knowledge but the solution should provide specific documentation.

Streaming services such as Flink should be amplified and better supported. 

There are not many connectors to MapReduce.

I have been using the solution for seven years. 

The solution is mature and stable compared to other products. 

The solution is scalable with no issues from a computer perspective.

I received support for initial challenges and it was very good. The support team was very professional and provided the answers I needed. 

I rate support an eight out of ten. 

Positive

I previously used Cloud-Bricks. 

The initial setup is easy for me because I access the solution on a web browser. 

Unilever had a specific team for implementing and managing the solution.

Walmart had a team of ten engineers for implementation and a couple of engineers for management. 

We receive an ROI for our batch constructs. 

The solution is a good value for batch processing and huge workloads. 

The price might be high for use cases that are for streaming or strictly data science. 

I have evaluated multiple options including Cloud-Brick and Dataproc for price versus performance, technical support, and CI/CD approach.

I started as a consumer and used the solution for on-premises deployment with Unilever from a data science perspective. At that time, the solution was in its beta stage but viewed as good, far ahead of its competition, and expensive. The key comparison used to be HDInsight or Adobe Cluster for cloud data and the solution was thought of as a cluster service rather than for unified analytics.

I moved along on my journey to Walmart where I was building their platform and compared it to the solution from a cloud perspective and a cluster service with notebooks. Consumers at the time were using Project Lightspeed and ATC for streaming. Spark was used as a micro-batching engine for machine learning, analytics, and big data processing. At some point, the solution became preferred and more than 100 staff members were leveraging its use.

I found that the solution had interesting features that I liked such as its notebook, interactive clusters with fast speed, and the ATC monitoring experience. I did not like the solution from a CI/CD perspective because it had a rigidity in terms of the approval process.

The solution grew from that original space and, by the time I had moved to Microsoft, was partnered with Microsoft Azure. An integration with ADF and other products solved the CI/CD issues for me.

I am now leading streaming platforms for Walmart so my interest is in the solution's streaming capabilities. I began building a streaming platform using Spark PM in Microsoft so the solution was its key competitor. Then the solution launched a vectorized machine on Photon for the Spark engine. Its performance was a key factor in moving from Microsoft because it performed much better than other products including opensource Spark, Microsoft Synapse Spark, and Dataproc.

It is important to do POCs and run tests to control the meter that also controls the price. The meter can go really high from a computing perspective if POCs and settings are not streamlined. 

I rate the solution an eight out of ten. 

We use this solution for finding anomalies and applying the rules to the streaming data.

There are around 50 people using this solution in my organization, including data scientists.

The ability to stream data and the windowing feature are valuable. There are a number of targeted integration points, so that is a difference between Stream Analytics and Databricks. The integrations input or output are better in Databricks. It's accessible to use any of the Python or even Java. I can use the third party, deploy it, and use it.

Support for Microsoft technology and the compatibility with the .NET framework is somewhat missing. There should be reliability between these two. Databricks is based on open sources. If it's more synchronous between the Microsoft technology and the programming languages, it'll be better. Python has better languages, but compatibility would be a great help.

I would like to have better support for Microsoft technology and better language components.

With Azure or Cosmo DB, I can store other data links or time series data tables. That would be a great help for analytics in real time.

I have been using Databricks for eight months.

The scalability is fine. We had thousands of devices and were sending data infrequently, so that worked for us. If the amount increases, the windowing function and job schedule may not perform as expected.

I would rate technical support 4 out of 5. We had some issues with setup, and they were finally solved but it was after following up a few times.

Azure Stream Analytics is easy to use and easy to deploy. It's a little bit better. Databricks is still having some stability issues. Azure Stream Analytics has a few input and output sources, and it's scalable to all types of third party or interfaces.

Setup was complex. There were some issues with setting up a database and installing the third party component on top of services. I would rate the setup 3 out of 5.

Implementation was done in-house.

The cost is around $600,000 for 50 users.

I would rate the price 2 out of 5.

I would rate this solution 8 out of 10.

We're using it to provide a unified development experience for all our data experts, including all data engineers, data scientists, and IT engineers. With the Databrick Platform we allows teams to collaborate easily towards building Data Science models for our clients. The development environment allows us to ingest data from various data sources, scale the data processing and expose them either trough API or through enriched datasets made available to web app or dashboard leveraging the serverless capacities of SQL warehouse endpoints.

Databricks allowed us to offer an homogeneous development environment accross different accounts and domains, and also across different clouds. The upskilling of our employees is far more linear and faster, while removing the complexity of infrastructure management. This lead to an increased collaboration between domain thanks to a better onboarding experience, more performant pipelines and a smoother industrialization process. Overall client satisfaction has increased and the time to first insight has been reduced.

The shared experience of collaborative notebooks is probably the most useful aspect since, as an expert, it allows me to help my juniors debug their books and their code live. I can do some live coding with them or help them find the errors very efficiently.

It has become very simple to set up thanks to its official Terraform provider and the open-source modules made available on GitHub.

I love Databricks due to the fact that we can now deploy it in 15 minutes and it's ready to use. That's very nice since we often help our clients in deploying their first Data Platform with Databricks.

The solution is stable, with LTS Runtimes that have proven to remain stable over the years. 

I would love to be able to declare my workflows as-code, in an Airflow-like way. This would help creating more robust ingestion python modules we can test, share and update within the company. 

We would also love to have access to cluster metrics in a programmatic way, so that we can analyse hardware logs and identify potential bottlenecks to optimize.

Lastly, the latest VS Code extension has proven to be useful and appreciated by the community, as it allows to develop locally and benefits from traditional software best-practices tools like pre-commits for example.

I've been using the solution for more than four years now, in the context of PoC to full end-to-end Data Platform deployment.

The product is very stable. I've been using it for three years now, and I have projects that have been running for three years without any big issues.

It's very scalable. I have a project that started as a proof of concept on connected cars. We had 100 cars to track at first - just for the proof of concept. Now we have millions of cars that are being tracked. It scales very well. We have terabytes of data every day and it doesn't even flinch.

I've had very good experiences with technical support where they answer me in a couple of hours. Sometimes it takes a bit longer. It's usually a matter of days, so it's very good overall. 

Even if it took a bit of time, I got my answer. They never left me without an answer or a solution.

Positive

The implementation is very simple to set up. That's why we choose it over many other tools. Its Terraform provider is our way-to-go for the initial setup has we are reusing templates to get a functional workspace in minutes.

Usually, we have two to five data engineers handling the maintenance and running of our solutions.

We deploy it in-house.

The solution is a bit expensive. That said, it's worth it. I see it as an Apple product. For example, the iPhone is very expensive, yet you get what you pay for.

The cost depends on the size of your data. If you have lots of data, it's going to be more expensive since your paper compute units will be more. My smallest project is around a hundred euros, and my most expensive is just under a thousand euros a week. That is based on terabytes of data processed each month.

We looked into Azure Synapse as an alternative, as well as Azure ML and Vertex on GCP. Vertex AI would be the main alternative.

Some people consider Snowflake a competitor; however, we can't deploy Snowflake ourselves just like we deploy Databricks ourselves. We use that as an advantage when we sell Databricks to our clients. We say, "If you go with us, we are going to deploy Databricks in your environment in 15 minutes," and they really like it.

Lately Fabric was released and can offer quite a similar product as Databricks. Yet, the user experience, the CI/CD capabilities and the frequent release cycle of Databricks remains a strong advantage.

We're a partner.

We use the solution on various clouds. Mostly it is Aure. However, we also have Google and AWS as well. 

One of the big advantages is that it works across domains. I'm responsible for a data engineering team. However, I work on the same platform with data scientists, and I'm very close to my IT team, who is in charge of the data access and data access control, and they can manage all the accesses from one point to all the data assets. It's very useful for me as a data engineer. I'm sure that my IT director would say it's very useful for him too. They managed to build a solution that can very easily cross responsibilities. It unifies all the challenges in one place and solves them all mostly.

I'd rate the solution nine out of ten.

We use the solution for reliability engineering, where we apply ML and Deep Learning models to identify the fear failure patterns across different geographies and products.

Databricks is hosted on the cloud. It is very easy to collaborate with other team members who are working on it. It is production-ready code, and scheduling the jobs is easy.

Databricks would have more collaborative features than it has. It should have some more customization for the jobs. Also, it has an average dashboarding tool. They can bring advanced features so we don't depend on other BI tools to build a dashboard. We are using Tableau to create a dashboard. If Databricks has more advanced features, we can entirely use Databricks.

I have been using Databricks for one year.

The product is stable. It has been giving consistent outputs without any major issues.

The solution is hosted on the cloud. It supports high scalability features.

10-20 users are using this solution.

There was a training session from Databricks where they explained how to use it. We never had to contact them because they had already given us proper training on the platform.

I have used Alteryx before. We switched to Databricks because it can compute and turn your code into production-ready code in very few seconds. Also, the stability is relatively high.

The initial setup is easy.

We have a dedicated team for the deployment.

Delta Lake is a free system. We practically work on the data that we get from Snowflake. Databricks are returned to the model outputs that are returned to Delta Lake. It is easy for us to collaborate using Delta Lake, and the computation speed is also quite high for Delta Lake.

The learning curve for Databricks is not very steep. It's pretty easy, and you will find a lot of materials online. So, if you are comfortable coding in Python, it's very straightforward. There is nothing to worry about when using Databricks.

Overall, I rate the solution a ten out of ten.

We use the solution for data analytics of industrial data.

We extensively use the product’s notebooks, jobs, and triggers. We can create activities. Wherever translation is required, we use Databricks. The product fulfills our customer requirements. It is a cost-effective solution.

The product should provide more advanced features in future releases.

I have been using the solution for six months.

Our data was not too huge. It worked well. It is easily adaptable.

The tool is scalable. We can make it available for a larger audience.

The initial setup is not that difficult. I rate the ease of setup a seven out of ten. The solution is cloud-based. We use native services like Data Factory for orchestration. Sometimes, the customers require us to use Amazon as the cloud provider instead of Azure.

The pricing is average.

There are many services which are coming up. They are still in the preview stage. Overall, I rate the product an eight out of ten.