From our experience, Azure Databricks could be improved with simpler cluster management and more predictable cost visibility and enhanced native monitoring for large enterprise environments. We would also like richer governance features, better debugging for distributed Spark jobs, and more granular controls for workload optimization over and across multiple teams, which we have at multiple customer environments and within our organization. In day-to-day operations, troubleshooting failed Spark jobs can still be time-consuming, especially in complex distributed workloads. We would like clearer root cause diagnostics and more actionable performance recommendations within Azure Databricks. Better cost optimization insights at the job and cluster level would also help us manage large multiple team environments more efficiently.
