Cloud-based data platforms can cost big bucks if you're not careful, fortunately with Databricks there are some very easy cost optimizations within reach. Cloud providers and Databricks both charge on a utilization basis. The quickest way to save money is to reduce usage, but for many organizations that is not an option. When reviewing "total cost of ownership" it is tempting to look at the cost of Databricks itself and attempt to trim that usage, but I firmly believe that a Databricks-based platform is cheaper than an AWS, GCP, or Azure-only data platform. In this post, we'll review five tips that will help you squeeze even more performance and value out of your data!
Use Photon
Databricks has a heavily optimized Spark runtime called Photon which can greatly improve the run times for heavy Spark operations. Photon is faster for most SQL, Python, or Scala workloads that run on Apache Spark. Less time to complete the job, means fewer compute hours to pay to the cloud provider!
Photon does not support every operation in Apache Spark and has an automatic fallback to the JVM-based Spark engine, which makes it safe to simply drop-in for many workloads. That said, not all Spark jobs will be faster with Photon! In our observations almost all DataFrame-based jobs will see a speed up, but RDD-based workloads or those which monopolize CPU on the driver node typically don't see an increase in performance with Photon.
When launching a cluster, Photon is a checkbox away, so it's worth trying out!
Spot only workers with Fleet Clusters
The typical Databricks cluster we will see in AWS is provisioned with the driver node using an "On Demand" instance, followed by some number of Spot instances using the "Fallback to On Demand" setting. For a lot of workloads we have observed it is cheaper and safe to only use Spot for workers.
This is specially valuable when coupled with Fleet Clusters which provide some extra magic to find suitable Spot capacity across availability zones allowing your workloads to find the cheapest possible compute available for your workloads.
In the cases where insufficient Spot capacity exists, the job will simply run with fewer resources. In our tests we have seen that it is cheaper to let a job run a little longer with fewer Spot instances, than to "fall back" to more expensive On Demand instances.
NOTE: Depending on the region of your workloads, you may see increased Spot
competition which can drive up
prices. We recommend
us-east-2 and us-west-2 for our North American customers.
Reduce additional EBS
For many of our customers the cost of EBS is almost as high as the compute resources used by their workloads! By default Databricks will provision two EBS volumes for every worker in a cluster:
- 30GB volume for the OS and Databricks services
- 150GB volume for the Spark worker and logs
Many Databricks users will inadvertently increase their costs by adding more EBS volumes in their configurations! Many workloads can be safely run without additional EBS volumes, which are typically used to handle workloads with excessive shuffles or other poor performance characteristics.
There is an argument to be made for adding additional EBS volumes to enable Delta caching, but I am of the opinion that the hit rates for most jobs are so low that Delta caching is not a performance gain and simply represents unnecessary cost.
Use SQL Serverless
In 2021 Databricks announced "serverless compute" for SQL workloads. Rather than compute running in your AWS account, SQL Serverless runs compute in Databricks' account. There are a number of reasons that this compute can be cheaper than running SQL workloads in your own AWS account, but SQL Serverless has a killer feature: extremely fast provision times.
For our customers we encourage setting up SQL Warehouses (formerly known as SQL Endpoints) with zero minimum capacity which costs them $0 most of the time. When a user starts running queries clusters are auto-provisioned in seconds on the Databricks backend, and can be configured to rapidly shutdown after use.
We expect Databricks to support more types of workloads in SQL Serverless in the future, which will allow many customers to take advantage of these rapid provisioning times, but also the discounts and optimizations a bigger cloud user like Databricks is able to command from AWS.
Avoid certain instances
Do not use i3 instances on AWS. There was a time when the i3 instance
family was a requirement to use Photon, since it's direct attached NVMe storage
allowed for high local disk performance. The downside of the i3 family is
that they are highly saught after for a number of disk heavy applications and
as such as one of the most frequently interrupted Spot instance families,
leading clusters trying to use those instance types to require On Demand
capacity.
If your Spark workloads benefit from fast locally attached storage, we
recommend m6d or c5d instance families instead. Consult the AWS Spot
instance advisor to get a
sense of which instance types to avoid in your region.
In our work we have found that relying on Databricks Fleet Clusters rather than specifying instance types directly can provide a simple and future-proof way to select the best priced instance types available!
Following these five tips can have a noticeable positive impact on your monthly data platform bills. As is typically the case, once you have made the easy changes, additional cost optimizations require a bit more measurement, analysis, and experimentation. If you'd like to take your cost optimization project to the next level, we can help! Drop me an email and we'll chat!!
