Lessons learned building delta-rs
Reviewing some of the lessons learned building the delta-rs tooling and community.
Big data adds significant value to your organization but can also add significant cost. Buoyant Data specializes in improving data infrastructure with high performance low-cost ingestion and transformation pipelines with Rust, Python, Databricks, and AWS.
Let's improve the ROI of your data platform!
Contact usAs creators of the deltalake Python and Rust packages,
we have been supporting Delta Lake applications since the beginning.
Buoyant Data offers one-time on-demand support as well as on-going
technical support subscriptions for your team!
With years of experience creating and deploying Rust data applications with delta-rs, kafka-delta-ingest, and more, Buoyant Data can help your organization adopt and excel with high-performance, low-cost data services or AWS Lambdas built with Rust.
Our expertise in leveraging Delta Lake includes both the Databricks Platform (Serverless, Unity, etc) and the AWS Data Platform (Glue, Athena, EMR). We can help design and implement a scalable and efficient data platform for your organization.
For organizations with existing data infrastructure and analytics, we can analyze and optimize in-place to squeeze faster queries and lower costs out of your existing data platform without substantial rearchitecture.
Expert insights on all things Delta Lake--including how to run batch and streaming jobs concurrently and accelerate the usability of your data. You'll also uncover how ACID transactions bring reliability to data lakehouses at scale.
Reviewing some of the lessons learned building the delta-rs tooling and community.
Serverless data ingestion can be extremely cost effective but limitations of AWS Lambda can result in transaction log bloat. In this post we'll discuss the "BUFFER_MORE" feature in sqs-ingest and how it helps get more bang for your Lambda buck.
Facing a large backlog of data it is tempting to horizontally scale Delta writers as much as compute and budget will allow. In this post we'll dive into how this can be counter propductive and actually slow throughput rather than accelerate it!
