I am pleased to share that Buoyant Data will be attending Data and AI Summit 2024. There are so many great sessions already on the agenda with a couple featuring Buoyant Data!
The past year since our last appearance at Data and AI Summit have been quite the whirlwind, with the contribution of schema evolution and change data capture to delta-rs along with the introduction of the oxbow lambdas for fast, lightweight, and cheap Lambda-based Delta ingestion on AWS. I also had the opportunity to contribute a chapter to the upcoming "Delta Lake: The Definitive Guide Book" discussing some of the novel ways in which Delta Lake can be used in Python or Rust to achieve great performance in a small package.
Below are a list of some of the sessions I will be participating in. If you're interested in talking more about Rust development, Delta Lake architecture, or highly efficient data platforms, please reach out!.
The following sessions we will be hosting or participating in on Thursday June 13th:
Delta Lake AMA
Have Delta Lake and Data Engineering questions? Join this Delta Lake Ask Us Anything with Delta Lake maintainers and contributors!
Fast, Cheap and Easy Data Ingestion with AWS Lambda and Delta Lake
In this session we will dive into examples of how to work with Delta tables from AWS Lambdas written in Python and Rust. For many ingestion, or lightweight data processing workloads AWS Lambda provides a fast, easy, and cheap execution environment. Lambda can be easily triggered from Kinesis, SQS, Kafka, S3 Event Notifications, and more, making it a powerful tool to consider when moving from the "nightly batch" to a more event-driven data platform.
Using the native Python or Rust libraries for Delta Lake we will explore the transaction log, write updates, perform table maintenance, and even query Delta tables in milliseconds from AWS Lambda. After this session attendees will be ready to incorporate Lambda into their existing AWS-based data platforms.
Let's Do Some Data Engineering With Rust and Delta Lake!
The future of data engineering is looking increasingly Rusty. By adopting the foundational crates of Delta Lake, data fusion, and arrow, developers can write high-performance and low-cost ingestion pipelines, transformation jobs, and data query applications. Attendees don't need to know Rust ahead of time; we will review some fundamental concepts of the language as they pertain to the data engineering domain. The main goal of this session is to provide attendees with a starting point to learn Rust by applying it to the real-world data problems they're already familiar with:
- Ingesting semi-structured data into Delta tables (e.g. CSV, JSON, etc.)
- Enriching data from multiple tables to create new silver/gold tables
- Performing table management (e.g. OPTIMIZE, VACUUM)
- Exporting data from Delta tables to external systems (e.g. Elastic/OpenSearch)
There are a ton of other great events at and around Data and AI Summit. If you find yourself in the San Francisco Bay Area, don't skip it!
