This talk is was given at the 2024 Data and AI Summit hosted by Databricks.
In this session we will dive into examples of how to work with Delta tables from AWS Lambdas written in Python and Rust. For many ingestion, or lightweight data processing workloads AWS Lambda provides a fast, easy, and cheap execution environment.
Lambda can be easily triggered from Kinesis, SQS, Kafka, S3 Event Notifications, and more, making it a powerful tool to consider when moving from the ""nightly batch"" to a more event-driven data platform. Using the native Python or Rust libraries for Delta Lake we will explore the transaction log, write updates, perform table maintenance, and even query Delta tables in milliseconds from AWS Lambda.
