Less is more: scaling streaming Delta Lake applications
Facing a large backlog of data it is tempting to horizontally scale Delta writers as much as compute and budget will allow. In this post we'll dive into how this can be counter propductive and actually slow throughput rather than accelerate it!