Inside DuckDB 2.0 🦆, ACID for AI Agents ✅, Variant Lands in Iceberg 🪆

DuckDB’s upcoming 2.0 release turns the database into a much broader platform, adding client/server operation, triggers, first-class semi-structured ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌  ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ 

Sign Up |Advertise|View Online TLDR TLDR Data 2026-08-20 📱 Deep Dives Agentic Transaction: Towards ACID-Compliant Agent Systems (22 minute read) Researchers apply database-style ACID guarantees to AI agents, treating multi-step workflows as transactions that validate before committing changes, isolate failed attempts, and preserve durable state. Its ACID-Agent implementation outperformed Claude Code by up to 10.6% on a data agent benchmark, suggesting reliability-oriented agent architecture can matter as much as model scale. Software engineering at a proprietary trading company: Optiver (17 minute read) Optiver's platform shows what happens when latency, reliability, and ML quality map directly to P&L. Roughly 950 engineers support 10M+ daily trades across a stack spanning C++, Python, Rust, Kafka, Postgres, Databricks, custom Linux kernels, and co-located hardware. The pattern is deep infrastructure ownership, not generic speed culture. Simplifying data and product integrations with a data abstraction layer (8 minute read) Uber's Data Abstraction Layer decouples product integrations from changing physical tables, schemas, and backends. A logical table plus FetchData API can resolve queries across real-time and daily tables, OLAP stores, Docstore, and Hive, then roll up metrics by requested dimensions. Advertiser reporting delivery fell from weeks to under two days. Behind the Scenes: Evolving Netflix's Ads Event Pipeline for Live (11 minute read) Netflix rebuilt its live ads event pipeline on Apache Flink, moving metadata enrichment out of the ad-serving critical path and into stateful streaming joins. The design handles region-aware routing, late events, hourly Spark recovery, best-effort deduplication, and dual-run validation that reached over 99.99% agreement before the legacy system was retired. 🚀 Opinions & Advice Postgres 19: How Our Advice Has Changed Since We Wrote It (16 minute read) Postgres 19 mostly strengthens proven practices rather than changing them, building on async I/O with autoscaling workers while adding more resilient COPY, LZ4 compression by default, improved indexing, and smoother partition management. The core advice remains the same: use COPY for bulk loads, model hot data structurally, choose indexes carefully, and partition primarily for lifecycle management. Spark Tips. Partition Tuning (12 minute read) Spark partition sizing is a throughput and stability lever: aim for enough partitions to use cores without creating tiny tasks, filter early, avoid unnecessary shuffles, and repartition around joins or writes when the current layout is wrong. The practical advice covers AQE, skew, spills, checkpoint barriers, and small-file tradeoffs. 💻 Launches & Tools A Preview of DuckDB v2.0 (16 minute read) DuckDB's upcoming 2.0 release turns the database into a much broader platform, adding client/server operation, triggers, first-class semi-structured VARIANT data, asynchronous I/O, a new SQL parser, and major storage and query-performance improvements. It also strengthens DuckDB for long-running transactional workloads and makes extensions easier to build, distribute, and maintain across versions. Semi-Structured Data in Apache Iceberg: Meet the Variant Type (3 minute read) Iceberg v3 added a Variant type for semi-structured data, so fast-changing JSON-like records can live in one column while preserving native types like timestamps, decimals, and binary values. It uses Parquet's binary encoding, works across supported engines including Spark and Flink, and can be queried from Spark SQL with variant_get. I spent 10 hours vibe-coding a tool to visualize any Parquet file's internals (2 minute read) A small browser tool parses Parquet files with Rust arrow-rs and exposes the internals that usually stay hidden: row groups, column chunks, pages, schemas, encodings, Bloom filters, sizes, and min/max statistics. It is a useful teaching and debugging aid for understanding why columnar files behave the way they do. 🎁 Miscellaneous (Re)Building a FAQ System for DataTalks.Club (12 minute read) DataTalks.Club rebuilt its FAQ bot around curated GitHub issues, Slack threads, YouTube transcripts, and explicit evaluation cases. The serverless design keeps retrieval simple enough for AWS Lambda, but puts effort into labels, review batches, and costly false-closure checks. It is a useful reminder that RAG quality starts with maintained data. We turned off Pub/Sub and nobody noticed (17 minute read) The team at Incident.io eliminated Google Pub/Sub as a single point of failure by adding NATS and building an active-active load balancer that automatically routes messages between both brokers. Production tests, including fully disabling Pub/Sub, caused no dropped messages or customer impact. ⚡ Quick Links We will ship the first Polars 2.0 release candidate next week (1 minute read) Polars 2.0 is nearing release, with the first release candidate due next week. DFlash 2: Keep Drafting Parallel (10 minute read) DFlash 2 makes speculative decoding more efficient, delivering roughly 3x faster LLM inference by accepting more tokens per pass while keeping output unchanged. Massively parallel Postgres backups (7 minute read) PlanetScale's sharded Postgres backup design turns each shard into a parallel worker that restores a good snapshot from S3, replays WAL, and freezes a consistent point-in-time copy. Want to advertise in TLDR? 📰 If your company is interested in reaching an audience of data engineering professionals and decision makers, you may want to advertise with us. Want to work at TLDR? 💼 Apply here, create your own role or send a friend's resume to jobs@tldr.tech and get $1k if we hire them! TLDR is one of Inc.'s Best Bootstrapped businesses of 2025. If you have any comments or feedback, just respond to this email! Thanks for reading, Joel Van Veluwen, Tzu-Ruey Ching & Remi Turpaud Manage your subscriptions to our other newsletters on tech, startups, and programming. Or if TLDR Data isn't for you, please unsubscribe.