litelink
An embedded Iceberg storage engine for append-only data.
What DuckDB is to query execution, litelink is to the durable write path. It is a Python library for high-throughput, append-only writes, and it needs no server, daemon, catalog service, or cluster.
Each append commits to a local SQLite buffer and returns once the row is durable. litelink then seals rows into sorted Parquet, compacts small files, publishes them to an Apache Iceberg v2 table on local disk or S3, and evicts the local copies it no longer needs.
Why it exists
Teams usually run a write path in one system and an analytical store in another, with a job copying data between them. Hand-rolled versions of that pipeline tend to fail the same ways. They produce floods of tiny objects and few-row Parquet files, compaction that never gets scheduled, and in-memory buffers that vanish in a crash.
litelink makes the write path and the analytical store one log, so the Iceberg table another engine reads is the same data the writer produced.
How it works
- Three tiers: a SQLite buffer, a local staging table, and the published table. Reads span all three, and each query's predicates decide which tiers it touches.
- Maintenance is explicit (
seal,compact,publish,evict) and not hidden in a background thread, which keeps it easy to reason about and to test. - Reads come back as Arrow record batches, through
scan()or through SQL via DuckDB. - Optional WAL replication with litestream covers recovery after losing a machine.
Built on litelink
streamcast uses it as its log, and agent-office runs on top of streamcast.