Publish Arc as an Apache Iceberg Table, Zero-Copy

Iceberg has quietly become the default table format for the analytics world. Snowflake reads it, Databricks reads it, Trino reads it, DuckDB reads it, BigQuery reads it. If your data is an Iceberg table, it's queryable by essentially every engine that matters: no custom connector, no proprietary API in the path.

Arc already writes Apache Parquet. Iceberg tables are made of Parquet. So the question we kept asking ourselves was: why should anyone have to re-export Arc's data into a second copy of Parquet just to get a lakehouse table? They shouldn't.

Starting in Arc 26.09.1, they don't. Flip one flag and Arc publishes its existing data as an Iceberg table, by reference, zero-copy. No data is rewritten. The same Parquet files that serve Arc's native SQL API become a standard Iceberg table that Spark, Trino, DuckDB, Snowflake, Dremio, and PyIceberg can all read.

Coming in September 2026. Iceberg export ships in Arc 26.09.1. Everything below was captured end-to-end on a preview build, so the commands and output are real, but the flag isn't in a public release yet.

This post is the full walkthrough. Real commands, real output, captured end-to-end on the 26.09.1 build.

Iceberg is a table format, not a file format

Worth being precise about, because it's the crux of why this is free.

Apache Iceberg is a metadata layer. It takes a pile of data files and turns them into a coherent, transactional table: a schema, snapshots, partition information, and a manifest of which files belong to the table right now. The data itself is still Parquet (or ORC/Avro). Iceberg just describes it.

Arc writes Parquet as it ingests. That means the expensive part, encoding columnar data files, is already done. All Iceberg export has to do is register Arc's existing files into an Iceberg table and keep that registration in sync as compaction and retention change the files underneath. No re-encoding, no second copy.

That's a real differentiator. Many streaming and time-series systems that offer "Iceberg output" re-export your data into new Parquet files, so you pay for the storage twice. Arc doesn't need to, because it already writes the open format.

How it works

A background reconciler runs on a timer (every 5 minutes by default). Each pass, for every measurement, it:

1. Walks the measurement's Parquet files on storage.
2. Diffs them against the Iceberg table's current file set.
3. Commits the delta in one Iceberg snapshot (adding files that were just written, removing files that compaction or retention deleted) without rewriting any data.
4. Expires old snapshots on a retention policy so metadata stays bounded.

Because it's driven by what's actually on storage rather than a transient event stream, it's self-healing: a missed or failed pass just converges on the next tick. And measurements whose files haven't changed since the last pass are skipped entirely, so steady state is cheap.

Crucially, this runs on a background timer and never touches Arc's write path. Ingestion throughput is unaffected.

Turn it on

One flag:

# arc.toml
[storage]
backend = "local"     # v1 requires a local backend (see Limitations)
 
[iceberg]
enabled = true        # default: false

Or via env var:

ARC_ICEBERG_ENABLED=true

Restart Arc. In the startup log you'll see the exporter come up:

{"level":"info","component":"iceberg","message":"Iceberg reconcile scheduler started","interval":300000}
{"level":"info","warehouse":"file:///var/lib/arc/data","interval_s":300,"message":"Iceberg export enabled"}

End-to-end: ingest, then read from three engines

Let's do the whole thing. I'll ingest telemetry, then read it back from Arc's native API, from DuckDB, and from PyIceberg. The exact same table, three different engines.

1. Ingest some data

Standard InfluxDB line protocol. 2,000 CPU samples across four locations and eight hosts:

curl -X POST "http://localhost:8000/write?db=telemetry" --data-binary @cpu.lp
# HTTP 204

Within one reconcile interval, the exporter registers the file. From the log:

{"component":"iceberg-exporter","database":"telemetry","measurement":"cpu","message":"Created Iceberg table"}
{"component":"iceberg-exporter","database":"telemetry","measurement":"cpu","added":1,"removed":0,"message":"Reconciled Iceberg table"}
{"component":"iceberg-scheduler","reconciled":1,"unchanged":0,"failed":0,"message":"Iceberg reconcile pass complete"}

The table lands at arc_telemetry.db/cpu/ under your storage root. Here's the whole warehouse, and note it's all metadata:

arc_telemetry.db/cpu/metadata/
├── 00000-…-.metadata.json          1107 B   ← table metadata
├── 00001-…-.metadata.json          2512 B
├── 20407294-…-m1.avro              4121 B   ← manifest (lists data files)
├── snap-4094801706336749915-….avro 2246 B   ← snapshot
├── v0.metadata.json                1107 B   ← reader-friendly copies
├── v1.metadata.json                2512 B
└── version-hint.text                  1 B   ← "current version" pointer

There is no Parquet in that directory. The data still lives exactly where Arc wrote it:

telemetry/cpu/2026/07/14/16/cpu_20260714_165536_340806000.parquet

The Iceberg manifest points at that path. Zero copies. The metadata is a few kilobytes regardless of how much data it describes.

2. Read it from Arc's native SQL API

Arc still owns the data and answers its own queries, unchanged:

curl -X POST http://localhost:8000/api/v1/query \
  -H "x-arc-database: telemetry" \
  -d '{"sql":"SELECT count(*) AS rows, count(DISTINCT host) AS hosts FROM cpu"}'

{"success":true,"columns":["rows","hosts"],"data":[[2000,8]],"execution_time_ms":4}

3. Read the same table from DuckDB

Point DuckDB's iceberg_scan at the table directory. Arc emits a version-hint.text, so directory-based readers resolve the current snapshot automatically, with no need to know the exact metadata filename:

INSTALL iceberg; LOAD iceberg;
 
SELECT count(*) AS total_rows
FROM iceberg_scan('/var/lib/arc/data/arc_telemetry.db/cpu');

┌────────────┐
│ total_rows │
├────────────┤
│       2000 │
└────────────┘

The schema comes through with proper Iceberg types. Note time maps to a timezone-aware timestamp:

┌─────────────┬──────────────────────────┐
│ column_name │       column_type        │
├─────────────┼──────────────────────────┤
│ host        │ VARCHAR                  │
│ location    │ VARCHAR                  │
│ mem_pct     │ BIGINT                   │
│ temperature │ DOUBLE                   │
│ load        │ DOUBLE                   │
│ time        │ TIMESTAMP WITH TIME ZONE │
└─────────────┴──────────────────────────┘

And a real analytical query runs directly against Arc's data, through Iceberg, in DuckDB, with no export step:

LOAD iceberg;
SELECT location,
       count(*)                   AS samples,
       round(avg(temperature), 2) AS avg_temp_c,
       round(max(load), 1)        AS peak_load_pct
FROM iceberg_scan('/var/lib/arc/data/arc_telemetry.db/cpu')
GROUP BY location
ORDER BY location;

┌──────────┬─────────┬────────────┬───────────────┐
│ location │ samples │ avg_temp_c │ peak_load_pct │
├──────────┼─────────┼────────────┼───────────────┤
│ dc-east  │     500 │       23.1 │          99.9 │
│ dc-west  │     500 │      22.78 │          99.9 │
│ edge-01  │     500 │      22.86 │         100.0 │
│ edge-02  │     500 │      23.28 │         100.0 │
└──────────┴─────────┴────────────┴───────────────┘

4. Read it from PyIceberg via the catalog

Arc keeps an Iceberg SQL catalog in its SQLite database, so PyIceberg can discover and load tables by name, with no file paths required:

from pyiceberg.catalog.sql import SqlCatalog
 
cat = SqlCatalog("arc", **{
    "uri": "sqlite:////var/lib/arc/data/arc.db",   # Arc's catalog DB
    "warehouse": "file:///var/lib/arc/data",        # storage root
})
 
print(cat.list_tables("arc_telemetry"))
# [('arc_telemetry', 'cpu')]
 
table = cat.load_table(("arc_telemetry", "cpu"))
print(table.schema())
df = table.scan().to_arrow()
print("rows:", df.num_rows)

[('arc_telemetry', 'cpu')]
table {
  1: host: optional string
  2: location: optional string
  3: mem_pct: optional long
  4: temperature: optional double
  5: load: optional double
  6: time: optional timestamptz
}
rows: 2000

Same table. Same 2,000 rows. Field IDs, timestamptz, the works. A fully standard Iceberg table.

It's a living table, not a one-time dump

The reconciler keeps the Iceberg table in sync with Arc. Ingest a second batch:

curl -X POST "http://localhost:8000/write?db=telemetry" --data-binary @cpu2.lp   # +1000 rows

Next reconcile pass picks it up as a new snapshot:

{"component":"iceberg-exporter","added":1,"removed":0,"message":"Reconciled Iceberg table"}

DuckDB, with no cache to bust, nothing to re-point, immediately sees the new total:

┌────────────┐
│ total_rows │
├────────────┤
│       3000 │   ← was 2000
└────────────┘

And the snapshot history reads exactly like a well-behaved Iceberg table:

format-version : 2
partition-spec : day(time)
snapshots      : 2
  snapshot 4094801706336749915  op=append  added-data-files=1  total-data-files=1  total-records=2000
  snapshot 5393730711960777602  op=append  added-data-files=1  total-data-files=2  total-records=3000

Two appends, a clean lineage, day(time) partitioning. This is what compaction and retention plug into: when compaction replaces ten small files with one big one, the reconciler's next pass removes the ten and adds the one in a single snapshot; when retention deletes an old day, the files leave the table the same way. Arc's storage lifecycle drives the Iceberg table.

Reading from other engines

• Spark: load the table directory with the Iceberg runtime:

  df = spark.read.format("iceberg").load("file:///var/lib/arc/data/arc_telemetry.db/cpu")
  df.createOrReplaceTempView("cpu")
  spark.sql("SELECT count(*) FROM cpu").show()

• Trino / Snowflake / Glue-based engines: these want a shared catalog service. Point a REST or JDBC catalog at the warehouse for broad multi-engine discovery. (PyIceberg and DuckDB work today against the SQLite catalog / table directory directly.)

Limitations in v1

Being upfront, this is a deliberately scoped first version:

• Local storage only. Iceberg export requires storage.backend = "local". Arc refuses to start if it's enabled with a non-local primary backend or with cold-tier tiering, because a file migrated to object storage would silently leave the Iceberg table. This is a guard, not a gotcha. It fails loud at startup.
• Eventual consistency. The Iceberg view reflects the last reconcile pass, so it lags live ingest by up to reconcile_interval. Expected for a lakehouse export.
• On-disk portability. Iceberg metadata references files by absolute path. Tables read fine on the same host; moving a local warehouse to a different path/host needs re-pointing. (Object-store warehouses, coming later, avoid this.)
• Single writer in a cluster. Exactly one node runs the reconciler; under the compactor failover lease that's guaranteed.

Backups already understand Iceberg: Arc's backup includes the warehouse metadata alongside the Parquet, so a restore keeps your tables intact.

Why we built it this way

The whole design goal was no second copy and no lock-in, taken one step further. Arc already stores open Parquet files you own. Iceberg export makes those same files a standard lakehouse table the entire ecosystem can read, without a proprietary API in the query path, without duplicating storage, and without touching the write path that keeps ingestion fast.

Your data goes in through Arc. It comes out through Arc's SQL API and through every Iceberg-aware engine on the planet, reading the exact same bytes.

Shipping in Arc 26.09.1 in September 2026. Full configuration reference and per-engine read instructions are in the Apache Iceberg integration docs.