DuckDB 2.0 File Format: Who Can Read Your Data?

#DuckDB#DuckDB 2.0#Parquet#open formats#data portability#vendor lock-in#storage#time-series database#Arc
Cover image for DuckDB 2.0 File Format: Who Can Read Your Data?

DuckDB 2.0.0 LTS shipped on October 21, 2026, and by DuckDB's own account it's their biggest feature release so far. It also changed the storage format. DuckDB's storage documentation is plain about it: "A file created with the v2.0.0 storage format cannot be opened by DuckDB versions prior to v2.0.0."

That's a reasonable engineering decision. The new format loads column metadata lazily, uses DICT_FSST string compression by default and validates more on read, and DuckDB 2.0 still reads and writes existing database files. But it's a good moment to ask a question that matters more for telemetry than for most data: who can read your data, and for how long?

Sensor data, fleet telemetry, flight test recordings and plant history are kept for years, often for decades. The engine you query them with today will be replaced, upgraded or retired several times over that span. The data has to outlive all of them.

So we took the same 511,000 rows, stored them two ways, and tried to open both with as many readers as we could install.

DuckDB 2.0 file vs Parquet: who opened what

One copy is a DuckDB 2.0 database file. The other is what Arc 26.09.4 wrote to disk for the same measurement: 39 ordinary Parquet files.

ReaderArc's Parquet (39 files)DuckDB 2.0 .duckdb file
DuckDB 2.0.0511,000 rows511,000 rows
DuckDB 1.5.5511,000 rowsIO Error: Trying to read a database file with version number 999, but we can only read versions between 64 and 68.
DuckDB 1.4.5511,000 rowsSame error, ending between 64 and 67.
Apache Spark 3.5.7511,000 rowscpu20.duckdb is not a Parquet file. Expected magic number at tail, but found [0, 0, 0, 0]
ClickHouse 26.11 (chDB 4.4.0)511,000 rowsThe table structure cannot be extracted from a Parquet format file.
Apache DataFusion 55.0.0511,000 rowsInvalid Parquet file. Corrupt footer
PyArrow 26.0.1511,000 rowsParquet magic bytes not found in footer.
pandas 3.1.1511,000 rowsSame error (pandas reads Parquet through PyArrow)
Polars 2.1.0511,000 rowsFile out of specification: The file must end with PAR1 or PARE

Three DuckDB versions plus six other engines and libraries all opened Arc's Parquet and counted the same 511,000 rows. The .duckdb file opened in one of them.

That those Parquet readers can't parse a .duckdb file is expected; they were never meant to. The real question is what it takes to open one from somewhere other than DuckDB itself.

What it takes to open a .duckdb file outside DuckDB

Other tools can open a DuckDB file, but only by bringing DuckDB with them: a Python package, a JDBC driver, a connector or a plugin. Each one pins its own DuckDB version, and since 2.0, that version decides whether the file opens at all. We tried the routes we could run, and checked the published versions of the ones we couldn't:

Tool (version tested or checked)Opens a DuckDB 2.0 file?What it takes
Polars 2.1.0Yes, with the right packageThe duckdb Python package, 2.0 or later. With 1.5.5 it fails.
Spark 3.5.7Yes, with the right driverAn 80 MB DuckDB JDBC 2.0 jar, plus a query that casts the TIMESTAMP WITH TIME ZONE column, which Spark rejects (UNRECOGNIZED_SQL_TYPE). With JDBC 1.5.6.0 it fails.
Trino 484NoNothing you can change: the latest Trino (October 16) bundles DuckDB JDBC 1.5.6.0. The same connector read the DuckDB 1.5 file fine.
Grafana DuckDB plugin v0.4.6No, going by the version it bundles (not tested)The latest release (October 22) bundles DuckDB 1.5.6.
pg_duckdb v1.2.0 (PostgreSQL)Experimental (not tested)It bundles DuckDB 2.0.0; the release notes say "Experimental, it is not ready for production use".

Every failure is the same message from an older DuckDB inside the tool. From the duckdb 1.5.5 Python package, and from the 1.5.6.0 JDBC driver in Spark and Trino:

IO Error: Trying to read a database file with version number 999, but we can only read versions between 64 and 68.
IO Error: Trying to read a database file with version number 69, but we can only read versions between 64 and 68.

So a month after DuckDB 2.0 shipped, the newest Trino can't open a DuckDB 2.0 file, the newest Grafana plugin bundles a DuckDB that can't either, and Spark gets there with a driver jar and a hand-written cast. Every route is the DuckDB library again, and each tool upgrades it on its own schedule. The same Spark read Arc's Parquet with one line: spark.read.parquet.

DuckDB storage format compatibility: the way out is an open format

DuckDB's storage documentation is candid about the limits of forward compatibility: "Forward compatibility is provided on a best effort basis. While stability of the storage format is important – there are still many improvements and innovations that we want to make to the storage format in the future. As such, forward compatibility may be (partially) broken on occasion."

Its documented way to move a database between storage formats goes through files. You run EXPORT DATABASE on the older version and IMPORT DATABASE on the newer one, and in between, in DuckDB's words, the export folder holds "the same data in a universal format as different files". Those files are CSV by default, or Parquet if you ask for it.

DuckDB gives you options. 2.0 can still write files older versions read, with an explicit STORAGE_VERSION. It reads and writes Parquet as a first-class format. And a single transactional database file is the right tool for plenty of jobs. Arc is built on DuckDB, and we're glad it keeps getting faster.

But when the data is years of plant or fleet history, we'd rather it live permanently in the kind of format DuckDB's own migration path passes through: files any tool can read.

Arc keeps your data in Parquet, not a database file

Arc runs DuckDB as a query engine, in memory, and never stores your data in a DuckDB database file. In the Arc 26.09.4 server, the two places that open DuckDB do it without a file path: the query engine (internal/database/duckdb.go) and the compaction subprocess, with sql.Open("duckdb", "") (internal/compaction/subprocess.go).

What Arc 26.09.4 left on disk in our test was two kinds of file:

  • Parquet, for the data. At ingest, Arc writes it with Apache Arrow's Go library (created_by: parquet-go version 18.6.0). At compaction, and when you delete rows, Arc rewrites files with DuckDB's Parquet writer, COPY ... (FORMAT PARQUET). Parquet is an open specification, not a DuckDB format.
  • SQLite, for Arc's own metadata. arc.db holds API tokens, RBAC, retention policies and continuous queries. The stock sqlite3 command-line tool lists its tables, like any SQLite file.

The engine version doesn't change that. We also had DuckDB 2.0 write the same rows with the options Arc's compaction uses (ZSTD, 122,880-row groups), the way a future Arc on DuckDB 2.0 would. That file opened in every reader in the table: DuckDB 1.4.5 and 1.5.5, Spark, ClickHouse, DataFusion, PyArrow, pandas and Polars all counted 511,000 rows.

So the files are the product, and the engine is replaceable. Arc has already swapped DuckDB versions under the same files, from 1.4 to 1.5 in 26.05.1, and we're building ArcX, Arc's own query engine, which reads the same Parquet. What an engine upgrade can change is SQL. One of DuckDB 2.0's deprecations, for example, disables the old single-arrow lambda syntax (x -> x + 1) by default, in favor of lambda x: x + 1. Arc 26.09.4 runs DuckDB 1.5.5, so that doesn't affect it today; when Arc moves to DuckDB 2.0, changes like this one will be called out in the release notes. The data on disk stays the same.

We made this argument in August when AWS announced it was acquiring DuckLabs, and in July when InfluxDB 3's new storage engine moved off Parquet to its own .pt files. The table above is what it looks like in practice. If Arc disappeared tomorrow, every tool on that list could still read your data from the Parquet files alone.

How we tested

We ran the released Arc 26.09.4 binary on an Apple Silicon Mac and wrote three hours of backdated telemetry into it with sustained_bench, our load generator. We copied the cpu measurement's 39 Parquet files out, then loaded the same rows into a .duckdb file with DuckDB 2.0.0. DuckDB, PyArrow, pandas, Polars, DataFusion and chDB ran from their PyPI packages, and Spark from the apache/spark:3.5.7-python3 image. Each reader got both copies, and the errors above are quoted from its output.

Arc stores Parquet in hour directories (2026/11/16/13/), not Hive-style key=value folders. Spark read the directory with recursiveFileLookup, and DataFusion, which doesn't descend into those directories, got the list of 39 files. For the .duckdb file we told DataFusion to accept the .duckdb extension, since by default it refuses any file not named .parquet. When a measurement gains columns over time, its files differ in schema; Arc reads them with union_by_name, and other tools need their own schema-merge option. Our test data had one schema.

For the routes into the .duckdb file, Polars 2.1.0 queried it over a connection from the duckdb Python package (1.5.5, then 2.0.0). Spark used the DuckDB JDBC jars from Maven Central (1.5.6.0 and 2.0.0.1) with duckdb.read_only. Trino ran from the trinodb/trino:484 image, with one DuckDB catalog per file. As a control, we loaded the same rows into a DuckDB 1.5.5 file; every route opened that one.

Three of the nine readers are DuckDB versions, and pandas uses PyArrow, so the list covers six independent Parquet implementations. This was one dataset on one machine. It shows which tools open the files Arc writes. It isn't a claim about every Parquet file in existence.

One more detail: Arc has an optional write-ahead log, off by default. When you turn it on, it holds rows in Arc's own format only until they're flushed to Parquet. At rest, your data is Parquet.

DuckDB 1.4 end of support: if you run Arc 26.04.1 or earlier

DuckDB's 2.0 announcement also set a support calendar. 2.0 is supported "until at least October 2027", and DuckDB 1.5's support was extended "until March 2027". And: "The previous LTS release, version 1.4, is still supported until November 17, 2026." That date has now passed.

Every Arc release pins one DuckDB version through the duckdb-go driver.

Arc releaseDuckDB insideDuckDB support
26.04.1 and earlier1.4.xEnded November 17, 2026
26.05.1 to 26.06.31.5.1Until March 2027
26.09.1 to 26.09.41.5.5Until March 2027

If you're on 26.04.1 or earlier, upgrade to 26.09.4. Because your data is Parquet, there is nothing to convert, export or re-ingest.

The short version

DuckDB 2.0's new database files open only through DuckDB 2.0 or later, and other tools get in only by bundling it; today, the latest Trino and Grafana plugin still bundle 1.5. Arc's data is Parquet, and all nine readers opened it, including Parquet written by DuckDB 2.0. And if you're on Arc 26.04.1 or earlier, DuckDB 1.4 support has ended: upgrade to 26.09.4, with nothing to migrate.

Ready to handle billion-record workloads?

Deploy Arc in minutes. Own your data in open files on your storage. Use for analytics, observability, AI, IoT, or data warehousing.

Get Started ->