Arc 26.09.1: Iceberg Export, Edge Sync, 34M Records/Sec

26.09.1 is the biggest Arc release we've shipped. Two new subsystems (Apache Iceberg export and edge-to-cloud sync, including a fully air-gapped transport), a 65% jump in sustained ingest throughput, a security hardening pass with two responsibly-disclosed findings closed, and the longest bug-fix list in any Arc release to date.

The headline numbers: sustained ingest moved from 20.6M to 34.0M records/sec on the IOT benchmark (2,043,451,000 records in a 60-second run). Your existing Parquet files can now be queried as a standard Iceberg table by Spark, Trino, Snowflake, or PyIceberg without copying a byte. And an Arc at the edge (a vehicle, a factory cell, a submarine) can now ship its data to a central Arc over the network or on a physical drive.

If you run Arc in production, update. There are a few default-behavior changes to read first (row ordering and Parquet dictionary encoding chief among them, all below), and clustered Enterprise deployments need a coordinated restart rather than a rolling one.

Apache Iceberg export: your Parquet is now a lakehouse table

Arc has always stored data as open Parquet files you own. 26.09.1 takes that promise to its strongest form: Arc can now publish those same files as an Apache Iceberg table, so any Iceberg-aware engine (Spark, Trino, DuckDB, Snowflake, PyIceberg) queries Arc's data directly, without going through Arc's API.

The design constraint was simple: don't touch the write path. Iceberg is a table format, a metadata layer over Parquet, not a new file format. A background reconciler periodically registers Arc's existing Parquet files into the Iceberg table by reference, using Iceberg's add-files mechanism. No data is copied or rewritten, so there is effectively no storage overhead beyond the small metadata, and nothing in the 34M records/sec ingest path is affected.

Enable it with one setting:

[iceberg]
enabled = true          # default false

Then point any engine at the table. With DuckDB:

INSTALL iceberg; LOAD iceberg;
SELECT * FROM iceberg_scan('/path/to/data/arc/arc_mydb.db/cpu');

The reconciler tracks compaction and retention, so the Iceberg table follows the underlying files as they change; snapshots are expired on a retention policy so metadata stays bounded; measurements that gain columns over time evolve the table schema automatically; and the Iceberg metadata (and SQL catalog) is included in Arc's backup/restore. It's read-tested against DuckDB, PyIceberg, and Apache Spark.

Two v1 scope notes: Iceberg export requires a local storage backend, and it's not compatible with cold-tier tiering (a file migrated to S3 would leave the table); Arc refuses to start with both enabled rather than serving a table with holes. The full engine-by-engine guide is in the docs under Integrations.

Edge-to-cloud sync: from a vehicle, a factory, or a submarine

Arc already runs at the edge as a standalone binary with local storage. What was missing was a first-class way to get that data to a central Arc. Backup is a full DR snapshot, not incremental sync, and re-ingesting Parquet through the write path double-counts on retry.

Edge sync ships immutable Parquet files from a spoke (edge Arc) to a hub (central Arc). The hub verifies every file's SHA256 before committing it, re-delivery of an already-received file is a no-op, and the same path arriving with different content is refused rather than overwritten, because one of the two copies is wrong, and silently replacing either destroys the evidence.

It ships with two transports, because "the edge" is not one thing:

• Network: the spoke initiates, so an edge behind NAT with no inbound reachability works. Discovery is a single batched round-trip regardless of backlog size: a spoke returning from a long outage with 5,000 pending files asks once, not 5,000 times. Transfers resume from a byte offset when a link drops (on hubs with local storage; an S3- or Azure-backed hub restarts the dropped file from zero), and a pass streams newest-first, so a contact window that closes mid-backlog has already delivered the freshest telemetry.
• Air gap: for a spoke with no network path at all, the spoke writes a signed bundle to removable media, the hub imports it after verifying the whole bundle (a tampered drive is refused and not one byte reaches storage), and a signed receipt travels back on the same drive. The receipt is what lets the spoke finally mark those files synced and prune its ledger; the box least able to receive a site visit no longer grows state without bound.

Authentication is layered: Arc's normal API tokens plus a per-spoke HMAC binding the spoke, the hub, the path, and the content digest. Spoke secrets are generated at registration, shown exactly once, and encrypted at rest. And compaction on a syncing spoke defers until data has been delivered, which makes hub-side duplication and compaction-caused data loss structurally impossible: the hub receives every row exactly once. The one caveat is historical: files compacted before this release sync once as a bounded one-time duplicate, never a loss.

Everything above is OSS and manual in this release; an operator triggers each step via the API. The scheduled agent that automates it is Enterprise and lands later.

Ingest: 20.6M to 34M records/sec

Two independent changes compound on the MessagePack columnar write path:

Parquet dictionary encoding at ingest time is now off by default (~26% by itself). Ingest files are transient staging: hourly and daily compaction rewrite them through DuckDB, which re-encodes every column with its own adaptive choices regardless of how the source was encoded. Dictionary-encoding at ingest paid a hash-table insert and an allocation per value to compress files that were about to be rewritten anyway. Long-term storage efficiency is unchanged; the tradeoff is a temporarily larger uncompacted hot partition (up to ~2x for highly repetitive data) until the next compaction pass. The keys to bring dictionaries back are in the behavior-changes section below.

Typed msgpack decode (+30% on top). Payloads now decode directly into typed column arrays, eliminating the per-value interface boxing that dominated decode CPU and fed GC pressure. There is no configuration and no wire-format change: unrecognized payload shapes transparently fall back to the previous decoder, so client-visible semantics are identical.

The result, on the IOT sustained-load benchmark (Apple M3 Max, 12 workers, 60-second runs):

The compressed variants ride the same path: gzip ingest moved 19.7M to 24.6M rec/s, zstd 20.2M to 24.9M.

Queries: smaller on the wire

Three changes for clients pulling large result sets over real networks:

• JSON and msgpack query responses honor Accept-Encoding. Send Accept-Encoding: zstd, gzip and Arc compresses the response: a 5M-row JSON result drops from 405.6MB to 128.1MB (-68%). curl, browsers, Python requests, and Grafana negotiate this automatically; clients that don't send the header get byte-identical responses to before.
• Arrow IPC egress gains opt-in dictionary encoding and buffer compression, per request via headers. Measured on a 500M-row table: 39.0 to 19.4 bytes per row with both enabled, which is 19.5GB to 9.7GB on the wire and roughly 2x faster end-to-end at 1Gbps. On loopback, leave them off; they trade CPU for bytes.
• The MessagePack query endpoint is now stable. Its response shape and type vocabulary are a published contract pinned by a golden test, so clients can bind to them. It's now the best general-purpose format for a client not using Arrow directly: columnar, typed, roughly 2-3x faster than JSON on large results, and it accepts SHOW statements, which the Arrow endpoint doesn't.

Security hardening

Four items, two of them responsibly disclosed by external researchers. Thank you both.

Replacement-scan RBAC bypass (GHSA-w8x2-cccw-25f7, reported by @arpitjain099). On an RBAC-enabled multi-tenant instance, an authenticated tenant could read other tenants' databases by putting a bare single-quoted path in table position: SELECT * FROM '/data/arc/db2/secrets/*.parquet'. DuckDB resolves that as a replacement scan with no function name, and both of Arc's query-authorization controls keyed on function names. The validator now rejects any single-quoted string standing where a table reference belongs, before any transform or RBAC check runs. This was a within-allow-list cross-database read, not a sandbox escape, and it only matters where RBAC is the tenant boundary.

Query-authorization hardening on the measurement endpoint (reported by zx (Jace), @manus-use). The GET-by-measurement endpoint now routes through the same validation as the raw query API, and the continuous-query and delete endpoints got the same input-validation pass. Full detail lands with the advisory once published.

Partition-pruner amplification DoS (#536). A single small query with a very wide time range (WHERE time >= '0001-01-01') forced Arc to materialize on the order of a million partition path strings and then glob or LIST every one of them: one HTTP request turned into large CPU, memory, and object-storage LIST billing. Path generation is now capped, the start date is floored at the epoch (lossless, since Arc has no pre-1970 data), and generation is cancelled on client disconnect. This one applies to every deployment mode.

Forwarding-header hardening (CVE-2026-45045 class). Dependabot flagged a GoFiber advisory around client-spoofable X-Real-IP handling. Arc is not affected (the vulnerable helper isn't in Arc's build graph, and Arc derives client IP exclusively from the TCP socket), but since no patched GoFiber v2 exists, we closed the vulnerability class directly: a clustered node now strips all client-supplied forwarding and identity headers before forwarding a request to a peer, and re-establishes trustworthy values itself. A related fix stops an authenticated caller from abusing the cluster loop-guard header to force a doomed local path; genuine routing loops now terminate with a clean 508.

Also in this pass: the Enterprise Raft consensus transport now authenticates every peer connection with a mutual HMAC handshake keyed by the cluster shared secret (it was the one cluster channel that didn't), and gRPC-Go moved past a scanner-flagged advisory even though Arc never starts a gRPC server or client.

The fixes operators will feel

The full notes list over thirty fixes. These are the ones most likely to have touched you:

Outer joins returned wrong answers. The query rewriter replaced LEFT JOIN, RIGHT JOIN, and FULL OUTER JOIN with a bare inner JOIN when rewriting table references: the query succeeded and silently returned only matched rows, the exact rows an outer join exists to preserve. Found via #586, with a related bare-join defect found and fixed by @schotime in #585. If you ever got a suspiciously small result from an outer join in Arc, this was it. Fixed, with the operator now emitted verbatim.

Hyphenated database names were unqueryable. SELECT * FROM "my-db".cpu returned zero rows, silently, because the quotes were spliced into the storage glob as literal characters. Quoted identifiers now resolve to their real names everywhere, including RBAC permission checks.

S3 and Azure queries no longer die an hour after startup. On EKS with IRSA (and on Azure managed identity), DuckDB resolves temporary credentials once at secret creation and never refreshes them, so query reads failed with expired-token errors roughly an hour after each process start, while ingest kept working and probes stayed green. Arc now manages these credentials itself through the cloud SDKs, re-issuing DuckDB session credentials before each expiry, covering IRSA, EC2 instance roles, EKS Pod Identity, SSO, and Azure managed identity. Verified end-to-end against live AWS STS and live Azure AAD through full token rotations. (#600, #605)

A dead read path can no longer hide behind green probes. The incident behind the credential work: a reader ran 21 hours READY 1/1 while every S3-backed query failed. /health now reports per-tier storage credential state (ok, degraded, expired) computed from in-memory refresher records (it never probes S3, so nothing flaps). An opt-in setting makes /ready fail while credentials are expired, so Kubernetes recycles the pod. (#603)

License-server outages no longer crash-loop Enterprise clusters. A transient failure reaching the license server during pod boot could drop a pod to OSS mode, which is fatal for enterprise-required configurations. One customer watched a writer accumulate 27 restarts during a brief license-server deploy. Boot now retries with backoff, distinguishes "the server said no" from "the server didn't answer," and falls back to a locally cached, signature-verified copy of the last license response, honored to the license's own expiry and never past it. Fully air-gapped environments can now use an offline license file with no network calls, ever.

Compaction got resource bounds. Each compaction subprocess inherited the full database memory limit and all CPU cores, so during backfill catch-up, compaction alone could reach 2x the configured memory limit on top of the main process. New compaction.memory_limit and compaction.threads keys bound this, with auto defaults that keep all concurrent jobs within roughly one memory limit total. Batch size is also configurable now (useful for edge deployments shipping files over constrained links), and a series of compaction fixes close double-compaction on storage errors, permanently-failing partitions without a time column, and stranded single-file remainders that never compacted.

WAL crash recovery is significantly more robust. Startup recovery could interfere with the active WAL file, and replayed data skipped parts of the live decode path (timestamp normalization, UTF-8 sanitization). Both fixed; a single bad WAL entry no longer stops replay of the rest of its file. If you run wal.enabled = true, take this release. (#594, #590)

Backups stream instead of loading whole files into memory (a moderately-sized backup could spike gigabytes of RSS), concurrent backup/restore requests are now refused with 409 instead of silently queueing (contributed by @mvanhorn in #622), a mostly-failed backup now fails loudly instead of reporting success, and continuous queries gained crash-safe dedup: duplicate windows now collapse at compaction time, automatically for CQs with no grouping, and opt-in for grouped CQs by declaring the new tag_columns field on the definition (a grouped CQ without it behaves exactly as before).

And DuckDB moves from 1.5.1 to 1.5.5, picking up four upstream releases of Parquet decompression hardening, which matters when your entire storage layer is Parquet.

Behavior changes to read before upgrading

Row order without ORDER BY is no longer insertion order. Arc previously forced DuckDB's insertion-order preservation; that's now configurable and off by default, following SQL semantics and unlocking memory savings and parallelism on large scans. Queries with an explicit ORDER BY are completely unaffected. If a dashboard relies on SELECT * FROM cpu LIMIT 10 returning oldest-first, add ORDER BY time (recommended) or set preserve_insertion_order = true under [database].

Ingest-time dictionary encoding is off by default (the 26% ingest win above). Long-term storage is unchanged because compaction re-encodes everything; the hot uncompacted partition can be temporarily larger until the next compaction pass. use_dictionary = true under [ingest] restores string-column dictionaries; adding numeric_dictionary = true restores the exact previous behavior.

Continuous queries that don't select a time column now stamp output with the window start instead of ingestion wall-clock time. This corrects a real bug (the old timestamps were wrong and made duplicates un-dedupe-able), but those destinations will show a timestamp discontinuity at the upgrade point. CQs that select a time column (the common case, like date_trunc('hour', time) AS time) are completely unchanged.

Operator action items

• Update. No config change required; existing arc.toml, tokens, and license keys work as-is.
• Clustered Enterprise: upgrade as a coordinated restart. Stop all nodes, upgrade every binary, restart all nodes. The authenticated Raft handshake is a hard cutover; a rolling restart will churn leader elections until the last node is upgraded. Single-node deployments are unaffected.
• Check dashboards and integrations for queries that rely on implicit row order without an ORDER BY (see above).
• If you run WAL-enabled ingest, RBAC multi-tenancy, or S3/Azure storage on temporary credentials, this release fixes real incidents in all three areas; prioritize it.
• Backups now stream through temp disk instead of memory: in containers with a small tmpfs /tmp, point TMPDIR at a volume with room for your largest Parquet file.
• If external tooling sets X-Forwarded or X-Real-IP headers on requests to a cluster and expects them to survive an inter-node forward, note they're now stripped on the forwarding hop. Client IP attribution never used them, so logs and audit are unchanged.

New: native Arch Linux packages

Starting with this release, Arc ships pacman packages (.pkg.tar.zst, x86_64 and aarch64) for Arch Linux and Arch-based distros like Omarchy, alongside the existing .deb and .rpm:

wget https://github.com/basekick-labs/arc/releases/download/v26.09.1/arc-26.09.1-1-x86_64.pkg.tar.zst
sudo pacman -U arc-26.09.1-1-x86_64.pkg.tar.zst
sudo systemctl enable --now arc

Same layout as the other system packages (/usr/bin/arc, /etc/arc/arc.toml, a systemd service running as a dedicated arc user), but done the Arch-native way: the service user and state directory are managed through sysusers.d/tmpfiles.d, and your edited arc.toml survives upgrades the pacman way (.pacnew on conflict). The package wraps the exact same cosign-signed binary the release publishes standalone, and release CI installs and boots the built package on an Arch userland before anything is published.

How to update

# Docker Hub
docker pull basekicklabs/arc:26.09.1
 
# or GitHub Container Registry
docker pull ghcr.io/basekick-labs/arc:26.09.1

On macOS, brew install basekick-labs/tap/arc (or brew upgrade arc if you have it). On Arch Linux and Omarchy, grab the .pkg.tar.zst from the release and install it with pacman -U (see above). For binary installations, download 26.09.1 from the GitHub releases page. For Kubernetes:

helm install arc https://github.com/basekick-labs/arc/releases/download/v26.09.1/arc-26.09.1.tgz

The full long-form release notes, with per-fix detail, the edge-sync threat model, HMAC key-separation design, and the complete configuration reference for Iceberg and edge sync, are in RELEASE_NOTES_2026.09.1.md on the release branch.

Get started:

• Arc documentation
• Star Arc on GitHub
• Join the Discord

Questions? Discord or GitHub Issues.