ArcX Update: Building Arc's Own Query Engine

When Arc turned one, we said the main engineering priority for year two would be ArcX, our own query engine. We're building it to replace DuckDB in Arc, with execution shaped around time-series data and the queries people run against it.

This post is an update on that work: how we're building it, what it can do today, the numbers we have so far, and when you'll be able to run it.

The short version: ArcX ships as an experimental engine in Arc 27.04.1, and becomes Arc's query engine in 27.08.1.

Why build our own engine

DuckDB has been a great foundation. Arc's query performance, including the ClickBench results against DuckDB itself, rests on it. But DuckDB is a general-purpose analytical engine, and Arc's workload is specific.

Arc stores data as Parquet files partitioned by time: database/measurement/year/month/day/hour. Most queries ask the same kinds of questions: a count or an average per hour, the latest values per host, the worst hosts by some metric over the last day. A general engine treats those files like any other Parquet. An engine built for Arc can use what Arc already knows about them.

Some of that knowledge is free. A count(*) per day over hourly files can be answered from Parquet metadata and the partition path without reading a single row. In one of our early measurements, that query went from 2.3 seconds on DuckDB to under a millisecond on ArcX.

Owning the engine also gives us control over the behavior that matters when a database runs every day: memory bounds, how a query fails, and what the operator sees when it does.

How we're building it: correctness first

A query engine that is fast and occasionally wrong is worse than a slow one. So the rule for ArcX is simple: it only answers a query when it can answer it exactly the way Arc answers it today. Everything else goes back to DuckDB.

That shapes the whole development process.

Every feature is tested against DuckDB. Each piece of SQL that ArcX learns, whether an aggregate function, a HAVING form, or an ORDER BY ... LIMIT, comes with differential tests that run the same queries on ArcX and on DuckDB over data designed to break things: NULL groups, NaN in both signs, positive and negative zero, values at the edges of 64-bit integers, ties at a LIMIT cut. We compare against DuckDB 1.5.6, the version Arc ships, and against DuckDB 2.0 as a second opinion. Where the two disagree, ArcX declines rather than pick one.

Every feature gets an adversarial review before we write it. Before implementing anything, a separate review tries to break the plan using DuckDB's actual behavior. Those reviews keep finding things. One example from this week: DuckDB evaluates BETWEEN with a single common type across the value and both bounds, so min(x) BETWEEN 9007199254740993 AND 1e20 can be true while the equivalent >= and <= pair is false. ArcX now reproduces that.

Every feature is fuzzed. For the latest slice, thousands of randomly generated HAVING expressions ran through ArcX and both DuckDB versions. Any query where ArcX returned a different answer, or answered where DuckDB returns an error, would fail the run.

Inside Arc, ArcX can run in shadow mode. In shadow mode, Arc answers every query with DuckDB as usual and runs ArcX alongside it, comparing the two results and logging any difference. This is how we validate each slice end to end before it serves a single query.

Some of DuckDB's answers aren't stable. A few aggregates in DuckDB accumulate floating-point sums in an order that depends on how many threads ran the query. On three million rows of a constant 0.1, avg(x) came out above max(x) with one thread, below it with four, and above it again with fourteen. Comparing those values to each other has no stable answer, so ArcX declines those comparisons today. Where DuckDB's own result is imprecise, as with skewness and kurtosis on data far from zero, ArcX computes the numerically stable value and the comparison records it as a known difference rather than a failure.

Where ArcX is today

ArcX covers the queries that make up most time-series dashboards:

• Scans with filters: comparisons, IN, BETWEEN, LIKE, IS NULL, arithmetic in WHERE, and time ranges that skip files and row groups. "Latest N rows" queries read from the newest files first.
• Aggregates: 59 of the 75 aggregate functions in our catalogue. That includes count, sum, avg, min/max, the variance and standard deviation family, correlation and regression, the Kahan sums, bitwise and boolean aggregates, count_if, median, quantile_cont/quantile_disc, mad, mode, skewness and kurtosis.
• Grouping: by a tag, by a date_trunc time bucket, or both.
• Map columns: OpenTelemetry-style attributes (attributes['service.name']) in filters and counts.
• Result shaping: aggregate aliases, ORDER BY an aggregate with LIMIT (top-N), and HAVING with arithmetic over aggregates, including aggregates that aren't in the select list.

The remaining aggregates are approximate sketches, first/last/any_value, and collection functions.

The numbers so far

These are end-to-end measurements through Arc's HTTP API on the same Arc binary: DuckDB with ArcX switched off, and ArcX with it switched on. The data is a synthetic telemetry corpus of 132 million rows in 41 hourly Parquet files with 1,000 hosts, on an Apple Silicon laptop, best of three warm runs.

Not every query is a win yet:

• HAVING host LIKE 'server1%' takes 41 ms on ArcX against 35 ms on DuckDB. DuckDB applies a filter on a grouping column before it aggregates. ArcX aggregates every group and then filters. Moving that filter earlier is on our list.
• product() is slower on ArcX (64 ms against 34 ms), so Arc doesn't route it to ArcX.
• Some shapes run at parity, like fsum/favg and the boolean aggregates.

We record these alongside the wins. When ArcX is slower, we either keep the query on DuckDB or track the fix, as with the LIKE case above.

What's next

The goal between now and full integration is that ArcX answers every query Arc users send. Today, a query ArcX can't answer goes back to DuckDB. Once ArcX is Arc's engine, every one of those cases needs an answer of its own. We're working through them by category:

• Ordering by hidden aggregates and expressions: ORDER BY count(*) DESC when count(*) isn't selected, or ordering by sum(bytes) / count(*). ArcX already computes these; the work is in validating them through Arc.
• Faster HAVING on grouping columns, so HAVING host LIKE ... filters before aggregating, as DuckDB does.
• Order statistics above memory limits: median and quantile over very large groups, computed in several passes over the files instead of holding every value in memory.
• Wide scans: streaming large results instead of declining past an in-memory budget.
• More grouping shapes: several tag columns at once, and more aggregate families grouped by tag.
• The rest of the SQL surface: joins, window functions, CTEs, and subqueries.
• The remaining aggregates: first/last, approximate sketches, and collections.

For each case, we decide whether ArcX should answer it or return a clear error, the same way we'd treat any other database behavior an operator depends on.

Release plan

Arc 27.04.1 includes ArcX as an experimental engine. You'll be able to turn it on in two ways:

• Shadow mode keeps DuckDB answering every query and runs ArcX alongside it, logging any difference. It's the safe way to see how ArcX handles your own queries and data.
• Serve mode lets ArcX answer the queries it supports and sends the rest to DuckDB.

If you run Arc on real workloads, this is where your help makes the biggest difference. Your queries, your schemas and your data distributions are exactly what our test corpus can't anticipate.

Arc 27.08.1 makes ArcX Arc's query engine. By then, the work above needs to be complete: every query Arc users run should have an answer from ArcX.

ArcX ships inside the Arc binary. There's nothing separate to install.

Follow along

We'll keep posting as ArcX develops, including the numbers that don't flatter us.

• Download Arc and run it on your own infrastructure.
• Join the conversation about ArcX on The Dive, under the arcx tag.
• Star Arc on GitHub and follow the releases.
• Talk to Basekick Labs if you'd like to test ArcX on a production workload ahead of 27.04.1.

A faster engine only matters if people can trust its answers. That's the standard we're holding ArcX to.