Arc Data Model: Five Decisions That Are Hard to Undo

Arc's data model is deliberately thin. A measurement is a table. Every value you write is an ordinary column next to a time column. There is no tag type, no series index, no schema you declare before writing.

Thin models are pleasant on day one and unforgiving on day four hundred. Because Arc doesn't impose structure, the structure is yours to get right, and a handful of those choices are effectively permanent once you have a few billion rows on disk.

This post covers the five decisions that are hard to undo, what Arc actually does with each one, and where three families of workloads diverge. Everything here is drawn from Arc's source rather than from intent; where the code does something surprising, I've said so plainly.

TL;DR

• Your only partition dimensions are database, measurement, and hour. Everything else is a column.
• Names are a policy boundary. Access control, retention, and tiering all key off database and measurement names. Wildcards are prefix/suffix only.
• Send microseconds. On the MessagePack path Arc infers the timestamp unit from the first value in a batch and applies it to all of them.
• Deduplication only happens if you write line protocol. MessagePack, MQTT, and WAL replay produce no dedup key.
• High tag cardinality is cheap. Unstable schemas are expensive. This is the opposite of the instinct most people bring from InfluxDB.

The layout, in one line

Every Parquet file Arc writes lands here:

{database}/{measurement}/{YYYY}/{MM}/{DD}/{HH}/{measurement}_{ts}_{nanos}.parquet

Hourly partitioning is fixed. It is not configurable, and there is no secondary partition dimension. A query bounded in time reads only the hour directories it needs; everything else is a column scan inside those files, with Parquet row-group statistics doing the fine-grained skipping.

That single line drives most of what follows. If a dimension isn't in that path, it cannot prune your query by directory. It can only be filtered as a column.

Decision 1: What earns a separate measurement

Since database and measurement are two of your three partition dimensions, deciding what becomes a measurement is the highest-leverage modeling choice you make.

Split a measurement out when it needs its own retention, access policy, or lifecycle. Those are the boundaries Arc can actually act on, so they're the ones worth encoding in the layout. Also split when schemas are genuinely unrelated: a measurement's column set is the union of everything ever written to it, so mixing unrelated shapes gives you wide, sparse files.

Keep things together when they're queried together. Arc reads only the columns a query touches, so an extra column costs far less than a join across measurements.

The common mistake is splitting by value rather than by kind: a measurement per device, per tenant, per satellite. It feels organized and it buys nothing. The value would have filtered just as well as a column, and now you have thousands of measurements, each with its own write buffer and its own small files. Split by kind, not by instance.

Decision 2: Names are a permanent policy boundary

This one surprises people, because naming feels cosmetic. It isn't. Three subsystems key off names, and all three are painful to retrofit.

Access control. Arc's RBAC matches database and measurement names with wildcards, but only *, prefix_*, *_suffix, and prefix*. There is no regex and no arbitrary matching. A policy like "this team reads all production telemetry" is only expressible if your production databases share a prefix. If you named them telemetry, metrics_prod, and prod-events, you cannot express that rule with one pattern, and you'll be enumerating names by hand forever.

Retention policies are per-database, optionally narrowed to a single measurement.

Tiering policies are per-database only. There is no per-measurement tiering policy. If two measurements in one database need different hot-storage windows, they need different databases.

One concrete trap: avoid hyphens in database names. Arc's create-time validation permits them, but the query router validates database identifiers as ^[a-zA-Z0-9_]+$, which is stricter. A hyphenated name parses differently in RBAC's db.table patterns than in the router, so the two can disagree about which database is being read. Use underscores.

Pick names on day one as though you'll be writing access policy against them, because you will be.

Decision 3: Timestamps, and the trap in the fast path

Arc's canonical unit is microseconds, stored as a UTC-zoned Parquet timestamp and read back as TIMESTAMP WITH TIME ZONE. How Arc learns your unit depends on the write path, and the difference deserves attention.

On the line protocol path you declare it: precision=ns|us|ms|s, defaulting to ns. Worth knowing that ns input is converted by integer division, so sub-microsecond precision is silently truncated. If you need finer resolution than a microsecond, put it in its own column.

On the MessagePack path, which is Arc's highest-throughput format and the one MQTT ingestion decodes, the unit is not declared at all. Arc reads the first element of the time column and picks a multiplier by magnitude:

That multiplier is then applied to every row in the batch.

The failure mode follows directly: a batch whose first timestamp sits in a different magnitude band than the rest gets silently misconverted. Not rejected, not warned about, just wrong. There's no way to override it either. The columnar payload has no unit field at all, and the _time_unit field on Arc's row-oriented record struct is never read by anything.

Send microseconds, and keep every batch internally consistent. This matters most for backfills from recorded telemetry, where a batch may not start where you assume.

Decision 4: Whether you get deduplication at all

Arc ingests append-only and de-duplicates during compaction. The dedup key comes from column names recorded in the Parquet footer under arc:tags. The key is (tag columns..., time), and one row survives per key.

The catch is which write paths record that metadata:

So Arc's fastest write path and the standard IoT path both produce data that is compacted but never de-duplicated. The third row matters too: data recovered from the write-ahead log after a crash carries no dedup key even if it was originally written via line protocol, which is precisely the moment duplicates are most likely.

This isn't a bug, but it's worth being precise about why. MessagePack columnar ingest hands Arc columns directly, and the format carries no tag/field distinction, so Arc has nothing to record as a dedup key. That's a gap in what the protocol declares, not a law of physics: the wire format could carry an explicit tag-column list, it just doesn't today. Either way, you have to choose deliberately:

• If duplicates are possible in your pipeline (at-least-once queues, MQTT redelivery, retried batches, replayed recordings), write line protocol and accept the lower ceiling.
• Or keep MessagePack and make duplicates harmless at query time, with SELECT DISTINCT or aggregates that tolerate them.

What you shouldn't do is assume compaction is cleaning up behind you. For most high-throughput setups, it isn't.

Decision 5: Schema stability beats cardinality

Here is where Arc most diverges from the mental model people bring from other time-series databases.

Arc's write buffers are keyed by database/measurement and nothing else. A million distinct tag values in one measurement still produce one buffer and one file per flush. There is no series index, no inverted index, no per-series accounting anywhere in the write path. High tag cardinality, the thing you were trained to fear in InfluxDB, costs you comparatively little here.

What costs you is schema churn. Arc tracks a per-buffer column signature: the sorted set of column names and their types. When that signature changes, Arc flushes the current buffer and starts a new one. Adding a column, dropping one, or changing a column's type all count.

That gives you two failure modes worth designing against:

1. Churn in tag keys, meaning new column names appearing and disappearing between writes, forces a flush each time and produces many small files. Note the distinction: many values in a stable column is fine; many column names is not.
2. Type instability. A column arriving as int64 in one batch and float64 in the next is a different signature. Send 1.0, not 1, for a float column that happens to hold a round number.

There is no upper bound on this. Nothing in Arc stops a writer from alternating schemas forever; it just quietly degrades into a pile of tiny files. Arc does have one guard, but it's narrower than it looks: if eight or more distinct schemas race the same buffer concurrently during a single flush, that write is rejected with HTTP 503 and no data loss. That's a concurrency signal. One writer flip-flopping between two schemas never trips it, so don't wait for an error to tell you this is happening. Watch your file counts.

Cardinality isn't entirely free. A high-cardinality column scattered across row groups gives Parquet statistics nothing to skip on. That's what sort keys are for. Configure ingest.sort_keys as measurement:col, and Arc sorts by that column before time within each partition, clustering equal values so row-group min/max statistics get tighter. Choose something that shows up in your WHERE clauses with far fewer distinct values than rows: host, site, subsystem. One caveat: a configured sort key naming a column that isn't in the batch is a hard error, so only use columns every write includes.

Worth flagging honestly, because it undercuts what I just told you: writing this post turned up a bug in that path. The implicit time append happens at ingest but not at compaction, so with cpu:host configured, compacted files come out ordered by host alone rather than (host, time).

That matters more than it sounds. Compacted files are where data spends nearly all of its life and the only ones that tier, so today configuring a custom sort key can leave time-range statistics on your long-lived files worse than on the raw files they replaced. Deployments on the default sort key are unaffected, since the list is identical either way. It's tracked as arc#792. Until it ships, treat custom sort keys as an ingest-side optimization and measure your time-range queries before and after turning them on.

Where workloads actually diverge

Most of the above is universal. Three families differ in which decision dominates.

Defense and aerospace: backfill and boundaries

These two share a shape: data is often recorded first and ingested later, whether from a platform that was off-network, a ground station pass, or an air-gapped transfer, and the access boundaries are the whole point.

The timestamp trap is your top risk. Backfills are exactly where a batch's first timestamp may not match your assumption, and Arc will apply that guess to the entire batch. Normalize to microseconds in the exporter, before the data ever reaches Arc.

Name for the boundary, not the convenience. RBAC's prefix/suffix wildcards are what you'll write classification and need-to-know policy against. Since tiering policy is per-database and retention is per-database-or-measurement, a program that needs its own retention schedule needs its own database. Decide that before ingest, not after.

Replayed recordings duplicate. If the same pass can be ingested twice, you want line protocol's dedup key, or an explicitly idempotent ingest step.

Manufacturing and IoT: duplicates and precision

These share the MQTT path and its consequences.

MQTT gives you no dedup key. QoS 1 is at-least-once by definition, so redelivery is normal operation, and the MQTT path decodes MessagePack/JSON, so there is no arc:tags and no dedup at compaction. Either route through line protocol, or design queries that tolerate repeats.

Don't store measurements as floats when exactness matters. Arc supports native Decimal128 columns, but they must be declared, because a decimal arrives on the wire as a number and is otherwise indistinguishable from a float. Declare them via ingest.decimal_columns before you start writing, not after.

Cardinality is your friend here. Per-device, per-sensor tag values are cheap in Arc. Model one measurement per kind of reading with device_id as a column, not one measurement per device.

Observability: churn is the whole problem

Observability is the one family where schema stability, not cardinality or duplicates, dominates.

Application metrics and structured logs are emitted by many independent services, and every deploy is an opportunity for a new label to appear or an old one to vanish. That is precisely the churn Arc charges for: each new column signature forces a flush, and enough alternation produces flush amplification and eventually 503s.

The fix is a boundary, not a bigger buffer. Normalize the column set before it reaches Arc. Fix a schema per measurement at the collector and drop or bucket unexpected labels, rather than passing whatever a service emits straight through. Separate measurements for genuinely different event shapes beats one wide measurement whose signature changes every deploy.

The checklist

Before your first production write:

1. Database names: underscores only, chosen so RBAC wildcards express your access policy.
2. Measurement split: by kind and lifecycle, never by instance value.
3. Column names: stable set per measurement; never reuse a name across a tag and a field.
4. Column types: one type per column, held stable; decimals declared up front.
5. Timestamps: microseconds, UTC, consistent within every batch.
6. Dedup: if duplicates are possible, use line protocol or tolerate them in queries.
7. Sort keys: a low-cardinality filter column ahead of time, present in every write.
8. Lifecycle: retention set, and daily compaction confirmed running if you rely on tiering.

One addition to that last point, because it's a quiet one: Arc's tiering only migrates files whose names end in _daily.parquet. Raw and hourly-compacted files stay on hot storage indefinitely. If daily compaction is disabled, or never reaches its file threshold for a low-volume measurement, that data never tiers, and you pay hot-storage prices for it forever without any error telling you so.

Further reading

The full reference version of this material now lives in the docs, with the exact validation rules, type mappings, and config keys:

• Arc data model guide
• Writing data
• Decimal precision
• Retention policies

And if you want the deeper treatment of how deduplication actually runs at compaction time, that has its own post.