Migrate InfluxDB to Arc, Straight Off Disk, with tsm2arc

The hardest part of leaving InfluxDB was never the query rewrite. It was getting your history out.

If InfluxDB is still running, the obvious move is to query it and re-write the data somewhere else. But that path has two problems. First, it's rarely the situation people are actually in: usually there's a volume from two years ago holding a few terabytes of TSM files, and the InfluxDB instance that wrote it is long gone (decommissioned, or on a version you'd rather not boot, or the box simply doesn't exist anymore). Second, and this catches people who do still have a live instance, querying it to drain years of history means hammering the same read path your production users and dashboards depend on. A bulk export is exactly the kind of heavy, wide scan that degrades everyone else's experience while it runs.

That's the case tsm2arc was built for. It reads InfluxDB's on-disk TSM and WAL files directly, decodes the block codecs natively in Go, reconstructs line protocol, and streams it into Arc, with no running InfluxDB required. Mount the volume read-only and go.

It handles InfluxDB 1.x (1.7/1.8) and 2.x (2.0-2.7). The on-disk format is the same across both; tsm2arc auto-detects the layout, and for 2.x it resolves bucket IDs to readable names from influxd.bolt and skips the internal system buckets. It's Apache-2.0 and feature-complete.

This post is the full migration walkthrough. Real commands, real behavior, and the handful of things worth getting right the first time.

Why "off disk" is the whole point

InfluxDB stores each field of a point as its own TSM key. cpu,host=a#!~#usage and cpu,host=a#!~#cores are two separate keyed streams, each with its own timestamps and values. A running InfluxDB rejoins them for you at query time. When you're reading cold files, nothing does that for you.

tsm2arc does it itself:

1. Parses the TSM index (header / index / footer) of every file.
2. Decodes every block with native Go implementations of the timestamp, float, integer, unsigned, boolean, and string codecs.
3. Rejoins fields by (series, timestamp) so multi-field points come back out as single line-protocol lines.
4. Emits line protocol with the original nanosecond timestamps (including pre-1970 timestamps), routed to the Arc database matching the source InfluxDB database.

Those codecs aren't reverse-engineered by eyeball. They're validated in unit tests against the real InfluxDB 1.7.11 encoder and cross-checked against real InfluxDB 2.7 data. The influxdb package is a test-only dependency (the oracle for round-trip tests) and never links into the shipped binary.

It doesn't touch InfluxDB's query path

This is the operational reason to prefer the file-layer approach even when your InfluxDB is very much alive.

A query-and-re-write migration drains history through InfluxDB's read path, the same path serving your dashboards, alerts, and users. Exporting years of data is a heavy, wide scan by definition, and running it against a live instance means the migration competes with production for CPU, cache, and I/O. Your users feel it while it runs.

tsm2arc sidesteps that entirely. It reads the TSM and WAL files at the filesystem level: it never issues a query, never opens a connection to influxd, never adds a single point of load to the engine. If the source is a running instance, point tsm2arc at its data directory. You can mount it read-only, or snapshot the volume and migrate from the snapshot so you're not even sharing disk I/O with production. (For 2.x bucket names, tsm2arc reads influxd.bolt from a copy, so a locked or read-only original is fine.)

So the migration runs as hard as you like, raising the workers and saturating the network, and your live InfluxDB users never notice it's happening.

Install

Grab a prebuilt binary from Releases (Linux/macOS/Windows, amd64/arm64), or:

# from source (Go 1.25+)
go install github.com/basekick-labs/tsm2arc/cmd/tsm2arc@latest
 
# container
docker run --rm ghcr.io/basekick-labs/tsm2arc:latest --version

Every release ships an SBOM (SPDX) and a checksums.txt; the container image is multi-arch on GHCR. Verify the checksum before you trust a binary you're about to point at production data.

Step 1: dry-run first, always

Never let the first thing tsm2arc does be a write. The first thing it should do is read: discover the shards, decode every block, reconstruct the points, and tell you exactly what it found, without touching Arc.

That's --dry-run:

# InfluxDB 1.x: point at the data dir (or its parent; layout is auto-detected)
tsm2arc --datadir /mnt/influxdb --dry-run --sample 10
 
# InfluxDB 2.x: point at the v2 root (~/.influxdbv2 or the mounted volume).
# engine/data, engine/wal, and influxd.bolt (for bucket names) are all auto-detected
tsm2arc --datadir /mnt/influxdb2 --dry-run --sample 10

A dry-run walks every shard, decodes every block, rejoins fields, and prints per-database/bucket point counts plus a sample of the actual line protocol it will send. If a codec were wrong or a shard unreadable, you'd see it here, before a single byte reaches Arc. This is your safe first contact with the source data, and it costs you nothing to run it twice.

You don't have to be precise about the path. tsm2arc auto-detects 1.x vs 2.x and resolves the rest:

• 1.x: the InfluxDB root (the directory containing data/), or .../data directly.
• 2.x: the v2 root (containing engine/ and influxd.bolt), or .../engine, or .../engine/data directly. Bucket names come from influxd.bolt; point --bolt at it if it lives somewhere unusual. Without it, buckets fall back to their 16-hex IDs as database names: readable, just not pretty.

Step 2: the WAL is not optional

This is the single most common way to lose data in a migration, so it gets its own section.

InfluxDB does not flush the write-ahead log to TSM on shutdown. Small or recently-written shards can live entirely in .wal files with no .tsm at all. On a cold, snapshotted volume (exactly the case tsm2arc targets) that's normal, not an edge case.

If you don't pass --waldir, tsm2arc only sees .tsm files. A WAL-only shard then has nothing for it to find, and that data silently never reaches Arc. No error. Just missing rows you won't notice until someone queries for last Tuesday.

So: pass --waldir.

tsm2arc \
  --datadir /mnt/influxdb/data \
  --waldir  /mnt/influxdb/wal \
  --dry-run --sample 10

For 2.x this is auto-detected from engine/wal, so you usually don't need the flag there, but it costs nothing to be explicit. When both sources are present, tsm2arc reads them and field-rejoins per shard. If a point exists in both a TSM file and the WAL (a partially-compacted shard), the WAL value wins: last-write-wins, matching how InfluxDB and Arc themselves resolve it.

If you can cleanly shut down a still-running InfluxDB before snapshotting, do it. A clean shutdown flushes WAL into TSM and simplifies your life. But the whole premise here is that you often can't, and --waldir is why that's fine.

Step 3: load into Arc

Once the dry-run looks right, drop --dry-run and give it Arc:

tsm2arc \
  --datadir /mnt/influxdb/data \
  --waldir  /mnt/influxdb/wal \
  --arc-url https://arc.example.net \
  --token   "$ARC_TOKEN" \
  --verbose

The token needs to be admin-tier, because Arc's import endpoint is admin-authenticated. Pass it with --token or the ARC_TOKEN env var; prefer the env var so the token never lands in your shell history.

Each source InfluxDB database routes to the Arc database of the same name. Data goes over the wire in gzipped chunks bounded at --chunk-bytes of raw line protocol. Arc's import endpoint caps a request at 500 MB of line protocol (enforced on both the compressed and the decompressed bytes), so the default of 450 MB leaves headroom. (Gzip saves bandwidth, not logical size; the cap that binds is on decompressed bytes, which is why the chunk threshold is set in raw LP.) Transient failures (429, 5xx, network blips) are retried with exponential backoff. A 4xx is treated as permanent and aborts the run, because a 4xx means you, not the network.

The flags you'll actually reach for:

--waldir DIR             InfluxDB WAL directory; read un-flushed data too
                         (auto-detected for 2.x as engine/wal)
--bolt PATH              InfluxDB 2.x influxd.bolt for bucket names (auto-detected)
--workers N              concurrent shards to migrate (default 2)
--db-map old=new         route a source DB/bucket to a different Arc DB (repeatable)
--database-filter db     migrate only this source DB/bucket (repeatable)
--chunk-bytes SIZE       raw LP per import request; bytes or a suffix like 450MB
                         (must be <500MB; default 450MB)
--checkpoint PATH        SQLite resume store (default tsm2arc.checkpoint.db)
--start / --end          RFC3339 UTC time filters
--precision ns|us|ms|s   precision sent to Arc (default ns; tsm2arc always emits ns)
--include-internal       also migrate 1.x's _internal database
--dry-run                extract + count, do not write to Arc
--verbose                per-shard / per-chunk logging

Renaming as you migrate is common, since you rarely want InfluxDB's bucket names verbatim in Arc:

tsm2arc ... --db-map telegraf=metrics --db-map sensors_v2=sensors

And you can migrate one database at a time to stage a large move:

tsm2arc ... --database-filter metrics

Step 4: it's resumable, so lean on that

A terabyte-scale migration runs for hours. Something will interrupt it: a killed process, a dropped connection, a rebooted host. tsm2arc is built so that this is a non-event.

Progress is tracked per shard, keyed on the stable source ID (the 1.x database name / 2.x bucket ID), in a SQLite checkpoint file (--checkpoint, default tsm2arc.checkpoint.db). A chunk's progress is committed only after Arc returns 2xx, and Arc's import handler flushes to storage before it returns, so 2xx means the data is durably persisted, not just accepted.

If a run is interrupted, re-run the exact same command. tsm2arc will:

• skip any shard already fully migrated, with no re-extraction and no re-send;
• for a partially-migrated shard, deterministically re-derive its chunks and resume from the first un-acknowledged one.

Because chunk boundaries are a pure function of a shard's extraction order and --chunk-bytes, a resumed shard produces byte-identical chunks, so the skip is exact. The only window for duplication is a crash between Arc persisting a chunk and tsm2arc recording it; on resume that one chunk is re-sent. Arc's compaction collapses the duplicate for tag-bearing series, and tagless series retain at most one chunk of duplicate rows per shard per crash. A clean, uninterrupted run produces zero duplicates.

Two things to know:

• Keep the checkpoint file. It's how resume works. Keep it next to the migration for the whole run. Delete it only when you want a full re-migration from scratch.
• Resume needs the same shaping flags. The checkpoint fingerprints --chunk-bytes, --start, --end, --db-map, and --precision. Change any of them and the chunk boundaries would misalign, so tsm2arc refuses to resume with a clear error instead of corrupting the migration. To change a shaping flag, start a fresh --checkpoint.

Sizing --workers: think about Arc, not your migration host

--workers N migrates N shards concurrently. Shards are fully independent (each has its own chunk sequence and checkpoint rows), so this scales cleanly, and resume and correctness don't care what N is.

The counterintuitive part is which machine is the constraint. It's almost never the migration host. Arc's import endpoint buffers each request fully in memory (roughly chunk-bytes decompressed plus parsed records), so peak transient memory on the Arc node is about workers × ~1-1.3 GB at the default 450 MB chunk. That's the binding limit. The big dedicated migration box you provisioned is rarely the bottleneck; the Arc node is.

So size --workers against Arc's free memory. The default of 2 is deliberately conservative. Raise it only when you know the Arc node has the headroom: 4 needs roughly 4-5 GB transiently free on Arc.

On the migration host, memory is modest and predictable. Extraction streams one series at a time (tsm2arc indexes the TSM key list up front, then reads, emits, and frees each series in turn), so extraction (and --dry-run) is near-constant memory bounded by the largest single series, not the shard. A multi-GB shard extracts in tens of MB. Load then adds the chunk buffers, so migration-host RAM is roughly workers × chunk-bytes (e.g. 4 × 450 MB ≈ 1.8 GB). Want less? Lower --chunk-bytes or --workers; both are safe, and resume and correctness are unaffected.

Step 5: verify the counts

The migration isn't done when the tool exits. It's done when you've reconciled the counts.

The clean way is to run the same dry-run twice and compare, because a dry-run reports authoritative per-database point counts straight from the source files:

# what's in the source (from the dry-run's per-DB totals)
tsm2arc --datadir /mnt/influxdb/data --waldir /mnt/influxdb/wal --dry-run

Then count the same measurements in Arc via its SQL API and reconcile against those numbers. Remember the two legitimate reasons a count can differ, so a small delta doesn't send you chasing ghosts:

• De-duplication. If a point existed in both TSM and WAL, tsm2arc already merged it (WAL wins), and Arc compaction collapses tag-bearing duplicates. Arc's total can be lower than a naive sum that double-counted overlap, and that's correct.
• System buckets. 2.x's _monitoring and _tasks are always skipped. 1.x's _internal is skipped unless you pass --include-internal. You almost never want those in Arc.

If the numbers reconcile, you're migrated.

Validate the tool itself, if you want proof

If you'd rather trust the codecs after watching them work than take a test suite's word, the repo ships a fixture that seeds a known dataset into a real InfluxDB and hands you an oracle to diff against:

cd fixture
docker compose up -d
./seed.sh                       # writes a known dataset: all field types, a
                                # tagless measurement, a pre-epoch timestamp,
                                # two databases, and prints the oracle
docker compose restart influxdb # clean shutdown flushes WAL → TSM
 
../tsm2arc --datadir ./data/influxdb/data --waldir ./data/influxdb/wal \
           --dry-run --sample 20

Every field type, a tagless measurement, and a pre-1970 timestamp (the exact corners that break naive TSM readers) round-trip and match what seed.sh told you to expect.

A migration checklist

The short version, in order:

1. Snapshot / mount read-only. tsm2arc never writes to the source, but mount it read-only anyway. Belt and suspenders.
2. Dry-run with --sample and eyeball the line protocol. Confirm databases, measurements, and counts look sane.
3. Include --waldir (1.x), or confirm it auto-detected (2.x). This is where silent data loss hides.
4. Provision an admin token via ARC_TOKEN, not --token on the command line.
5. Map and filter with --db-map / --database-filter if you want clean Arc names or a staged move.
6. Start with --workers 2. Raise it only against Arc's free memory, deliberately.
7. Keep the checkpoint file. If anything interrupts the run, re-run the identical command.
8. Reconcile counts between the dry-run totals and Arc before you call it done.

Reusable beyond Arc

Worth saying plainly: the extraction half of tsm2arc produces standard InfluxDB line protocol. The Arc sink is just the first sink. The whole TSM/WAL decoder is Apache-2.0 and can be reused to migrate InfluxDB 1.x/2.x data into other systems: ClickHouse, QuestDB, TimescaleDB, whatever you like. New sinks are welcome; see CONTRIBUTING.md.

We built it because leaving a database shouldn't require asking permission from the database you're leaving. Your data is on disk, in a format we can read. That's enough.

The full design (the verified Arc ingest constraints, the resume protocol, the on-disk format notes) is in docs/DESIGN.md. The tool is on GitHub. Point it at a cold volume and see what comes back.