Arc 26.03.1: Backup & Restore, TLE Ingestion, and 13 Bugs That Won't Bother You Anymore

Two weeks since 26.02.2 and I almost didn't believe the changelog when I put it together. Three new features, thirteen bug fixes, a comprehensive code review across eleven components, a Go upgrade, and a bunch of performance wins that you get for free just by updating.

This is the most complete release we've shipped. Not the flashiest — that was probably the MQTT and InfluxDB compatibility release — but the one where we went deep on the things that matter when you're running Arc in production. Backup and restore. Proper null handling. Fixing that time_bucket bug that was making Grafana dashboards timeout. The kind of work that doesn't make for exciting demos but makes the difference between a database you're testing and one you're trusting.

Let's get into it.

Backup & Restore API

Since day one, the question I kept hearing was: "What happens if something goes wrong?"

Fair question. You're running a database. You need to know your data is safe. Until now, if you wanted backups, you had to roll your own — snapshot the storage, copy the SQLite file, hope you got everything in a consistent state. It worked, but nobody liked it.

Arc now has a full backup and restore system via REST API. One call to start a backup, poll for progress, and restore when you need to — with granular control over what gets restored.

# Start a backup
curl -X POST "http://localhost:8000/api/v1/backup" \
  -H "Authorization: Bearer $ARC_TOKEN"
 
# Poll progress
curl "http://localhost:8000/api/v1/backup/status" \
  -H "Authorization: Bearer $ARC_TOKEN"
 
# {"operation": "backup", "backup_id": "backup-20260211-143022-a1b2c3d4",
#  "status": "running", "total_files": 1200, "processed_files": 450,
#  "total_bytes": 5368709120, "processed_bytes": 2147483648}

Backups capture everything: Parquet data files, SQLite metadata (auth tokens, audit logs, MQTT config), and your arc.toml configuration. The operations run asynchronously in the background with a two-hour timeout, so you're not holding a connection open while gigabytes get copied.

When it's time to restore, you choose what comes back:

curl -X POST "http://localhost:8000/api/v1/backup/restore" \
  -H "Authorization: Bearer $ARC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "backup_id": "backup-20260211-143022-a1b2c3d4",
    "restore_data": true,
    "restore_metadata": true,
    "restore_config": false,
    "confirm": true
  }'

Want to restore data but keep your current config? Done. Just metadata? Done. The existing files get copied with a .before-restore suffix before anything gets overwritten, so you always have a way back.

It works across all storage backends — local filesystem, S3, Azure Blob. No additional configuration beyond enabling the backup path. Point your cron job at the backup endpoint and sleep well.

Line Protocol Bulk Import

You've got an InfluxDB export sitting on disk. Maybe it's a 500MB .lp file from influx_inspect, maybe it's a gzipped dump from a data pipeline. Until now, getting that into Arc meant writing a script to stream it through the write API batch by batch.

Not anymore.

curl -X POST "http://localhost:8000/api/v1/import/lp" \
  -H "X-Arc-Database: mydb" \
  -F "file=@export.lp"

One command. The data flows through Arc's high-performance columnar pipeline — the same ArrowBuffer → ArrowWriter → Parquet → storage path used by streaming ingestion — so bulk imports get the same throughput and sort optimization as live data.

Gzip compression is auto-detected. Timestamps respect the precision parameter (ns, us, ms, s), so you don't lose fidelity when importing from InfluxDB's nanosecond-default exports. A single file can contain multiple measurements, and they all get routed correctly.

# Import with second-precision timestamps
curl -X POST "http://localhost:8000/api/v1/import/lp?precision=s" \
  -H "X-Arc-Database: mydb" \
  -F "file=@export_seconds.lp"

If you've been putting off migrating from InfluxDB because the data migration felt like too much work — this is your window.

TLE Ingestion: Native Satellite Data

If you've seen the satellite tracking demo on our site — 14,000+ objects orbiting Earth on a 3D globe, color-coded by orbit type, with trajectory trails when you click on one — this is how the data gets in.

TLE (Two-Line Element) is the standard format used by Space-Track.org, CelesTrak, and every ground station pipeline to describe satellite orbits. It's a 60-year-old format, and until now, ingesting it into a time-series database meant writing a custom parser.

Arc now speaks TLE natively. Two endpoints:

Streaming — for continuous feeds, cron jobs, and real-time updates:

curl -X POST "http://localhost:8000/api/v1/write/tle" \
  -H "X-Arc-Database: satellites" \
  --data-binary @stations.tle

Bulk import — for historical backfill from Space-Track catalog dumps:

curl -X POST "http://localhost:8000/api/v1/import/tle" \
  -H "X-Arc-Database: satellites" \
  -F "file=@catalog.tle"

The parser is pure Go, handles both 2-line and 3-line TLE formats (including mixed files), validates checksums, and derives orbital metrics automatically — semi-major axis, period, apogee, perigee, and orbit classification (LEO/MEO/GEO/HEO). Bad entries get skipped with warnings, not fatal errors.

The performance story is interesting. TLE ingestion uses a typed columnar fast path that bypasses the []interface{} intermediary used by generic ingestion. The parser operates directly on []byte input with contiguous record allocation and single-pass typed column construction: ~3.5M records/sec on commodity hardware.

Once it's in, it's just SQL:

SELECT object_name, orbit_type, period_min, perigee_km, apogee_km
FROM satellite_tle
WHERE orbit_type = 'LEO'
ORDER BY period_min

This is the kind of domain-specific feature that makes Arc different. Not every database needs to parse satellite orbital data. But if you're in aerospace, defense, or space situational awareness — and you need to store, query, and analyze orbital elements at scale — it's built in.

Bug Fixes

Thirteen fixes. Some subtle, some not.

Null Handling in Line Protocol (#202)

This one was reported by https://github.com/bjarneksat — thank you.

If you ingested line protocol with different field sets to the same measurement (line 1 has field1, line 2 has field2), the missing fields were stored as 0 instead of NULL. That's a huge difference when you're doing aggregations. AVG() on a column full of zeros gives you a wrong answer. AVG() on a column with proper NULLs gives you the right one.

The root cause was deep in the pipeline: the type conversion step was discarding null information, and the Parquet writer wasn't passing validity bitmaps to Arrow. We introduced a TypedColumnBatch that carries validity bitmaps through the entire path — merge, sort, slice, write.

Stale Cache After Compaction (#204)

After compaction merged old Parquet files and deleted them from S3, queries would intermittently fail with 404 errors. DuckDB's cache_httpfs was caching directory listings that still referenced the deleted files.

Arc now clears all relevant caches — glob, parquet metadata, partition pruner, and SQL transform — immediately after each compaction job.

For enterprise clustering, where compaction runs on a dedicated node, the Compactor now broadcasts cache invalidation to all cluster peers via an internal endpoint. Fire-and-forget with a 5-second timeout. If a reader is temporarily unreachable, its cache expires naturally via TTL.

Actual Error Messages (#207)

Query endpoints used to return "Query execution failed" no matter what went wrong. Now you get the actual DuckDB error: Parser Error: syntax error at or near "SELEC". Obvious in hindsight. Makes SQL debugging in Grafana dramatically easier.

time_bucket and date_trunc (#212)

This one was painful. time_bucket() and date_trunc() GROUP BY queries were returning one row per unique second instead of proper time buckets. A 7-day hourly query returned 604,801 rows (16.9MB) instead of 169 rows (5KB). Grafana dashboards got painfully slow on wider time ranges — they'd eventually load, but nobody wants to wait thirty seconds for a chart that should render instantly.

The root cause was a subtle difference between DuckDB and PostgreSQL. Arc's query rewriter converts time-bucketing SQL to epoch-based arithmetic for performance. The rewritten SQL used DuckDB's / operator, which performs float division on integers (unlike PostgreSQL's integer division). So (epoch(time)::BIGINT / 3600) * 3600 returned the original value — no bucketing happened at all.

Fix: change / to // (DuckDB's integer division operator). Three lines of code. Weeks of headaches.

WAL Recovery After Flush Failure (#218)

When an S3 flush failed (timeout, network blip), WAL recovery would replay all rotated WAL files — including the ones whose data was already successfully written to Parquet. This caused data duplication on storage, inflated query results, and increased costs.

The fix adds a purge step before recovery in the failure branch, limiting the replay window to data that actually needs replaying.

Self-Adjusting Flush Timer (#142)

The periodic flush used a fixed-period ticker, which meant buffers created just after a tick waited up to 1.5x the configured age before flushing. Replaced with a self-adjusting timer that fires exactly when the oldest buffer is due. A signal channel recomputes the deadline on every new buffer.

MQTT CleanSession (#239)

CleanSession was hardcoded to true, which told the broker to discard unacknowledged messages when Arc disconnected. Combined with QoS=1, this meant at-least-once delivery quietly became at-most-once across reconnects. Messages sent during the reconnection window were silently dropped.

Now configurable per subscription, defaulting to false. If you're running MQTT ingestion in production, this one matters.

Delete API Honesty (#235)

The delete endpoint used to return success: true even when some files failed to rewrite. Now it returns HTTP 207 with success: false, a list of failed_files, and an error summary. Successfully processed files still get reported. The API tells the truth.

Compaction Manifest Cleanup (#240)

Three related bugs in the compaction manifest system that could leave orphaned input files alongside compacted output files, causing queries to return duplicated data. Stale manifests now follow the full recovery path, manifest deletion is deferred until all input files are removed, and filtering errors skip the partition instead of processing blind.

And a Few More

Replication metrics (#237) — two new Prometheus metrics for monitoring replication: dropped entries and sequence gaps.

Orphaned hot files (#236) — a reconciliation pass now detects and removes files that were migrated to cold storage but never deleted from hot.

Cache TTL tuning (#214) — DuckDB's glob, metadata, and file handle caches are now properly unified. Glob TTL fixed at 10 seconds (directory listings change during compaction), metadata/handle TTLs aligned with parquet immutability. Cache sizes scale proportionally with configured limits.

Code Quality & Security

We paused to do a two-pass code review across eleven Arc components: Scheduler, Clustering, Backup, MQTT, Querying, Compaction, Line Protocol, MsgPack, CSV/Parquet Import, and TLE.

On the security side: write endpoints (/write, /api/v2/write, /api/v1/write/line-protocol, CSV, Parquet, MsgPack) now validate database names to prevent path traversal. Backup restore writes files with 0600 permissions instead of 0644 — the config and SQLite database should never be world-readable.

We also fixed a scheduler goroutine leak on shutdown, an MQTT subscription TOCTOU race that could start duplicate subscribers, and unbounded memory growth in the cluster router's connection map.

On performance: CSV, Parquet, and TLE imports now share a pooled gzip reader (3-5x faster decompression). The Line Protocol parser got a single-pass unescape() function replacing three sequential strings.ReplaceAll calls. The SQL safety validator went from 7 regex passes to 1.

And some cleanup: the CSV and Parquet import handlers shared ~80% identical code and are now consolidated. The LP parser's duplicate splitLine() and splitOnComma() functions were extracted into a single shared function.

Go 1.26

Free performance. Upgraded from Go 1.25.6 to Go 1.26.

• Green Tea GC — 10-40% reduction in garbage collection overhead. Enabled by default.
• 30% faster cgo calls — every DuckDB query and SQLite operation benefits.
• 2x faster io.ReadAll — improves S3/Azure storage reads and HTTP response parsing.
• Stack allocation for slice backing stores — compiler can stack-allocate more slices, reducing heap pressure in hot paths.

No code changes needed. Just upgrade and the runtime is faster.

Get It

docker run -d \
  -p 8000:8000 \
  -e STORAGE_BACKEND=local \
  -v arc-data:/app/data \
  ghcr.io/basekick-labs/arc:latest

• GitHub: github.combasekick-labs/archttps://github.com/basekick-labs/arc
• Documentation: https://docs.basekick.net/arc
• Discord: https://discord.gg/nxnWfUxsdm
• Python SDK: https://pypi.org/project/arc-tsdb-client/