Disaster Recovery¶

Arca writes a .meta sidecar file alongside every blob, so that the object index can be rebuilt from the filesystem if the metadata database is lost. This page explains the recovery architecture, the tools, and what a rebuild from sidecars can and cannot restore.

Known limitations of arca recover

Until TD-014 and TD-032 are fixed, a database rebuilt by arca recover is not equivalent to the one it replaces:

Only buckets, objects and credentials are restored. Users, teams, grants, every bucket configuration (versioning, encryption, lifecycle, Object Lock, policy, notifications, replication), bucket and object tags, retention and legal hold are lost, and every object's owner is set to root.
The oldest version of each key wins, not the newest (see Duplicate keys and versions).
Multipart objects are dropped, and their parts reappear as bogus key#uploadId#n objects.
Compressed objects are skipped unless you pass --skip-verify.
It always writes a SQLite database, even on a PostgreSQL deployment.

Treat arca recover as a last resort for salvaging object data, and prefer restoring a backup of the metadata database (see Backup Strategy).

Filesystem as Source of Object Data¶

Every object stored in Arca produces two files:

Blob file: the object bytes (encrypted and/or compressed when those features are enabled), named by UUID
.meta sidecar: a JSON file describing the object (bucket, key, size, ETag, content type, metadata, encryption and compression parameters)

The deliberate write order ensures recoverability:

1. Write blob file  →  2. Write .meta sidecar  →  3. Insert into the metadata database

If the server crashes at any point:

After step 1 only: orphaned blob, no sidecar — arca fsck detects and reports it
After step 2: blob + sidecar exist but DB doesn't know — arca recover rebuilds the DB entry
After step 3: fully consistent

Sidecar `.meta` Format¶

Each sidecar file is a JSON document:

{
  "bucket": "my-bucket",
  "key": "photos/2024/vacation.jpg",
  "size": 1234567,
  "etag": "d41d8cd98f00b204e9800998ecf8427e",
  "content_type": "image/jpeg",
  "last_modified": "2026-02-27T14:30:00Z",
  "metadata": {"x-amz-meta-author": "pietro"}
}

Field	Description
`bucket`, `key`	Object location
`size`	Plaintext size in bytes
`etag`	S3 ETag (the MD5 of the plaintext for single-part objects)
`content_type`	Content type, or `null`
`last_modified`	RFC 3339 timestamp
`metadata`	User and system metadata (`x-amz-meta-*`, `cache-control`, ...); empty when absent
`encryption`	Optional. Present on encrypted blobs: `algorithm` (`AES256` or `SSE-C`), `encrypted_dek`, `dek_nonce`, `nonce_prefix`, `key_id`
`compression`	Optional. Present on compressed blobs: `algorithm`, `chunk_size`, `original_size`, `compressed_size`
`version_id`	Optional. Version ID of an object written to a versioned bucket
`composite`	Optional. Present on a multipart object assembled without copying its parts: the list of parts (`blob_id`, `plaintext_size`, `plaintext_etag`, optional `encryption`). No blob file exists for a composite sidecar; reads stream from the parts.

Backup Strategy¶

To back up an Arca instance, you need:

What	Path	Why
Blob files + sidecars	`{data_dir}/blobs/`	Object data + metadata for recovery
Metadata database	`{data_dir}/arca.db` (SQLite), or your PostgreSQL database	Everything else: users, teams, grants, credentials, bucket configuration, tags, versions, retention, and the object index
Config file	`/etc/arca/config.toml`	Server settings

Tip

Back up the metadata database together with the blobs directory. arca recover can salvage the objects from the blobs directory alone, but most of the database content cannot be rebuilt from sidecars (see the warning at the top of this page).

`arca recover`¶

Rebuild the database from sidecar files.

arca recover [--config-path <PATH>] [--dry-run] [--skip-verify]

Step-by-Step Process¶

Walks {data_dir}/blobs/ recursively, reading all .meta sidecar files
Verifies each blob file exists and its MD5 matches the sidecar ETag (unless --skip-verify)
Preserves credentials from the existing SQLite database at {data_dir}/arca.db (if any)
Deletes that SQLite database and creates a fresh one (which contains only the root user and the built-in grants)
Restores the saved credentials
Recreates every bucket found in the sidecars (unversioned, with no configuration) and inserts one object per sidecar, owned by root

arca recover always works on the SQLite file in data_dir, regardless of metadata_backend. On a PostgreSQL deployment it does not touch the PostgreSQL database: it writes an SQLite file the server will not use (TD-032).

What Is Restored and What Is Lost¶

Restored	Lost
Buckets (names only)	Users, teams and grants (only the root user and the built-in grants of a fresh database exist afterwards; credentials of other users are restored, their owning users are not)
Objects: bucket, key, size, ETag, content type, last-modified, metadata, encryption parameters	Bucket configuration: versioning, encryption, lifecycle, Object Lock, policy, notifications, replication
Credentials, when the old SQLite database is still readable	Bucket and object tags
	Retention and legal hold
	Object ownership (sidecars do not carry the owner: every recovered object is owned by `root`)
	Older versions of versioned objects, and version IDs
	Multipart (composite) objects

Options¶

Option	Description
`--dry-run`	Print what would be recovered without modifying the database
`--skip-verify`	Skip MD5 checksum verification of blob files (faster, and required to keep compressed objects)
`--config-path`	Path to the configuration file (default: `/etc/arca/config.toml`)

Usage¶

Always start with a dry run:

# Preview recovery
arca recover --dry-run

# Full recovery with checksum verification
arca recover

# Fast recovery (skip checksums — useful for very large datasets)
arca recover --skip-verify

Credential Preservation¶

During recovery, credentials are read from the existing database before it is replaced. If the old database is readable, credentials survive the recovery. If the database is completely lost, you'll need to let Arca generate new root credentials on the next startup.

Duplicate Keys and Versions¶

When several sidecars claim the same bucket + key (the versions of a versioned object, or overwritten objects whose old blob was not yet reclaimed), the oldest one wins. This is a known bug (TD-032 in Technical Debt): sidecars are inserted newest first by last_modified, the buckets are re-created unversioned, and every insert overwrites the previous one, so the last insert (the oldest sidecar) is what remains. The version IDs are discarded.

Edge Cases¶

Multipart objects: a multipart object assembled as a composite has a sidecar but no blob file, so it is reported as an orphaned sidecar, skipped with a warning, and missing from the rebuilt database. Its part blobs carry ordinary sidecars whose key is {key}#{upload_id}#{part_number}, so each part is re-created as a separate, bogus object under that key (TD-014 in Technical Debt). Other objects with a multipart ETag (containing -) skip checksum verification, because the composite ETag is not the MD5 of the blob.
Encrypted objects: checksum verification is skipped (with a note), because the ciphertext on disk cannot be compared with the plaintext ETag
Compressed objects: the ETag is the MD5 of the plaintext but recover hashes the compressed bytes on disk, so verification fails and the object is skipped with a checksum-mismatch warning. Run with --skip-verify to keep them (TD-032).
Orphaned sidecars (no blob file): skipped with a warning
Malformed JSON: skipped with a warning
Checksum mismatches: skipped with a warning (blob may be corrupted)

`arca fsck`¶

Check consistency between the database and the filesystem, and with --repair fix what it finds.

arca fsck [--config-path <PATH>] [--verify-checksums] [--grace-seconds <SECONDS>]
          [--repair [--yes] [--delete] [--abort-uploads-older-than <DAYS>]]

The check types, the repair of each one, the options and the exit codes are in the CLI reference. In short:

fsck understands multipart objects: the parts of a completed upload and of an upload still in progress are live data, exactly as arca gc sees them, so the two tools agree.
On a running server, files younger than --grace-seconds (default one day) are skipped: they may belong to an upload in flight.
Unfinished multipart uploads are listed with their age and size. They are the first thing to look at when a disk fills up for no visible reason (Troubleshooting).
--repair asks before each fix, can be interrupted with Ctrl+C and simply run again, quarantines orphaned files in <data_dir>/lost+found/ unless --delete, and deletes the row of an object version only when its data is really gone.

Usage¶

# Quick consistency check
arca fsck

# Full check including blob checksums (slow for large datasets)
arca fsck --verify-checksums

# Repair, asking before each fix
arca fsck --repair

# With the server stopped: repair everything, abort uploads older than 7 days
arca fsck --repair --yes --grace-seconds 0 --abort-uploads-older-than 7

Data Integrity¶

arca fsck relies on two checks:

What	How	Check
Blob content	MD5 of the blob file compared with the ETag stored in the database (the ETag of a single-part object is its MD5 hex digest, as the S3 protocol requires)	`CORRUPT` (only with `--verify-checksums`)
`.meta` sidecar	Field-by-field comparison of `bucket`, `key`, `size`, `etag` and (when the sidecar carries one) the version id with the database record; an unreadable or malformed sidecar is reported apart	`SIDECAR_MISMATCH`, `UNREADABLE_SIDECAR`

Note

These checks require the database to be intact. During arca recover, sidecars are trusted by necessity since there is nothing to compare against.

Disaster Recovery¶

Filesystem as Source of Object Data¶

Sidecar .meta Format¶

Backup Strategy¶

arca recover¶

Step-by-Step Process¶

What Is Restored and What Is Lost¶

Options¶

Usage¶

Credential Preservation¶

Duplicate Keys and Versions¶

Edge Cases¶

arca fsck¶

Usage¶

Data Integrity¶

Sidecar `.meta` Format¶

`arca recover`¶

`arca fsck`¶