Skip to content

Replication

Arca replicates objects asynchronously from a source bucket to a destination bucket on any S3-compatible endpoint. The destination can be another Arca instance, AWS S3, MinIO, or any server that speaks the S3 API and signs with SigV4.

Replication is one-way per rule, but two-way mirrors are supported safely by configuring symmetric rules on both sides — the loop-prevention contract (see below) guarantees no ping-pong.

What gets replicated

Event Replicated?
PutObject Yes
CompleteMultipartUpload Yes
CopyObject (destination side) Yes
PutObjectTagging / DeleteObjectTagging Yes
Delete marker on versioned bucket Yes (opt-in per rule, default on)
Hard delete of a specific version No (AWS semantics)
Object Lock / retention changes Not yet (tracked for a later phase)
ACLs Not yet

The destination bucket must already exist — Arca does not auto-create it.

Prerequisites

  • Versioning must be Enabled on the source bucket. Replication operates on object versions; PutBucketReplication returns InvalidRequest otherwise. Enable versioning first (console → Bucket Settings → Versioning → toggle).
  • A destination credential stored in the source instance's server settings, under replication.credentials.<name>. The console's rule editor can create one inline; the admin API accepts direct POSTs (see below).

Quick Start — console

  1. Open Buckets → bucket name → Settings on the source instance. Scroll to the Replication card.
  2. If the card shows an amber "Versioning required" banner, click Enable versioning (the link scrolls to the Versioning card), toggle it on and reload.
  3. Click + Add Rule.
  4. Fill in:
  5. Rule ID: any stable string (auto-generated by default).
  6. Prefix (optional): replicate only objects whose key starts with this string.
  7. Tag filter (optional): click + Tag to require one or more key=value tags. An object must carry all listed tags to match. Combine with prefix to narrow further — both conditions must hold.
  8. Destination Bucket, Endpoint URL (e.g. https://replica.example.com), Region.
  9. Destination credentials: pick an existing one, or choose + New, enter a credential name plus the access-key / secret, and save.
  10. Replicate delete markers: default on.
  11. Click Add rule.

The replication worker picks up new objects on its next tick (default 15 s). Each object's x-amz-replication-status header moves PENDING → COMPLETED on the source and arrives as REPLICA on the destination.

Quick Start — CLI / direct API

1. Store the destination credential

curl -X POST "http://localhost:9000/admin/replication/credentials/replica-creds" \
  --aws-sigv4 "aws:amz:us-east-1:s3" \
  --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
  -H "Content-Type: application/json" \
  -d '{"access_key_id":"AKIA...","secret_access_key":"..."}'

2. PUT the replication configuration

aws s3api put-bucket-versioning --bucket source-bucket \
  --versioning-configuration Status=Enabled \
  --endpoint-url http://localhost:9000

cat >/tmp/repl.xml <<'EOF'
<ReplicationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Role></Role>
  <Rule>
    <ID>to-replica</ID>
    <Status>Enabled</Status>
    <Priority>1</Priority>
    <Filter><Prefix></Prefix></Filter>
    <Destination>
      <Bucket>replica-bucket</Bucket>
      <Endpoint>https://replica.example.com</Endpoint>
      <Region>us-east-1</Region>
      <CredentialRef>replica-creds</CredentialRef>
    </Destination>
    <DeleteMarkerReplication><Status>Enabled</Status></DeleteMarkerReplication>
  </Rule>
</ReplicationConfiguration>
EOF

curl -X PUT "http://localhost:9000/source-bucket?replication" \
  --aws-sigv4 "aws:amz:us-east-1:s3" \
  --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
  -H "Content-Type: application/xml" \
  --data-binary @/tmp/repl.xml

Filter shapes

The <Filter> block accepts three forms (matching the AWS ReplicationConfiguration surface):

  • Prefix only — replicate every object under the prefix: 
    <Filter><Prefix>logs/</Prefix></Filter>
  • Single tag, no prefix — replicate every object that carries this exact tag: 
    <Filter><Tag><Key>env</Key><Value>prod</Value></Tag></Filter>
  • Prefix plus one or more tags — replicate only objects matching all listed conditions, via <And>: 
    <Filter>
  <And>
    <Prefix>data/</Prefix>
    <Tag><Key>env</Key><Value>prod</Value></Tag>
    <Tag><Key>team</Key><Value>core</Value></Tag>
  </And>
</Filter>

Tags are evaluated against the object's tag state at the moment of the replicated operation. Tagging an existing object via PutObjectTagging emits a separate tag-replication event; it does not retroactively replicate the object itself (AWS CRR semantics — use S3 Batch Replication for backfill).

3. Upload an object and verify

aws s3 cp ./hello.txt s3://source-bucket/hello.txt --endpoint-url http://localhost:9000

# Replication status on the source
aws s3api head-object --bucket source-bucket --key hello.txt \
  --endpoint-url http://localhost:9000 \
  | jq -r '.ReplicationStatus'
# COMPLETED   (after the worker's next tick)

# On the destination, the replicated object carries status REPLICA
aws s3api head-object --bucket replica-bucket --key hello.txt \
  --endpoint-url https://replica.example.com \
  | jq -r '.ReplicationStatus'
# REPLICA

The loop-prevention contract

Replication extensions Arca adds on top of the standard S3 replication surface:

  • Every outbound request from the replication worker carries a custom header x-amz-arca-replication-source: <source_endpoint_id>. The value comes from [replication].source_endpoint_id in the TOML config (default arca; set a unique stable string per deployment in production).
  • When an Arca instance receives a PutObject / DeleteObject / PutObjectTagging whose request carries that header, it:
  • Stamps the object's replication_status as REPLICA instead of computing a PENDING emit.
  • Skips the journal emit entirely — no journal row is inserted.
  • When the emit logic checks a rule's filter, it short-circuits on REPLICA objects, so they are never re-emitted by a rule configured on the receiving side.

This is what makes two-way mirrors safe. Configure the same-named bucket on both Arcas with symmetric rules: a client-originated PutObject on A fires exactly once on A's journal, shows up on B as a REPLICA (no emit), and stays. No ping-pong. The test_mirror_does_not_loop integration test asserts this invariant.

Caveat: destinations that are not Arca (AWS S3, MinIO, …) ignore the custom header, so two-way mirrors against non-Arca destinations are not supported — use one-way only.

Two-way mirror setup (Arca ↔ Arca)

  1. On instance A, create the destination credential (pointing at B) and a rule source-bucket → source-bucket with Endpoint=B.
  2. On instance B, create a destination credential (pointing at A) and a rule source-bucket → source-bucket with Endpoint=A.
  3. Enable versioning on source-bucket on both sides.

Writes on either side now reach the other; convergence is eventual.

Replication journal

Every pending / in-flight / failed delivery lives in the replication_journal table. The admin console has a dedicated Replication view at #/replication — it's the admin's single pane of glass:

  • Created / Bucket / Key / Rule / Event type / Destination / Status / Tries / Actions columns with inline column-header filters.
  • Color-coded status chips, pulsing on in_flight entries.
  • Side panel with a dedicated "Flow" summary (source ➜ destination), the last error, and a per-entry Retry action.

Programmatic access:

# List journal entries
curl "http://localhost:9000/admin/replication/journal?status=failed" \
  --aws-sigv4 "aws:amz:us-east-1:s3" \
  --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
  | jq

# Re-queue a specific entry
curl -X POST "http://localhost:9000/admin/replication/retry/<entry-id>" \
  --aws-sigv4 "aws:amz:us-east-1:s3" \
  --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY"

Retry policy

  • The worker runs on a timer (poll_interval_seconds, default 15 s) and claims up to batch_size pending rows per cycle (default 100).
  • On delivery failure, the entry's attempts is incremented and next_retry_at is set to now + retry_base_seconds * 2^(attempts-1) (capped at 1 hour).
  • After max_retries failures (default 10) the entry is marked failed and the source object's replication_status is stamped FAILED.
  • Transient errors don't stamp FAILED — only the terminal retry does. Clients polling HEAD won't see spurious FAILED blips during a brief network hiccup.

Conflict resolution

Before every PUT the worker does a HEAD on the destination:

  • If the destination object's Last-Modified is greater than or equal to the source's, the PUT is skipped (destination-wins). The journal entry is still marked completed and the source is stamped COMPLETED.
  • If the HEAD fails (network issue, 4xx), the worker proceeds with the PUT — a destination that rejects the write will surface as a normal failure and retry.

This matches S3's eventual-consistency semantics: replicated writes never silently overwrite a newer destination version, and there is no cross-instance coordination protocol.

Journal retention

The existing retention-purge worker prunes completed journal rows older than journal_retention_days (default 30). A hard cap journal_max_age_days (default 90) prunes rows of any status, so even a destination that has been offline indefinitely won't blow up the journal. Both thresholds are exposed in the console Settings page next to the other retention knobs.

TOML configuration reference

[replication]
poll_interval_seconds  = 15     # worker tick
batch_size             = 100    # rows claimed per tick
max_retries            = 10     # before FAILED
retry_base_seconds     = 5      # exponential base; capped at 1h
request_timeout_seconds = 60
source_endpoint_id     = "arca-east"   # stable per-deployment; loop-prevention marker
journal_retention_days = 30     # completed rows purged after
journal_max_age_days   = 90     # any row purged after

Or start a development replica alongside the primary:

bin/arca start -d --dev --replication   # boots arca on :9000 and arca-replica on :9001
bin/test replication                    # runs the 4 boto3 integration tests
  • Access Control — destination credentials are just another SigV4 access-key/secret pair.
  • Monitoring & Logging — /admin/metrics exposes journal counters alongside the rest.