Roadmap¶

MVP¶

0

1

2

3

4

5

6

7

8

9

10

11

The Arca MVP (Minimum Viable Product) is complete. All 12 phases (0–11) have been implemented, tested, and verified. The server implements 15 S3 operations with 100% pass rate on implemented features against the Ceph s3-tests compatibility suite (270/829 passing — all 468 failures are in unimplemented feature categories).

Full Product Roadmap¶

The post-MVP roadmap covers the path from a functional single-node S3 server to a production-grade, enterprise-ready storage platform. Phases are numbered from 12 onward, continuing from the MVP phases (0–11).

Priority Legend¶

Tag	Meaning
P0	Critical — required for production deployment
P1	High — expected by most users
P2	Medium — improves completeness and operational maturity
P3	Low — advanced features, long-term vision

Progress Overview¶

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

29.1

30

31

Dependency Overview¶

graph LR
    12["12 TLS"] --> 15["15 Presigned URLs\n+ SSE-C"]
    13["13 SSE-S3"] --> 14["14 SSE-KMS\nVault/OpenBAO"]
    14 --> 15
    13 --> 16["16 Access Control\n+ Policies"]
    13 --> 19["19 Object\nTagging"]
    19 --> 20["20 Lifecycle\nRules"]
    16 --> 17["17 Object\nVersioning"]
    17 --> 21["21 Object Lock\nWORM"]
    20 --> 25["25 Notifications\n+ Events"]
    25 --> 26["26 Notification\nConnectors"]
    13 --> 27["27 Transparent\nCompression"]
    17 --> 28["28 Replication"]
    13 --> 23["23 Performance\n+ Hardening"]
    24["24 PostgreSQL\nBackend"] --> 28
    28 --> 29["29 Multi-Node\nHigh Availability"]
    29 --> H291["29.1 HA\nHardening"]
    29 --> 30["30 Migration\n+ Maintenance"]
    13 --> 30
    24 --> 30
    18 --> 31["31 OpenTelemetry\nIntegration"]

    style 12 fill:#c62828,color:#fff
    style 13 fill:#c62828,color:#fff
    style 14 fill:#c62828,color:#fff
    style 15 fill:#c62828,color:#fff
    style 16 fill:#e65100,color:#fff
    style 17 fill:#e65100,color:#fff
    style 18 fill:#e65100,color:#fff
    style 19 fill:#2e7d32,color:#fff
    style 20 fill:#2e7d32,color:#fff
    style 21 fill:#2e7d32,color:#fff
    style 22 fill:#2e7d32,color:#fff
    style 23 fill:#2e7d32,color:#fff
    style 24 fill:#2e7d32,color:#fff
    style 25 fill:#2e7d32,color:#fff
    style 26 fill:#1565c0,color:#fff
    style 27 fill:#1565c0,color:#fff
    style 28 fill:#1565c0,color:#fff
    style 29 fill:#1565c0,color:#fff
    style H291 fill:#e65100,color:#fff
    style 30 fill:#1565c0,color:#fff
    style 31 fill:#1565c0,color:#fff

    18["18 Monitoring\n+ Audit"]
    22["22 S3 API\nCompleteness"]

Legend: P0 Critical · P1 High · P2 Medium · P3 Low — Arrows indicate dependencies

Phase Summary¶

Phase	Name	Priority	Dependencies	Version	Status
12	TLS/SSL and Transport Security	P0	—	`v0.2.0`	✔
13	Server-Side Encryption: SSE-S3	P0	—	`v0.3.0`	✔
14	SSE-KMS with HashiCorp Vault/OpenBAO	P0	13	`v0.5.0`	✔
15	Presigned URLs, Query-String Auth, and SSE-C	P1	12, 13	`v0.6.0`	✔
16	Access Control and Bucket Policies	P1	13	`v0.7.0`	✔
17	Object Versioning	P1	16	`v0.8.1`	✔
18	Monitoring, Metrics, and Audit	P1	—	`v0.9.0`	✔
19	Object Tagging	P2	13	`v0.12.0`	✔
20	Lifecycle Rules	P2	19, 18	`v0.13.0`	✔
21	Object Lock (WORM Compliance)	P2	17	`v0.13.0`	✔
22	S3 API Completeness	P2	—	`v0.14.0`	✔
23	Performance and Hardening	P2	13	`v0.14.0`	✔
24	PostgreSQL Backend	P2	—	`v0.16.1`	✔
25	Notifications and Event System	P3	20	`v0.18.0`	✔
26	Notification Connectors	P3	25	`v0.20.0`	✔
27	Transparent Compression	P2	13	`v0.21.0`	✔
28	Replication	P3	17, 24	`v0.23.0`	✔
29	Multi-Node High Availability	P3	All prior	`v0.25.0`	✔
29.1	HA Hardening	P1	29	`v0.26.0`	✔
30	Migration and Maintenance	P3	13, 24, 29	`v0.28.0`	✔
31	OpenTelemetry Integration	P3	18

Phase 12 — TLS/SSL and Transport Security [P0]¶

Native TLS termination in the Arca binary via tokio-rustls, enabling encrypted transport without requiring a reverse proxy.

Phase 13 — Server-Side Encryption: SSE-S3 [P0]¶

Transparent at-rest encryption using AES-256-GCM with per-object data encryption keys (DEKs). Local master key from config provides a bootstrap mode before Vault integration.

Phase 14 — SSE-KMS with HashiCorp Vault/OpenBAO [P0]¶

Master key fetched from HashiCorp Vault or OpenBAO (100% API-compatible) KV v2 secrets engine at startup, cached in memory. Vault is only needed at startup — not a runtime dependency. Encryption pipeline (EncryptingBlobStore, DEK wrap/unwrap, streaming) unchanged.

Depends on: Phase 13 (encryption pipeline and bucket encryption config)

Phase 15 — Presigned URLs, Query-String Auth, and SSE-C [P1]¶

Enable URL-based authentication for direct browser downloads and customer-provided encryption keys.

Depends on: Phase 12 (TLS recommended for production presigned URLs), Phase 13 (encryption pipeline for SSE-C)

Phase 16 — Access Control and Bucket Policies [P1]¶

Role-based access control with users, teams, and policy-based authorization.

Depends on: Phase 13 (bucket_config table)

Phase 17 — Object Versioning [P1]¶

Full object versioning with version IDs, delete markers, and version-specific operations.

Depends on: Phase 16 (access control interacts with versioning)

Phase 18 — Monitoring, Metrics, and Audit [P1]¶

Operational visibility through metrics, audit logging, instance-wide settings, and region support.

Depends on: None

Phase 19 — Object Tagging [P2]¶

S3-compatible object and bucket tagging with key-value metadata.

Object tagging: PutObjectTagging / GetObjectTagging / DeleteObjectTagging. New object_tags table (migration v11). Max 10 tags per object, key max 128 chars, value max 256 chars
Bucket tagging: PutBucketTagging / GetBucketTagging / DeleteBucketTagging. New bucket_tags table
Tags on PutObject via x-amz-tagging header. Tags on CopyObject via x-amz-tagging-directive
Version-aware tagging: tags tied to specific object versions when bucket versioning is enabled
Cascade deletes: object/bucket tags cleaned up on object/bucket deletion
(Console) Object tagging UI (view/edit key-value pairs in detail panel), bucket tags in bucket settings

Depends on: Phase 13 (bucket_config table)

Phase 20 — Lifecycle Rules [P2]¶

Automated lifecycle management for storage hygiene with configurable expiration and cleanup rules.

Lifecycle rules: PutBucketLifecycleConfiguration / GetBucketLifecycleConfiguration / DeleteBucketLifecycleConfiguration. XML format (S3 compatible). Rules stored as JSON in bucket_config
Expiration: delete objects after N days, with prefix and tag-based filtering (single tag, And filter with prefix + tags)
NoncurrentVersionExpiration: hard-delete old versions after N days
Abort incomplete multipart uploads after N days
Configurable evaluation interval via lifecycle_evaluation_interval admin setting (default: 3600s / hourly)
Background lifecycle worker using existing BackgroundWorker::spawn_periodic framework. Batch processing (100 objects/rule/cycle), audit logging for all lifecycle actions
(Console) Lifecycle rules editor in bucket settings: add/remove/save rules with prefix filter, expiration days, noncurrent days, abort upload days

Depends on: Phase 19 (tagging for tag-based filtering), Phase 18 (background worker framework)

Phase 21 — Object Lock (WORM Compliance) [P2]¶

Write-Once-Read-Many compliance for regulatory and data protection requirements.

Object Lock config: PutObjectLockConfiguration / GetObjectLockConfiguration. Per-bucket default retention mode (GOVERNANCE / COMPLIANCE) and period (days or years). Auto-enables versioning, prevents suspension. Stored as JSON in bucket_config
Per-object retention: PutObjectRetention / GetObjectRetention. Mode + retain-until-date per version. COMPLIANCE can only be extended. GOVERNANCE requires bypass header to modify. Default retention applied on PutObject from bucket config
Legal hold: PutObjectLegalHold / GetObjectLegalHold. Binary ON/OFF flag per object version. Requires Object Lock enabled on bucket
Enforcement: locked objects cannot be hard-deleted (version-specific DELETE blocked). GOVERNANCE mode allows bypass with x-amz-bypass-governance-retention: true + s3:BypassGovernanceRetention permission. COMPLIANCE mode: no bypass until retention expires. Legal hold: blocks deletion when ON. Delete marker creation always allowed. Lifecycle worker respects locks
Response headers: x-amz-object-lock-mode, x-amz-object-lock-retain-until-date, x-amz-object-lock-legal-hold-status on GET/HEAD
Schema migration v12: retention_mode, retain_until_date, legal_hold_status columns on objects table
7 new S3 policy actions including s3:BypassGovernanceRetention
(Console) Object Lock card in bucket settings: enable with mode/days, status indicator. Versioning suspend disabled when locked

Depends on: Phase 17 (versioning — Object Lock operates on object versions)

Phase 22 — S3 API Completeness [P2]¶

Fill remaining gaps in the S3 API surface to maximize compatibility.

ListParts: GET /{bucket}/{key}?uploadId=X with pagination (max-parts, part-number-marker)
GetObjectAttributes: GET /{bucket}/{key}?attributes with x-amz-object-attributes header (ETag, Checksum, ObjectParts, StorageClass, ObjectSize)
Checksum algorithms: store and return client-provided x-amz-checksum-sha256, x-amz-checksum-crc32, x-amz-checksum-crc32c, x-amz-checksum-crc64nvme on PutObject/GetObject/HeadObject. Schema migration v13
Storage classes: storage_class field on ObjectRecord (default STANDARD), accept x-amz-storage-class header on PutObject, return in list and head responses
Chunked transfer with SigV4 payload signing (STREAMING-AWS4-HMAC-SHA256-PAYLOAD) — already implemented in body.rs
Resolves: TD-002 (storage class), TD-008 (content-type source)

Phase 23 — Performance and Hardening [P2]¶

Production-grade limits, caching, and graceful operations.

Request size limits: configurable max body size (default 5 GB for PutObject). [server.limits] TOML section with max_body_size, streaming LimitedByteStream wrapper, Content-Length fast-reject, EntityTooLarge S3 error
Rate limiting: per-credential and per-IP via governor crate (GCRA algorithm). SlowDown (503) S3 error with Retry-After header. Configurable rates and burst in [server.limits], disabled by default (rate = 0)
In-memory LRU cache for metadata lookups (bucket existence, HEAD) via moka crate. CachingMetadataStore wrapper with configurable size and TTL in [server.cache]. Write-through invalidation on create/delete/put
Graceful rolling upgrades: drain mode via tokio::sync::watch channel. Health endpoint returns 503 {"status":"draining"} during configurable drain window (drain_timeout_seconds). Load balancers stop routing traffic before connections are closed
Performance benchmarking suite: HEAD and DELETE benchmarks, JSON output (--json), baseline comparison (--baseline FILE), p50/p95/p99 latency reporting
Security hardening: request validation middleware (header count limit, null byte rejection, user metadata size limit). Configurable via max_header_count and max_metadata_size in [server.limits]

Depends on: Phase 13 (encryption pipeline)

Phase 24 — PostgreSQL Backend [P2]¶

Alternative metadata backend for deployments requiring a shared database.

PgStore implementing all 8 store traits (MetadataStore, CredentialStore, UserStore, TeamStore, GrantStore, AuditStore, MetricsStore, ServerConfigStore) via sqlx-core/sqlx-postgres
Config switch: [storage] metadata_backend = "sqlite" | "postgres" with [storage.postgres] connection settings
Docker Compose overlay (docker-compose.postgres.yml) + --postgres flag for bin/arca start and bin/test postgres
PostgreSQL schema migration runner with consolidated initial schema (equivalent to SQLite v1-v13)
(Console) Server info panel shows database backend type, /admin/info returns metadata_backend field Depends on: Phase 13 (encryption pipeline)

Phase 25 — Notifications and Event System [P3]¶

S3-compatible bucket notifications for event-driven architectures.

Benefits from: Phase 20 (lifecycle worker framework)

Phase 26 — Notification Connectors [P3]¶

Modular delivery connectors for the notification system. Phase 25 established a trait-based connector architecture with webhook as the first implementation. This phase adds 13 additional connectors across four categories.

Queue connectors: Kafka (rdkafka), AMQP/RabbitMQ (lapin), Redis Pub/Sub (redis), NATS (async-nats), MQTT (rumqttc)

Database connectors: PostgreSQL (sqlx-postgres, zero new deps), MySQL/MariaDB (sqlx-mysql), MongoDB (mongodb), Elasticsearch (reqwest, zero new deps)

Protocol connectors: gRPC (tonic), SMTP (email notifications), Syslog RFC 5424 (syslog)

Testing architecture: Each connector's tests run against a real backend in a temporary Docker container, started one at a time (not all at once). bin/test connectors <type> and bin/test connectors all subcommands. Connector tests are a separate test stream, not included in bin/test or bin/test all.

Depends on: Phase 25 (notification system and connector trait)

Phase 27 — Transparent Compression [P2]¶

Server-side transparent object compression with pluggable algorithms, per-bucket configuration, and offline migration tools. CompressingBlobStore follows the EncryptingBlobStore wrapper pattern: compress, then encrypt, then store. ETag computed on original data, Content-Length reports original size. Mixed-mode coexistence allows compressed and uncompressed objects to coexist transparently.

Depends on: Phase 13 (encryption pipeline for correct stacking order)

Phase 28 — Replication [P3]¶

Asynchronous cross-instance replication for disaster recovery and geographic distribution.

Depends on: Phase 17 (versioning), Phase 24 (PostgreSQL for production)

Phase 29 — Multi-Node High Availability [P3]¶

High availability beyond single-node: a symmetric, self-configuring, fully-replicated cluster. See the High Availability guide.

Multi-node clustering: symmetric nodes, service discovery (mDNS / static / DNS), real-time replication of the full data and control plane
Consistency modes: quorum (CP, majority to write) and available (AP); last-writer-wins with a deterministic blob_id tiebreak
Self-healing: anti-entropy reconciliation (objects + control plane), tombstones (no delete resurrection), proactive blob repair, composite-aware blob GC
Cluster-aware storage capacity (smallest node bounds the cluster) + 507 InsufficientStorage write guard; config-drift detection
Console cluster topology, GET /admin/cluster, GET /admin/health?verbose=1, bin/cluster, and production deploy manifests (Kubernetes StatefulSet + HAProxy/keepalived)

Deferred to future phases (the cluster ships with full replication, not sharding): erasure coding (data + parity shards, e.g. EC:4+2), consistent-hashing / sharding for capacity beyond full replication, S3 Batch Operations API, and SelectObjectContent (SQL queries on CSV/JSON).

Depends on: All prior phases. Major architecture evolution.

Phase 29.1 — HA Hardening [P1]¶

Remediation of all findings from an in-depth post-release review of the Phase 29 HA cluster (design and implementation, v0.24.0 → v0.25.1). The full working plan — fixed design decisions, per-milestone detail, and a finding-by-finding traceability table — is published as a living document: see the HA Hardening plan. The review it stems from is in the repository: arca-phase-29-ha-review.md. The whole phase ships as a single release when all nine milestones are done (plan decision H11).

R1 — P0 correctness: true write quorum (ACK counting), commit-ordered PostgreSQL manifest cursor, tombstone-first control merge
R2 — Cluster test infrastructure: real network partitions, available-mode suite, control-plane catch-up test, flakiness fixes
R3 — Membership and quorum integrity: peer authentication (H12: a challenge-response on the cluster secret — signed ping, fresh nonce, HMAC(secret, nonce) — ships for every cluster, plain-HTTP included, as the baseline layer; the R4 mutual TLS adds the independent second factor; fan-out, anti-entropy pulls, quorum and capacity minimum all count only authenticated + config-aligned peers), config drift excluded from quorum, public health minimised to {status, node_id}, fail-closed cluster-size write gate (H6), tombstone-GC liveness guard, membership pruning, 2-strike failure detector
R4 — Inter-node transport security: verified mutual TLS with a shared cluster CA (H12 confirmed: CA provided via config — [cluster.tls], required over HTTPS, fail-closed; arca tls generate-cluster tooling shipped; client certificates enforced on /cluster/v1/*; resolves TD-015), ±15 min anti-replay window, 2 MiB body cap on the JSON cluster endpoints, secret-strength enforcement (16-char floor, placeholders refused), dual-secret rotation (secret_previous)
R5 — Reconcile completeness: every control-plane family now in the snapshot merge — grant attachments, memberships, bucket config keys, bucket tag sets, server settings (resolves TD-016) — plus multipart uploads/parts with close tombstones and concat fetch-from-peer, Object-Lock changes visible to anti-entropy, identical-row no-op ending the steady-state manifest churn, SSE-C cluster read-repair (§3.6 spike: fixed in scope, no new debt), export/import node-identity guard
R6 — Cluster-aware workers: the lifecycle evaluator is leader-gated (lowest node_id among the eligible nodes — the quorum's own predicate, so a rogue cannot steal the role; automatic failover, worker_leader exposed on /admin/cluster); the per-node workers (metrics, retention, notification delivery) and the external-replication worker stay un-gated by design — the replication journal is node-local and deliveries were already exactly-once, the review's duplicate-delivery premise did not hold
R7 — Synchronization state and operability: syncing readiness gate (a re-entering node answers 503 on the LB health check until its first anti-entropy pass toward every eligible peer completes — no stale 404s/partial listings served from rotation), per-peer sync state on /admin/cluster (HWM cursor, lag, first_pass_done, skipped entries), backup-restore rewind detection (automatic HWM reset off the ping's seq cursor), stuck-entry skip after 5 failed passes, budget-bounded resumable blob repair (blob_repair_budget), Retry-After pinned on every retriable 503, console syncing/lag indicators — plus finding N2: an equal-timestamp LWW tie let a restarted node's stale manifest re-pull silently erase Object-Lock state cluster-wide; lock changes now carry their own lock_updated_at LWW dimension on both backends
R8 — Console: node selector for the node-local views behind the load balancer (audit, metrics history, notification events, replication journal) via a server-side ?node= proxy over the signed cluster transport (eligible peers only — every H12 gate applies; clear 404/503/502 errors), including an "All nodes" merged view (parallel fan-out, rows merged newest-first and labeled with their source node, per-source result report; pagination is per-source-page by design) and per-node chart series in Monitoring; every response is labeled with the node that answered, so the default LB view always says what it shows (decision H9)
R9 — Documentation, deploy and closure: write-aware health check (?writable=1 answers 503 read_only while the write gate is closed — a separate LB write pool drops read-only nodes, commented reference backends in both HAProxy configs), §5.4 integration tests (proactive blob repair observed on the node volume without a masking GET; tombstone-GC liveness guard blocking, no-resurrection at re-entry, release — on a tombstone_grace_seconds test overlay), honest consistency docs (available-mode silent LWW losers + RPO window, the real read-staleness window under partitions, the 503-behind-LB cost and mitigations, the WORM-in-cluster trust model), six operational runbooks (dead-node replacement, restore from backup, cluster resize, secret rotation, coherent backups, non-empty merge), deploy alignment (HAProxy fall/rise motivated per environment + sticky example; k8s readiness 5s/2 and liveness moved off the syncing path)

It also closes a security workstream (review §3.7): the cluster authenticates the sender of every replication request but never the receiver of a fan-out, so a rogue peer discovered over mDNS can receive all newly written data without holding the shared secret. R3 (peer authentication, public-health minimisation) and R4 (verified inter-node TLS, secret-strength enforcement, anti-replay) close it; on an untrusted cluster network this carries P0 urgency.

Depends on: Phase 29. Critical for production [cluster] deployments: it closes the gap between the consistency guarantees the cluster declares and those it enforces, and the inter-node trust gaps found in review §3.7.

Phase 30 — Migration and Maintenance [P3]¶

In-place migration and maintenance tooling — change the metadata backend, the encryption state of existing objects, or the cluster topology without migrating to a new instance — built on a new console-driven maintenance-jobs subsystem. See the Migration & Maintenance guide.

Maintenance-jobs subsystem (M1): long-running operator operations tracked as maintenance_jobs rows (progress persisted across restarts), one job at a time, live vs maintenance mode (the latter drains the S3 API on the node via the health check so the load balancer stops routing, while the admin API and worker stay live), pause / resume / cancel with state-guarded transitions, a single background worker that is leader-gated in a cluster (only the worker-leader node runs it). New JSON admin API under /admin/maintenance/jobs and a console Maintenance page.
Hot copy-on-write re-encryption (M2) — arca encrypt-existing / arca decrypt-existing: encrypt existing plaintext objects to SSE-S3, or decrypt them back, in place without re-uploading. Runs as an encrypt / decrypt maintenance job in live mode (copy-on-write + compare-and-swap + optional byte/sec throttle, zero downtime) or maintenance mode (drained, full speed); ETag and Last-Modified preserved. Also available as an offline CLI escape hatch for disaster recovery. SSE-C and multipart/composite objects are skipped (TD-014).
arca migrate-db --to <sqlite|postgres> (M3): metadata migration between SQLite and PostgreSQL backends in place. Copies every metadata table via a generic, type-aware, FK-safe column copier with per-table row-count reconciliation; blob files are untouched. Available as an offline CLI escape hatch and as a maintenance-mode migrate-db job. After a run, switch metadata_backend and restart.
arca migrate-topology --to-cluster | --to-single (M4): guided in-place transition between standalone and HA cluster. The cluster is fully replicated (not sharded), so no data redistribution: --to-cluster emits a ready-to-paste [cluster] stanza and seeds the object_seq counter; --to-single purges cluster-only tombstones and VACUUMs SQLite. CLI-only by design (both directions are operator+restart actions). Full 3-node bringup runbook lives in the HA guide.

Descoped (Pietro's decision — these duplicate mc / aws s3 and add a client-mode surface Arca does not need to own):

~~Remote S3 client mode: arca s3 ls/cp/mv/rm/sync/presign~~ — use mc or aws s3 against the endpoint instead.
~~Profile management: arca profile add/list/remove/use (~/.arca/profiles.toml)~~ — client-mode only; descoped with the S3 client.
~~Interactive shell mode: arca shell~~ — client-mode only; descoped with the S3 client.

Depends on: Phase 13 (encryption pipeline for encrypt/decrypt), Phase 24 (PostgreSQL for migrate-db), Phase 29 (multi-node for topology migration)

Phase 31 — OpenTelemetry Integration [P3]¶

Export traces, metrics, and logs via the OpenTelemetry Protocol (OTLP) for integration with observability platforms (Grafana, Datadog, Jaeger, etc.).

Distributed tracing: instrument request handling with tracing + opentelemetry-otlp. Each S3/admin request produces a trace span with operation, bucket, key, status, latency. Spans propagate traceparent (W3C Trace Context) for end-to-end visibility
OTLP metrics export: export the existing Prometheus counters, histograms, and gauges via OTLP gRPC/HTTP alongside the /admin/metrics scrape endpoint
OTLP log export: forward structured log entries (and optionally audit records) via OTLP for centralized log aggregation
Config: [monitoring.otlp] section with endpoint, protocol (grpc | http), per-signal enable flags (traces, metrics, logs), service_name, optional headers/auth
Docker Compose overlay with OpenTelemetry Collector + Jaeger for local development and testing
(Console) Trace ID in audit log entries, linkable to external trace viewer

Depends on: Phase 18 (monitoring and audit infrastructure)

Admin & Server Enhancements¶

Independent of numbered phases — general improvements to the admin API and server capabilities.

Configuration export/import API: GET /admin/export and POST /admin/import for full instance config (settings, users, teams, grants, credentials, buckets, bucket configs). Supports section filtering, secret masking, skip/overwrite/dry_run modes (P2)
Per-bucket quota: hard/soft caps on bucket size (bytes) and/or object count, enforced at PutObject/CompleteMultipartUpload with QuotaExceeded-style S3 errors. Stored in bucket_config, live-updatable via the admin API (P2)

Console Enhancements¶

Independent of server phases — can ship at any time.

MVP Implementation Plan¶

Each phase built on the previous one and ended with verification: unit tests, boto3 integration tests, and manual aws s3 CLI checks — all inside Docker containers.

Progress Overview¶

0

1

2

3

4

5

6

7

8

9

10

11

Phase	Name	Status
0	Project Skeleton	✔
1	Configuration & Storage Foundation	✔
2	Bucket Operations	✔
3	Core Object Operations	✔
4	CopyObject + ListObjectsV2	✔
5	Multipart Upload	✔
6	AWS SigV4 Authentication	✔
7	Disaster Recovery + Polish	✔
8	Admin API	✔
9	Web Console	✔
10	S3 Compatibility Hardening	✔
11	Documentation	✔

Phase 0 — Project Skeleton¶

Set up the Cargo workspace with all 5 crates and the basic infrastructure.

Initialize Cargo workspace: arca-core, arca-auth, arca-proto, arca-storage, arca-server
Scaffold Axum router with stub handlers returning S3Error::NotImplemented
Wire XmlErrorResponse so all requests return valid S3 XML errors
Dockerfile (multi-stage: rust:1.85-bookworm builder + debian:bookworm-slim runtime), docker-compose.yml
LICENSE, CLAUDE.md, config/default.toml

Verify: docker compose up starts server; aws s3 ls --endpoint-url http://localhost:9000 returns valid S3 XML error (NotImplemented).

Phase 1 — Configuration & Storage Foundation¶

Establish the configuration model and SQLite database foundation. MinIO-like approach: static server settings in a TOML config file, dynamic data (credentials, users) in the database.

Config file at /etc/arca/config.toml (default), config/default.toml as repo template, --config-path override
SQLite database initialization in arca-storage (tokio-rusqlite, WAL mode)
Schema: credentials table (access_key_id, secret_access_key, created_at, active)
arca credential add/list/remove CLI subcommands (direct SQLite access)
Auto-generate root credential on first startup if none exist, print to stdout
Unit tests for config loading, credential CRUD, auto-generation
Integration test: server starts, prints generated credentials

Verify: bin/arca start -d --build starts server and prints auto-generated credentials; arca credential list shows the generated credential.

Phase 2 — Bucket Operations¶

Implement the 4 bucket operations with SQLite metadata storage.

SQLite schema (buckets table), SqliteMetadataStore bucket methods
Bucket name validation (S3 naming rules) in arca-core, handlers call MetadataStore directly
Axum handlers in arca-proto/handlers/bucket.rs with AppState injection
XML response types: ListAllMyBucketsResult
Unit tests for validation (~22 tests), SqliteMetadataStore (~7 tests), XML types (~3 tests)
Integration test: test_buckets.py with boto3 (~10 tests)

Verify: aws s3 mb, aws s3 ls, aws s3 rb all work.

Phase 3 — Core Object Operations¶

Implement PutObject, GetObject, HeadObject, and DeleteObject with streaming I/O.

FsBlobStore: UUID path generation, configurable prefix depth (1–4, default 2), streaming write with concurrent MD5 computation, atomic rename (temp file + rename), sidecar .meta JSON write
FsBlobStore.get(): streaming read via tokio::fs::File, byte range support (seek + take)
SQLite objects table, SqliteMetadataStore object methods (put returns old record for cleanup), bucket_is_empty check
Axum handlers with streaming request/response bodies (handlers call BlobStore + MetadataStore directly)
DeleteBucket now checks bucket is empty before deleting (returns BucketNotEmpty 409)
Unit tests for FsBlobStore (~12 tests), SqliteMetadataStore objects (~7 tests)
Integration test: test_objects.py (~12 tests)

Verify: aws s3 cp localfile s3://bucket/key and back, range requests, ETags, Content-Type.

Phase 4 — CopyObject + ListObjectsV2¶

Add object copying and listing with prefix/delimiter/pagination support.

CopyObject: PUT /{bucket}/{*key} with x-amz-copy-source header, stream-through copy via BlobStore get → put
SqliteMetadataStore::list_objects: SQL with key > ? for pagination, key LIKE ? for prefix, delimiter handling for common prefixes
Continuation token: base64-encoded last key (opaque to client)
XML response: ListBucketResult with Contents, CommonPrefixes, IsTruncated, NextContinuationToken
Integration tests: test_list.py (~12 tests), TestCopyObject (~6 tests), unit tests for XML builders and list_objects

Verify: aws s3 ls s3://bucket/prefix/, aws s3 cp s3://bucket/a s3://bucket/b.

Phase 5 — Multipart Upload¶

Full multipart upload lifecycle with composite ETag computation.

SQLite tables: multipart_uploads, parts
MetadataStore multipart methods: create/get/delete upload, put/list parts
Parts stored as normal blobs via BlobStore::put(), stream-concatenated at complete time
Composite ETag: hex(MD5(binary_MD5(part1) || binary_MD5(part2) || ...))-{count}
XML: InitiateMultipartUploadResult, CompleteMultipartUploadResult, parse CompleteMultipartUpload request body
Query-param dispatch in object/multipart handlers
Part size validation at complete time (non-last parts >= 5 MB)
Integration test: test_multipart.py (~12 tests)

Verify: aws s3 cp largefile s3://bucket/key (triggers multipart in aws-cli), ETag format is "<hex>-<count>".

Phase 6 — AWS SigV4 Authentication¶

Full AWS Signature V4 implementation with auth middleware. Credentials loaded from the SQLite database (managed via arca credential CLI from Phase 1).

arca-auth: Full SigV4 implementation (canonical request, string-to-sign, signing key derivation, signature verification)
Test against AWS SigV4 test vectors (downloadable test suite)
Constant-time signature comparison via subtle
Auth middleware in arca-proto: parse Authorization header, look up credential in DB, verify signature
Virtual-hosted-style middleware: rewrite bucket.s3.domain/key -> /bucket/key
Trailing-slash normalization with original URI preservation (mc compatibility)
Environment variable credential override (ARCA_ROOT_ACCESS_KEY/ARCA_ROOT_SECRET_KEY) for testing
Integration tests: valid credentials succeed, bad credentials get SignatureDoesNotMatch / InvalidAccessKeyId

Verify: Set up credentials via arca credential add, all operations require auth, unauthenticated requests rejected.

Phase 7 — Disaster Recovery + Polish¶

Recovery tools, operational logging, and graceful shutdown.

arca recover CLI: walk data/ tree, read all .meta sidecar files, rebuild SQLite DB from scratch. Supports --dry-run for preview and --skip-verify to skip MD5 checksum verification. Preserves credentials across DB rebuild. Skips orphaned/malformed/corrupt sidecars with warnings.
arca fsck CLI: compare DB records against filesystem, report orphaned blobs / missing blobs / sidecar mismatches / orphaned sidecars / stale temp files. Optional --verify-checksums for MD5 verification of every blob. Exit code 0 = clean, 1 = issues found.
Structured logging with tracing (--log-format text|json on serve command)
Graceful shutdown (finish in-flight requests on SIGTERM/SIGINT) Verify: Delete SQLite DB, run arca recover, verify all data accessible again.

Phase 8 — Admin API¶

JSON-based administration API for the web console and other management tools. Endpoints live under /admin/* on the same port (9000), using SigV4 auth.

Health, info, stats endpoints (GET /admin/health, /admin/info, /admin/stats)
Credential CRUD (GET/POST /admin/credentials, DELETE /admin/credentials/{access_key_id})
Unit + integration tests

Verify: Admin API responds to health/info/stats requests; credential CRUD works via API.

Phase 9 — Web Console¶

Web-based administration console and bucket browser, deployed as a separate application (console/ directory) that communicates with Arca exclusively via S3 API and Admin API.

Web console app: single-file Alpine.js + Tailwind CSS, "The Vault" dark theme, nginx:alpine Docker image (superseded: the console was later split into ES modules under console/js/, and the image rebased on plain alpine + the nginx package — see the console guide)
Admin dashboard: bento-grid layout with server info, storage stats, SVG donut chart, health indicator, auto-refresh
Bucket browser: list/create/delete buckets, prefix navigation with breadcrumbs, upload/download/delete objects, detail panel, treemap visualization
Credential management: card grid, create with reveal-once secret, delete with confirmation, lockout warning
Browser SigV4 signing via Web Crypto API, CORS middleware in Arca
Admin flag on credentials: admin boolean field, migration v5, CLI --admin flag (superseded: the flag was removed in migration v26 — privileges come from the owning user, root implicitly and everyone else through grants)
Admin-only access to Admin API endpoints (non-admin gets 403)
Role-aware console: admin sees dashboard + credentials + buckets; non-admin sees buckets only
Lockout prevention: cannot delete the last active credential, nor the last active credential belonging to a root user

Verify: Web console connects to Arca and provides dashboard and bucket browsing.

Phase 10 — S3 Compatibility Hardening¶

Run industry-standard compatibility tests and harden edge cases.

Set up Ceph s3-tests in Docker (docker/s3-tests/Dockerfile + s3tests.conf, bin/s3-tests runner)
Run test suite, triage failures — 232 pass / 506 fail / 91 skip (see s3-tests/TRIAGE.md)
Fix compatibility issues: x-amz-request-id/x-amz-id-2/Server headers, GetBucketLocation, CreateBucket idempotency, ListObjectsV1, empty delimiter handling, whitespace-preserving DeleteObjects XML parser, unimplemented PUT bucket ops return 501
Track pass/fail list (s3-tests/passlist.txt), HTML compatibility dashboard (s3-tests/report.html)
Performance testing with concurrent requests + large files (tests/perf/perf_test.py, bin/perf-test)

Verify: Ceph s3-tests running in CI, pass/fail list tracked, no regressions.

Phase 11 — Documentation¶

Comprehensive manuals and guides for users, administrators, and operators.

Restructured documentation into grouped sections (Getting Started, User Guide, Reference, Operations)
Installation guide and Quick Start
CLI reference (extracted from configuration page)
Web Console guide with automated screenshots
Disaster Recovery guide (extracted from configuration + architecture)
Monitoring & Logging guide
Production Deployment guide
Automated screenshot tool (bin/screenshots, Playwright + Docker)

Verify: All manuals published on GitHub Pages, covering installation through production operations.

Post-MVP — Technical Debt¶

The MVP includes several workarounds and hardcoded values that pass compatibility tests but need proper implementation for production use. These are tracked in TECH_DEBT.md with unique IDs (TD-XXX) referenced in source code comments.

Remaining items:

~~Ownership model (TD-001)~~: Resolved — Owner ID derived from credential's user; buckets/objects track creator's username
Storage classes (TD-002): Always STANDARD — no storage tiering
~~Versioning (TD-003)~~: Resolved — Full object versioning implemented in Phase 17
~~Region support (TD-004)~~: Resolved — Configurable region in [server] TOML or Admin API, per-bucket via bucket_config. Phase 18
Audit write contention (TD-005): Per-request SQLite inserts may bottleneck under heavy load — batch via mpsc channel if needed
~~Request ID consistency (TD-005)~~: Resolved — error XML and response header now match
~~Encryption (TD-006)~~: Resolved — SSE-S3 with AES-256-GCM implemented in Phase 13
Unimplemented ops (TD-007): ~33 bucket operations return 501
Multipart Content-Type (TD-008): Captured at init time — verify against AWS semantics
SSE-C multipart (TD-010): SSE-C headers rejected on multipart uploads — needs per-part encryption tracking
Composite blobs in recover / fsck (TD-014): the multipart Complete optimisation produces composite sidecars with no on-disk blob file. arca recover aborts on them as orphans, arca fsck reports false-positive orphaned_sidecars. Runtime S3 reads/writes are unaffected — only the recovery and integrity-check tools need teaching how to walk composites.
time pinned to =0.3.47 (TD-017): time 0.3.48 introduces From impls that clash (E0119 coherence) with rcgen's blanket conversions when time's parsing/formatting features are enabled in the graph. No security exposure (0.3.47 already contains the CVE-2026-25727 fix — the old TD-011, resolved by the 2026-06 dependency refresh together with TD-012, the rustls-pemfile retirement). Unpin when rcgen or time fixes the conflict upstream.
Re-encryption skips multipart/composite (TD-018): the encrypt/decrypt jobs and CLI skip multipart/composite objects (detected by their <hex>-<n> ETag) and SSE-C objects, so a store with multipart objects is only partially re-encrypted. Related to TD-014 (composite blobs have no single on-disk file to rewrite copy-on-write). Phase 30
migrate-db online direction limited to postgres→sqlite (TD-019): the online maintenance-job direction can only run postgres→sqlite (the config auto-detects PostgreSQL as the running backend); sqlite→postgres is done with the offline CLI. Phase 30
Deferred integration tests (TD-020): cold-CLI re-encryption and the 3-node cluster behaviours of re-encryption / migrate-topology round-trips are covered by unit tests and the online single-node suite only — no dedicated cold-CLI or cluster integration phase yet. Phase 30
Cluster re-encryption defers old-blob reclaim to GC (TD-021): after a copy-on-write re-encryption swap the old blob is deleted eagerly on a single node but left for the grace-bounded anti-entropy GC in a cluster, so reclaim never races the blob-repair pulls peers issue while converging. Correctness is unaffected — content and lock state converge on independent LWW dimensions (content_updated_at vs lock_updated_at, migrations sqlite v25 / pg 0013), so a concurrent lock op can no longer revert blob_id. Residual: a transient extra copy of each re-encrypted object's old blob per node until the next GC pass. Phase 30
migrate-db job not cancellable mid-copy (TD-022): the online migrate-db job does not poll the pause/cancel flag between tables, so a cancel only takes effect once the whole copy finishes; on error the destination is left partial and must be dropped before retry. Phase 30
Offline migrate-db has no crash-safe checkpoint (TD-023): the offline CLI copies all tables in a single process with no resume marker; a crash mid-run leaves the destination partial and the operator must drop it before retrying. migrate-topology is unaffected (its operations are idempotent/in-place). Phase 30
rustls-webpki pinned to a vulnerable version transitively (TD-024): rumqttc 0.25.1 (MQTT connector) hard-pins rustls-webpki = "^0.102.8", which carries 4 published advisories (1 high, 1 medium, 2 low). The vulnerable code is unreachable from Arca: rumqttc uses the crate only for one error variant and never configures a CRL, while MQTTS certificate validation runs through rustls 0.23.40 on the already-patched rustls-webpki 0.103.13. No upstream fix exists (0.25.1 is the latest release, rumqtt master still pins 0.102.8, and a [patch.crates-io] override cannot satisfy ^0.102.8). Dependabot alerts dismissed as not_used; track upstream rumqttc releases or evaluate its native-tls backend.
Cluster conditional writes (CAS) are exact per node, not cluster-wide (TD-025): If-Match / If-None-Match / x-amz-if-match-* preconditions are evaluated authoritatively only against the serving node's own state — put_object/delete_object commit locally, then fan out, with no per-key authoritative node to arbitrate across peers. Two conditional writes racing the same base ETag on two different nodes can both pass their own local check and both succeed, converging afterward by last-writer-wins. Single-node CAS (SQLite and PostgreSQL) is unaffected and fully atomic. Mitigated operationally with sticky load-balancer routing (see documentation/docs/guide/ha.md). Fix path: owner-node forwarding for conditional writes.
bin/test tls cannot create certs/ on Linux (TD-026): the self-signed flow creates the bind-mount directory on the host (mkdir -p certs) and then writes into it from the tls-init container, which runs as 65532:65532; on a real Linux host the directory belongs to the invoking user (0755) and arca tls generate fails with EACCES. Invisible on macOS, where Docker Desktop fakes bind-mount ownership. Fix by generating into a named volume (as bin/test tls-permissions does) or by running tls-init as the invoking uid/gid
~~Cluster inter-node TLS (TD-015)~~: Resolved — verified mutual TLS with an operator-distributed cluster CA ([cluster.tls], required when the cluster runs over HTTPS), danger_accept_invalid_certs removed, client certificates enforced on /cluster/v1/*, material minted by arca tls generate-cluster. HA hardening R4 (Phase 29.1)
~~Partial cluster control-plane reconcile (TD-016)~~: Resolved — the anti-entropy snapshot merge now covers every control-plane family (grant attachments, memberships, bucket config keys, bucket tag sets, cluster-wide server settings, plus multipart uploads and parts) with per-row LWW timestamps, deletion tombstones and parent-dead filtering, so a returning node fully self-heals. HA hardening R5 (Phase 29.1)