Changelog

All notable changes to this project are documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

0.34.1 — 2026-10-30

v0.34.0 was tagged but never published: the scans of its release run found the advisories under Security, so no image, binary or GitHub release exists for it. 0.34.1 is the first release with the changes below.

Added

• A benchmark suite that compares Arca with MinIO: bin/perf-test suite. It runs the same warp scenarios against both servers on the same host and storage: small PUT, GET and HEAD at several concurrencies, 64 MiB PUT and GET, ranged GET, multipart upload, a mixed workload, DeleteObjects, ListObjectsV2 and a versioned workload. Each server runs in three configurations: plain, SSE-S3 and compression. The suite also times ListObjectsV2 walks over a seeded key tree, with and without a delimiter, on both servers, and runs the SDK-side boto3 suite. For Arca it adds the offline maintenance commands (fsck, fsck --verify-checksums, gc, compress-existing, recover) on a seeded store, with each command's peak memory.
• Isolation. Every server starts fresh, in a compose project of its own with no fixed host port, so the suite never touches a running bin/arca start.
• Output. Everything goes into one run directory under perf-results/, which is git-ignored. It holds the raw warp benchdata and logs, the JSON results, the server metrics and logs, a summary.json, and report.html, which links every raw file.
• Options. --profile quick|full, --targets, --variants, --scenarios, --storage volume|tmpfs and --release <tag> (benchmark a published image).
• Comparison with a baseline. --baseline <run-dir> compares the report with an earlier run, and bin/perf-test report <run-dir> [--baseline <run-dir>] regenerates the report of a finished run in a plain Python container, with no server and no compose project. For each scenario and server of the run, the report adds the baseline's figure as a third bar between Arca and MinIO, with the change since then. When the run did not measure MinIO, as in a targeted run of Arca only, it takes MinIO's figures from the baseline. The listing, maintenance and boto3 tables gain the baseline's figures and the change too. The report warns when the two runs were not made the same way (host, Docker resources, profile, duration, storage, MinIO image, dataset sizes).
• Documentation and tests. See the new Performance Testing page under Operations. bin/test perf-report tests the report generator and the runner; bin/test scripts tests the orchestration.
• Licensing. warp and MinIO run as separate test-only containers: neither is a dependency of Arca. The default MinIO image is Chainguard's build of upstream MinIO, because MinIO no longer publishes server images on Docker Hub.
• bin/perf-test reports the aggregate request rate. Its ops_per_sec divides the operations by the sum of their latencies, which is the rate of a single client, not the rate the server sustained under concurrency. A new wall_ops_per_sec divides the operations by the elapsed time of the run. ops_per_sec keeps its old meaning, so old baselines still compare. perf_test.py --json-out <path> writes the JSON results to the given path.

Changed

• The console dashboard recognises TLS terminated by a reverse proxy. Behind a proxy that holds the certificates, Arca listens on plain HTTP and the Transport field showed "HTTP", although the browser reached Arca over HTTPS. When the console's Arca endpoint is https:// and Arca has no TLS of its own, the field now shows "TLS · proxy" in green, with a tooltip saying the last hop, from the proxy to Arca, is plain HTTP.
• A GET of a whole SSE-S3 object is several times faster. The decrypting read pulled the file in 4 KiB reads, each a round trip to tokio's blocking pool (16 per 64 KiB chunk), where a plain GET reads 64 KiB at a time. It now reads one whole on-disk chunk at a time and decrypts each chunk in place, without copying it twice. GET of a 64 MiB SSE-S3 object, interleaved A/B on the suite's reference host: 12.0 to 67.5 objects/s with 4 clients, 14.9 to 159.5 with 16 (MinIO in the reference suite run: 45 and 117). Ranged GETs, which already read their range in one go, are unchanged.
• A GET of a whole compressed object costs half as much CPU. It opened the blob twice, once for the footer's chunk count and once for the chunks, which doubled the file operations, each a round trip to tokio's blocking pool; and zstd created and freed a decompression context for every chunk, which cost more than decoding a small chunk. A whole-object GET now reads the blob once: the object size in the sidecar says where the chunks end, and the footer is checked against the chunks as the read reaches it, so a truncated or corrupted blob is still an error. zstd keeps one decompression context per thread. GET of a 1 KiB object in a zstd bucket, interleaved A/B on the suite's reference host: 19,700 to 29,800 objects/s with 16 clients, 20,400 to 43,300 with 64, the same as an uncompressed bucket (30,200 and 44,800). GET of a 64 MiB compressed object is unchanged, and so are ranged GETs.
• A compressed PUT spends less CPU on zstd. Every chunk created a zstd compression context, sized it for a stream of unknown length (most of the cost: zeroing tables far larger than a small chunk) and freed it, about 18% of the server's CPU on 1 KiB PUTs where the compression itself took under 1%. zstd now keeps one compression context per thread and encodes each chunk with its size known up front; the frames are standard zstd frames, and blobs written before read as they did. PUT 1 KiB to a zstd bucket with [storage] fsync = false, interleaved A/B on the suite's reference host: 5,383 to 5,890 objects/s with 16 clients, 5,200 to 5,506 with 64 (uncompressed: 6,372 and 5,676). With the default fsync = true the disk's fsync is the limit and the rate is unchanged; so is PUT of a 64 MiB object.
• Small requests are several times faster. On the suite's reference host (14 CPUs, Docker volume storage), GET of a 1 KiB object with 16 clients went from 2,660 to 31,794 objects/s (MinIO: 33,350) and HEAD from 3,214 to 44,234 (MinIO: 41,915). The time went into per-request overhead, not into storage:
• The shared server state is one pointer. Every middleware and handler cloned an 840-byte struct of about 30 reference-counted fields on every request.
• mimalloc is the global allocator. The static musl binary used musl's allocator, which serialises every allocation on one global lock. mimalloc (MIT) keeps per-thread heaps; THIRD-PARTY-NOTICES attributes the crate and its vendored C sources.
• SQLite reuses its prepared statements. Every call compiled its SQL again; each connection now caches up to 256 statements, and listing page sizes are bound parameters, so one statement serves every page size.
• Identity lookups are cached for 5 seconds. Every authenticated request read the credential, its user and the user's policies from the database, and the audit log read the credential again. Any change to credentials, users, teams or grants empties the cache at once, on every cluster node it replicates to, so a revoked credential is refused on the next request. Only a change written by another process, such as the offline arca credential and arca user commands run against a live server, waits for the entries to expire.
• A GET reads the blob's sidecar once, instead of once per layer of the blob stack (compression, encryption, file store, cluster read repair), and once more per inner range read of a compressed object.
• A GET or HEAD counts the object's tags in the object query instead of a second query, for x-amz-tagging-count.
• Unversioned PUTs write less. An unversioned PUT replaces every row of its key, so the new row is already the latest one: the is_latest fix-up that rewrote it twice is skipped, on both backends. The fix-up also rewrote every version of the key on each versioned PUT; it now touches only the rows that change.
• PostgreSQL writes to different keys no longer wait for each other on a single node. Every object write took the same counter row as its first statement and held it until commit, so the whole database wrote one object at a time: 790 PUTs/s with 16 clients and the same with 64. That counter orders the changes a cluster node hands to its peers, which a single node does not need. A single node now locks only the key it writes (a transaction-level advisory lock, so conditional writes stay exact) and numbers its rows from a sequence (PostgreSQL migration 0018, applied at startup). PUT 1 KiB: 798 to 1,808 objects/s with 16 clients, 792 to 2,340 with 64; the commit's WAL flush is now what PUTs wait for. DeleteObjects applies its whole request in one transaction here too: 660 to 15,235 objects/s. Cluster mode is unchanged. When the store opens, both numbering schemes are raised above every row, so switching between single node and cluster stays safe. fsck, gc, encrypt-existing and decrypt-existing now open the database in the node's mode ([cluster] enabled), as the server and the other commands already did, so a command running next to a cluster node takes the same lock as the server. See decision H13 in the HA design decisions.
• A PUT with tags is one transaction. PutObject with x-amz-tagging, CopyObject (its tags resolved before the write, from the request under REPLACE or from the source under COPY) and CompleteMultipartUpload (the tags given at CreateMultipartUpload) wrote the object row, then its tags in a second transaction: a second commit, a second fsync. They are now written together, on both backends. PUT 1 KiB with tags, durable SQLite, 16 clients: 486 to 892 objects/s, as fast as without tags. The version is never visible without its tags, also to notifications and outbound replication. A failed tag write now fails the request with 500, where it was logged and answered 200 with the object left untagged. In a cluster the tags still replicate to peers as a separate best-effort message (TD-033).
• Concurrent writes on SQLite share one commit (group commit). The single SQLite writer committed every write on its own, and with [storage] fsync = true every commit is an fsync of the WAL, so small writes were capped at one per fsync of the database disk whatever the number of clients (TD-048). The request-path writes (PUT, CopyObject, DELETE, DeleteObjects, the multipart writes, object tags, replication journal entries and notification events) now queue while a commit is in flight and the queued ones commit together in one transaction, one fsync for the group. Each write runs under its own savepoint, so a refused precondition or an Object Lock refusal still fails that request alone, and each request is answered only once the transaction holding it is durable. There is no waiting for a group to fill: a lone write commits at once, as before. Durable SQLite, on the suite's reference host: PUT 1 KiB with 16 clients 924 to 1,509 objects/s, with 64 clients 888 to 2,293, PUT 64 KiB with 16 clients 805 to 1,139, DeleteObjects 4,522 to 8,750; a single client is unchanged. The gain grows with the disk's fsync latency.
• DeleteObjects applies the whole request in one transaction on SQLite, one commit (one fsync) instead of one per key, with a savepoint per key, so a key that fails (a precondition, an Object Lock) fails alone: 966 to 4,781 objects/s. A cluster still deletes key by key, because each key is replicated with its own quorum.

Fixed

• A SQLite write under heavy read load no longer fails with "database is locked". The single writer opened its transactions as DEFERRED, and most of them read before they write, so they asked for the WAL write lock as read transactions, a case where SQLite returns SQLITE_BUSY at once instead of waiting busy_timeout (5 s). A reader holds that lock for an instant when it reads a wal-index header the writer is updating, so under enough concurrent reads a write occasionally failed with 500 (seen as 7 failed PUTs out of 260,000 in a perf suite seed). The writer now opens its transactions as IMMEDIATE, which takes the lock at BEGIN, where busy_timeout applies; with a single writer this costs nothing.
• A copy of an object onto itself keeps its tags. CopyObject with the default COPY tagging directive read the source's tags after writing the destination; when both are the same unversioned object (a copy that changes the metadata), the write had already removed the old version's tags, so the copy came out untagged. The source's tags are now read before the write. A REPLACE copy with an invalid x-amz-tagging is now refused with InvalidTag before anything is written, as PutObject does; it used to succeed and leave the copy untagged.
• Prefix filters on SQLite match case exactly. ListObjects, ListObjectVersions, ListMultipartUploads and the lifecycle rules selected keys with SQL LIKE 'prefix%', which SQLite compares ignoring ASCII case: a listing with prefix photos/ also returned Photos/ and PHOTOS/ keys, and a lifecycle rule for logs/ also expired Logs/ objects. The filter is now a key range (key >= prefix AND key < next), which matches bytes as S3 does and is an index seek instead of a scan. PostgreSQL's LIKE was already case-sensitive.
• PostgreSQL lists in S3 byte order whatever the database locale. On a database initialised with a glibc locale (en_US.UTF-8, the default of the Debian-based postgres image and of most distribution packages) keys, version ids, upload ids and bucket names sorted by that locale, which ignores case and punctuation at first: a_c came before A/x and B. Listings were out of S3 order, and ListObjects with a delimiter, which compares keys in Arca's own code, could skip or repeat entries across pages. PostgreSQL migration 0017 (applied at startup) sets COLLATE "C" on those columns. On such a database the listing order changes to the S3 one. The migration rebuilds the indexes of the objects, multipart_uploads and buckets tables, so the first start takes longer on a large database. bin/test postgres now runs on the Debian image, where this kind of bug shows up; the Alpine image collates by bytes and hid it.
• Lifecycle rules delete only what S3 deletes, and all of it.
• Every match is processed. Each rule handled the first 100 matches per cycle (one cycle per hour by default) with no cursor: a bucket with many expired objects took weeks to catch up, and 100 locked noncurrent versions at the start of the bucket stopped the rule for good. The evaluator now pages through every match each cycle.
• AbortIncompleteMultipartUpload honours the rule's prefix. It aborted the stale uploads of the whole bucket, also those outside the rule's scope.
• NoncurrentDays counts from when the version became noncurrent, the time the next newer version or delete marker was written, not from the version's own creation: a version created a year ago and replaced yesterday was deleted at once, where S3 keeps it for NoncurrentDays more.
• NewerNoncurrentVersions is enforced. It was stored and returned but ignored, so the newest noncurrent versions the rule keeps were deleted too.
• The tag filter applies to noncurrent versions. NoncurrentVersionExpiration ignored it and expired the versions without the tag as well.

Each of these deleted data that S3 keeps; after the fix a rule may delete less than before, never more. Two gaps remain, both on the side of deleting less: expired object delete markers and noncurrent delete markers are never removed (TD-056). - ListObjectVersions honours delimiter. It was read and ignored, so a delimited listing returned every version of every key and no CommonPrefixes. The versions of the keys that have the delimiter after the prefix now roll up into one common prefix, which counts once towards max-keys, the response echoes Delimiter, and a page that ends on a common prefix returns it as NextKeyMarker with no NextVersionIdMarker, as S3 does. - The per-IP rate limit tells clients apart. The server never passed the TCP peer address to the request handlers, so every client without an X-Forwarded-For header shared one limiter bucket: with rate_limit_per_ip_per_second set, the whole server was limited as if it had a single client. The audit log also recorded no source address for direct clients; it now records the peer address. X-Forwarded-For is still trusted from any client, so a client that reaches Arca directly can evade the per-IP limit (TD-055). - A delimited listing reads only what its page returns. ListObjects (V1 and V2) and ListObjectVersions with a delimiter read every key, or every version, under each common prefix before moving to the next one: a listing at the root of a bucket read the whole bucket, however few prefixes it returned, and one request could keep the database busy for as long as the bucket was large. Once a listing reaches a common prefix, its next query now starts past every key under it, so a page costs at most one query of 100 rows per item it returns. The three listings share the same grouping code. - DeleteBucket no longer counts the bucket's objects to see whether it is empty (nor does enabling Object Lock on an existing bucket). The check was COUNT(*) ... LIMIT 1, which counts every row before applying the limit; it is now an EXISTS that stops at the first row. - The documented default of server.runtime.max_blocking_threads was wrong. The configuration guide and the code comments said 0 meant num_cpus; it leaves tokio's own default, a pool of up to 512 threads that grows on demand, which is what the server has always run with. The guide now says so, and advises against a low cap: every file operation runs in that pool.

Security

• musl CVE-2026-107891 in both images. The console's Alpine base shipped musl 1.2.6-r2; it moves from Alpine 3.24.1 to 3.24.3, which ships musl 1.2.6-r3, openssl 3.5.9-r0 and zlib 1.3.2-r1, so the console Dockerfile no longer needs its openssl and zlib constraints and has none. The arca binary links musl statically, where the image scan cannot see it because the scratch image has no package database, and the Rust builder image still ships 1.2.6-r2: the builder now requires musl-dev >= 1.2.6-r3. Earlier binaries and images embed musl 1.2.6-r2.
• rustls 0.23.46 (Cargo.lock only, a compatible point release) for GHSA-c8m5-c952-6cm4 (MEDIUM: the Debug output of a session exposed TLS secrets) and three LOW advisories: GHSA-h2v8-394m-p7vv (Brotli certificate decompression could allocate an attacker-chosen amount of memory), GHSA-vp88-798v-h67f (an ECH rejection could expose buffered application data) and GHSA-w44r-93j7-qv28 (an unauthenticated ServerHello could poison the per-name key-exchange record).

0.34.0 — 2026-10-30 [YANKED]

Tagged but never published; its changes ship in 0.34.1.

0.33.0 — 2026-10-17

Added

• Run the published images without building. The images on ghcr.io are public, so bin/arca start --release <tag>, bin/console start --release <tag> and bin/cluster up --release <tag> pull latest, X, X.Y or X.Y.Z (no login, no Rust toolchain, no build) instead of building the working tree. The compose files read the tag from ARCA_RELEASE; bin/arca saves it in .arca-env so the console and bin/arca stop follow it. --release cannot be combined with --build or --dev, and bin/build never tags a local build with a release tag.
• The publish workflow verifies the images are public. After tagging, every tag of both images is fetched from the registry API with an anonymous token (no docker login, plugins or builders involved) and must list linux/amd64 and linux/arm64; a package that went private fails the release. bin/build --push runs the same check, after both images are pushed.
• A "Choosing a tag" guide in the installation documentation: latest for the newest release, X.Y for fixes only, X from 1.0, X.Y.Z for business-critical production. The Kubernetes manifests and the deployment guide now use latest and point at it.

Changed

• bin/console start and bin/cluster up follow the same rule (run the existing image, else pull latest, --dev builds), and the console warns when it runs a different version from the server. bin/cluster does not remember --release.
• bin/arca start no longer builds a production image implicitly. It used to let Compose build when the image was missing; it now pulls the published latest instead, as docker run does with a missing image (--dev has no published image, so a missing one is built).
• bin/test integration has no known failures left. The cluster, replication and notification-delivery tests need names (arca-1, arca-replica, webhook-receiver) that exist only in the compose project of their own mode, so against the developer's server 61 of them failed (and 7 cluster tests passed against a single node standing in for the load balancer). They now skip unless their mode set the variable that points at those services, and run in bin/test cluster, bin/test replication and bin/test notifications. bin/test cluster joined the other self-contained modes: its own compose project, ephemeral loopback ports (it no longer collides with a running server on port 9000), and bin/cluster partition/heal follow it through ARCA_CLUSTER_PROJECT. New bin/test full runs unit, integration, notifications, replication and cluster as separate stages and names the ones that failed.
• The console image now listens on ports 8080 (HTTP) and 8443 (HTTPS) instead of 80 and 443: remap your port mappings and proxies (TD-027). HTTP is now on 8080 (was 80) and HTTPS on 8443 (was 443), the nginx binary no longer carries the cap_net_bind_service file capability, and the container user is renumbered from 100:101 to 10001:10001, so the container runs with every capability dropped and the Kubernetes manifest adds none back. This is incompatible for anyone running the published console image with their own setup: a port mapping -p X:80 / X:443 must become -p X:8080 / X:8443, a reverse proxy pointing at console:80 must point at console:8080, a pod that sets runAsUser: 100 and runAsGroup: 101 must use 10001 for both, and a manifest that declares containerPort: 80 must declare 8080. The bundled Compose files (host ports 9080 and 9443 are unchanged), deploy/kubernetes/arca-console.yaml (Service still on port 80) and the documentation are already updated. TLS key access is unaffected: the console still reads the key through Arca's service GID 65532. trivy fs now reports no misconfiguration or vulnerability; KSV-0125 (ghcr.io is not in the check's registry allow-list) and KSV-0013 on the cluster manifest (latest is its documented evaluation default) are accepted in .trivyignore with their reasons.
• The console is documented like the server. Every way of running Arca now has its console counterpart with the same structure: the published image with bin/console start -d --release latest or a plain docker run (no repository needed, ARCA_ENDPOINT, ports 8080/8443 inside the container, 9080/9443 on the host), the HTTPS variant, the Compose service and how to stop it. The documentation also told readers to run bin/console start --tls, a flag that does not exist: the console inherits TLS from the Arca server it follows.

Fixed

• A completed multipart upload gets an object checksum, and keeps its part layout (TD-046). CompleteMultipartUpload stored no checksum and compared nothing: not the part checksums of the XML, not a whole-object x-amz-checksum-* header. CreateMultipartUpload now takes x-amz-checksum-algorithm and x-amz-checksum-type (COMPOSITE or FULL_OBJECT; CRC64NVME, the AWS CLI default, only has the full-object form, SHA1 and SHA256 only the composite one; contradictory or unknown values answer 400 InvalidRequest). UploadPart computes the checksum when the client declares none and refuses another algorithm's. Complete builds the checksum from the part checksums, without reading the object (a CRC combination for FULL_OBJECT, checked against the Ceph s3-tests vectors), and refuses with 400 BadDigest a whole-object header or a part checksum that does not match, and with 400 InvalidRequest a wrong x-amz-mp-object-size or x-amz-checksum-type, all before anything is assembled. The checksum, with x-amz-checksum-type, is returned by Complete, HeadObject and GetObject (checksum mode) and GetObjectAttributes. Every object completed from parts also keeps the layout of its parts, which GET and HEAD with ?partNumber=N (206, x-amz-mp-parts-count, the part's checksum) and the ObjectParts of GetObjectAttributes (paginated, per-part checksums) now serve; it is read on demand, so a GET, HEAD, listing or PUT costs the same whatever the number of parts. GetObjectAttributes also named the part count TotalPartsCount instead of PartsCount, so SDKs never read it. CopyObject keeps a FULL_OBJECT checksum and drops a COMPOSITE one (the copy has no parts). Uploads created without a checksum algorithm still get no object checksum, and objects completed before this release have no part layout (?partNumber answers 400 InvalidPart for them). Adds the objects.parts and multipart_uploads.checksum_type columns (SQLite migration 28, PostgreSQL 0016, applied automatically). Measured against the previous release on the project's perf suite, no request type is slower (all within 1%).
• The chunks and the trailer of signed aws-chunked uploads are verified (TD-045). For STREAMING-AWS4-HMAC-SHA256-PAYLOAD bodies (and the -TRAILER variant) the chunk-signature= of each chunk and the x-amz-trailer-signature were read and thrown away, so only the headers were authenticated: a client that sends no digest, such as mc over plain HTTP, had its body checked by nothing, and anyone able to alter the stream could change what was stored. PutObject and UploadPart now verify the signature chain as the body streams (each chunk signs its data and the previous signature, the first chain on the request signature). A modified byte, a bad or missing signature, a reordered or replayed chunk, a body cut anywhere (the signed final chunk is required) or a signed-chunk body under a presigned URL answers 403 SignatureDoesNotMatch before the blob is committed: nothing is stored and an existing object is left untouched. Clients that sign chunks (MinIO clients over plain HTTP) keep working; boto3 and the AWS CLI use STREAMING-UNSIGNED-PAYLOAD-TRAILER, which has no chunk signatures and still relies on the declared checksum (TD-034). Checked against the AWS chunked-upload example vectors, an independent signer in the new integration tests and mc. Verifying the signatures hashes every byte of a signed-chunk upload: measured on the project's perf setup, such an upload (mc over plain HTTP) runs about 16% slower, 392 against 465 MB/s on a 128 MiB body; every other upload and request is unaffected.
• Objects, delete markers, buckets and multipart uploads record the user who created them (TD-042). PutObject, CopyObject (the copier, not the source owner) and CompleteMultipartUpload (the user who completes it) stored root as owner whatever the credential; delete markers, buckets and multipart uploads did the same. Owner in ListObjects (V1 now returns it too, as AWS does), ListObjectsV2 with fetch-owner and ListObjectVersions, and Owner/Initiator in ListParts and ListMultipartUploads now show that user's name. The userIdentity.principalId of notification events carries it as well (it was empty). Objects, buckets and uploads created before this release keep root: the real writer was never recorded and cannot be recovered. Lifecycle delete markers and arca recover still use root. Multipart uploads gain an initiator column (SQLite migration 27, PostgreSQL 0015, applied automatically). The MetadataStore trait gains create_bucket_owned and a marker_owner argument on delete_object_if. In a cluster, nodes on an older version keep writing root for the writes they serve.
• The Kubernetes console manifest and the tooling Dockerfiles are tightened for the trivy misconfiguration scan (TD-027, TD-028). deploy/kubernetes/arca-console.yaml pins the console image to the release tag, runs with a read-only root filesystem (emptyDir volumes for nginx's writable paths, an init container seeding index.html and http.d) and drops every capability (possible thanks to the unprivileged ports under Changed). The screenshots image runs as the unprivileged pwuser, and the screenshots, perf, test and s3-tests images declare HEALTHCHECK NONE. The pinned tag is now a version location to bump at every release. The port and UID/GID findings are closed by the unprivileged-port change under Changed.
• An abort racing CompleteMultipartUpload no longer destroys the completed object's data (TD-053). Complete committed the object row and only afterwards consumed the upload, so an AbortMultipartUpload, a lifecycle AbortIncompleteMultipartUpload rule or arca fsck --repair --abort-uploads-older-than landing in between deleted the parts the new object was made of: Complete answered 200 and the object could not be read (InternalError). The object row and the consumption of the upload are now one transaction in SQLite and PostgreSQL, so whichever commits first wins: an abort that wins makes the Complete answer NoSuchUpload and write nothing, and a Complete that wins leaves the abort with no parts to delete. A refused If-Match / If-None-Match still leaves the upload intact for a retry. On an HA cluster the guarantee holds per node.
• A version-id-marker deleted between two pages no longer skips the rest of its key (TD-054). ListObjectVersions looked the marker version up to find where to resume, and when it was gone resumed after the whole key. The internal full listing (re-encryption scan, arca fsck) now pages by the position of the last row it read, so it cannot skip a version even under concurrent deletes (for example lifecycle noncurrent expiry). On S3, a version-id-marker that does not exist (never did, or deleted by a client purging the bucket page by page) restarts the listing from the beginning of the key-marker's key and answers 200: a version of that key can be listed twice, none is skipped.
• UploadPartCopy honours the copy-source conditional headers (TD-035). x-amz-copy-source-if-match, -if-none-match, -if-modified-since and -if-unmodified-since were evaluated by CopyObject only, so a part was copied even when the condition failed. They now answer 412 PreconditionFailed and store no part.
• Virtual-hosted-style addressing works (TD-037). With [server] domain set, bucket.<domain>/key was routed as bucket key (and bucket.<domain>/ as ListBuckets) because the rewrite ran after routing. It now runs before routing, in the same layer that saves the signed URI, so the signature still verifies; the bucket root maps to the bucket, and a virtual-hosted dir/ stays a directory-marker key. The host is read from Host or, on HTTP/2, the URI authority. New self-contained bin/test virtual-host.
• Blob ids from cluster peers are validated (TD-041). An id inside a replicated object row, a composite part of a received sidecar or an anti-entropy manifest row was used as a file name unchecked, so a hostile or corrupt peer could make a node address paths outside its blob tree or panic on a short id. Such ids are now refused: the cluster endpoints answer 400, anti-entropy skips the row, and the blob store itself rejects any id that is not a UUID. Files under blobs/ whose name is not a blob id (a stray file, a copied backup) are no longer treated as blobs: arca gc used to delete them as orphans, now it leaves them alone, arca fsck lists them as FOREIGN_FILE with their size, and arca gc --reclaim --delete-foreign-files removes them on request.
• Config-drift detection compares cluster_size (TD-036). The fingerprint nodes exchange hashed the write quorum but not cluster_size, so a resize between 2 and 3 (or 4 and 5) left a node with a stale size unflagged. A second fingerprint now includes it (quorum mode only: in available mode the setting has no effect). The ping sends both, and a node compares the new one only with a peer that sends it, so a rolling upgrade from an older version shows no drift and loses no write quorum; cluster_size drift is caught between upgraded nodes.

0.32.1 — 2026-10-15

Added

• arca gc reports the space it frees: the dry run says how much --reclaim would free, and --reclaim how much it freed (Reclaimed 0 orphan blob(s) and 118 stale temp file(s), freeing 419.0 MiB.). An orphan counts with its sidecar, which is deleted with it; only the files actually deleted count toward the freed space.

Changed

• Plain HTTP is served by the same accept loop as HTTPS (serve.rs) instead of axum::serve, so the two listeners behave alike: the bounded shutdown above, the [server.http] HTTP/2 settings (which only HTTPS applied), hyper's 30-second limit for receiving a request's headers, which also closes a keep-alive connection left idle for 30 seconds (only HTTPS had it; clients reconnect on their own), and, when accepting fails for a reason that is not about one connection (EMFILE, out of file descriptors), a 1-second pause before retrying instead of a busy loop (which only plain HTTP had).

Fixed

• Shutdown is bounded: a stopping server no longer hangs until it is killed. When the drain period ended, the HTTPS listener waited for every open connection to end by itself without telling any of them to close, so a single idle keep-alive connection kept the server running: in production, 137 idle client connections held a stopping server until systemd SIGKILLed it at TimeoutStopSec, skipping the rest of the shutdown. Plain HTTP closed idle connections but waited for running requests with no limit, so one stalled upload did the same. Every connection is now told to close when the drain ends: an idle one closes at once, a busy one after its current request (HTTP/2 sends GOAWAY), and those still running 10 seconds later are dropped, so a stop takes at most drain_timeout_seconds + 10 seconds. The deployment guide also wrongly advised a drain_timeout_seconds at least as long as the Kubernetes termination grace period: the grace period must be the longer of the two.
• The HTTPS listener no longer keeps every closed connection until shutdown. The task of each connection stayed in the listener's set after the connection closed, so memory grew with every connection accepted since the server started (and the connection count logged at shutdown was inflated by all of them). Closed connections are now dropped as new ones are accepted.

0.32.0 — 2026-10-15

Added

• arca fsck --repair: fixes what the check finds, asking before each fix (y, n, a for the rest of the category, q to stop; --yes answers yes to all). Stale temp files are deleted without asking; orphaned blobs and sidecars are moved to <data_dir>/lost+found/<timestamp>/ (or deleted with --delete); the row of a version whose data is gone is deleted; a sidecar that disagrees with its row is rewritten from it, keeping its encryption, compression and part list; with --abort-uploads-older-than <days>, old unfinished uploads are aborted. It may run against a live server: files younger than the grace period are never touched (on a cluster node the grace is at least the tombstone grace), every item is checked again right before its fix (a row is deleted only if it still points at the missing blob, checked inside the delete's transaction), and the referenced set is recomputed before orphans are moved. Database fixes run first and are followed by a rescan, so the files they leave behind are reclaimed in the same run. Ctrl+C exits at once, or after the current fix if one is running, and a re-run continues, with no journal. Aborting an upload that a client is completing at that very moment loses the completed object, as AbortMultipartUpload does (TD-053): abort only uploads old enough to be abandoned. It refuses to start while an offline rewrite is unfinished, and on an HA cluster node it only repairs files: a missing blob is left to anti-entropy, which has it on a peer, and uploads are aborted through the S3 API.
• arca gc reclaims stale temp files (TD-051): the <blob>.tmp and sidecar temp files of interrupted writes, older than the grace period, are reclaimed by arca gc --reclaim, the single-node GC worker and cluster anti-entropy GC; the recovery files of an in-place rewrite (*.rewrite.tmp, *.meta.pending.tmp) are never touched. arca gc also reports the unfinished multipart uploads and the bytes they hold, which it does not reclaim.
• arca fsck reports unfinished multipart uploads (TD-052): each upload that was never completed nor aborted is listed with its key, upload id, start date, part count and size, and the summary gives their total and the oldest one. They are valid state, so they do not make the exit code 1; the summary points at AbortMultipartUpload and the AbortIncompleteMultipartUpload lifecycle rule.
• arca fsck --grace-seconds (default 86400, as arca gc): files written within the grace period are skipped and counted apart instead of reported, so a check against a running server no longer flags the blobs and temp files of uploads still in flight. Use 0 only with the server stopped.
• arca fsck summary: one line per category with its count and, where it matters, the bytes involved (orphaned blobs, stale temp files, unfinished uploads).
• arca fsck and arca gc support PostgreSQL: both open the configured metadata backend. fsck used to open a SQLite file whatever the configuration said, and gc refused PostgreSQL.

Fixed

• Interrupted writes no longer leave temp files behind (TD-051). A PutObject or UploadPart whose client disconnected left its partial <blob>.tmp: the cleanup ran only on an error the write returned, never when the request was dropped. The fallback copy of CompleteMultipartUpload (mixed or compressed parts) also left its temp file on any error, a missing part included, and every writer left one when the final rename failed. A guard now owns the temp file from its creation to the rename that publishes it, and removes it on every other way out.
• arca fsck understands multipart objects (TD-014, the fsck half). A completed multipart upload is stored as a composite: a sidecar listing its part blobs and no blob file of its own. fsck reported every multipart object as MISSING_BLOB plus ORPHANED_SIDECAR, and every part, completed or still uploading, as ORPHANED_BLOB: on a production server, 91,867 false findings that hid the real ones. fsck now computes what is referenced exactly as arca gc does (object rows, in-progress parts, and the parts of every live composite), so the two tools agree; a composite with a missing part is reported as MISSING_BLOB with the missing part ids, and the sidecar and parts of a composite whose object is gone as orphans. Unreadable sidecars get their own UNREADABLE_SIDECAR category, and the recovery files of an unfinished in-place rewrite (*.rewrite.tmp, *.pending.tmp) are reported as REWRITE_RECOVERY instead of stale temp files.
• ListObjectVersions pagination no longer skips versions. Both metadata backends ignored version-id-marker: a page that ended in the middle of a key's versions resumed at the next key, so the remaining versions of that key were never listed, and versions sharing a timestamp had no stable order across pages. The listing now resumes right after the marker version (null for the null version) in a total order. The same bug made the encrypt / decrypt maintenance jobs and arca fsck skip versions; all of them now share one paging helper.
• arca fsck --verify-checksums no longer reports compressed objects as corrupt (the ETag hashes the plaintext, the file holds the compressed bytes), and no longer walks the whole blobs directory once per object to find each blob file.
• Offline tools refuse a missing SQLite database instead of creating an empty one: fsck, gc, encrypt-existing and decrypt-existing now stop with Database does not exist, so a wrong data_dir cannot make every blob look unreferenced.

0.31.2 — 2026-10-07

Added

• arca --version (or -V) prints the version of the binary, so a host running a release binary can tell which one it has.

Fixed

• Large multipart objects can be downloaded again on a server with a low open-files limit. A GET of a multipart object opened the file of every part it covered before sending the first byte, so it needed one file descriptor per part: a 200 GB object of 5995 parts failed every download with 500 Internal Server Error (open part blob: No file descriptors available (os error 24)) on a server whose soft RLIMIT_NOFILE was 1024, and fewer, smaller downloads in parallel could exhaust a higher limit just as well. The data on disk was intact. Plain and SSE-S3/KMS composite reads, ranged or not, now open one part at a time, each when the previous one has been streamed, and the decrypt-and-re-encrypt fallback of CompleteMultipartUpload (mixed or compressed parts under encryption) reads its parts the same way. The first part is still opened before the response starts, so a missing part remains an error response; a part that disappears later aborts the transfer. The Compose file shipped in docker/ already raises nofile to 65535; deployments using their own should do the same.

Security

• zlib CVE-2026-85091 (heap buffer overflow, MEDIUM) in both images. The console's Alpine base ships zlib 1.3.2-r0, and so does the newest Alpine 3.24 point release: the console Dockerfile now requires zlib>=1.3.2-r1, next to the existing openssl constraint. The arca binary links zlib statically (librdkafka, through libz-sys), where the image scan cannot see it because the scratch image has no package database: the builder now requires zlib-dev / zlib-static >= 1.3.2-r1, which a build from a cached layer did not guarantee. Earlier binaries and images may embed the vulnerable 1.3.2-r0; zlib is reached there only by the Kafka notification connector's message compression.

0.31.1 — 2026-10-06

Added

• GitHub releases. Every release tag now gets a GitHub release, created by bin/release (step 12 of RELEASING.md): the notes are derived from the tagged CHANGELOG entry (the section headings and the bold title of every bullet, then links to the full entry, the compare view and the container images), previewed as GitHub renders them, and published only after confirmation. v0.31.0 is the first one.
• Linux release binaries. Every GitHub release now carries the server binary for linux/amd64 and linux/arm64 (arca-X.Y.Z-linux-<arch>.tar.gz) and their SHA256SUMS. The binary is the static (musl) one of the release image, archived by the Publish images workflow from the very image it scanned, so it runs on any Linux of that architecture with no dependency; each archive also carries LICENSE-AGPL-3.0, NOTICE and THIRD-PARTY-NOTICES.md. bin/release downloads them from the tag's workflow run, verifies the checksums and attaches them; its preview shows the assets as GitHub will.

Fixed

• The server no longer panics at startup when the system has no CA certificates (TD-049). The webhook and Elasticsearch connectors built their HTTP client at startup, whether or not any notification was configured, and aborted the server when the platform trust store could not be loaded (No CA certificates were loaded from the system): the published image was not affected (it ships a CA bundle), but a bare binary on a host or in an image without one was. The server now starts, logs one warning per connector, and those connectors fail each delivery and connection test with HTTP client unavailable: <cause>.

0.31.0 — 2026-10-06

Added

• Published multi-arch container images. Every release is now published to the GitHub Container Registry as ghcr.io/dxc-technology/arca and ghcr.io/dxc-technology/arca-console, each a single multi-arch image for linux/amd64 and linux/arm64, so the same tag runs on x86 servers and on Apple Silicon / ARM hosts. A release is tagged X.Y.Z, plus X.Y, X and latest only when it is the newest release of its minor line / major line / overall — publishing a fix for an older line never moves them backwards — and a prerelease gets its exact tag only; the major-only X tag is not published while Arca is at 0.x (a minor bump may break compatibility, so :0 would promise too much) and appears by itself from 1.0.0. Publishing is automatic on every vX.Y.Z tag push (.github/workflows/publish-images.yml): each platform builds natively on its own runner (an emulated Rust release build would take hours), is scanned by Trivy, and is pushed by digest, untagged; only when all four builds pass are the digests merged into one index per image and tagged, so a scan failure on any platform publishes nothing. The Trivy gate fails on any finding and, for arca, also scans Cargo.lock: the scratch image holds no package database and a plain Rust binary, so an image scan alone has nothing to check. The workflow also refuses a tag that disagrees with Cargo.toml or the console version. bin/build --push runs the same pipeline from a local machine as a fallback. All of it lives in one library, bin/lib/images.sh, covered by 45 tests (bin/test scripts, docker mocked), so CI and the fallback cannot drift apart.

• OCI image metadata. Both production Dockerfiles now carry org.opencontainers.image.* labels (title, description, source, url, licenses); version and revision are added at publish time, and every label is copied to the multi-arch index as an annotation, which is where GHCR reads the package description from. source links each package to this repository. docker inspect on a running container now tells which release and commit it was built from.

• arca tls ensure keeps a local CA and a server certificate valid, (re)generating only what is needed: a missing, unreadable or inconsistent CA (key not matching its certificate), or one expiring within the renewal window (30 days by default), is replaced; a server certificate that is missing, expiring, issued for other SANs or not signed by the current CA is reissued with the same CA, so a client that trusts the CA once keeps trusting every renewal. The CA lasts ten years, the server certificate one year (well inside the 398/825-day limits browsers and macOS apply to server certificates). The CA private key lives in its own directory and never reaches the one mounted in containers — Arca's certificate auto-detect would otherwise see two private keys and refuse to start. Idempotent: a run with nothing to do writes nothing. arca tls generate is unchanged. Reading certificates back uses x509-parser, already in the dependency tree through rcgen (whose x509-parser feature is now enabled to sign with an existing CA): no new crate.

• HTTPS without certificates of your own. bin/arca start --tls used to fail when certs/ was empty. It now decides at every start: your certificates in certs/ (any .pem/.crt/.cert/.key file, symlinks followed) are served as before; with none, a local CA and a server certificate for localhost, 127.0.0.1, ::1 and arca are created in certs/local/ on the first start and kept valid by arca tls ensure on every later one (plus any names in ARCA_TLS_SANS, from the environment or docker/.env). Dropping your certificates in wins at the next start and removing them falls back to the local ones; nothing in certs/ itself is ever written. The CA key lives in certs/local-ca/, which no container mounts. The files are created as the host user with Arca's service group, so the host user can manage them and Arca and the console read the 0640 key through the group, as with any certificate. The choice is saved in .arca-env for bin/console. The TLS guide documents trusting the local CA once on macOS, Debian/Ubuntu or in the S3 clients.

• [storage] fsync setting (default true). With true the server fsyncs blob data, sidecars and new shard directories and commits SQLite with synchronous=FULL, so an acknowledged write survives a power loss or kernel crash (TD-038). false skips every fsync and uses synchronous=NORMAL, the behaviour of previous releases: faster on slow disks, but objects acknowledged shortly before a power loss can be lost or truncated (writes stay atomic: a sidecar is never half written). The server logs a warning at startup when it is off. It applies to the live server only: the offline tools (recover, gc, migrate-db, compress-existing, ...) are always durable. With PostgreSQL the database side is PostgreSQL's own synchronous_commit.

Changed

• The Ceph s3-tests report counts documented AWS divergences as passed. docker/s3-tests/report.py has a table of tests where Ceph expects RGW's answer and Arca deliberately gives AWS's; such a test counts as passed only when it fails on exactly the listed assertion (every earlier check in it passed), never on any other failure or error, and the HTML report and summary.json (aws_divergences) list them with the reason. Two entries so far: test_object_checksum_sha256 and test_object_checksum_crc64nvme, which expect BadDigest for a malformed checksum value where AWS answers InvalidRequest. Latest run: 371/825 (was 369), 0 unexpected failures. Covered by docker/s3-tests/report_test.py.
• Audit-log entries are committed in batches of up to 1 second. The audit writer shares the metadata database, so with SQLite FULL every audit commit was an fsync, paid even by a GET. It now commits when 1024 entries are queued or 1 second after the first one, and at once when the server starts a graceful shutdown. A crash can lose up to the last second of audit entries, which until then were only held in memory.
• Image names. The production images are now named after their registry repository everywhere: the Compose files, bin/test, the Kubernetes manifests (deploy/kubernetes/, which now reference the published images) and the documentation. Local builds are tagged after their build target, ghcr.io/dxc-technology/arca:production / :development and ghcr.io/dxc-technology/arca-console:production, never with a release tag, so a working-tree build can neither pass for a release nor shadow the published latest. Images built locally under the old arca / arca-console names are no longer used and can be removed with docker rmi arca arca-console. Test and tooling images keep their names and are never published.
• Documentation audited against the code. The architecture reference is rewritten from the current code: real crate graph and middleware order (there is no use-case layer), every storage trait and decorator and how they are stacked, the table list with what a cluster replicates, the write path, multipart composite blobs, how conditional writes are made atomic on each backend, and a clustering section with the design rationale. A new HA Design Decisions page records decisions H1 to H12 (what the code does today, why, the alternatives rejected), which code comments cite as "decision Hn". The S3 API reference gains a conditional-writes section and two documented divergences from AWS (a refused conditional CompleteMultipartUpload is always 412, never 409; If-None-Match accepts a specific ETag). The HA guide is corrected where it promised more than the code does: the drift fingerprint covers the write quorum, not cluster_size; only object writes are ACK-counted in quorum mode; object tags do not self-heal; sync state is read from GET /admin/cluster, not arca cluster status; TD-025 needs per-key load-balancer routing. The Admin API, CLI, Disaster Recovery, Configuration and TLS pages are completed or corrected likewise (health states, /admin/cluster, ?node=, five missing CLI commands, what arca recover really restores). The help of arca cluster status and the instructions printed by arca migrate-topology now point at GET /admin/cluster for sync state. Bugs found during the audit are tracked as TD-029 to TD-042.

Removed

• The "HA Hardening Plan" documentation page and the .claude/ planning documents. Plans and reviews were working documents for a single task. Their durable content (design decisions and rationale, invariants, known limits) now lives in the architecture reference, the HA Design Decisions page, the HA guide and TECH_DEBT.md, and code comments no longer cite IDs defined only there.

Fixed

• The offline compress-existing, decompress-existing, encrypt-existing and decrypt-existing can no longer turn good data into bad on a power cut. They rewrote blobs that were already durable with no fsync, renamed the new blob over the old one and only then wrote the sidecar with a plain fs::write: a power loss during or shortly after a run could leave a zero-length blob, or a blob that no longer matched its sidecar (a compressed file described as uncompressed, which reads back as garbage and is compressed a second time by a re-run). For encrypt-existing the window was worse: the freshly generated wrapped key existed only in memory until the sidecar was written, so a failure there left ciphertext whose key was lost for good, and a re-run skipped the blob as "already encrypted". The new blob is now written to a temp file and fsynced, the new sidecar is written durably to a journal file, and only then are the blob and the journal renamed into place, each rename followed by a directory fsync (arca_storage::inplace; the durability helpers sync_dir, rename_durable and write_file_atomic are now public). Running the same command again after a crash discards a rewrite that never replaced the blob, or completes one that did (including the database row for the encryption tools), before looking at the sidecar. Leftover temp files are named *.tmp so GC and arca fsck ignore or report them; the encryption temp files used to end in .aenc-tmp / .plain-tmp, which GC and fsck took for blobs. fsync itself cannot be observed by a test; the ordering, the cleanup and the recovery of every crash point are covered by unit tests. A crash between the blob rename and the sidecar rename leaves a mismatched pair until the tool is run again, so every real run now writes a durable marker (<data_dir>/.offline-rewrite-in-progress: command, re-run arguments, start time, pid) before touching any blob and removes it only after a clean finish: arca serve refuses to start while it exists (no override flag), recover, gc, migrate-db and migrate-topology refuse, fsck warns, another rewrite command is refused until the interrupted one is re-run, and the same command clears it. Residual: the database row update of the encryption tools is still after the sidecar, repaired only when the crash happened inside the journaled swap. Behaviour change for scripts: encrypt-existing and decrypt-existing now exit NON-ZERO when the run ends with per-object errors (the summary is still printed, the marker is kept, and the error names the command to re-run); they used to print the errors and exit 0. A clean run and --dry-run still exit 0. compress-existing and decompress-existing already aborted with a non-zero exit on the first failing object, so they are unchanged.
• Keep-alive GETs over plain HTTP no longer take about 40 ms each (TD-047). The plain-HTTP listener left Nagle's algorithm on (the TLS listener already disabled it), so a response written as headers then body, which is every GetObject, waited for the client's delayed ACK: on a keep-alive connection every GET after the first took about 40 ms more than it should (measured 3 ms for the first, 42 ms for each following one, 1 KiB objects, Linux). Every accepted plain-HTTP connection now gets TCP_NODELAY. All real S3 clients use keep-alive, so this affected every plain-HTTP deployment, for example one behind a TLS-terminating reverse proxy. PutObject was not affected (its response has no body).
• Acknowledged writes now survive a power loss (TD-038). Blob and sidecar writes never called fsync, and sidecars were written with a plain, non-atomic fs::write, so a power cut after a 200 OK could lose or truncate the data. The blob temp file is now fsynced before the rename (plain put, replicated raw writes, the concat fallback, re-encryption), newly created shard directories are fsynced, and sidecars are written to a temp file, fsynced, renamed and the directory fsynced. A blob is durable once its sidecar has been written: the sidecar lives in the blob's directory, so its directory fsync also covers the blob's rename (3 fsyncs per small PUT, not 4), and every handler now fails the request, discarding the fresh blob, when the sidecar write fails instead of logging a warning and committing the row (UploadPart and the re-encryption job did that). SQLite now commits with synchronous=FULL instead of NORMAL, so the metadata row of an acknowledged write survives a power loss too (in WAL mode NORMAL can lose the last commits). The new [storage] fsync setting (default true) turns all of this off for deployments that prefer throughput over durability (see Added). Measured with warp, 1 KiB PUTs at concurrency 16 (A = no fsync, the old behaviour; B = file fsyncs only; C = file fsyncs + SQLite FULL, the new default): local NVMe (AWS instance store, 0.16 ms per synchronous write) B -35%, C -37%; gp3 EBS (3.3 ms) B -53%, C -78%; Hyper-V virtual disk (3 ms) B -49%, C -77%. 1 MiB objects lose 6-20% on NVMe and concurrent gp3, large objects (512 MiB) 0-2%. On a slow disk C caps small PUTs at about 200 per second whatever the concurrency, because every SQLite commit is an fsync on the single writer connection (TD-048).
• UploadPartCopy writes a part sidecar for every part (part of TD-014). A plain (unencrypted) copied part used to get none, so CompleteMultipartUpload copied every byte instead of assembling a composite, an encrypted store failed it with "missing sidecar", and arca recover could not see the part. It now writes the same sidecar as UploadPart.
• arca compress-existing no longer damages encrypted blobs or aborts on multipart ones (TD-044). It compressed the ciphertext file of SSE-S3/SSE-KMS blobs in place, leaving the objects unreadable (parse header: invalid magic bytes), and aborted the whole run on composite (multipart) blobs, which have no file of their own. It now skips encrypted blobs (including SSE-C) and composite blobs, and decompress-existing skips composite and encrypted blobs, each with a SKIP message and without stopping the run. If compress-existing was already run on a store with encryption enabled, some objects may be unreadable; the data and keys are intact and recoverable, the repair is still tracked in TD-044.
• A failing bin/test step no longer leaks its test project. The self-contained modes kept their compose command in a function-local COMPOSE; when set -e ended the script at a failing step, the EXIT trap ran after the function was gone, hit COMPOSE: unbound variable and never issued down -v, so the arca-test-<mode> project and its volumes stayed behind. The compose command (and, for tls-permissions, the names of the resources to remove) now live in script-level variables the cleanup can always read. Covered by 7 new tests in bin/test scripts that make a test step fail and check the cleanup, the failing exit code and the single cleanup of a passing run.
• Upload bodies are now verified against the digest the client sends, as on AWS (TD-034). PutObject and UploadPart accepted and stored any body: Content-MD5 was never read, x-amz-checksum-* values were stored without being compared with the data, and a hex x-amz-content-sha256 was signed but never checked, so a corrupted upload was acknowledged. A client that sends a wrong digest now gets 400 as on AWS and nothing is stored (an existing object or part is left untouched): BadDigest for a wrong Content-MD5 or x-amz-checksum-*, InvalidDigest for a malformed Content-MD5, InvalidRequest for a malformed checksum or two checksums at once, XAmzContentSHA256Mismatch for a wrong hex x-amz-content-sha256. All five S3 checksum algorithms are verified (CRC32, CRC32C, CRC64NVME, SHA1, SHA256; SHA1 was not recognised before), whether sent as a header or as an aws-chunked trailer. Trailing checksums, which boto3 / botocore 1.36+ send by default over HTTPS, were silently dropped and are now recorded and returned with x-amz-checksum-mode: ENABLED; UploadPart returns its checksum and ListParts reports it when CreateMultipartUpload named an algorithm. Clients that send correct digests (boto3, AWS CLI, MinIO mc and minio-py, over HTTP and HTTPS) are unaffected. Still not verified: aws-chunked chunk signatures (TD-045) and multipart object checksums at CompleteMultipartUpload (TD-046). No new dependency crates: crc, sha1 and sha2 were already in the dependency tree.

• A failed upload no longer leaves a temporary file behind. When a PutObject or UploadPart body failed mid-stream (body larger than max_body_size, client disconnect, malformed aws-chunked encoding), FsBlobStore returned the error but left the partial {id}.tmp file in the blobs directory, where arca fsck reported it as stale. It is now removed.

• Compressed objects could not be read on a server with encryption enabled (TD-029). On a bucket with compression configured, a GetObject (full or ranged) of a compressed object on a server with SSE-S3 enabled, globally or for that bucket, failed with 500 InternalError for practically every compressible object larger than 64 KiB, and for every object whose compressed frame came out larger than the object itself (incompressible bodies). Compression sits above encryption, so the encryption layer encrypts the compressed frame, but it sized its ranged reads with the object's uncompressed size from the sidecar instead of the frame's length; it now uses the frame's length (compressed_size). Only reads were wrong: objects already stored compressed and encrypted are intact and read correctly after upgrading, no repair is needed. bin/test compression now also runs the suite on an encrypted server, which has new tests for incompressible bodies, open-ended and suffix ranges, multipart uploads and copies.

• PostgreSQL conditional writes now have a regression test (TD-039). bin/test postgres also runs test_conditional_writes.py against the PostgreSQL-backed server, so a reordering of the compare-and-swap read relative to the transaction's first locking statement (required under READ COMMITTED) no longer goes unnoticed.

• Test runs no longer destroy the developer's server (TD-030). Every self-contained bin/test mode (tls, encryption, compression, per-bucket-encryption, recrypt, kms, postgres, migrate-db, migrate-topology, replication, notifications, every connector) and bin/perf-test --tls/--encryption ran in the developer's own compose project: they recreated the running arca container with their own config, overwrote config/.generated.toml, and finished with down -v, deleting the arca-data volume, while .arca-env went on describing a server that no longer existed. Each one now runs in a compose project of its own, arca-test-<mode>, so its containers, network and volumes are separate and its down -v removes only them; it generates its own config/.generated-test-<mode>.toml (mounted through ARCA_GENERATED_CONFIG); and every host port its services publish becomes an ephemeral loopback port, because the suites reach their services over the compose network and a fixed 9000 (or 8200, 5432, 8765, 9001) would collide with the server already running. The OpenBAO, PostgreSQL and webhook-receiver ports gain OPENBAO_HOST_PORT, POSTGRES_HOST_PORT and WEBHOOK_RECEIVER_HOST_PORT like the existing ARCA_HOST_PORT, and the TLS overlay no longer publishes a fixed 9000 next to ARCA_HOST_PORT. A running dev stack is left exactly as it was, and suites of different modes can run side by side. bin/test presigned, bin/test ssec and bin/perf-test without features, which start no server of their own, used a plain compose run that could recreate the running server from the base compose file: they now run against it with --no-deps, like bin/test integration. Covered by 33 new bin/test scripts tests, with docker mocked.

• bin/test integration skipped the encryption and KMS suites (TD-031). test_encryption.py, test_per_bucket_encryption.py, test_recrypt.py and test_kms.py only run when an environment variable says the server supports them, and only their dedicated modes set it, so against a server started with --encryption or --kms they were silently skipped. bin/test integration now sets them from the features recorded in .arca-env: global encryption runs the encryption suite, KMS also the KMS suite, per-bucket encryption (local key or KMS) the per-bucket and re-encryption suites.
• bin/test integration tested a different server from the one you started. It ran docker compose run --rm test, whose depends_on recreated the running arca container from the base compose file alone: the generated config.toml and the certificate mount were dropped, so after bin/arca start --tls --encryption the suite silently ran against a plain, unencrypted server (and, since the image rename above, against the stale :production image instead of the dev one). It now runs with --no-deps, never touching the server, reads .arca-env to reach it the way it was started — plain HTTP, HTTPS verified against the local CA, or, with your own certificates, HTTPS under a name the served certificate covers (its first plain DNS SAN, a wildcard made concrete as arca.<domain>, or ARCA_TLS_HOSTNAME from docker/.env), mapped to the arca container and verified against the system CAs — and stops with a clear error when no server is running.
• The TLS test suites no longer touch certs/, and work on Linux (TD-026). bin/test tls generated its self-signed certificates into the repository's certs/ bind mount and deleted them afterwards — destroying any file of the user's with the same names — and on a real Linux host it failed outright, because the tls-init container runs as 65532 and cannot write the invoking user's directory (invisible on macOS, where Docker Desktop fakes bind-mount ownership). bin/test tls and bin/perf-test --tls now generate a throwaway CA + server certificate into a named volume (arca-tls-test-certs), serve it through the same compose overlay, and remove it at the end. bin/perf-test --tls also stops disabling certificate verification: its client now trusts the throwaway CA, and with --tls-fqdn your own certificates are used in place instead of being symlinked into certs/. The tls-init compose service, which was the only writer into certs/, is removed.
• A request right after an early reject could be misread on the same connection. When Arca answered a request before reading its body — any authentication or authorization reject in the S3 auth middleware, a missing bucket, a failed precondition — it kept the HTTP/1.1 connection alive although the body was still owed. With Expect: 100-continue, which botocore sends on every PutObject, the client, having received a final status instead of 100 Continue, legitimately never sends that body, and the server cannot tell the bytes that arrive next from the owed body: hyper sometimes swallowed the first bytes of the next request as the body and parsed the rest as a mangled request line. A DeleteObject sent after a denied PutObject then arrived as TE /bucket/key and failed with SignatureDoesNotMatch instead of AccessDenied — the intermittent failure of test_readonly_user_can_read_not_write, reproduced on the wire in 14% of raced follow-up requests. A new outermost layer (CloseOnUnreadBodyLayer) now wraps every request body, records whether it was read to the end, and adds Connection: close to any response that leaves it unread, as RFC 9110 §10.1.1 asks and AWS S3 does — whichever middleware or handler produced the response, and whatever its status. It replaces the per-call-site close_after that covered only PutObject's own early rejects (commit eb7eb6e) and left the auth middleware, UploadPart and the configuration PUTs exposed. HTTP/2 is left alone: each stream is framed, so an unread body cannot desync the next request. http-body (MIT, already in the dependency tree through axum and hyper) becomes a direct dependency of arca-proto.

Security

• Dependency updates for five advisories (all compatible point releases, Cargo.lock only): rustls 0.23.40 → 0.23.45 (GHSA-2mjx-qc3c-rqvc, TLS 1.3 handshake messages accepted across encryption-level boundaries; pulls rustls-webpki 0.103.15 and aws-lc-rs 1.18.1 / aws-lc-sys 0.45.0), hickory-resolver, hickory-proto and hickory-net 0.26.1 → 0.26.2 (GHSA-5j98-2g5x-46v6, GHSA-6w6g-hm98-mhgm, GHSA-6f2x-v7q7-m7m5: hidden DNSSEC validation failures, an unbounded truncated-response retry loop, irrelevant CNAME records followed), xxhash-rust 0.8.15 → 0.8.19 (GHSA-6g2r-675j-hx59). trivy fs --scanners vuln Cargo.lock reports 0 findings; the transitive licence audit is unchanged (no copyleft) and THIRD-PARTY-NOTICES.md is regenerated.

0.30.2 — 2026-09-23

Security

• arca tls generate wrote every file world-readable, private keys included. All four outputs of arca tls generate — and the CA plus every per-node key of arca tls generate-cluster — went through std::fs::write, which creates a file 0666 & ~umask, i.e. 0644 under the usual umask. The private keys were therefore readable by every local user on the host and by every process in any container sharing that directory. Modes are now explicit and split by role: certificates 0644 (public material), server and node private keys 0640, the CA private key 0600. The server key is deliberately group-readable rather than owner-only, because the group is the access channel: Arca runs as 65532:65532 and the web console runs as a different user (100:101), and in the Compose dev setup both read the same key — the console reaches it through a supplementary group, never through world-readable bits. The CA key is 0600: nothing reads it at runtime, it only signs. The mode is applied at creation time (OpenOptions::mode) so a key is never even briefly world-readable, and re-applied afterwards for two reasons — the creation mode is masked by the process umask, which must not be allowed to strip the group bit a key needs, and it does not apply at all to a file left behind by an earlier run, which is what makes re-running arca tls generate tighten keys an older version wrote loose. Action required: keys generated by an earlier version keep their old mode until they are regenerated or chmod'd by hand; treat a key that has been sitting world-readable on a shared host as disclosed and rotate it.

Fixed

• The Kubernetes console deployment could not start. deploy/kubernetes/arca-console.yaml set runAsUser: 101, but 101 is the nginx group — the image's user is uid 100. Running as 101 fails at startup: the entrypoint cannot rewrite index.html (sed: can't create temp file) and nginx cannot create /var/lib/nginx/tmp/client_body, because the image pre-owns those paths as 100:101. Corrected to runAsUser: 100, runAsGroup: 101, and the misleading comment on that line with it. The pod also gained fsGroup: 65532 so a mounted certificate Secret is readable when the console terminates TLS itself rather than at the ingress; arca-cluster.yaml, which was already correct at 65532 throughout, is unchanged.

• The console never enabled HTTPS with certificates produced by arca tls generate. Its entrypoint skips CA material when auto-detecting the certificate and key, and did so by matching the bare substring ca against the file name — which matches arca-server.crt and arca-server.key, inside the word "arca". Every file the tls-init service writes was therefore skipped, no certificate was found, and nginx quietly kept serving plain HTTP while port 9443 refused connections. The match is now against a ca word in the base name (ca.pem, arca-ca.crt, client-ca.pem, ca-bundle.crt), so CA files are still excluded and server files are not. The bug was masked for anyone whose certs/ directory also held real certificates under names like fullchain.pem/privkey.pem, which the old glob happened not to match. The entrypoint additionally no longer degrades silently when the key is present but unreadable: it logs the uid and groups it is running as and how to grant the missing one, instead of leaving "port 443 is closed" as the only symptom.

Changed

• The console reaches Arca's TLS private key through a shared group instead of world-readable bits. Arca's images run as 65532:65532, the console as 100:101, and the Compose dev flow bind-mounts one certs/ directory into both — no owner and mode can grant read to both users without also granting it to everyone. The key is now group-readable by 65532 (Arca's service GID), and the console is given that GID as a supplementary group: group_add: ["65532"] in docker/docker-compose.tls.yml, fsGroup: 65532 in deploy/kubernetes/arca-console.yaml — the same mechanism Kubernetes provides natively. Note that 65532 is the distroless nobody GID and is documented as "the identity Arca runs under", not as a privilege boundary. The TLS guide gained a File Permissions section stating the rule once (group-readable, never other-readable), the expected mode of each file, the Compose and Kubernetes snippets, and — the part that actually bites in production — a certbot --deploy-hook example, because a renewal rewrites privkey.pem as root:root 0600 and silently reverts a one-time chgrp, breaking TLS 60-90 days after the change that caused it. setfacl -m g:65532:r is documented as the alternative where group ownership cannot be changed.

Added

• publiccode.yml — the metadata file that describes Arca to the Developers Italia open source software catalogue, written against version 0.7.0 of the publiccode.yml standard and validated with the official parser (italia/publiccode-parser-go). It declares the software as a stable standalone backend owned by DXC Technology, under AGPL-3.0-or-later, and carries a user-facing description, the feature list, the documentation links and the console screenshots. Two of its keys, softwareVersion and releaseDate, go stale at every release and are now part of the version-location checklist in RELEASING.md; releaseDate in particular holds no version string, so the usual "grep for the old version" sweep does not catch it. Note that the catalogue's crawler only scans repositories registered against a publisher, so the file alone does not list Arca in the catalogue.

• AI-generated code disclosure. A new NOTICE file at the repository root states that substantially all of this repository - source code, tests, configuration, scripts and documentation - was generated with the assistance of AI coding tools, under human direction and review, and that the disclosure does not alter the AGPL-3.0-or-later terms. The notice is surfaced wherever a reader is likely to look rather than repeated in every one of the ~200 source files: an info callout in the License section of README.md and of the documentation home page, the MkDocs copyright footer so it renders on every documentation page, a //! line in each of the five crate roots (so it also appears in cargo doc output), and a comment in console/index.html and console/js/app.js.

• Trivy filesystem-scan configuration (trivy.yaml, trivy-secret.yaml, .trivyignore) — trivy fs . previously reported a fixed root/test credential (AKIA5B6BSHJA8CIHZSVG and its secret key) used throughout tests/, the Compose test services, and the console login placeholder, as 9 leaked-secret findings; it is a value Arca itself generates for local containers, never a real AWS credential, so trivy-secret.yaml now allow-lists it by exact value wherever it appears. The scan also reported the 4 rustls-webpki advisories already tracked and dismissed as unreachable in TD-024; .trivyignore suppresses those same 4 GHSA IDs, pointing back at that entry. Two remaining findings — the console Kubernetes manifest and a handful of test-only Dockerfiles — are real, minor gaps left visible and tracked as TD-027/TD-028 rather than suppressed.

• bin/test tls-permissions — a self-contained suite for the permission semantics above, because nothing else could catch a regression in them. The existing TLS tests run against bind mounts, and Docker Desktop on macOS fakes bind-mount ownership: a 0600 key owned by the host user appears owned by whatever UID the container runs as and reads fine, so any uid/gid assertion built on one would pass vacuously on a developer Mac and fail only on a real Linux host. The new suite stages its material in a named Docker volume, where ownership is real on every host, and runs nine checks: the four modes and the owning GID that arca tls generate writes, seen from inside a container rather than from the host; Arca (65532) serving HTTPS from the 0640 key; the console (uid 100) serving HTTPS with the service GID; and — so the suite has teeth — the console without it failing to enable TLS, plus the bare read of the key failing and succeeding in the same two configurations.

0.30.1 — 2026-09-14

Changed

• Base images of every distributed image are now pinned to exact versions. rust:alpine → rust:1.98.1-alpine3.24 (builder, and the dev stage that inherits it), debian:stable-slim → debian:13.6-slim (development runtime) and alpine → alpine:3.24.1 (console); production stays FROM scratch, which has no tag to pin. Each pin is digest-identical to what the floating tag resolved to when it was set, so this changes nothing about the images today — it stops them changing underneath us tomorrow, which matters because the in-Dockerfile package pins (libcrypto3>=3.5.8-r0, libssl3>=3.5.8-r0, libuuid>=2.42.3-r1) are each relative to a specific base image version and go stale silently when the base moves. The Rust tag deliberately keeps its -alpine3.24 suffix, which fixes the apk repository branch the builder's C dependencies (ring, rdkafka) come from, not just the compiler version; Debian uses the numeric point release rather than stable-slim because the numeric tag also pins the apt suite to the codename (trixie), so apt-get install in that stage cannot silently cross a Debian major release. Note that pinning the Alpine rootfs does not freeze the apk index: the v3.24 branch keeps publishing updates, which is why alpine:3.24.1 still installs openssl 3.5.7-r0 while the pinned constraint pulls 3.5.8-r0 from the repository. Images used only for testing and tooling (python:alpine, python:3-slim, the Playwright screenshot image, and the third-party services in the connector/KMS/cluster compose overlays) stay on floating tags on purpose: they are never distributed. AGENTS.md gained a "Base Image Pinning and Upgrades" section with the upgrade procedure — how to find and verify a new tag, and, critically, how to re-check every package pin against the new base and remove the ones it has overtaken instead of leaving a constraint whose CVE list is no longer true. The FROM lines are the single source of truth for the tags in force: the documentation states the pinning principle and points at the Dockerfiles rather than repeating version numbers that would need rewriting at every bump, and this changelog is the only file where exact versions are recorded.

Security

• Console image rebuilt on plain alpine + the nginx package instead of nginx:alpine. The official image ships the image-filter, xslt, geoip, njs and acme modules, none of which the console loads (it serves static files and has no load_module directive anywhere), and they drag in libgd → libxpm → libxt → libsm → libuuid along with tiff, freetype, libx11, libavif, libxml2, libxslt, curl and geoip. That transitive libuuid (util-linux) is what a container scan flagged: CVE-2026-53612, CVE-2026-53613, CVE-2026-53614, CVE-2026-76642, CVE-2026-78408, CVE-2026-78409, CVE-2026-78410 (High) and CVE-2026-27456 (Medium), all against 2.42.1-r0. Rather than upgrade the package, the whole unused dependency chain is gone: the image drops from 70 packages / ~65 MB to 18 packages / ~13 MB, and libuuid is no longer installed at all, so future util-linux advisories cannot reach the console. Alpine's nginx includes /etc/nginx/http.d/*.conf rather than conf.d, so the server config and the entrypoint's TLS rewrite moved to that path. Arca's own image is unaffected (FROM scratch, no OS packages).

• All four images now scan clean (trivy image, 0 findings). Two gaps remained after the rebuilds above. The console's base still shipped openssl 3.5.7-r0, leaving 20 findings (libcrypto3/libssl3, fixed in 3.5.8-r0) that the previous nginx:alpine base carried too — the earlier scan had simply not listed them, reporting only libuuid. And both Python images shipped pip, whose vendored msgpack and setuptools copies trail their own advisories (GHSA-6v7p-g79w-8964, CVE-2025-47273, CVE-2026-59890); nothing needs pip at runtime, so it is removed once the requirements are installed. Remediation is by pinned point upgrade, never a blanket apk upgrade: each Dockerfile names the individual package and the advisory that forced it (apk add 'libcrypto3>=3.5.8-r0' 'libssl3>=3.5.8-r0', apk add 'libuuid>=2.42.3-r1'), so a build pulls a known set of versions and the diff records why. >= rather than = because Alpine drops superseded versions from its repo index, which would turn an exact pin into a build failure the day a newer build lands. Where no fix exists yet the finding is accepted and tracked in TECH_DEBT.md rather than papered over with a blanket upgrade. For reference, trivy on the pre-change console reported 52 findings across curl, libcurl, libcrypto3, libssl3, libuuid and libexpat. The console image grows 12.6 → 18.4 MB (replacing base packages rewrites them in a new layer); still well under a third of the original 65.5 MB.

• Connector-test receiver images rebuilt on the same hygiene rules. webhook-receiver and grpc-receiver carried the same libuuid advisories. Unlike the console, the dependency is genuine there — CPython's _uuid extension links libuuid.so.1 — so the package is upgraded (a pinned apk add 'libuuid>=2.42.3-r1', from the base image's 2.42.1-r0) rather than removed, and both images move to the current python:alpine. grpc-receiver additionally dropped the gcc/g++/musl-dev/libffi-dev/linux-headers toolchain it kept in the runtime layer: musl wheels now exist for grpcio, so pip install --only-binary=:all: never compiles from source, and stub generation moved to a builder stage that keeps grpcio-tools out of the runtime — 820 MB → 90 MB. grpcio 1.66.2 → 1.84.0 and protobuf 5.29.6 → 7.36.1 along the way. Neither image is ever distributed; both exist only for bin/test notifications and bin/test connectors grpc, which pass unchanged (20 and 7 tests). The webhook-receiver compose service also gained an explicit image: arca-webhook-receiver, matching every other locally built service — without it compose derived the name from the project directory and an image built under the old name lingered as an untracked orphan, which is how a stale copy stayed on disk long enough to be scanned.

Fixed

• Console container reported unhealthy in every mode. The HEALTHCHECK probed http://localhost:80/, and busybox wget resolves localhost to ::1 first while nginx listens on IPv4 only, so the probe was always refused. It now targets 127.0.0.1 and, because TLS mode replaces the server config with an HTTPS-only one that stops answering on port 80, falls back to https://127.0.0.1:443/ (certificate validation skipped: a liveness probe, not an integrity check). Verified healthy in both plain-HTTP and TLS modes.

• Cluster 507 test could not start its third node. The InsufficientStorage integration phase replaces arca-3's data volume with a 16 MiB tmpfs, and Docker gives a tmpfs the familiar world-writable 1777 only when it creates the mount point — mounted over a directory the image already ships (/data, mode 755) it inherits root:root 755 instead. That was harmless while the images ran as root, but since they were switched to a non-root user in 0.30.0 the node died at startup with unable to open database file: /data/arca.db, haproxy then failed to resolve the missing container, and the phase timed out waiting for three live nodes. The overlay now sets the mode explicitly. Note the YAML octal prefix (mode: 0o1777): Compose reads a bare 1777 as decimal and mounts the tmpfs as mode 3361, which fails the same way. Test-harness only, no change to any distributed image; the perf overlay's tmpfs (a local-driver volume rather than a bind-style tmpfs mount) was checked and is unaffected.

0.30.0 — 2026-09-01

Fixed

• Conditional writes (If-Match, If-None-Match, x-amz-if-match-last-modified-time, x-amz-if-match-size) are now atomic compare-and-swap, not check-then-commit. Every conditional-write path evaluated the precondition against a snapshot read long before the write actually committed — for PutObject and CompleteMultipartUpload the gap spanned the entire body upload / part assembly, so two concurrent conditional writes to the same key both routinely passed their check and both got 200 OK, silently degrading to last-writer-wins on exactly the primitive S3 clients use for coordination. DeleteObject (plain and version-specific) and the DeleteObjects batch had the same shape with a narrower window. Preconditions are now re-evaluated authoritatively inside the same transaction that performs the write, on both backends (SQLite and PostgreSQL): MetadataStore gained put_object_if, delete_object_if and delete_object_version_if, and every handler commits through them instead of a plain read-then-write. The early snapshot check is kept as a fast-path optimisation (fail fast before uploading a doomed body) but is no longer authoritative. Behaviour change: a concurrent conditional write that previously raced to 200 OK now correctly receives 412 PreconditionFailed — this is the fix, not a regression; no correct client could depend on the old racy 200.
• DeleteObjects batch conditional deletes on a specific version checked the bucket's latest object instead of the targeted version. A version-specific entry with an ETag/LastModifiedTime/Size precondition was validated against get_latest_object, so deleting an older version whose ETag matched could be wrongly refused (or, on a versioned bucket with tags, another version's stale metadata could wrongly pass). It now checks the exact version being deleted, matching the plain (non-batch) DeleteObject path.
• PostgreSQL backend: conditional writes were not actually serialized under concurrency. The new authoritative CAS check (above) ran its precondition read before next_object_seq() — the single statement in each transaction that actually contends under PostgreSQL's default READ COMMITTED isolation, since a plain SELECT does not block concurrent transactions. Concurrent writers for the same key could all read the same pre-write snapshot, all pass their precondition, and commit one after another, reproducing the exact check-then-commit race the fix above exists to close — on PostgreSQL only (SQLite's single-connection actor already serialized every statement regardless of order). Fixed by taking the seq first in put_object_if, delete_object_if and delete_object_version_if, turning it into a serialization gate so each transaction's precondition check is guaranteed to see genuinely current state. This also closes a gap in delete_object_if's non-cluster unversioned-delete path, which previously took no serializing statement at all.
• Known limitation, tracked as TD-025: in a cluster, conditional-write atomicity is exact per node, not cluster-wide — put_object/delete_object commit locally then fan out, with no per-key authoritative node to arbitrate across peers. Two conditional writes racing the same base ETag on two different nodes can both be accepted, converging afterward by last-writer-wins. Mitigated by pinning a key's conditional-write traffic to one node via sticky load-balancer routing; see the HA guide.

Added

• license field in the Cargo manifests. The workspace declared no licence at all in its package metadata, so cargo metadata, cargo package and any downstream tooling reading the manifests saw Arca as unlicensed even though the repository has always carried the licence file and the notices in README.md. [workspace.package] now sets license = "AGPL-3.0-or-later" and all five crates inherit it with license.workspace = true. Metadata only: no code, dependency or behaviour change.

Security

• Dependency vulnerability remediation. quick-xml 0.40.1 → 0.41.0 (RUSTSEC-2026-0194, RUSTSEC-2026-0195), h2 0.4.14 → 0.4.17 (RUSTSEC-2026-0258, transitive via hyper), crossbeam-epoch 0.9.18 → 0.9.20 (RUSTSEC-2026-0204, transitive via moka), event-listener 5.4.1 → 5.4.2 (RUSTSEC-2026-0221, transitive via async-lock/lapin/moka/sqlx-core). The grpc-receiver connector-test tooling's protobuf bumped 5.28.2 → 5.29.6 (CVE-2025-4565, CVE-2026-0994) — library-only, never part of the distributed arca binary. rustls-webpki 0.102.8 (4 published advisories, transitive via rumqttc's MQTT connector) remains unfixed: rumqttc 0.25.1, the latest published release, hard-pins rustls-webpki = "^0.102.8" with no newer version available — tracked as TD-024 rather than patched with an unvetted override.
• All Docker images now run as non-root by default. The production (scratch) and development (debian:stable-slim) Arca runtimes, the nginx-based console, and every auxiliary test/tooling image (Dockerfile.test, Dockerfile.perf, grpc-receiver, webhook-receiver, s3-tests) drop root and run as a fixed non-root user — except s3-tests, which runs as the invoking host user (docker/docker-compose.yml's S3TESTS_UID/S3TESTS_GID, set by bin/s3-tests) since it writes results into a bind-mounted host directory. HEALTHCHECK instructions were added to every long-running server image that lacked one. See the TLS guide for a permissions caveat this introduces for bind-mounted certificate files on Linux hosts.
• Kubernetes manifests hardened. deploy/kubernetes/arca-cluster.yaml and arca-console.yaml gained pod- and container-level securityContext: runAsNonRoot, explicit runAsUser/runAsGroup matching each image's built-in user, dropped Linux capabilities, seccompProfile: RuntimeDefault, and (for the arca container, whose only writable path is the /data volume) readOnlyRootFilesystem: true.
• THIRD-PARTY-NOTICES.md regenerated. It had silently gone stale since the license field was added in a prior commit: cargo-about's rule for skipping Arca's own crates keys on publish = false, which that commit never set, so every generation attempt since has been failing at the root-cause level rather than just being out of date. Fixed by adding publish = false to [workspace.package] and publish.workspace = true to all five crates.
• Bootstrap credential no longer uses AWS's published SigV4 documentation example. Every dev/test default (Docker Compose, s3-tests config, console login-form placeholder, screenshot automation, storage unit-test fixtures, integration/perf test defaults, and the admin-API docs example) used AKIAIOSFODNN7EXAMPLE / wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY, which secret scanners and password managers flag as a known/compromised credential even though it was never a real secret. Replaced with a freshly generated random credential everywhere it served as a bootstrap default. The same string still appears, deliberately unchanged, in crates/arca-auth's SigV4 conformance tests, where it is AWS's own published test vector with hardcoded expected signatures rather than a credential.

0.29.0 — 2026-07-28

Security

• Anti-replay window on header-signed S3 and admin requests. The Authorization-header SigV4 paths (S3 API and /admin/*) never validated x-amz-date against the server clock, so a captured signed request — sniffed off a plain-HTTP hop, or lifted from an access log or proxy trace — stayed replayable forever. Both paths now reject a request whose x-amz-date lies more than 15 minutes from the server clock (the AWS SigV4 convention, and the same window /cluster/v1/* already enforced since 0.26.0), returning 403 AccessDenied. The signature covers x-amz-date, so an attacker cannot re-date a captured request without the secret key. Presigned (query-string) requests were already bounded by their own X-Amz-Expires. Clients whose clock drifts beyond ±15 minutes from the server's will now be refused — run NTP.
• Auth fails closed when a credential's user no longer exists. A valid credential whose user_id did not resolve to a users row was silently promoted to a synthetic root identity (a leftover backward-compatibility accommodation for pre-users-table credentials), which meant a dangling credential — one left behind after its user was deleted — authenticated as root and bypassed all policy evaluation. Both the S3 and admin auth paths now deny such a request (403 AccessDenied), and a genuine metadata-backend failure is a 500 InternalError instead of being collapsed into the same root fallback. To remove the source of dangling credentials, UserStore::delete_user now deletes the user's credentials in the same transaction on both backends (SQLite and PostgreSQL), so a credential can no longer outlive its user. This is not merely defensive: DELETE /admin/users/{id} refuses with 409 Conflict while credentials exist (unchanged), but two paths reach the store without that check and could mint a dangling credential that authenticated as root — the arca user delete CLI, which validated only "not root", and the cluster control-plane apply of a UserDelete op, where the deleting node's check cannot see a credential another node created concurrently. In the cluster the cascade is also what converges the two nodes on "user gone, its keys gone" instead of leaving one peer with a permanently denied key. arca user delete now prints each access key it revokes rather than dropping them silently. Credentials predating the users table are unaffected: the migration that added credentials.user_id backfilled them to the root user, which exists.

Fixed

• Admin access follows the user, not the credential. A credential carried its own admin boolean, and the /admin/* gate required BOTH a root user AND credential.admin = true. Two paths minted credentials that could never satisfy it: POST /admin/users/{id}/credentials (the console's "create credential" flow) hardcoded admin = false, so a second credential created on the root user signed in as a plain user with no admin reach at all, while POST /admin/credentials honoured the request's admin field — the same identity therefore got different privileges depending on which endpoint created its key. Privileges are a property of the identity, so the flag is gone: the gate now allows a request when the credential's user is_root, or when the user's effective grants allow the arca:* action mapped from the admin path (which also, for the first time, lets a non-root user reach the specific admin endpoints its grants cover). The credentials.admin column is dropped (SQLite migration v26 / PostgreSQL 0014), admin is removed from the Admin API request and response bodies (replaced by user_id, the owning user), from arca credential add --admin, and from the console's create-credential form and card badge (now the owner's username, linking to that user). An incoming admin field is silently ignored rather than rejected, so an older cluster peer can still replicate credentials to an upgraded node. The reverse direction is not supported: a 0.29 node replicates credentials without the field, and a pre-0.29 peer requires it, so a cluster must not be left with mixed versions across this release. The lockout guards on delete/deactivate now key on "last active credential belonging to a root user".

Removed

• admin flag on credentials. Dropped from the core type, the database (SQLite migration v26 / PostgreSQL 0014), the Admin API request/response bodies, the arca credential add CLI and the console. See the entry above for the reasoning; incoming admin fields are ignored rather than rejected, so older clients and export files keep working, but a cluster must not be left running mixed versions across this release.

0.28.0 — 2026-07-24

Added

• Guided topology transition migrate-topology (Phase 30, M4). Scripts the two in-place single-node ⇄ HA-cluster topology changes so the operator does not hand-assemble the config and the small DB chores. The cluster is fully replicated (not sharded), so neither direction redistributes data. arca migrate-topology --to-cluster [--output <file>] turns a standalone instance into the first node of a new cluster: it refuses if already clustered, reconciles this node's object_seq write counter to MAX(seq) (so the first clustered write cannot skip a pre-cluster object), and prints a ready-to-paste [cluster] stanza with a generated cluster_id, a strong random secret (32 bytes hex, reusing the ring-backed RNG), mode = "quorum", cluster_size = 3, discovery = "mdns", and commented templates for static/DNS discovery and [cluster.tls] (pointing the operator at arca tls generate-cluster for the inter-node CA). Restarting with the stanza enables cluster (tombstone) mode automatically; empty joiners then converge via anti-entropy — no data copy. arca migrate-topology --to-single --force collapses a cluster back to one node on the surviving authoritative node: it purges the cluster-only state (object tombstones + control-plane tombstones) and VACUUMs SQLite (PostgreSQL autovacuum needs none), then the operator strips [cluster] and restarts standalone; --force is the explicit confirmation that every peer is synced and stopped (collapsing while a peer is behind loses its un-replicated writes). CLI-only by design (no console job): both directions are inherently operator+restart actions an online job cannot perform. New runbook in the HA guide.
• Metadata-backend migration migrate-db (Phase 30, M3). Copy ALL metadata from one backend to the other (SQLite ⇄ PostgreSQL) in place, without re-uploading data: blob files are filesystem-resident and untouched, only the metadata database moves. After a run the operator switches metadata_backend (and [storage.postgres]) in the config and restarts onto the new backend. Available two ways: an offline CLI escape hatch arca migrate-db --to <sqlite|postgres> [--force] (server stopped, no writes race the copy), and an online maintenance-mode job (type: "migrate-db", params { "target", "force" }) that drains the S3 API on its node while it runs. The copier is a single generic, type-aware column-walker driven by a static description of all 24 data tables (name + ordered columns + each column's logical kind) rather than per-table structs; it reads/writes every backend's native representation (SQLite TEXT timestamps ⇄ PG TIMESTAMPTZ, SQLite INTEGER booleans ⇄ PG BOOLEAN, SQLite TEXT JSON ⇄ PG JSONB), copies tables in foreign-key-safe order, and reconciles source/destination row counts per table (a mismatch aborts naming the table). Refuses a destination that already holds operator data unless --force (which fully replaces it); the always-present schema baseline (root user, built-in grants, seq counter) is ignored. The object_seq write counter and the audit/metrics surrogate sequences are migrated/reseeded so writes on the new backend don't collide.
• In-place re-encryption jobs (Phase 30, M2). Encrypt existing plaintext objects to SSE-S3 (AES-256), or decrypt them back, without re-uploading and without migrating to a new instance. Runs as a maintenance job (console or admin API, type: "encrypt" / "decrypt") in either live mode (copy-on-write + compare-and-swap + an optional byte-rate throttle, zero downtime) or maintenance mode (S3 drained on the node, full speed). Copy-on-write is used in both modes: a new encrypted blob is written and the object row is atomically swapped to it, so concurrent readers always see a consistent blob and the cluster's blob-id immutability is preserved — the new blob reaches peers via read-repair / anti-entropy and the orphaned old blob is reclaimed by GC. The ETag and Last-Modified are preserved (no lifecycle-clock impact); cluster convergence rides a dedicated content_updated_at LWW dimension (separate from Object Lock's lock_updated_at) so a re-encrypted row is adopted deterministically without ever clobbering a concurrent lock change, and vice versa. SSE-C objects (customer-held keys) and multipart/composite objects are skipped (TD-014). An offline CLI escape hatch (arca encrypt-existing / arca decrypt-existing) performs the same transform in place for disaster recovery when the server is stopped. Console Maintenance page gains the encrypt/decrypt launchers (bucket/prefix filter, throttle).
• Maintenance jobs subsystem (Phase 30, M1). Long-running, operator-launched maintenance operations are tracked as maintenance_jobs rows (SQLite migration v24 / PostgreSQL 0012) so progress survives restarts and is observable from the console. A single background worker processes one job at a time, committing progress per item so pause / cancel / restart are always safe; in a cluster the worker is leader-gated (only the worker-leader node runs it). A job in maintenance mode drains the S3 API on its node for its lifetime — every S3 request is refused with 503 ServiceUnavailable (+ Retry-After) AND the health endpoint reports draining so the load balancer stops routing — while the admin API and worker stay live (the graceful-shutdown drain stays non-blocking, by design). New JSON admin API under /admin/maintenance/jobs (create / list / get-with-logs / pause / resume / cancel / clear-history, single-job lock, state-guarded transitions) and a console Maintenance page (launch panel, live active-job card with progress / ETA / rate, log panel, paginated job history with a clear-history action). The re-encryption (M2) and migration (M3/M4) job types build on this foundation.

Fixed

• Re-encryption no longer clobbers Object Lock state during cluster convergence (Phase 30). In-place re-encryption changes an object's blob/algorithm/key WITHOUT bumping last_modified — exactly like a retention / legal-hold change. Both used to reuse the single lock_updated_at LWW dimension, so during anti-entropy a re-encryption could revert a newer lock state (or a lock change revert a newer re-encryption) whenever the two raced on different nodes. Re-encryption now stamps its own content_updated_at dimension (SQLite migration v25 / PostgreSQL 0013), and apply_remote_object merges the content column group (blob/algorithm/key, by content_updated_at) and the lock column group (retention/legal-hold, by lock_updated_at) independently, so the two converge without ever overwriting each other regardless of apply order.
• migrate-db maintenance job requires maintenance mode. The online migrate-db job copies every metadata table across separate read transactions, so a concurrent write mid-copy would land in some already-dumped tables but not others, producing a torn destination that per-table row-count reconciliation cannot detect. A live-mode request is now rejected with a clear message pointing at maintenance mode (or the offline CLI); previously it was accepted and could silently produce an incomplete copy.

0.27.1 — 2026-07-07

Added

• Per-run blob GC log summary. Each blob GC pass (the single-node background worker and the cluster anti-entropy worker) now emits a concise INFO summary — blob GC pass started and blob GC pass complete scanned=… candidates=… reclaimed=… failed=… elapsed_ms=… — so a scheduled run is visible even when it reclaims nothing (previously only a non-zero reclaim was logged). A pass skipped by the fail-safe (an enumeration error) is logged at WARN. arca gc likewise prints the scanned/orphan counts.

0.27.0 — 2026-07-06

Added

• arca gc — reclaim orphaned blob files on a single node. Orphan blobs (on disk, referenced by no live object row, in-progress part, or non-orphan composite sidecar) accumulate from interrupted uploads, overwrites, crashes between the metadata and blob delete, and swallowed blob-delete failures. Until now the only reclamation path was the cluster anti-entropy worker, which is spawned only when clustering is enabled — a single-node deployment never reclaimed them and they grew unbounded (arca fsck could report them but not remove them). The new offline command reuses the same composite-aware, fail-safe selection as the cluster worker: it previews by default (dry run) and deletes only with --reclaim. --grace-seconds (default 86400) protects freshly-written blobs whose object row may not be committed yet; drop it to 0 when the server is stopped. Schedule it from cron or run it on demand. The shared selection logic now lives in one place (blob_gc::collect_reclaimable_blobs), used by both the CLI and the anti-entropy worker.
• Opt-in single-node blob GC worker. For hands-off operation, a background worker can reclaim orphans on a schedule without an external cron job. Off by default; enable with [storage] blob_gc_enabled = true (plus blob_gc_interval_seconds, default 3600, and blob_gc_grace_seconds, default 86400). It is skipped under clustering (the anti-entropy worker already reclaims orphans) and exposes the arca_blobs_reclaimed_total metric. On very large stores prefer cron-ing arca gc so the full-store scan runs out of the serving process.
• arca_blob_delete_failures_total metric. Counts blob-delete failures during object deletion. When a blob delete fails after its metadata row is already gone, the object is genuinely deleted (a subsequent GET returns 404), so DeleteObjects/DeleteObject still report success per S3 semantics — but an orphan blob is left on disk. This counter makes the orphan-creation rate observable (previously visible only in warn! logs), so operators can see whether arca gc needs to run.

Fixed

• Console: shared bucket links now land on the bucket after login. Opening a deep-link URL (e.g. a bucket link sent by another user) and then logging in previously always dropped you on the dashboard (admin) or the buckets list (non-admin), discarding the link. The console now preserves the deep link and routes to it after login, only defaulting to home when no destination was given. If the target bucket is inaccessible or missing, the user is bounced home with an explanatory toast — 403 → "You don't have access to bucket …", 404 → "Bucket … does not exist". Because S3 returns no error for a non-existent prefix, a missing sub-folder (a non-empty prefix whose listing is entirely empty, with no folder marker) is inferred client-side and returns the user to the bucket root with a "Folder … does not exist" toast; genuinely empty folders still open normally.

0.26.1 — 2026-06-12

Security

• time 0.3.41 → 0.3.47 — CVE-2026-25727 fix picked up (resolves TD-011). The Docker builder moved from rust:1.85-alpine to rust:alpine (currently Rust 1.96), unblocking the time versions that contain the stack-exhaustion fix; the MSRV pins (home, serde_with, darling) became unnecessary and were removed from the Dockerfile. time 0.3.48 remains excluded for a non-security reason (a coherence clash with rcgen — tracked as TD-017).
• rustls-pemfile retired (resolves TD-012, RUSTSEC-2025-0134). PEM parsing migrated to rustls-pki-types (PemObject), which is maintained as part of the rustls project; the unmaintained dependency is gone from the workspace.

Changed

• Full dependency refresh — every crate brought to its latest version. The lockfile had never been systematically refreshed since the project started; this pass updates all ~540 locked packages and bumps 23 declared majors/minors, with the code migrated where APIs changed:
  • sqlx 0.8 → 0.9 (PostgreSQL/MySQL): dynamically composed queries now go through the new AssertSqlSafe audit opt-in (all of Arca's dynamic SQL is fixed fragments + bind parameters, audited by construction).
  • redis 0.27 → 1.2: connection-info overrides moved to the new builder setters.
  • lapin 2.5 → 4.10 (AMQP): native tokio integration — the tokio-executor-trait/tokio-reactor-trait shims are gone from the workspace.
  • tonic/prost 0.12/0.13 → 0.14 (gRPC): prost codegen moved to the new tonic-prost/tonic-prost-build crates; TLS features renamed (tls-ring).
  • quick-xml 0.36 → 0.40: entity references now arrive as separate reader events — the manual DeleteObjects parser accumulates text fragments and resolves character/predefined entities itself.
  • RustCrypto digest 0.11 ecosystem (sha2 0.11, md-5 0.11, hmac 0.13), rand 0.10, rcgen 0.14 (issuer-based signing API), reqwest 0.13 (TLS roots via the platform verifier), brotli 8, lz4_flex 0.13, governor 0.10, async-nats 0.49, rumqttc 0.25, rdkafka 0.39, mongodb 3.7, thiserror 2, toml 1.
  • Known constraints handled: rusqlite stays at 0.37 (the latest tokio-rusqlite requires ^0.37); time pinned at 0.3.47 (TD-017, see Security).
  • The integration-test image moved to python:3-slim; its Python dependencies and the console's CDN libraries already float to latest by construction.
• Process-default crypto provider hardened. The refreshed graph compiles more than one rustls crypto provider (lapin's TLS stack brings aws-lc-rs alongside Arca's ring), so every constructor that builds TLS machinery (cluster client, membership prober, Vault client, replicator, webhook/Elasticsearch connectors, the rustls config loader) now installs the ring process default idempotently — previously only main() did, which left non-main entry points (and unit tests) able to hit the multi-provider ambiguity panic.

Fixed

• Generated certificates now pass modern strict TLS verification. OpenSSL in strict mode — the default for Python 3.13+ clients — refuses CA-issued certificates without an Authority Key Identifier and CA certificates without the KeyUsage (keyCertSign) extension. arca tls generate and arca tls generate-cluster now emit both, so freshly minted material works with current AWS SDKs, curl and browsers out of the box. (Surfaced by the refreshed integration-test image; previously generated certificates keep working with clients that do not enforce strict mode — regenerate them to be future-proof.)
• Cold Docker builds produced empty workspace crates. The builder's dependency-caching layer compiles dummy crates whose cargo fingerprints are newer than the COPY'd real sources' mtimes, so on a cold build (fresh cache, first build, right after docker builder prune) cargo considered the real sources unchanged and linked the empty dummies — hundreds of unresolved import errors. The real sources are now touched after the COPY; warm builds are unaffected (unchanged context = layer-cache hits all the way). Reproduced on both the old and new toolchain — a latent defect exposed by the cache prune, not by the dependency refresh.
• Integration tests aligned with current S3/SDK semantics (no server changes — the server was right): checksum assertions now request ChecksumMode=ENABLED on HEAD/GET as S3 requires; suspending versioning on an Object-Lock bucket expects AWS's real 409 InvalidBucketState (not 400); the notification event-log test signs with S3SigV4Auth like every other admin test (the generic SigV4Auth signs the payload hash over http but never sends x-amz-content-sha256, which the server rightly rejects).

0.26.0 — 2026-06-12

Security

• Verified inter-node mutual TLS with a cluster CA (Phase 29.1 R4, review §3.7(C), decision H12 — resolves TD-015). Inter-node TLS no longer accepts invalid certificates: the danger_accept_invalid_certs accommodation is gone. A new [cluster.tls] section distributes an operator-owned cluster CA plus this node's CA-signed cert/key (mint everything with the new arca tls generate-cluster): outbound cluster clients verify peer certificates against the CA and present the node cert as their client identity, while the listener requests client certificates — optional at the TLS layer, because S3 clients share the single port — and /cluster/v1/* refuses any request whose connection did not present one signed by the cluster CA. This is the independent second factor on top of the R3 challenge-response (an attacker now needs the secret AND a CA-signed key), and it closes the passive-sniffing / active-MITM exposure on the cluster network. Breaking: a cluster running over HTTPS now refuses to start without [cluster.tls] (fail closed — there is no insecure fallback); plain-HTTP clusters are unaffected (the R3 peer authentication remains their baseline; use TLS on any network you do not fully trust).
• Anti-replay window on cluster requests (Phase 29.1 R4, review §3.1/§3.7(C)). /cluster/v1/* now rejects signed requests whose x-amz-date lies more than 15 minutes from the node's clock (the SigV4 convention). Previously a captured signed request — e.g. a blob GET sniffed off a plain-HTTP segment — remained replayable forever; the signature covers the timestamp, so it cannot be refreshed without the secret. Cluster nodes already require NTP.
• Cluster secret strength enforced (Phase 29.1 R4, review M5/§3.7(B)). Startup now refuses a [cluster].secret shorter than 16 characters and the placeholders shipped in the reference configs (dev-cluster-secret-change-me, CHANGEME-CLUSTER-SECRET), with a warning when the secret looks low-entropy — generate one with openssl rand -hex 32. The secret keys all inter-node authentication (SigV4 + the peer challenge), so a guessable value hands an attacker the whole cluster. The dev/test cluster configs now ship a floor-compliant dev secret; deployments still on a placeholder fail at startup with a clear message until it is replaced.
• Cluster endpoint defense-in-depth (Phase 29.1 R4, review §3.4, M6, D12.1). The JSON cluster endpoints (object, object/delete, op, manifest, control-snapshot, ping) now cap request bodies at an explicit 2 MiB (blob transfer streams to disk and stays uncapped); the blob_id path parameter must parse as a UUID before it reaches the blob layer, where it becomes a file name; and the replication receive side drops node-local server_config keys (the node identity), so not even a buggy or malicious peer can rewrite who a node is — the filter previously existed only sender-side.
• Cluster peers are now authenticated, not just cluster requests (Phase 29.1 R3, review §3.7(A)/D1, decision H12). Inbound /cluster/v1/* requests were always SigV4-verified, but the outbound direction trusted any endpoint that answered a public health probe: a rogue process on the cluster network registering itself over mDNS with the right cluster_id would be counted alive, receive the replication fan-out of every new write with no knowledge of the cluster secret, and sustain a ghost write-quorum. Membership now probes peers with a signed GET /cluster/v1/ping carrying a fresh nonce, and a peer is authenticated only if its response proves possession of the cluster secret (nonce_mac = HMAC(secret, nonce) — answering 200 proves nothing, a recorded MAC is useless against a fresh nonce). Replication fan-out (objects, blobs, control plane), anti-entropy pulls (manifest, control snapshot, blob repair — also a data-poisoning vector), the write-quorum count and the cluster-minimum capacity now all consider eligible peers only: alive AND authenticated AND config-aligned. Legacy pre-upgrade peers (no ping route) stay visible as alive but are excluded until upgraded — complete a rolling upgrade promptly. On plain-HTTP clusters this closes the rogue-peer attack; passive sniffing / active MITM still require the verified inter-node TLS arriving with R4.
• Public cluster health minimized (Phase 29.1 R3, review §3.5/§3.7(B)). The unauthenticated GET /cluster/v1/health now returns only {status, node_id}. It previously exposed the cluster config fingerprint — an offline brute-force oracle for the cluster secret (truncated SHA-256 over mostly-guessable inputs) — plus disk capacity. That detail moved to the authenticated ping, where only secret holders can read it.

Added

• Write-aware health check (Phase 29.1 R9, review D5). GET /admin/health?writable=1 answers 503 {"status":"read_only"} (with Retry-After) while the node's cluster write gate is closed — write quorum lost or cluster size exceeded — and 200 otherwise (single-node deployments and available mode are always writable). A node that lost the quorum deliberately stays 200 on the default health check (it serves every read); pointing a separate load-balancer write pool at the write-aware variant routes write methods only to nodes that would accept them, instead of letting ~1/N of the writes fail with a client-visible 503 while the cluster is degraded. Both shipped HAProxy configs carry the write pool as a commented reference backend.
• [cluster] tombstone_grace_seconds (Phase 29.1 R9). Advanced seconds-granularity override of tombstone_grace_days (≥ 1), for tests and demos that must observe the tombstone-GC liveness guard within seconds; production deployments keep sizing the grace in days.
• Cluster operational runbooks and honest consistency docs (Phase 29.1 R9, review D3/D5/D7/D10/D11/D12.4, §5-doc). The HA guide gains an Operational Runbooks section — replacing a dead node (an empty disk is safe by construction), restoring a node from backup (and when not to: a backup older than the tombstone grace), resizing the cluster (cold procedure; why rolling cannot work), rotating the cluster secret, taking coherent backups (filesystem snapshots vs stopped-node copies, SQLite WAL caveat), and forming a cluster from non-empty nodes (a silent union+LWW merge — discouraged, with the safe alternative). The consistency documentation now states plainly what was previously only implied: in available mode concurrent writes to the same key are both acknowledged but only the LWW winner survives (the loser is discarded with no error to the client) and durability is single-copy until the fan-out lands (RPO > 0); in quorum mode reads are not linearizable (R = 1 by design) and a minority node serves reads that age for a partition's whole duration; the WORM/Object-Lock trust model in a cluster (verbatim remote applies; immutability rests on the secret, the cluster CA and OS access on every node); and the clock-skew test gap is declared instead of implied away.
• Tombstone-GC and blob-repair integration tests (Phase 29.1 R9, review §5.4). New cluster test phase on a dedicated overlay: the proactive blob-repair sweep is proven by deleting an object's payload file straight from a node's volume and watching the file reappear (observed on the volume — a client GET would mask the sweep with lazy read-repair) and serve intact bytes; the §3.2 GC liveness guard is proven end-to-end — an object deleted while a node is down beyond the grace blocks tombstone GC on both survivors, the returning node learns the deletion (no resurrection anywhere), and the guard releases.
• Per-node admin views behind the load balancer (Phase 29.1 R8, review D6, decision H9). The audit log, metrics history, notification event log and replication journal are strictly node-local (each node records what it served), so behind a load balancer their console views showed whichever node luck routed to — with nothing saying which. The four list endpoints (GET /admin/audit, /admin/metrics/history, /admin/notifications/events, /admin/replication/journal) now accept ?node=<node_id>: the serving node proxies the query server-side to that peer over the signed cluster transport (new POST /cluster/v1/admin/* receive routes, behind the same peer authentication, mTLS enforcement and body caps as every cluster endpoint — browsers never talk to nodes directly and the cluster credential never leaves the server side). Only eligible peers are valid targets (unknown node → 404, dead/drifted → 503, unreachable → 502); every response carries a top-level node label, including the default LB path. ?node=all returns the merged view: a parallel fan-out to every eligible node, rows merged newest-first, each labeled with its source node, plus a per-source result report so a partially-failed fan-out is visible. Pagination of the merged view is per-source-page (each node is asked for the same window and the newest rows across nodes are kept) — an approximation by design, documented instead of hidden behind cross-node cursors. The console gains a node selector in all four views (hidden when clustering is off): default "This node (via LB)" with a badge always naming the node that answered, per-node options, and "All nodes" with per-row source badges; Monitoring draws one chart series per node in the merged view. Clear All and journal Retry are disabled while a node is selected — they operate on the node serving the request, not the viewed one.
• Syncing readiness gate for re-entering cluster nodes (Phase 29.1 R7, review D2). A node returning from downtime (or replaced with an empty disk) used to enter the load-balancer rotation immediately and answer 404 / partial listings for data it had not pulled yet — wrong answers given with full confidence, on ~1/N of the routed traffic, with nothing signalling it. A clustered node now tracks the completion of its first anti-entropy pass toward every eligible peer since startup, and until then the plain GET /admin/health answers 503 {"status":"syncing"} (with a Retry-After hint; a shutdown drain takes precedence), so the LB and the Kubernetes readiness probe keep it out of rotation exactly while it could mislead clients. The verbose health stays inspectable (200 with status: "syncing"); with no eligible peers the node reports ok (degraded-but-serving, there is nothing to sync from). New cluster test phase: a node restarted after downtime must hold the 503 until caught up, and the moment it reports ready the data written during its absence must already be readable on it.
• Per-peer sync state exposed (Phase 29.1 R7, review D2/M1). GET /admin/cluster now reports the node-level syncing flag plus, for every peer, this node's pull state toward it: the hwm manifest cursor, the lag still to pull (against the peer's ping-reported write cursor), last_reconcile, first_pass_done and skipped_entries. The console topology card shows a syncing notice and amber per-node sync lag / skipped indicators when non-zero.
• Restore-from-backup rewind detection (Phase 29.1 R7, review D3c). Each node advertises its object write cursor on the authenticated ping; a peer reporting a cursor below what this node already consumed from it was restored from a backup, and its post-restore writes (whose seq values re-use the rewound range) would have stayed invisible to the incremental sync until this node restarted. The high-water mark is now reset automatically (one idempotent full re-pull, logged as a warning). A freshness guard — the report must be newer than the last cursor advance — keeps sustained writes from false-alarming.
• [cluster] blob_repair_budget (Phase 29.1 R7, review M2). The proactive blob-repair sweep now attempts at most this many peer fetches per anti-entropy tick (default 100, ≥ 1) and resumes where it left off on the next tick — previously a node missing many blobs repaired them all in one pass, monopolizing the anti-entropy worker for hours on a large store. Between budget slices, lazy read-repair still covers anything a client actually asks for.
• Complete control-plane reconcile (Phase 29.1 R5, review D9 — resolves TD-016). The anti-entropy snapshot merge now covers EVERY control-plane family, not just the original five: grant attachments (user_grants, team_grants), team memberships, per-bucket config keys (versioning, encryption, …), bucket tag sets, and cluster-wide server_config settings. Each family carries a per-row LWW timestamp (migrations sqlite v22 / pg 0010) and deletions leave tombstones, so a node that was down when a grant was revoked, a membership removed or a bucket setting changed now fully self-heals at re-entry — previously those families replicated in real time only and a missed change persisted until the next write touched it. Child families are parent-filtered in the merge: rows whose parent (user, team, grant, bucket) resolved deleted are never adopted, replacing per-child cascade tombstones. Bucket tag sets reconcile as ONE entity per bucket (matching the replace-all semantics of PutBucketTagging).
• Multipart uploads reconcile across the cluster (Phase 29.1 R5, review D4). In-progress multipart uploads and their part rows are now part of the control snapshot: a node that was down during CreateMultipartUpload/UploadPart learns them at re-entry, and a multipart tombstone recorded at Complete/Abort time stops a closed upload from resurrecting off a stale peer. CompleteMultipartUpload on a node that has the part rows but is missing some part bytes (their fan-out never reached it) now repairs them from a peer before assembling instead of failing.
• SSE-C objects get cluster read-repair (Phase 29.1 R5, review §3.6). SSE-C blobs always replicated (the sidecar write ships the customer-key-encrypted bytes verbatim — peers never see the key), but a GET on a node that had the object row and not yet the bytes failed until the next anti-entropy pass, because the SSE-C read path bypassed the cluster wrapper. SSE-C reads now repair the missing bytes from a peer synchronously, like every other object.
• arca tls generate-cluster (Phase 29.1 R4). Mints the cluster CA plus one CA-signed certificate per node (--node name=san1,san2,..., SANs covering every name/IP peers dial; serverAuth + clientAuth EKUs so the same pair serves as listener certificate and inter-node client identity), and prints the [server.tls] / [cluster.tls] snippets to paste on each node. Operators with their own PKI can supply equivalent material instead.
• Zero-downtime cluster secret rotation (Phase 29.1 R4, review D3b, decision H8). New optional [cluster] secret_previous: inbound cluster authentication accepts the current secret first, then the previous one, while outbound signing, the challenge MAC and the config fingerprint always use the current secret (the ping answers with the secret the request verified under, so probes keep verifying across the window). Runbook: set secret_previous = old and secret = new on every node, rolling restart, then remove secret_previous. During the window expect transient drift flags (the fingerprint includes the secret): nodes on the new secret form the writable side once they reach a majority.
• HA Hardening plan (Phase 29.1). The remediation plan for the findings of the in-depth Phase 29 HA review is now part of the official roadmap: a new Phase 29.1 section (progress bar, dependency graph, summary table, milestone checklist) plus the full working plan — fixed design decisions, per-milestone detail, and a finding-by-finding traceability table — published as a living page on the documentation site.
• Cluster-size write guard (Phase 29.1 R3, review D3a, decision H6). In quorum mode, if MORE eligible nodes than the configured cluster_size are observed, every node refuses writes with 503 (fail-closed, no config escape hatch) and /admin/cluster reports size_exceeded: true. The write majority is derived from cluster_size, so an over-sized membership (e.g. a 4th node started with the 3-node config) would let two disjoint "majorities" accept conflicting writes — split-brain. Resizing has a documented cold procedure (R9 runbook).
• Tombstone-GC liveness guard (Phase 29.1 R3, review §3.2). Anti-entropy now skips the tombstone purge while any known peer has been unreachable beyond the grace window, with a loud warning and a tombstone_gc_blocked: true flag on /admin/cluster — purging would erase the very tombstones that returning peer needs to learn its missed deletions, resurrecting deleted data. Membership pruning (below) eventually evicts a never-returning peer so GC cannot stay blocked forever.
• Cluster membership pruning (Phase 29.1 R3, review M3). Peers unreachable beyond [cluster] peer_prune_days (default: tombstone_grace_days) are evicted from membership (and from the admin/console topology). A pruned node that later returns is re-discovered and re-synced like a new candidate; returning beyond the tombstone grace is the documented residual resurrection risk.
• /admin/cluster topology detail (Phase 29.1 R3). Per-node authenticated flag (did the node prove possession of the cluster secret on its last probe), cluster-level eligible_node_count (what the write quorum is measured against, vs. the visibility-only live_node_count), size_exceeded and tombstone_gc_blocked.
• [lifecycle] worker interval (Phase 29.1 R6). New optional [lifecycle] interval_seconds (default: 3600, the previous fixed value) controls how often the background worker evaluates bucket lifecycle rules.

Changed

• Cluster deploy references aligned (Phase 29.1 R9, review §5-deploy). The production HAProxy config, the test/demo one and the HA-guide snippet now agree: production health checks use fall 3 rise 2 (ride out a transient blip), the test/demo config keeps fall 2 rise 1 for fast convergence and says so, and both carry commented balance source (sticky read-your-writes) and write-aware-pool examples. The Kubernetes readiness probe tightens to periodSeconds: 5, failureThreshold: 2 (~10 s worst-case window of traffic to a crashed pod) and its comments now describe the syncing semantics introduced in R7.
• The lifecycle evaluator is now a cluster singleton (Phase 29.1 R6, review §3.3, decision H5). The object table is fully replicated, so every node used to evaluate the same lifecycle rules and delete the same expired objects — N× the delete fan-out, duplicate audit entries and races between concurrent deleters. Only the worker leader now runs the evaluation tick (expirations, noncurrent-version deletes, stale-multipart aborts): the node with the lowest node_id among the eligible nodes, the same authenticated-and-config-aligned predicate that gates the write quorum — so an unauthenticated rogue cannot steal the role and silence the workers. No election protocol: each node computes the role locally from its membership view, and failover to the next-lowest node is automatic at the following membership tick (a brief double-execution window during membership disagreement is accepted — the work is idempotent). /admin/cluster and /admin/health?verbose=1 report the node's claim as worker_leader; in a stable cluster exactly one node says true. The per-node workers (metrics snapshot, retention purge, notification delivery) and the Phase 28 bucket-replication worker stay un-gated by design: they operate on strictly node-local state — in particular the replication journal records each client write only on the node that served it, so deliveries to external destinations are already exactly-once cluster-wide (the review's duplicate-delivery premise did not hold; gating that worker would instead strand the entries of non-leader nodes).
• Configuration export omits the node identity, import refuses it (Phase 29.1 R5, review D12.2). GET /admin/export no longer includes the node-local node_id settings key, and POST /admin/import skips it even when present in the document (e.g. an export taken before this change) — importing another node's identity would corrupt cluster loop-prevention. Double defense, mirroring the existing replication-side filter.
• Cluster failure detector tolerates one missed probe (Phase 29.1 R3, review D12.3). A peer is declared dead after 2 consecutive probe failures (and alive again at the first success) instead of flapping on a single missed probe — a routine GC pause or dropped packet no longer bounces a node out of the quorum. Dead peers now also keep their last_seen timestamp in /admin/cluster (how long a node has been unseen is what the GC guard and pruning reason about).
• Config drift now excludes a node from the write quorum (Phase 29.1 R3, review D1/H7). A peer whose cluster-critical config differs (config_ok: false — wrong secret, different master key, different mode) stays visible in the topology but no longer counts toward the write quorum and no longer receives replication fan-out: it could not store the replicas correctly anyway. With 1 of 3 nodes drifted the cluster keeps writing (2 eligible ≥ quorum 2); with 2 of 3 drifted the aligned node refuses writes with 503. Drift detection itself no longer depends on the (removed) public fingerprint: a wrong-secret node is detected by its 403 on the authenticated ping, an aligned-secret/wrong-key node by the fingerprint now exchanged inside the ping.
• Real-network-partition test infrastructure (Phase 29.1 R2, review D8/§5.1/§5.2). The 3-node cluster suite grows from 11 to 24 tests and now exercises the scenarios the consistency modes exist for, not just crash-stops. bin/cluster partition <n> / heal <n> cut a node off the inter-node network while its process stays alive (the cluster compose now uses two networks, so the test runner keeps reaching the isolated node). New phases verify: in quorum mode an isolated node refuses writes with 503 immediately — inside the window where its membership still believes the peers are alive, the missing fan-out ACKs close it (the R1 true-quorum fix at work) — while reads keep being served, the majority side keeps writing, and the healed node converges and becomes writable again; in available mode (new config overlay) a split-brain write to the same key is accepted on both sides and exactly one LWW winner survives everywhere after heal, and a 1-of-3 minority node still accepts writes. The catch-up phase now also covers the control plane: a bucket and a credential created while a node is down appear on it after re-entry.
• True write quorum in cluster quorum mode (Phase 29.1 R1, review §2.1). A replicated write (PutObject, CompleteMultipartUpload, DeleteObject, delete markers, version deletes) is now acknowledged to the client only when at least write_quorum nodes durably hold it at ACK time: the local copy plus every peer whose fan-out response certified "row applied AND blob present". Previously the quorum was only an admission gate on membership liveness — a write whose fan-out failed on every peer was still acknowledged and could exist on a single machine. On a quorum shortfall the client now receives 503 ServiceUnavailable with a Retry-After header; the local copy is not rolled back (as in any quorum system without distributed transactions, the error means "not acknowledged as replicated", not "undone") and anti-entropy either propagates it or a retry overwrites it. available mode is unchanged.
• Parallel cluster fan-out (Phase 29.1 R1, review §2.4). Replication to peers (object rows, blobs, version deletes, control-plane ops) is now sent to all live peers concurrently instead of one at a time, so a slow peer no longer adds its latency to every write and the ACK counting above sees all responses in one round-trip.

Fixed

• Kubernetes liveness probe could kill a node mid catch-up (Phase 29.1 R9). Since the R7 syncing gate, /admin/health answers 503 while a returning node re-pulls what it missed — which is exactly right for readiness, but the shipped manifest pointed the liveness probe at the same path: a catch-up outlasting three probe periods would get the pod restarted in a loop precisely while it was healing. The liveness probe now uses ?verbose=1, which always answers 200 on a live process.
• Object-Lock state could be silently erased cluster-wide by anti-entropy (Phase 29.1 R7, finding N2). Retention/legal-hold changes keep last_modified unchanged (matching S3), so two replicas of the same object version could differ only in lock state while their last-writer-wins key tied — and the tie was resolved in favor of whichever copy happened to be applied last. The losing sequence: a retention is set while one node is down; another node later restarts and re-pulls that peer's full manifest (its sync cursor is in-memory), re-applies the stale lock-free copy over its own newer lock state, and the rewrite's fresh manifest cursor propagates the regression to every node — the retention silently disappears cluster-wide, with no error anywhere. For a WORM feature this is data loss of the protection itself. Object rows now carry a lock_updated_at timestamp (sqlite migration v23 / PostgreSQL 0011), stamped by every retention/legal-hold change, and the replication apply guards order equal-timestamp copies by it — a stale copy can no longer clobber a newer lock state, in either direction. Found by the new cluster catch-up test; root-caused via seq-counter forensics on the preserved node databases.
• A poison manifest entry no longer blocks a peer's incremental sync forever (Phase 29.1 R7, review M1). A row that persistently failed to apply stopped the anti-entropy cursor toward that peer at the failing entry, freezing the whole incremental sync (everything behind it included) until a restart. After 5 consecutive failed passes on the same entry the node now skips it with a warning and counts it as skipped_entries in /admin/cluster — operator evidence that the key may not converge locally until it changes again on the peer.
• Unreadable blob mtimes no longer silently disable blob GC (Phase 29.1 R7, review M8). When a blob file's mtime cannot be read, the GC walk falls back to "just written" — the safe direction (never reclaim), but previously a silent one: on a filesystem that cannot report mtimes, orphan blobs would simply never be collected with no trace of why. The fallback now logs a warning naming the file.
• Object-Lock changes were invisible to cluster anti-entropy (Phase 29.1 R5, finding N1). PutObjectRetention / PutObjectLegalHold updated the object row without stamping a fresh manifest cursor (seq) on either backend, so a peer that was down during a lock change never received it — only the real-time fan-out carried it, and a dead node misses that by definition. For a WORM feature this silently weakened the guarantee on the returning node. Lock updates now stamp a fresh seq, so the changed-since manifest re-delivers the row with its lock columns at re-entry.
• Steady-state anti-entropy churn: identical rows were endlessly redelivered (Phase 29.1 R5, review M7). apply_remote_object rewrote the row (stamping a fresh seq) even when the incoming copy was byte-identical to the local one. Two caught-up nodes therefore kept re-delivering their whole object tables to each other on every anti-entropy pass, forever — each apply bumped a seq the peer then saw as new. An identical incoming row is now a no-op (the equal-timestamp guard still lets through rows that differ only in lock columns, which N1 above relies on), so a converged cluster reaches a quiet fixed point.
• PostgreSQL: rows could be skipped by cluster incremental sync (Phase 29.1 R1, review §2.2). The PG backend assigned the anti-entropy manifest cursor (seq) from a PostgreSQL SEQUENCE, which is not transactional: a row could commit with a lower seq after a peer had already advanced its cursor past it, so the incremental sync silently never delivered it. The sequence is replaced by a commit-ordered single-row counter (object_seq, pg migration 0009, mirroring the SQLite design): the counter's row lock serializes assignment until commit, so cursor order now equals commit order and no committed row can be skipped.
• Control-plane merge could resurrect deleted entities on crash (Phase 29.1 R1, review §2.3). The anti-entropy control-plane merge executed entity deletions before adopting the corresponding tombstones, each in its own transaction: a crash between the two left "row deleted, no tombstone", and the next reconcile with a peer still holding the live row re-created it — for a revoked credential, a security hole. Tombstones are now adopted first, so an interrupted merge always converges to the deletion.
• Cluster test flakiness (Phase 29.1 R2, review §5.3). Two latent races in bin/test cluster: the post-convergence wait was shorter than HAProxy's eviction time (fall 2 inter 2s ≈ 4 s), so a test could hit the load balancer while a stopped node was still in rotation; and the 507 phase could read the cluster-minimum free space before the tiny-disk node was alive in the topology (its stats are cleared while it is considered dead). The wait now gives the LB a 5 s margin and the 507 precondition requires all 3 nodes alive.

0.25.1 — 2026-06-10

Changed

• Performance: per-bucket replication-config cache. Object writes (PutObject, CompleteMultipartUpload, delete markers, PutObjectTagging) no longer read the bucket's replication_configuration from the metadata store on every request. The result is cached per bucket for 30 seconds — mirroring the existing per-bucket encryption cache — so buckets without replication (the common case) skip the database query entirely on the write hot path. The cache is invalidated immediately on PutBucketReplication, DeleteBucketReplication, and the admin credential-cascade disable, so a configuration change still takes effect at once.
• S3 compatibility scope. Ceph s3-tests for RGW-only extensions (x-rgw-* headers, ?list-type=unordered, account usage) — which are not part of the AWS S3 API — are now excluded from the reported compatibility denominator. A new "Intentional divergences from AWS S3" section in the API reference documents Arca's deliberate allowance of enabling Object Lock on an existing empty bucket (AWS permits it only at bucket creation).

Fixed

• Intermittent 400 Bad Request on a keep-alive connection reused immediately after a rejected conditional PutObject. When a PutObject was rejected before its request body was read (a failed If-Match / If-None-Match precondition → 412, or NoSuchBucket / NoSuchKey / EntityTooLarge / invalid tagging), the unconsumed request body could desync the HTTP/1.1 keep-alive connection, so the next request parsed on it intermittently failed with a protocol-level 400. These early-reject responses now send Connection: close (matching AWS S3), so the client never reuses a possibly-desynced connection.

0.25.0 — 2026-06-03

Added

• High-availability clustering (Phase 29). Arca can now run as a symmetric, self-configuring, fully-replicated multi-node cluster — every node is identical, holds the full dataset, and serves reads and writes. Enable it with a single [cluster] config section (or bin/arca --cluster for local dev); there is no special "primary" and no separate coordinator.
• Real-time replication of the whole data and control plane: objects (including multipart uploads, object tags, retention and legal hold), buckets and bucket config, and identity/authorization (credentials, users, teams, grants) and server settings. A write to any node fans out to its peers, so a client behind a load balancer can read its write from any node.
• Two consistency modes. quorum (default, CP): a write needs a majority of nodes; if a node loses the majority it goes read-only and rejects writes with 503 ServiceUnavailable rather than risk divergence. available (AP): always writable, best-effort replication. Last-writer-wins conflict resolution with a deterministic blob_id tiebreak.
• Self-healing. A returning or lagging node catches up automatically: an anti-entropy worker reconciles objects (changed-since manifests) and the control plane (full-snapshot merge), proactively repairs blobs whose bytes a node is missing, and garbage-collects orphaned blobs (composite-multipart-aware). Tombstones ensure a delete is never resurrected by a peer that missed it.
• Peer discovery via mDNS (LAN), static seed list, or DNS (e.g. a Kubernetes headless Service).
• Cluster-aware storage capacity. Because every node holds the full dataset, the cluster's capacity is bounded by its smallest node; the dashboard reports the cluster-minimum free space and a write larger than that is rejected with 507 InsufficientStorage, even if the node serving it has room.
• Config-drift detection. A node whose cluster-critical config (cluster id / secret / mode / size / master key) differs from its peers is flagged on /admin/cluster and in the console, without the cluster refusing to run.
• GET /admin/cluster — cluster topology, consistency mode, write-quorum status, per-node liveness and config alignment, and cluster-effective disk capacity ({"enabled": false} on a single node).
• GET /admin/health?verbose=1 — health plus the cluster snapshot for operators; the default (non-verbose) health shape consumed by load balancers is unchanged.
• (Console) Cluster topology on the Dashboard: a dedicated topology card showing the consistency mode, write status, and a live node list (online/offline, "this node", endpoints, last-seen, config-mismatch flags). The card is now shown on single-node deployments too, where it presents the single local node — the dashboard layout is identical whether or not clustering is enabled.
• Tooling and deployment. bin/cluster manages a local 3-node cluster (compose + HAProxy); deploy/kubernetes/arca-cluster.yaml (StatefulSet + headless-Service DNS discovery) and deploy/haproxy/ (HAProxy + keepalived floating VIP) provide production templates; a High Availability guide documents the design, trade-offs, and operations.

0.24.0 — 2026-05-21

Added

• (Console) Keyboard navigation across the file list, turning the side panel and the fullscreen preview into a slideshow. ↑ / ↓ walk the list and reload the inline preview in place if it is open; ← / → step through the fullscreen modal. The fullscreen header gains a position / total counter and ‹ › arrows for mouse users, and the next two images are prefetched so navigation feels instant. Editable fields (search, tags) keep their native arrow behaviour.
• (Console) The sidebar identity block now shows the authenticated user's username next to the Admin / User badge, with the access key on a dedicated muted line underneath instead of a truncated access key. Works for non-admin users too.
• GET /admin/me is now identity-only: any valid Arca credential can read its own username, with no arca:* grant required. Previously the username probe needed arca:ViewServerInfo, which forced admins to attach AdministratorAccess to every user just to populate the sidebar — and that side-effect also exposed the empty Users / Teams / Grants / Audit Log pages where every call inside was a 403. The grant gate on the rest of the admin API is unchanged.
• (Console) "Open in new tab" button in the fullscreen preview header. Useful when the in-iframe renderer is weak — most notably macOS Safari, which displays PDFs as a tiny thumbnail strip in iframes but renders them correctly at top level (where macOS users can also hand off to Preview.app).

Changed

• (Console) Server card on the Dashboard now sits in a stable 3×2 grid with a new Topology tile (currently "Single node", will reflect multi-node in Phase 29). Longer labels like Per-bucket (Vault) no longer wrap.
• (Console) The drag-and-drop zone in the bucket browser now stretches to the bottom of the file area instead of hugging the first few rows. Anywhere under the file list is droppable; the sidebar is not.

Fixed

• (Console) The Effective Grants tab on the user-detail page now refreshes automatically after attaching / detaching a direct grant or joining / leaving a team. Previously the tab kept showing the stale state until the user navigated away and back.

0.23.1 — 2026-05-07

Fixed

• Plain PutObject and UploadPart no longer block tokio workers on MD5. The previous pipeline interleaved hashing and disk writes on the same async task, so the CPU-bound MD5 work serialized the runtime that was supposed to be pulling network bytes — leaving plain uploads slower than encrypted ones after v0.23.0 moved AEAD off the runtime (the symptom that triggered this release). MD5 and write now run on a dedicated blocking worker fed by a bounded channel.

Single-stream (1 × 1 GiB, tmpfs): plain steady at ~266 MB/s; encrypted 188 → 235 MB/s (+25%). Concurrent (4 × 256 MiB): plain ~833 → ~862 MB/s (+3.5%), encrypted 594 → 706 MB/s (+19%) — the largest win where the runtime was most contended (encrypted, low parallelism), exactly the PBM-backup workload. - (Console) Admin error toasts now show the backend's descriptive message (e.g. Username "root" already exists) instead of the bare HTTP category (e.g. Conflict). The S3 endpoints that were still surfacing Error ${status} were converted to use the existing <Message> extractor at the same time.

Added

• (Console) Soft duplicate-description warning on Create / inline-edit forms for users, teams, grants and credentials. Advisory only — descriptions remain free-form, the save path is unchanged.

0.23.0 — 2026-05-06

Fixed

• Encrypted multipart uploads no longer pay the 2× decrypt+re-encrypt tax on CompleteMultipartUpload. The default concat implementation used to decrypt every part and re-encrypt the whole final blob, so a Percona PBM MongoDB backup that took 1h plain took 2h with SSE-S3. The encrypted store now produces a composite sidecar that points at the still-on-disk encrypted parts (no copy, no extra crypto). Reads only decrypt the chunks that overlap the requested range; deletes cascade to the parts. Sidecar schema is additive, so old sidecars still deserialize.
• Plain multipart uploads also benefit from the composite sidecar. Same trick when every part is plain (no encryption, no compression), eliminating the byte-by-byte copy that previously dominated CompleteMultipartUpload. Encrypted-or-compressed parts still take the existing copy + MD5 path.
• HTTP/2 keep-alive no longer panics. keep_alive_interval on the hyper auto-builder requires a Timer; the TLS server now installs one explicitly so internal pings don't abort the runtime.

Bench (4 concurrent multipart uploads, 256 MiB each, 8 MiB parts, tmpfs)

Scenario	Throughput	CompleteMultipartUpload p50
Plain baseline	527.8 MB/s	0.71s
Encrypted baseline	207.7 MB/s	3.03s
Plain after fix	836.9 MB/s	0.003s (≈237× faster)
Encrypted after fix	594.9 MB/s	0.004s (≈758× faster)

Changed

• AEAD chunk crypto runs off the tokio async runtime. The encrypting stream now batches 4 × 64 KiB chunks and hands each batch to a spawn_blocking worker; up to 4 batches fly per stream so network reads, disk writes and CPU-bound AEAD overlap instead of serializing on the same worker thread. Tag and ciphertext are produced in place — one allocation per batch instead of two per chunk.
• Tokio runtime is now built explicitly instead of via the #[tokio::main] macro. New [server.runtime] config section: worker_threads, max_blocking_threads (both default to num_cpus).
• HTTP/2 server tunables exposed via [server.http]: max concurrent streams, keep-alive interval/timeout, initial stream/connection window. The TLS auto-builder is now constructed once and shared across connection tasks.
• Process-wide SystemRandom for DEK / nonce generation (previously a fresh instance per call).

Docker

• docker-compose.yml now sets nofile=65535 and net.core.somaxconn=4096 on the arca service so connection fan-out (multipart, parallel PBM workers) doesn't hit a hidden FD ceiling. New docker-compose.perf.yml overlay (tmpfs /data for reproducible perf measurements) is layered on automatically by bin/perf-test --tls / --encryption.

Benchmarks

• perf_test.py gained a parallel-multipart scenario that issues N concurrent multipart uploads and reports CompleteMultipartUpload time separately. Used to detect the encrypted-concat regression and verify the fix.

Added

• (Console) New 2d timeframe button in the Monitoring view, between 24h and 7d.
• (Console) Replication credentials are now created and managed from a dedicated modal flow with inline location hints, instead of the old free-form text fallback.

Documentation

• (Console manual) New screenshots covering the replication setup and journal flow.
• (Project report) Refreshed with v0.23.0 metrics: ~23 full-time-equivalent days, 285 commits, 72,651 LoC, 2,152 tests, 28/31 phases done.

Tech debt

• TD-014 (recorded, not yet fixed): arca recover and arca fsck do not understand composite blobs and currently flag them as "orphaned sidecar". Runtime S3 reads/writes are unaffected. Proper fix sketched in TECH_DEBT.md.

Console (other)

• Anchor monitoring chart edge x-ticks so the leftmost / rightmost labels can no longer clip out of the SVG viewport.

0.22.0 — 2026-04-22

Added

• Phase 28 — Replication (P3). Asynchronous cross-instance replication for disaster recovery and geographic distribution. One-way per-rule with explicit loop prevention, so two-way mirrors converge instead of bouncing forever. Destination is any S3-compatible endpoint (Arca, AWS S3, MinIO, …) and outbound requests are signed with full AWS SigV4. Versioning is required on the source bucket (AWS CRR semantics).
• Change journal records every pending / in-flight / failed delivery in a replication_journal table (SQLite v17 / Postgres 0004). Per-event-type dispatch (PUT / delete-marker / tag), last-writer-wins conflict resolution against the destination's Last-Modified, exponential backoff capped at one hour, and a FAILED state stamped only after the terminal attempt so transient failures don't leak to clients.
• Loop prevention: every outbound request carries x-amz-arca-replication-source: <source_endpoint_id>. Receiving Arca tags the object REPLICA and skips journal emission. Mirror configurations are integration-tested end-to-end: a single PUT on the source reaches the mirror exactly once and never bounces back.
• x-amz-replication-status response header on the relevant object handlers, with values PENDING / COMPLETED / FAILED / REPLICA.
• Admin API for managing the journal (list with filters, retry an entry) and destination credentials (list, create, delete, query usage). Deleting a credential cascade-disables every rule that references it so the worker can't loop forever against a vanished reference. /admin/info gained a replication_enabled flag.
• PutBucketReplication / GetBucketReplication / DeleteBucketReplication S3 handlers with Arca-flavoured XML extensions (<Endpoint>, <CredentialRef>, <Region> alongside the standard <Bucket>).
• [replication] config section with poll interval, batch size, retry budget, request timeout, source endpoint id, and journal retention windows (30 days for completed rows, 90 day hard cap so the table stays bounded during an extended outage).
• Docker replication test mode: bin/arca start --replication boots a second arca-replica instance on :9001; bin/test replication runs the full two-instance integration suite (4 boto3 tests: PUT, delete-marker, tag, mirror no-loop).
• (Console) Per-bucket Replication card in Bucket Settings with an amber "Versioning required" banner that links to the versioning card, rule rows showing source ➜ destination with inline Enabled/Disabled toggles and tag-filter chips, and a modal editor (Prefix + Tag filter, Destination, Credential Ref dropdown with an inline "+ New" flow).
• (Console) New admin view at #/replication — Destination Credentials card above the Replication Journal. The journal has inline column-header filters, color-coded status chips with a pulse dot on in_flight, a side detail panel, pagination, auto-refresh every 30 s, and per-row / per-panel "Retry delivery now" CTAs on failed rows.
• (Docs) New guide/replication.md covering setup, the loop-prevention contract, two-way mirrors, destination-credential management, journal and retry workflow.

Changed

• ObjectRecord gained a replication_status field; wire format and existing migrations are unchanged.

0.21.0 — 2026-04-20

Added

• Phase 27 — Transparent at-rest compression (P2). CompressingBlobStore wraps FsBlobStore/EncryptingBlobStore to compress plaintext before encryption and disk write, while keeping the wire format unchanged (ETag is still MD5 of plaintext; Content-Length reports plaintext size). Six algorithms ship: zstd, lz4, snappy, gzip, brotli, xz. An auto mode picks per-object via a deterministic rule table based on Content-Type and size (no sampling or ML). Compression is per-bucket, console-managed — the wrapper is always installed and activates only when a bucket has compression configured via the console or the Arca-specific PUT/GET/DELETE /{bucket}?compression subresource. Baked-in MIME and size filters skip already-compressed content; skipped reasons are surfaced as Prometheus counters. Chunked frame format with a footer chunk index enables O(1) ranged reads. Mixed-mode: compressed and plain blobs coexist transparently via sidecar metadata.
• arca compress-existing / arca decompress-existing CLI: offline atomic retrofit of an existing data directory (resumable via sidecars, --dry-run, --bucket, --algorithm overrides). arca fsck already recognizes .compressing.tmp orphans.
• Prometheus series: arca_compression_plaintext_bytes_total{algorithm}, arca_compression_compressed_bytes_total{algorithm}, arca_compression_skipped_total{reason}, arca_storage_compression_ratio.
• (Console) Per-bucket Compression card in bucket settings — matches the Object Lock card pattern: inline algorithm dropdown + level input + single Enable/Disable button, with an "Enabled" status badge and current algorithm/level displayed when active.
• (Console) Inline help system: every setting carries a "?" trigger next to its label. Hovering shows a short one-line hint tooltip; clicking opens a modal with the full explanation, bullets, rows, notes and code samples. Backed by a single Alpine component (helpTrigger(topicId)) and a global modal listening for arca:open-help custom events. All 13 starter topics live in console/js/help.js; adding help to a new setting is one topic entry plus a six-line snippet next to the label.
• (Console) Help coverage pass: bucket-settings cards (encryption, compression, versioning, object-lock, lifecycle, notifications), Settings page (log level, retention windows, S3 region, preview size limits) and the dashboard bucket-size widget (now also shows compression and object-lock badges, previously missing).
• (Console) Create-bucket dialog gained an Enable Object Lock checkbox. When ticked, the PUT /{bucket} request sends x-amz-bucket-object-lock-enabled: true so Object Lock and versioning are enabled atomically with the bucket. The API client's request helper now forwards caller-supplied headers through SigV4 signing.
• (Console) Irreversible bucket-setting toggles (first-time Versioning enable, Object Lock enable) now require typed bucket-name confirmation in a modal before firing. Prevents accidental clicks in production (versioning cannot be disabled, only suspended; Object Lock cannot be disabled at all).
• (Docs) New guide/compression.md, new Compression note in guide/configuration.md, new CLI entries for compress-existing/decompress-existing. Console user guide updated for the new Object Lock enablement flow, confirmation modals and card intros.

Changed

• Object Lock can now be enabled on empty existing buckets, not only at bucket creation. PutObjectLockConfiguration used to reject every call on a bucket whose object_lock config was absent with InvalidBucketState (strict S3 semantics). The handler now accepts the request when the bucket still has no objects and auto-enables versioning as before; buckets that already contain objects are still rejected, with a clearer message ("can only be enabled on empty buckets or at bucket creation"). Three new integration tests cover the accepted-empty / rejected-non-empty / accepted-at-creation matrix.
• (Console) Buckets list cards: creation date moved to its own line above the capability badges, and the badge row now wraps so four or more indicators (encrypted + versioned + locked + compressed) no longer overflow the card.
• (Console) Bucket settings page normalized: every card (Encryption, Compression, Versioning, Object Lock, Lifecycle, Event Notifications) now opens with a short plain-English intro paragraph explaining what the setting does and whether it can be turned off. Versioning and Object Lock intros carry the amber "This action cannot be undone." red-thread, matching the Object Lock card that set the pattern.
• (Console) Compression and Object Lock inline controls (dropdowns, inputs, buttons) harmonized with the global Settings page baseline (text-sm px-3 py-1.5); no more size discrepancy between pages.

0.20.0 — 2026-04-18

Added

• Kafka notification connector: produce S3 events to Kafka topics via rdkafka (librdkafka). Supports SASL authentication, configurable security protocol, and custom topics
• AMQP notification connector: publish S3 events to RabbitMQ via lapin (pure Rust AMQP 0-9-1). Supports custom exchanges, routing keys, durable queues, and publisher confirms
• Elasticsearch notification connector: index S3 events as documents via REST API (reqwest). Supports custom indices and basic auth. Zero new dependencies
• Syslog (RFC 5424) notification connector: send S3 events as syslog messages over UDP or TCP. Configurable facility, severity, and app name. Zero new dependencies
• SMTP notification connector: deliver S3 events as e-mail via lettre over SMTP or SMTPS. Supports STARTTLS, PLAIN authentication, custom subject/sender, and configurable timeout (smtp_timeout_seconds, default 15 s). Body is the same S3 event JSON the webhook connector posts
• gRPC notification connector: deliver S3 events as a unary arca.notifications.v1.NotificationService/Notify RPC via tonic + prost. Supports h2c and TLS, Bearer token via gRPC metadata, custom CA certificates for self-signed servers, and arbitrary user metadata forwarded through the proto map. Configurable timeout (grpc_timeout_seconds, default 10 s)
• Connector configuration guide (guide/connectors.md): single-page reference for all 13 connectors with destination URL formats, properties, XML examples, and timeout tuning
• (Console) Connector-specific form fields for Kafka, AMQP, Elasticsearch, Syslog, SMTP, and gRPC (all now active in the connector picker)
• (Docs) Rewrote the Event Notifications section of the console user guide with dedicated Webhook / SMTP / gRPC form examples and three new screenshots

Fixed

• Kafka/AMQP connector integration tests: resolved TD-013. The AMQP tests used to fail because wait_for_amqp_receiver ran docker exec rabbitmq-diagnostics as root in a tight loop, racing with RabbitMQ's .erlang.cookie initialization and crashing the container (EACCES). Switched the AMQP and Kafka readiness helpers to docker inspect-based health polling so they never exec into the receiver. Also fixed the Kafka test subscriber (manual partition assignment + seek_to_end avoids consumer-group coordination races) and raised the SigV4 admin-API default timeout so the test-connector failure path for Kafka's unreachable host does not time out the HTTP client before Arca can respond

Removed

• AMQP DNS pre-resolution workaround: the resolve_uri helper that rewrote AMQP URIs to use a tokio-resolved IP was added under the false TD-013 diagnosis. With the real fix in place, lapin/tcp-stream resolves hostnames correctly from the Alpine/musl arca binary, so the workaround is gone and the connector passes the URI straight to Connection::connect
• ONVIF connector: dropped from Phase 26 before any implementation shipped. The ConnectorType::Onvif enum variant, the console picker entry, and the roadmap checklist item have been removed. ONVIF integration was judged too niche to justify its maintenance cost; an external adapter can bridge ONVIF events into Arca via the webhook or gRPC connector when needed
• "Coming soon" paths in the console: since every connector is now implemented, removed the dead active flag on CONNECTOR_TYPES, the soon label + disabled-tile styling in the picker, and the conditional templates that were only reachable for unimplemented connectors

0.19.0 — 2026-04-15

Added

• Configuration export API: GET /admin/export returns full instance configuration as JSON (settings, users, teams, grants, credentials, buckets, bucket configs). Supports sections query parameter for selective export and include_secrets for secret key visibility (masked by default)
• Configuration import API: POST /admin/import applies exported JSON to a running instance. Three conflict modes: skip (default), overwrite, and dry_run. Processes sections in dependency order, automatically skips masked credentials
• (Console) Export/Import buttons in Settings page with polished modals: section checkboxes with aligned labels, secrets toggle with warning, drag-and-drop file upload, conflict mode selection, per-section result display
• Configurable log level with runtime reload: log level can be changed via PUT /admin/settings/log_level and takes effect immediately without restart
• Git commit hash at startup: compile-time embedded git hash logged on server start for traceability
• SQLite read connection pool: separate read-only connection pool for concurrent query execution
• BlobStore::concat: efficient multipart assembly without intermediate copies
• Batched audit writes: dedicated writer task with channel-based batching to reduce per-request overhead
• Admin & Server Enhancements section in roadmap for tracking features outside numbered phases

Changed

• Request logging moved from INFO to DEBUG level to reduce noise

Fixed

• Unused imports in multipart.rs and validate.rs tests
• Timeline divider label in project report (changed from "MVP" to "NOW")

0.18.2 — 2026-04-11

Changed

• Performance: release profile optimization — enable opt-level = 3, thin LTO, codegen-units = 1, and symbol stripping for significantly faster crypto, hashing, and XML parsing
• Performance: TLS session resumption — enable server-side session cache (256 entries) to avoid full handshakes on client reconnections
• Performance: TCP_NODELAY on TLS connections — reduce latency on small responses (HEAD, DELETE, errors)
• Performance: SQLite tuning — set synchronous=NORMAL (safe with WAL), 64 MB page cache, 256 MB mmap, temp_store=MEMORY, 5 s busy timeout
• Performance: 64 KB ReaderStream chunks — increase from 8 KB default, reducing syscalls by ~8x on large object downloads
• Performance: eliminate double allocation in decryption — reuse the in-place decryption buffer instead of copying plaintext
• Performance: compact sidecar JSON — drop pretty-printing for smaller sidecar files and faster I/O

Fixed

• Unused imports in arca-proto middleware (std::net::IpAddr, http::StatusCode)

Added

• Perf-test summary table: all test results in a single table with throughput and latency percentiles
• Global Performance Index: weighted geometric mean score for run-to-run comparison
• Perf-test --tls flag: self-contained TLS testing with self-signed certificates
• Perf-test --tls-fqdn flag: TLS with existing certificates and FQDN verification
• Perf-test --encryption flag: self-contained encryption testing
• Perf-test -q/--quiet flag: show only the summary table

0.18.1 — 2026-04-10

Added

• NATS notification connector: delivers S3 event notifications by publishing JSON payloads to a NATS subject. Supports custom subject names via subject property (default arca.notifications), optional token or user/password authentication, and configurable connection timeout (nats_timeout_seconds). Includes admin API connectivity test via POST /admin/notifications/test-connector with connector_type=nats
• (Console) NATS connector enabled in notification editor with subject, token, and user/password configuration fields
• MQTT notification connector: delivers S3 event notifications by publishing JSON payloads to an MQTT topic. Supports custom topic names via topic property (default arca/notifications), configurable QoS level (0/1/2, default 1), optional username/password authentication, and configurable connection timeout (mqtt_timeout_seconds). Includes admin API connectivity test via POST /admin/notifications/test-connector with connector_type=mqtt
• (Console) MQTT connector enabled in notification editor with topic, username, and password configuration fields
• Database notification connectors (PostgreSQL, MySQL, MongoDB): delivers S3 event notifications by inserting rows/documents into a database table or collection. Shared db_common module provides DDL generation and event field extraction. Tables/collections are auto-created on first delivery
• PostgreSQL connector: sqlx-postgres driver, TIMESTAMPTZ columns, supports table and schema properties (default: arca_notifications)
• MySQL connector: sqlx-mysql driver, DATETIME(6) columns, supports table property (default: arca_notifications)
• MongoDB connector: mongodb driver, BSON documents, supports database (default: arca) and collection (default: arca_notifications) properties
• (Console) PostgreSQL, MySQL, and MongoDB connectors enabled in notification editor with database-specific configuration fields
• Deployment files: Kubernetes manifests (Deployment, Service, PVC, ConfigMap), systemd unit file, and sample configuration file for production deployments
• Troubleshooting guide: documentation page with common issues, diagnostic steps, and solutions

Fixed

• SIGHUP killing the process when TLS is not enabled: signal handler crashed on unwrap() of the TLS reloader when TLS was not configured
• (Console) STORAGE SIZE chart Y-axis showed "undefined" labels when all values were 0 (fractional tick values caused negative index in formatBytes)

0.18.0 — 2026-04-09

Added

• Redis Pub/Sub notification connector: delivers S3 event notifications by publishing JSON payloads to a Redis Pub/Sub channel. Supports custom channel names via channel property, optional password authentication, and configurable connection timeout (redis_timeout_seconds). Includes admin API connectivity test via POST /admin/notifications/test-connector with connector_type=redis
• Dedicated connector test stream: bin/test connectors redis (and future bin/test connectors all) runs connector integration tests against real Docker-based receivers, separate from the normal development test flow
• Presigned URL tracking: generated presigned URLs are now tracked in the database (metadata only, not the URL itself, for security). Active shares are visible in the console with a permanent blue share icon next to shared files. Clicking the icon shows a detail modal with method, duration, remaining time, creator, and a button to remove the record. Expired records are purged automatically by the retention worker
• GET /admin/presigned-urls?bucket=X: lists active (non-expired) presigned URL records for a bucket
• DELETE /admin/presigned-urls/:id: removes a presigned URL tracking record (the URL itself remains valid until expiry)
• POST /admin/presign response now includes an id field for tracking reference (SQLite migration v16, PostgreSQL migration v3)
• (Console): Monitoring chart time range buttons expanded with 6m and 1y presets, plus a custom date range picker (From/To datetime inputs)
• bin/seed-metrics: development utility to generate fake metrics data for testing monitoring charts

Fixed

• Monitoring chart time ranges: 24h, 7d, and 30d views only showed ~8 hours of data because the API returned LIMIT 500 most-recent rows. Fixed with server-side downsampling using ROW_NUMBER() CTE to return evenly spaced points across the full range (both SQLite and PostgreSQL)
• (Console): X-axis labels now adapt to actual data span (time-only for short ranges, date+time for days, month+day for weeks, month+year for long ranges) instead of being hardcoded per button
• Ceph s3-tests: 18 additional tests now pass (lifecycle, object lock, versioning, encryption)

0.17.1 — 2026-04-03

Added

• Modular notification connector architecture: trait-based NotificationConnector in arca-core with ConnectorRegistry for pluggable delivery backends. Webhook is the first implementation; 9 additional connectors (Kafka, AMQP, Redis, NATS, MQTT, PostgreSQL, MySQL, MongoDB, Elasticsearch) planned for Phase 26
• ConnectorType enum: all 10 connector types defined with display_name(), category(), all() helpers for UI enumeration
• WebhookConnector: extracted webhook delivery logic into a standalone connector implementing the NotificationConnector trait, with Bearer token authentication support via auth_token property
• Webhook auth token: DestinationConfig now supports properties map (Arca extension) for connector-specific settings. Webhook connector reads auth_token and sends Authorization: Bearer <token> header
• POST /admin/notifications/test-connector: generalized admin endpoint for testing any registered connector type; existing test-webhook endpoint preserved as backward-compatible alias with auth_token support
• (Console): Notification editor redesigned as a modal dialog (replacing inline lifecycle-style forms) with connector type selector grid showing all 10 types grouped by category (Functions, Queue, Database), SVG icons, and "coming soon" badges for unimplemented connectors
• Roadmap Phase 26: Notification Connectors phase added (9 connectors, modular per-connector Docker test infrastructure)

Changed

• (Console): Settings page reorganized: "Monitoring" section renamed to "Data Retention" with all retention policies grouped together (Audit Log, Notification Events, Metrics Snapshots), ordered by sidebar position
• Notification event retention is now a server setting (notification_retention_days) manageable from the console, following the same TOML > DB > default precedence as audit and metrics retention (default: 7 days)
• Notification worker refactored to dispatch via ConnectorRegistry instead of hardcoded webhook delivery. Retry logic (exponential backoff) is now generic across all connector types
• DestinationConfig extended with connector_type (default: Webhook) and properties (HashMap) fields, both Arca extensions preserved in JSON storage and XML serialization
• NotificationEventRecord extended with connector_type field (SQLite migration v15, PostgreSQL migration v2)
• (Console): Section title changed from "Notification Webhooks" to "Event Notifications"; notifications list shows connector type badge and icon
• Webhook test receiver (docker/webhook-receiver/server.py) now supports AUTH_TOKEN env var for Bearer token validation (returns 401 on mismatch)

Fixed

• (Console): Monitoring chart line clipping at maximum values: Y-axis scale now always extends above the data maximum, preventing the line from being drawn outside the SVG viewBox

0.17.0 — 2026-04-02

Added

• Phase 25: Notifications and Event System
• PutBucketNotificationConfiguration / GetBucketNotificationConfiguration: S3-compatible bucket notification configuration via ?notification query parameter. Accepts all three S3 destination types (TopicConfiguration, QueueConfiguration, CloudFunctionConfiguration), treating all as webhook destinations
• Event emission: s3:ObjectCreated:Put, s3:ObjectCreated:Copy, s3:ObjectCreated:CompleteMultipartUpload, s3:ObjectRemoved:Delete, s3:ObjectRemoved:DeleteMarkerCreated events emitted from handlers via non-blocking mpsc channel
• Webhook delivery worker: background worker consumes events, matches per-bucket notification rules (event type + prefix/suffix filters), delivers HTTP POST to configured webhook URLs with exponential-backoff retry (configurable max retries, base delay, timeout)
• Notification event persistence: notification_events table (SQLite migration v14, PostgreSQL schema update) stores delivery records with status tracking (pending/delivered/failed)
• Per-webhook Enabled/Disabled toggle: Arca extension persisted via <Enabled> XML element, allowing temporary webhook suspension without removal
• NotificationStore trait: CRUD + purge for notification event records, implemented for both SQLite and PostgreSQL backends
• Admin API: GET /admin/notifications/events (list with filters), GET /admin/notifications/events/count, DELETE /admin/notifications/events (clear with confirmation), POST /admin/notifications/test-webhook (connectivity test)
• [notifications] config section (optional): tuning parameters for channel size, retry behavior, webhook timeout, and event retention days (default: 7)
• Event retention: notification events automatically purged by the retention worker based on event_retention_days
• Notification config caching: in-memory cache with 60s TTL in the delivery worker to avoid per-event DB reads
• (Console): Notification event log viewer with audit-style inline column header filters (Time, Event, Bucket, Key, Destination, Status), pagination, side panel detail, clear-all with confirmation
• (Console): Per-bucket notification rules editor in bucket settings (lifecycle-style inline forms, Enabled/Disabled toggle, prefix/suffix filters, test webhook button)
• (Console): Sidebar navigation link for Notifications (between Audit Log and Monitoring)
• Docker webhook receiver service (docker/webhook-receiver/) for integration testing
• bin/test notifications mode with webhook receiver compose overlay
• 35 unit tests (XML parsing/roundtrip, event matching, filter matching, validation, NotificationStore CRUD)
• ~18 integration tests (configuration CRUD, webhook delivery, event format, filters, batch delete, admin API)

0.16.1 — 2026-04-01

Fixed

• PostgreSQL backend not activating: when using --postgres with other features (e.g. --encryption), the metadata_backend key from the postgres config fragment landed under the wrong TOML section due to naive fragment concatenation, silently falling back to SQLite
• Auto-detect metadata_backend = "postgres" from presence of [storage.postgres] config section, removing the need for an explicit key in the fragment

0.16.0 — 2026-03-31

Added

• Configurable preview size limits: 3 new server-side settings (preview_max_size_mb, preview_max_text_mb, preview_max_video_mb) to control the maximum file size for object previews in the console
• (Console): New "Console" section in the Settings page to manage preview thresholds per file type (images/PDF/HTML, text/Markdown, video), with 0 = unlimited
• (Console): Preview logic reads limits from server settings instead of using hardcoded values
• 2 new integration tests for preview settings CRUD and validation

0.15.0 — 2026-03-31

Added

• Phase 24: PostgreSQL Backend
• PgStore: PostgreSQL metadata backend implementing all 8 store traits (MetadataStore, CredentialStore, UserStore, TeamStore, GrantStore, AuditStore, MetricsStore, ServerConfigStore) via sqlx-core/sqlx-postgres
• Config switch: [storage] metadata_backend = "sqlite" | "postgres" with [storage.postgres] section for connection string and pool settings
• Docker overlay: docker-compose.postgres.yml with PostgreSQL 17 Alpine, --postgres flag for bin/arca start and bin/test postgres
• Migration runner: consolidated initial schema (equivalent to SQLite v1-v13) applied automatically at startup
• /admin/info: returns metadata_backend field ("sqlite" or "postgres")
• (Console): Database indicator in server info panel showing current backend type
• 20 PostgreSQL-specific integration tests covering buckets, objects, multipart, versioning, tags, lifecycle, copy, range reads

0.14.0 — 2026-03-31

Added

• Phase 22: S3 API Completeness
• ListParts: GET /{bucket}/{key}?uploadId=X with pagination
• GetObjectAttributes: GET /{bucket}/{key}?attributes with ETag, Checksum, ObjectParts, StorageClass, ObjectSize
• Checksum algorithms: store and return client-provided x-amz-checksum-sha256/crc32/crc32c/crc64nvme on PutObject, GetObject, HeadObject
• Storage classes: storage_class field on ObjectRecord, accept x-amz-storage-class header
• SQLite migration v13: storage_class, checksum_algorithm, checksum_value on objects; checksum_value, last_modified on parts; checksum_algorithm on multipart_uploads
• Resolves TD-002 (storage class) and TD-008 (content-type source)
• Phase 23: Performance and Hardening
• Request size limits: [server.limits] TOML section, streaming LimitedByteStream, Content-Length fast-reject, EntityTooLarge error (default 5 GB)
• Rate limiting: per-IP and per-credential GCRA via governor crate, SlowDown 503 with Retry-After (disabled by default)
• Metadata cache: in-memory LRU via moka crate, CachingMetadataStore for bucket existence and object HEAD, configurable size/TTL in [server.cache], write-through invalidation
• Graceful shutdown: drain mode via tokio::sync::watch, health endpoint returns 503 during configurable drain window
• Request validation middleware: header count limit, null byte rejection, user metadata size limit
• Performance benchmarks: HEAD/DELETE benchmarks with --json output and --baseline comparison

0.13.0 — 2026-03-26

Added

• Phase 21: Object Lock (WORM Compliance)
• 6 new S3 operations: PutObjectLockConfiguration, GetObjectLockConfiguration, PutObjectRetention, GetObjectRetention, PutObjectLegalHold, GetObjectLegalHold
• Retention modes: GOVERNANCE (bypassable with permission) and COMPLIANCE (absolute protection) with retain-until-date
• Legal hold: per-object ON/OFF flag, independent of retention
• Default retention: bucket-level config applied automatically on PutObject
• Enforcement: blocks hard-deletion of locked versions, delete markers always allowed, auto-enables versioning (prevents suspension)
• SQLite migration v12: adds retention_mode, retain_until_date, legal_hold_status columns
• 7 new S3 policy actions including s3:BypassGovernanceRetention
• Lifecycle worker respects Object Lock
• Console: Object Lock — lock badge in bucket list and breadcrumbs, retention mode/period in bucket settings, versioning suspend disabled when locked
• Phase 20: Lifecycle Rules
• 3 new S3 operations: PutBucketLifecycleConfiguration, GetBucketLifecycleConfiguration, DeleteBucketLifecycleConfiguration
• Rule features: object expiration after N days, noncurrent version expiration, abort incomplete multipart uploads
• Filters: by prefix, tag, or combined (And filter). Rules stored as JSON in bucket_config table (no schema migration)
• Background worker: periodic evaluation across all buckets, configurable interval (default 1 hour), batch processing (100 objects per rule per cycle), audit logging for all lifecycle actions
• Console: lifecycle rules editor — add/remove rules with prefix filter, expiration days, noncurrent version days, abort upload days
• Admin setting: lifecycle_evaluation_interval — configurable evaluation interval (60–86400 seconds) via GET/PUT/DELETE /admin/settings/lifecycle_evaluation_interval

Fixed

• Tagging: tag validation on PutObject — invalid x-amz-tagging headers now rejected with 400 before writing the blob
• Tagging: x-amz-tagging-count header — GET and HEAD responses now include tag count when the object has tags
• Tagging: multipart upload tags — CreateMultipartUpload now captures x-amz-tagging and applies tags at CompleteMultipartUpload
• Versioning: delete marker detection — GET/HEAD on a delete marker now returns 404 with x-amz-delete-marker: true and x-amz-version-id
• Versioning: CompleteMultipartUpload x-amz-version-id — response now includes version ID when versioning is enabled
• Versioning: UploadPartCopy with versioned source — now uses ?versionId= from copy source header instead of always fetching latest
• Versioning: conditional DELETE with delete markers — conditional headers on DELETE now correctly evaluate against latest version including delete markers
• S3 compatibility: +18 Ceph s3-tests passing, 338/829 (40.8%), up from 320/829 (38.6%)
• Encryption: key mismatch returns 403 — wrong master key now returns AccessDenied instead of InternalError

0.12.0 — 2026-03-24

Added

• Phase 19: Object Tagging
• 6 new S3 operations: GetBucketTagging, PutBucketTagging, DeleteBucketTagging, GetObjectTagging, PutObjectTagging, DeleteObjectTagging
• Inline tags: x-amz-tagging header on PutObject, x-amz-tagging-directive on CopyObject
• Limits: max 10 tags per object/bucket, key max 128 chars, value max 256 chars
• Version-aware: tags tied to specific object versions, cascade deletes on object/bucket removal
• SQLite migration v11: new object_tags and bucket_tags tables
• Console: tag editor — view, add, and remove object tags in the detail side panel
• Roadmap restructured — Phase 19 split into Object Tagging (19) and Lifecycle Rules (20), subsequent phases renumbered

0.11.0 — 2026-03-24

Added

• Console: search and filter — real-time debounced search bar on all list views (Buckets, Users, Teams, Grants, Credentials, Bucket Detail), filters files and folders, updates treemap and select-all in sync
• Console: audit operation filter — smart presets (S3 Read, S3 Write, All S3, All Admin, Data Changes) with category chips, drill-down, and individual operation checkboxes
• Console: responsive mobile layout — full support down to 375px, collapsible sidebar, responsive grid, touch-friendly controls, horizontal-scrolling tables, viewport-safe popovers

Fixed

• Removed unused Rust imports (delete in router, MetadataStore and admin_settings in worker) to eliminate compiler warnings

0.10.0 — 2026-03-23

Added

• Console: object preview — collapsible preview panel with fullscreen modal. Supports images (JPEG, PNG, GIF, WebP, SVG, AVIF), video (MP4, WebM, MOV with native controls), text/code with syntax highlighting, Markdown, HTML in sandboxed iframe, and PDF
• Environment variable overrides — ARCA_SERVER_BIND, ARCA_SERVER_PORT, ARCA_STORAGE_DATA_DIR override config file values

0.9.1 — 2026-03-22

Added

• Console: inline column header filters — Time (date range popover), Bucket (value list with counts), Key (text search), User (value list with counts), Status (value list with color-coded codes), and Operation (tag-based include/exclude popover) filters are built into the table column headers
• Console: clear audit log — "Clear All" button with confirmation modal requiring explicit CLEAR AUDIT LOG text input. DELETE /admin/audit backend endpoint
• Console: date range filter — From/To datetime pickers for the Time column, server-side filtered
• Console: status column filter — filterable by HTTP status code via dropdown with checkboxes and counts
• Console: sticky filters — all audit log filters (column headers, operation tags, page size) persist across navigation via sessionStorage

Changed

• Console: audit log layout — filter bar removed, all filters moved into column headers with consistent UX (funnel icon on hover, cyan badge when active, X to clear)
• Console: fixed sidebar — sidebar no longer scrolls with page content

Fixed

• Console: multi-file upload — fixed regression where only the first file was uploaded (live FileList invalidated during async iteration)
• Audit log feedback loop — read-only monitoring operations (Health, Metrics, ListAudit, etc.) are no longer logged to prevent audit entries from generating more audit entries
• Audit log missing identity — access key and user ID now correctly extracted from the Authorization header before auth middleware consumes the request

0.9.0 — 2026-03-21

Added

• Phase 18: Monitoring, Metrics, and Audit
• Prometheus metrics endpoint: GET /admin/metrics (unauthenticated) with request counters by operation/status, latency histograms (10 buckets), and gauges (active connections, buckets, objects, storage size)
• Audit logging: every S3 and admin operation recorded in audit_log table with timestamp, operation, identity, status, duration. GET /admin/audit with filters (bucket, operation, user, time range) and pagination. Tag-based operation filter with include/exclude modes in console
• Instance-wide settings: GET/PUT/DELETE /admin/settings/{key} for runtime-configurable settings (region, retention days). TOML config takes precedence when set (read-only in console)
• Configurable region: region in [server] TOML section or via Admin API. Per-bucket region via bucket_config. Resolves TD-004
• Metrics history: periodic gauge snapshots stored in metrics_snapshot table. GET /admin/metrics/history with time range filter. SVG sparkline charts with labeled axes
• Retention management: configurable audit log and metrics retention (days). Background worker purges old records hourly
• Background worker framework: reusable BackgroundWorker::spawn_periodic for periodic tasks (metrics snapshots, retention purge)
• Console: Settings page — instance-wide settings with lock indicators for TOML-set values, auto-save for editable fields
• Console: Audit Log page — filterable table with tag-based operation filter, pagination with first/last buttons, page size selector, slide-in detail panel, auto-refresh
• Console: Monitoring page — SVG sparkline charts for objects, storage, buckets, connections with time range selector and labeled axes
• Console: fixed sidebar — sidebar no longer scrolls with content, stays pinned to viewport
• Database migration v10: server_config, audit_log, and metrics_snapshot tables
• Integration tests: 27 new tests for Prometheus, audit, metrics history, settings, and region

Fixed

• Multi-file upload: fixed regression where uploading multiple files or folders in the console only uploaded the first file (FileList reference invalidated by input reset during async iteration)

0.8.1 — 2026-03-21

Fixed

• DeleteObjects with VersionId: batch delete (POST /{bucket}?delete) now supports <VersionId> per object, enabling hard-deletion of specific versions and delete markers. Previously, VersionId was ignored and versioned deletes only created more delete markers, making buckets impossible to empty.
• Console: object list now refreshes after deleting a specific version from the version history panel.
• Ceph s3-tests: 50 additional tests now pass (270 → 320), primarily in Versioning and Bucket categories.

Changed

• bin/s3-tests: report generation no longer skipped when pytest exits non-zero (test failures). The script now correctly captures the exit code and proceeds to generate the HTML report, SVG badge, passlist, and summary.json.

0.8.0 — 2026-03-20

Added

• Phase 17: Object Versioning
• Bucket versioning config: PutBucketVersioning / GetBucketVersioning with three states (Disabled / Enabled / Suspended)
• Version IDs: every PutObject on a versioned bucket generates a UUID version ID, returned via x-amz-version-id header
• Delete markers: DeleteObject on versioned buckets creates a delete marker instead of removing the object; x-amz-delete-marker: true header
• Version-specific operations: GetObject?versionId=X, HeadObject?versionId=X, DeleteObject?versionId=X for accessing or permanently deleting specific versions
• ListObjectVersions: rewritten with real version data, delete markers, and Owner elements (replaces TD-003 fake implementation)
• CopyObject with source versionId: x-amz-copy-source: bucket/key?versionId=X copies a specific version
• Suspended versioning: new writes get null version ID, existing real versions preserved
• Batch DeleteObjects: safe on versioned buckets (creates delete markers, no panic on empty blob_id)
• SQLite migration v9: objects table recreated with version_id, is_latest, is_delete_marker columns, partial unique index
• Error codes: NoSuchVersion (404), MethodNotAllowed (405) for GET on delete markers
• arca recover: sorts sidecars by last_modified to recover newest version; SidecarMeta gains version_id field
• arca fsck: handles all object versions and skips blob checks for delete markers
• Web console: bucket versioning toggle (Enable/Suspend) matching encryption switch style
• Web console: versioning indicators (clock icon) in dashboard, bucket list, and bucket detail breadcrumbs (blue=Enabled, amber=Suspended)
• Web console: version history panel in object detail with download/delete per version, Latest badge, delete marker indicators
• Web console: "Show deleted" toggle in bucket browser header for versioned buckets, showing deleted files and directories with strikethrough + red badge
• Web console: scrollable breadcrumbs for deep directory paths with auto-scroll to deepest level
• Web console: centralized SVG icons (icons.encryptionShield, icons.versioningClock, icons.versioningSuspended) in app.js
• 13 new unit tests for versioning (put/get/delete versioned, delete markers, suspended mode, stats)
• 20 versioning integration tests (config, PUT, GET, HEAD, DELETE, batch delete, ListVersions, CopyObject, backward compat)
• Resolves: TD-003

Fixed

• File re-upload after delete now works (reset file input value after upload)
• Batch DeleteObjects no longer panics on versioned buckets (skip blob deletion for delete markers)

0.7.0 — 2026-03-18

Added

• Phase 16: Access Control and Bucket Policies
• RBAC foundation: users, teams, grants (policy documents) with full CRUD via Admin API
• Policy evaluation engine: AWS IAM-style policy documents with Effect/Action/Resource matching, wildcard support, deny-overrides evaluation
• S3 authorization middleware: non-root users are authorized against effective policies for every S3 operation (action + resource ARN matching)
• Admin authorization: non-root users need arca:* grants to access admin endpoints (e.g., AdministratorAccess grant)
• Identity resolution: credentials linked to users, users belong to teams, grants attach to users or teams. Effective policies = direct grants + team-inherited grants
• Built-in grants: AdministratorAccess, S3FullAccess, S3ReadOnlyAccess created during migration
• Owner model: buckets and objects track their creator's username (resolves TD-001)
• Admin API endpoints: /admin/me, /admin/users, /admin/teams, /admin/grants with full CRUD, membership management, grant attachments, effective grant queries
• CLI arca user subcommand: create, list, delete for offline user management
• CLI arca credential add --user: associate new credentials with specific users
• Web console: Users, Teams, Grants management views with dual-list shuttle components for membership and grant assignment
• Web console: inline editing of names and descriptions for users, teams, grants, and credentials (save-on-change)
• Web console: credential activate/deactivate toggle on credential cards
• Admin API: PUT /admin/credentials/{id} for updating credential active status and description
• Admin API: PUT /admin/users/{id} now accepts optional username field for renaming
• Admin API: PUT /admin/teams/{id} now accepts optional name field for renaming
• Documentation: Access Control guide page with diagrams explaining the identity model, auth flow, and effective grants
• Documentation: database ER diagram (Mermaid) in architecture page showing all 12 tables and relationships
• Documentation: full Admin API reference (37 endpoints), updated CLI reference, console manual with 19 screenshots
• Ceph s3-tests: machine-readable summary.json output after test runs
• SQLite migration v8: users, teams, grants, team_members, user_grants, team_grants tables; owner fields on buckets/objects; user_id on credentials
• 41 RBAC integration tests (user/team/grant CRUD, attachments, effective grants, E2E access control)
• 90 new unit tests (47 policy evaluator, 37 RBAC store implementations, 6 credential/user/team update)

0.6.0 — 2026-03-16

Added

• Phase 15: Presigned URLs, Query-String Auth, and SSE-C
• Query-string SigV4 authentication: presigned URLs with X-Amz-Algorithm, X-Amz-Credential, X-Amz-Date, X-Amz-Expires, X-Amz-SignedHeaders, X-Amz-Signature query parameters
• Presigned URL generation: generate_presigned_url() in arca-auth for server-side URL creation
• Admin presign endpoint (POST /admin/presign): generate presigned GET/PUT URLs with configurable expiry, optional endpoint override for correct public URL, percent-encoded paths
• SSE-C (Server-Side Encryption with Customer-provided keys): AES-256-GCM encryption where the customer provides the key on each request. Supported for PutObject, GetObject, HeadObject, and CopyObject (including cross-mode: SSE-C to plain, plain to SSE-C, SSE-C to SSE-C with different keys)
• Console: Share button on objects with presigned URL generation, expiry selector (1h/6h/1d/7d), and copy-to-clipboard
• SSE-C CORS headers for cross-origin browser access
• 17 presigned URL integration tests (GET, PUT, HEAD, DELETE, security, admin presign, special chars, endpoint override)
• 17 SSE-C integration tests (put/get roundtrip, error handling, head, copy, range reads, delete, validation, multipart rejection)
• 18 new unit tests (12 auth, 6 storage) for presigned URL parsing/signing and SSE-C encryption

0.5.0 — 2026-03-16

Added

• Streaming archive endpoint (POST /admin/archive): download multiple objects as a tar.gz archive streamed on-the-fly, with no temporary files on the server
• Console: directory upload via file picker and drag-and-drop, preserving directory structure
• Console: multi-select with checkboxes, select-all, and floating action bar
• Console: batch delete with recursive directory support and confirmation dialog (warnings for multiple items and recursive directory wipes)
• Console: batch download as streaming tar.gz archive
• CHANGELOG.md with full release history
• RELEASING.md with release procedure checklist

0.4.1 — 2026-03-16

Added

• Plugin system: composable features via --flags (--tls, --encryption, --kms), replacing combinatorial config/compose files
• Binary extraction: bin/build --binary [--arch amd64|arm64] extracts a standalone Linux binary to build/
• Console: disk usage shown in dashboard stats
• Console: documentation link in sidebar
• --help flag on all bin/ scripts
• Auto-create master key in Vault/OpenBAO at startup if missing

Changed

• Centralized all builds in bin/build with --dev, --console, --binary flags
• Consolidated version in workspace Cargo.toml (single source of truth, inherited by all crates)
• Use debian:stable-slim instead of bookworm-slim for dev image

Fixed

• Foreground mode (bin/arca start) not stopping associated services on Ctrl+C
• Treemap button not rendering on click in console

0.4.0 — 2026-03-15

Added

• Phase 14: SSE-KMS with Vault/OpenBAO — fetch master encryption key from Vault/OpenBAO KV v2 at startup, [encryption.kms] config section, reqwest with rustls-tls backend
• Per-bucket encryption with bucket settings console page
• Database schema diagram in architecture documentation (SVG for dark mode)

Changed

• Adopted semver 0.x.x pre-release versioning (from 1.2.0 to 0.3.0)
• Console uses standard ports (80/443) instead of 3000
• Replaced RUST_LOG with ARCA_LOG for log level configuration
• Replaced ICO favicon with transparent PNG for Safari/Chrome compatibility

0.3.0 — 2026-03-13

Added

• Phase 13: Server-Side Encryption (SSE-S3) — AES-256-GCM via ring, chunk-based streaming (64 KiB), envelope encryption (random DEK per object wrapped by master KEK), ETag computed on plaintext, mixed-mode coexistence, bucket_config table for per-bucket settings
• --config (-c) option for custom config files in bin/arca
• Combined TLS + encryption config for development
• Version number in web console sidebar

0.2.0 — 2026-03-13

Added

• Phase 12: Native TLS/SSL — tokio-rustls + hyper_util, ring crypto backend, single port for all traffic including health checks, ArcSwap for cert hot-reload on SIGHUP
• Console HTTPS support
• Full post-MVP product roadmap (Phases 12–26) with dependency diagram

Fixed

• HTTP/2 SigV4 host header (:authority pseudo-header handling)

0.1.0 — 2026-03-10

Added

• Phases 0–11: complete MVP — 15 S3 operations, single-node server
• S3 API: CreateBucket, HeadBucket, ListBuckets, DeleteBucket, PutObject, GetObject, HeadObject, DeleteObject, DeleteObjects, CopyObject, ListObjectsV2, CreateMultipartUpload, UploadPart, CompleteMultipartUpload, AbortMultipartUpload
• Five-crate Cargo workspace: arca-core, arca-auth, arca-proto, arca-storage, arca-server
• Streaming-first architecture: never buffers full objects in memory
• AWS SigV4 authentication with credential CRUD via Admin API
• Disaster recovery: sidecar .meta files, arca recover for full DB rebuild, arca fsck for integrity checking
• Web console: Alpine.js + Tailwind CSS dashboard, bucket browser, credential management, role-aware UI
• Admin API: health, info, stats, credential CRUD under /admin/*
• S3 compatibility hardening: 270/829 Ceph s3-tests passing (32.6%), MinIO Client support
• Documentation site: MkDocs with Material theme, architecture docs, user guides
• Scratch-based production Docker image (8.6 MB)