Technical Debt¶
Workarounds, hardcoded values, and temporary fixes that pass S3 compatibility tests or satisfy client expectations but need proper implementation post-MVP.
Each entry has a unique ID referenced in the source code via // TECHDEBT(TD-XXX):
comments. Run grep -r 'TECHDEBT' crates/ to find all markers.
Active Items¶
| ID | Area | Workaround | Proper Fix | Tests Affected | Files |
|---|---|---|---|---|---|
| TD-007 | Unimplemented ops | ~33 bucket-level GET/PUT operations return 501 NotImplemented (ACLs, CORS, policies, etc.) | Implement each feature as needed post-MVP | All tests in skipped/unimplemented categories | bucket.rs |
| TD-010 | SSE-C multipart | SSE-C headers on CreateMultipartUpload and UploadPart are rejected with InvalidArgument. SSE-C + multipart requires encrypting each part independently and tracking per-part nonce prefixes |
Implement SSE-C for multipart: per-part encryption, nonce tracking, assembly of encrypted parts at complete time | test_ssec.py::TestSsecMultipartRejection |
multipart.rs, object.rs |
| TD-014 | recover composite blobs |
Composite blobs (the result of the optimised CompleteMultipartUpload) live as a sidecar that lists the part blob ids — no on-disk file exists at the composite's blob path. recover::process_sidecar rejects them as "orphaned sidecar (blob file missing)"; the caller prints a WARNING and skips the sidecar, so every multipart object is silently dropped from the rebuilt database. Worse, the retained part blobs carry ordinary sidecars whose key is {key}#{upload_id}#{part_number} (handlers/multipart.rs), so recover re-creates each part as a bogus object under that key. (Closed parts: parts produced by UploadPartCopy used to get a sidecar only when encrypted; UploadPartCopy now writes the same part sidecar as UploadPart for every part, so copied parts take the composite fast path and share the {key}#{upload_id}#{part_number} problem above. The fsck half is resolved: arca fsck computes the referenced set exactly as blob GC does, blob_gc::referenced_blob_ids / add_live_composite_parts, and reports a composite with an absent part as MISSING_BLOB with the missing part ids.) recover therefore misbehaves on any data directory that has experienced multipart uploads after the composite-concat optimisation. Runtime S3 reads/writes are unaffected — the request path fully handles composites. |
recover: when a sidecar has composite: Some(parts), skip the blob-existence check, walk the part list, and verify each referenced part blob exists. Insert one ObjectRecord per composite that points at the composite blob_id (the part records are not surfaced to the metadata DB — they are a storage-layer detail), and do not re-create part sidecars ({key}#{upload_id}#{n}) as objects. Unit tests with fixtures that mix composite and ordinary blobs (the fsck fixtures in fsck.rs are a model). |
None at the runtime level; recover on data with composites drops every multipart object |
crates/arca-server/src/recover.rs:206 |
| TD-017 | time pinned to =0.3.47 |
time 0.3.48 adds From impls that clash (E0119 coherence error) with rcgen's blanket From conversions whenever time's parsing/formatting features are enabled elsewhere in the graph (they are, via transitive deps). 0.3.47 already contains the CVE-2026-25727 fix, so no security exposure — this is a freshness pin only |
Unpin when rcgen or time resolves the conflict upstream (track rcgen releases past 0.14.8) | None | Cargo.toml |
| TD-018 | Re-encryption skips multipart/composite | The encrypt / decrypt maintenance jobs (and the encrypt-existing / decrypt-existing CLI) skip multipart/composite objects: they are detected by their <32 hex>-<n> ETag and excluded from the candidate scan, so a store with multipart objects is only partially re-encrypted/decrypted. Related to TD-014 — composite objects live as a composite sidecar listing part blob ids, with no single on-disk blob to rewrite via copy-on-write. SSE-C objects are also skipped (the server never holds the customer key). |
Teach re-encryption to walk a composite's part list, rewrite each part blob copy-on-write, and rebuild the composite sidecar with the new part ids and (de/en)cryption state — sharing the composite-walking logic added for TD-014. Add unit + integration coverage with mixed plain/multipart fixtures. | None at the runtime level; mixed-content stores re-encrypt only their single-blob objects | crates/arca-server/src/maintenance.rs (is_recrypt_candidate, is_multipart_etag), crates/arca-storage/src/recrypt.rs |
| TD-019 | migrate-db online direction limited to postgres→sqlite |
The online (maintenance-job) migrate-db can only run postgres → sqlite: Arca auto-detects the running backend from the config (PostgreSQL when [storage.postgres] is present), so a live SQLite instance has no PostgreSQL section to point the source at and cannot launch a sqlite→postgres job online. The sqlite→postgres direction is done with the offline CLI (arca migrate-db --to postgres), which can open both backends explicitly. |
Either accept the target [storage.postgres] connection parameters in the job params so the online direction can target an external PostgreSQL without it being the running backend, or document the offline CLI as the supported sqlite→postgres path (current state). |
None | crates/arca-server/src/maintenance.rs (process_migrate_db), crates/arca-server/src/migrate_db.rs |
| TD-020 | Deferred integration tests: cold-CLI re-encryption + 3-node cluster round-trips | The offline-CLI re-encryption path (encrypt-existing / decrypt-existing against a stopped server) and the 3-node cluster behaviours of re-encryption (one-time full re-replication of rewritten blobs via anti-entropy) and migrate-topology (single→cluster→single round-trip across real nodes) are currently covered by unit tests and the online single-node integration suite only — there is no dedicated cold-CLI or cluster integration phase for them. |
Add a cold-CLI integration phase (stop server, run the CLI, restart, verify) and a cluster integration phase (re-encrypt on the worker-leader, assert convergence on peers; migrate-topology round-trip). | None (covered by unit tests meanwhile) | tests/ (no cold-CLI/cluster phase yet), crates/arca-server/src/{recrypt_existing,migrate_topology}.rs |
| TD-021 | Cluster re-encryption defers old-blob reclaim to GC | After a copy-on-write re-encryption swap, the old blob is deleted eagerly on a single node but left for the grace-bounded anti-entropy GC in a cluster, so out-of-band reclaim never races the blob-repair pulls peers issue while converging onto the re-encrypted row. Correctness is not affected (content and lock state now converge on independent LWW dimensions — content_updated_at for blob/algorithm/key vs lock_updated_at for retention/legal-hold — so a concurrent lock op can no longer revert blob_id; migrations sqlite v25 / pg 0013). The only residual is a transient extra copy of each re-encrypted object's old blob on every node until the next GC pass reclaims it. |
Once the 3-node cluster re-encryption phase (TD-020) confirms it, consider reclaiming the old blob eagerly in the cluster path too (the new blob is already CAS-committed and propagates on its own dimension), or trigger a targeted GC sweep at job completion instead of waiting for the periodic pass. | None (unit tests cover the independent-merge convergence: apply_remote_reencryption_and_lock_change_merge_independently, apply_remote_reencrypted_row_converges_via_content_updated_at) |
crates/arca-server/src/maintenance.rs (recrypt_one, RecryptCtx::clustered), crates/arca-core/src/types.rs (resolve_replicated) |
| TD-022 | migrate-db online job is not cancellable mid-copy (there is no online migrate-topology job; that is CLI-only) |
The migrate-db maintenance job copies every metadata table in one pass; unlike the re-encryption jobs it does not check the pause/cancel flag between tables, so a cancel request only takes effect once the whole copy finishes (or the job errors). The destination is left with a partial copy on error, which the operator must drop before retrying. |
Check the job status between tables (and ideally between row batches) and abort cleanly, truncating or dropping the partially-populated destination tables on cancel. Surface a "not resumable, will restart from scratch" note in the console. | None | crates/arca-server/src/maintenance.rs (process_migrate_db) |
| TD-023 | Offline migrate-db has no crash-safe checkpoint |
The offline arca migrate-db CLI copies every table to the destination in a single process with no checkpoint; a crash or SIGKILL mid-run leaves the destination partially written and the operator must drop it before retrying. Row-count reconciliation at the end catches a torn copy, but only if the process survives to print it. (The migrate-topology CLI is not affected: its operations are idempotent/in-place — tombstone purge is safely re-runnable, VACUUM is a no-op on restart.) |
Write a resume marker (last-completed table / cursor) to the destination or a sidecar file and skip already-copied tables on restart; or wrap the whole copy in a single destination transaction where the backend allows it. | None | crates/arca-server/src/migrate_db.rs, crates/arca-storage/src/migration.rs |
| TD-024 | rustls-webpki 0.102.8 pinned transitively (vulnerable code paths unreachable) |
rumqttc 0.25.1 (MQTT notification connector) hard-pins rustls-webpki = "^0.102.8", which carries 4 published advisories, tracked upstream as GHSA-82j2-j2ch-gfr8 (HIGH, DoS panic on a malformed CRL BIT STRING, fixed in 0.103.13), GHSA-pwjx-qhcg-rvj4 (MEDIUM, CRLs not treated as authoritative by Distribution Point, fixed in 0.103.10), GHSA-xgp8-3hg3-c2mh and GHSA-965h-392x-2mh5 (LOW, name constraints wrongly accepted for wildcard and URI names, fixed in 0.103.12); these are the same four issues previously recorded here by their RUSTSEC IDs (RUSTSEC-2026-0104/0049/0098/0099). The vulnerable code is not reachable from Arca. Cargo.lock carries two copies of the crate: 0.102.8, whose only dependent is rumqttc, and 0.103.13, which is already patched for all four and is what rustls 0.23.40, async-nats, rustls-connector and rustls-platform-verifier resolve to. rumqttc's entire use of its 0.102.8 copy is the WebPki(#[from] webpki::Error) variant in rumqttc/src/tls.rs; it never configures a CRL, so both CRL advisories are dead code, and MQTTS certificate path validation actually runs through tokio-rustls -> rustls 0.23.40 -> rustls-webpki 0.103.13. No fix is available upstream: 0.25.1 is the newest published rumqttc, bytebeamio/rumqtt master still declares rustls-webpki = "0.102.8", no patched 0.102.x release exists, and a [patch.crates-io] override to 0.103.13 would be reported as unused because it cannot satisfy ^0.102.8 (forcing it would need a rumqttc fork). Dependabot alerts #1 to #4 on dxc-technology/arca were dismissed as not_used on 2026-08-22, referencing this entry. As of 2026-09-23 the same four GHSA IDs are also suppressed in trivy fs output via .trivyignore (repo root), which points back at this entry. |
Track rumqttc releases that relax the rustls-webpki requirement and bump as soon as one lands; if upstream stays dormant, evaluate rumqttc's native-tls backend or a maintained fork. Re-assess immediately if the MQTT connector ever starts configuring CRLs, or if rumqttc begins calling into webpki for path validation, since either would make the advisories reachable — at that point also remove the .trivyignore entries. |
None (MQTT connector integration tests pass; the vulnerable code paths are unreachable as analysed above) | Cargo.toml, Cargo.lock, .trivyignore |
| TD-025 | Cluster conditional writes (CAS) | Cluster-wide conditional-write atomicity is not provided: If-Match / If-None-Match / x-amz-if-match-* preconditions are evaluated authoritatively only against the serving node's own state (put_object_if/delete_object_if/delete_object_version_if commit locally, then fan out — there is no per-key authoritative node to arbitrate across peers). Concurrent conditional writes for the same key landing on two different nodes at once can both pass their own local CAS and both be accepted (200 OK); the row then converges by last-writer-wins, the same as any other conflicting write. Single-node CAS (the common case, and the whole point of the conditional-write-atomicity fix) is unaffected — see crates/arca-storage/src/{sqlite,pg}/metadata.rs. Mitigated operationally by pinning a given key's conditional-write traffic to one node (a sticky/session-aware load balancer — see documentation/docs/guide/ha.md). |
Owner-node forwarding for conditional writes: the node that receives a conditional PUT/DELETE for a key it doesn't own forwards it to that key's designated owner node instead of committing locally, giving cluster mode the same per-key authoritative arbitration single-node CAS already has. | None (cluster CAS boundary is pinned by tests/integration/test_cluster.py::test_conditional_put_is_per_node_not_cluster_wide; single-node CAS is fully covered by tests/integration/test_conditional_writes.py) |
crates/arca-server/src/cluster/cluster_meta.rs (put_object_if, delete_object_if, delete_object_version_if) |
| TD-032 | arca recover rebuilds a wrong database |
Besides TD-014: (1) entries are sorted by last_modified descending and buckets are re-created unversioned, so each put_object overwrites the previous one and the oldest version of every key wins, while the code comment says the newest is inserted last; (2) only buckets, objects and credentials are restored — users, teams, grants, every bucket_config (versioning, encryption, lifecycle, lock, policy, notifications, replication), bucket/object tags, retention and legal hold are lost, owner is forced to root, and the checksum and part layout of multipart objects (TD-046) are not restored, since sidecars carry neither; (3) compressed objects fail checksum verification (the ETag is the plaintext MD5, recover hashes the on-disk bytes) and are skipped unless --skip-verify; (4) it always opens a SQLite store, so on a PostgreSQL deployment it writes an unused SQLite file; (5) PutObject, CopyObject and CompleteMultipartUpload write the sidecar with version_id: None before the metadata commit assigns the version, so sidecars never carry it and a rebuild cannot restore version ids. operations/recovery.md documents none of this and claims the opposite for (1). |
Insert newest-last (and re-create versioned buckets as versioned, keeping version_id), verify compressed objects against the decompressed stream, support the configured backend, and document precisely what a sidecar rebuild can and cannot restore. Unit tests with multi-version, compressed and composite fixtures. |
None (no integration coverage of recover on such data) |
crates/arca-server/src/recover.rs, documentation/docs/operations/recovery.md |
| TD-033 | Object tags never converge in a cluster | put_object_tags / delete_object_tags replicate only through a best-effort ControlOp::ObjectTags fan-out (cluster_meta.rs). They bump no seq, and object tags are not part of ControlSnapshot, so anti-entropy never repairs them: a node that was down or unreachable during a tag change keeps the old tags until the object itself is rewritten. guide/ha.md lists tags among the data that self-heals. |
Carry object tags in anti-entropy, either by bumping the object row's seq and shipping tags with the manifest entry, or as a per-object-version LWW family in ControlSnapshot. Cover with a cluster test that tags an object while a node is down. |
None | crates/arca-server/src/cluster/cluster_meta.rs, crates/arca-core/src/cluster.rs (ControlSnapshot), crates/arca-storage/src/{sqlite,pg}/metadata.rs |
| TD-040 | Cluster maintenance jobs can stall on a non-leader node | Maintenance jobs are stored in each node's local MaintenanceStore (not cluster-wrapped, not part of ControlSnapshot), while only the worker-leader (lowest eligible node_id, decision H5) runs the maintenance worker. A job created through the load balancer on any other node is never picked up until that node happens to become leader; the API accepts it and it stays pending. guide/maintenance.md and guide/console.md do not mention this. |
Route job creation (and pause/cancel) to the worker-leader over /cluster/v1/*, or replicate the jobs table as a control-plane family and let the leader claim jobs. Until then, document that jobs must be created on the leader node. |
None | crates/arca-server/src/maintenance.rs (spawn_maintenance_worker), crates/arca-proto/src/handlers/maintenance.rs, crates/arca-server/src/main.rs (store wiring) |
| TD-043 | Remaining test tooling that disturbs the developer's stack | After TD-030 the self-contained bin/test modes run in their own compose projects, but: bin/screenshots still runs in the developer's project (overwrites config/.generated.toml, up -d arca, down, and its cleanup deletes .arca-env); arca-tls-test-certs is one fixed-name volume shared by bin/test tls and bin/perf-test --tls, so the two cannot run concurrently. (The cluster suite already runs in its own project on ephemeral loopback ports. bin/s3-tests wiping docker_arca-data is deliberate: the Ceph suite needs a clean state; --no-clean skips it.) |
Apply use_test_project to bin/screenshots; name the TLS test volume per project. |
None | bin/screenshots, bin/lib/compose.sh (make_test_certs) |
| TD-044 | compress-existing compresses ciphertext of encrypted blobs (guard shipped, repair open) |
Guard shipped (Unreleased): compress-existing now skips encrypted (SSE-S3/KMS and SSE-C) and composite blobs, and decompress-existing skips composite and encrypted blobs, each with a SKIP message, never aborting the run. Remaining: repairing stores damaged by the old behaviour. Original problem: arca compress-existing (run_compress_existing) never checks meta.encryption (nor composite blobs): on an SSE-S3 encrypted blob it compresses the ciphertext file in place and records compression in the sidecar, producing ACMP(AENC…), the reverse of the live compress-then-encrypt order, with original_size set to the plaintext size although the frame holds ciphertext. Such objects are unreadable (parse header: invalid magic bytes); the bytes and DEK are intact, so they are recoverable. decompress-existing cannot undo it (its decompressed-size check against original_size fails, and the error aborts the whole run), and either tool aborts on composite blobs, which have no file. Found 2026-10-01 while fixing TD-029. |
Repair only (the going-forward guard is done; routing encrypted blobs through the real store stack is optional future work): for sidecars with both encryption and compression whose raw file starts with ACMP, decompress the raw file back to the AENC ciphertext (verify against the ciphertext length, not original_size), then clear compression or re-layer properly. Tests with encrypted and composite fixtures. |
None | crates/arca-server/src/compress_existing.rs |
| TD-050 | Small-PUT throughput plateau on ARM, not yet explained | On an ARM single-board computer (RK3588, 8 cores) the no-fsync baseline saturated at about 200-270 small (1 KiB) PUTs per second whatever the concurrency, with the CPU mostly idle (2026-10-02). The same build does about 400 PUT/s at concurrency 1 on an x86 VM (4 vCPU) and about 2,400 at concurrency 16 on an AWS c8id.xlarge, so it is not a general limit. Not investigated. Leading hypothesis to verify first: one saturated thread (the single SQLite writer, which until TD-038 also committed one audit batch per request) looks like an idle machine on 8 cores. | Profile a small-PUT run on ARM (per-thread CPU with top -H / perf, SQLite writer queue time) with the audit batching now in place; group commit (TD-048, resolved 2026-10-20) has since lifted the single-writer commit cap, so re-measure on ARM first; if the plateau remains, track the real cause here. |
None (throughput only) | Unknown until investigated |
| TD-052 | Unfinished multipart uploads are unbounded, and only the CLI reports them | The parts of a multipart upload that is never completed or aborted are live data (by design), and nothing aborts them unless the operator adds a lifecycle AbortIncompleteMultipartUpload rule. Seen 2026-10-14 on a production single-node server: nightly backups of a backup tool that uploads in parts failed with write blob: No space left on device, the client never aborted its uploads, and the leftovers (about 450 GB in 179 uploads, 12 more per night) kept the disk full, so no backup was restorable for 14 days while the buckets held under 500 GB of objects. Arca answered every failed part with a 500 as it should; the gap is that nothing bounds what the client leaves behind. Done: arca fsck reports each unfinished upload (UNFINISHED_UPLOAD: age, parts, size) and is composite- and parts-aware, so it agrees with arca gc; the list_object_versions pagination is fixed in both stores; arca gc prints the count and size; arca fsck --repair --abort-uploads-older-than <days> aborts them. Still open: no metric, nothing in the console, no default bound. |
(1) A metric with the count, bytes and age of the oldest unfinished upload, and the same in the console (bucket view and dashboard). (2) Consider a safe default for AbortIncompleteMultipartUpload, or a prominent warning when none is set and unfinished uploads are older than N days. |
None | crates/arca-server/src/worker.rs (metrics), console/ |
| TD-055 | Per-IP rate limit trusts X-Forwarded-For from any client |
extract_client_ip keys the per-IP limiter on the first X-Forwarded-For address whenever the header is present, and only falls back to the TCP peer (ConnectInfo, inserted by serve.rs) without it. A client that talks to Arca directly can send a different fake address on every request and never be limited; the audit log records the same unverified header |
First step: record the TCP peer address in the audit log next to the header-reported one (a new audit column, shown by the admin API and the console), so a forged header can no longer hide the real origin. Then a [server] trusted_proxies list (addresses or CIDRs): honour X-Forwarded-For / X-Real-IP only when the peer is a trusted proxy, taking the right-most untrusted address; default none, so direct deployments use the peer address |
rate_limit.rs / audit.rs unit tests |
crates/arca-proto/src/middleware/rate_limit.rs, crates/arca-proto/src/middleware/audit.rs |
| TD-056 | Lifecycle never removes delete markers | Expiration with ExpiredObjectDeleteMarker is accepted, stored and returned by GetBucketLifecycleConfiguration, but the evaluator skips it, so an expired object delete marker (a marker with no noncurrent version left behind it) stays forever. NoncurrentVersionExpiration hard-deletes noncurrent versions but never noncurrent delete markers. Nothing is lost, the markers only pile up in ListObjectVersions |
Implement ExpiredObjectDeleteMarker (remove a latest delete marker once it is the only version of its key) and include noncurrent delete markers in NoncurrentVersionExpiration, as S3 does; both delete things that are kept today, so release-note the change | test_lifecycle.py |
crates/arca-server/src/worker.rs |
| TD-057 | Fresh blob orphaned after a failed metadata write | PutObject, CopyObject, UploadPart, UploadPartCopy and CompleteMultipartUpload write the blob and its sidecar, then the metadata row. When that write fails with a database error, the handler answers 500 and keeps the blob: only a refused precondition (412, NoSuchKey, NoSuchUpload), which is known to have written nothing, deletes it. Deleting on every error would be unsafe: a cluster quorum shortfall (503) comes after the local commit, which is not rolled back (decision H1), and an error at COMMIT itself (a dropped PostgreSQL connection, a SQLite I/O error) leaves the outcome unknown, so the row may exist. The orphan wastes space until gc reclaims it after its grace period, and until then arca recover, which rebuilds rows from sidecars, would bring the failed write back as an object |
The stores report a definite "not committed" (an error raised before COMMIT, with the transaction rolled back) as its own error kind, through the cache and cluster decorators, and the five handlers delete the fresh blob (the assembled one for Complete, never its parts) only on that kind | none yet: needs a store that fails on demand | crates/arca-proto/src/handlers/object.rs, crates/arca-proto/src/handlers/multipart.rs |
Legend¶
- Tests Affected: Ceph s3-tests that are impacted (either currently failing, or passing due to the workaround)
- Files: Source files containing the workaround (abbreviated — full paths under
crates/) - Items are moved to the Resolved table when properly implemented
How to Use¶
- When adding a new workaround, assign the next
TD-XXXID - Add a
// TECHDEBT(TD-XXX): brief descriptioncomment in the source code - Add a row to the Active Items table above
- When implementing the proper fix, remove the code comment and move the row to the Resolved table below
Resolved Items¶
| ID | Area | Resolution | Status |
|---|---|---|---|
| TD-001 | Owner identity | Owner ID derived from credential's user. Buckets and objects store creator's username (correction 2026-10-01: this was untrue until TD-042, which made both buckets and objects record the requester). Migration v8 adds owner columns and user_id on credentials. Phase 16 |
✔ |
| TD-002 | Storage class | storage_class field added to ObjectRecord, DB migration v13, x-amz-storage-class header accepted on PutObject, returned in list and head responses. Phase 22 |
✔ |
| TD-003 | Versioning | Full object versioning: PutBucketVersioning/GetBucketVersioning, version IDs, delete markers, version-specific GET/HEAD/DELETE/COPY, ListObjectVersions with real data. Migration v9 adds versioning columns. Phase 17 | ✔ |
| TD-004 | Region | Region configurable in [server] TOML section or via Admin API (/admin/settings/region). Per-bucket region via bucket_config. HeadBucket and GetBucketLocation return effective region. Phase 18 |
✔ |
| TD-005 | Request ID | Error XML placeholder <RequestId> replaced by the request-ID middleware with the real x-amz-request-id value |
✔ |
| TD-006 | Encryption config | PutBucketEncryption / GetBucketEncryption / DeleteBucketEncryption implemented with SSE-S3 (AES-256-GCM). Per-bucket config in bucket_config table, global default from [encryption] config section. Phase 13 |
✔ |
| TD-008 | Content-Type source | Verified: Content-Type captured from CreateMultipartUpload matches AWS behavior. Clients set it at init time. Phase 22 | ✔ |
| TD-009 | Audit write contention | Audit writes are batched through a bounded mpsc channel (capacity 10,000) with a dedicated writer task doing bulk inserts in a single transaction (middleware/audit.rs, state.rs); since TD-038 the writer also commits in 1 s batches. |
✔ |
| TD-011 | time crate CVE |
Docker builder moved from rust:1.85-alpine to rust:alpine (currently 1.96); time updated to 0.3.47, which contains the CVE-2026-25727 fix. The MSRV pins for home/serde_with/darling in the Dockerfile became unnecessary and were removed. (0.3.48+ stays excluded for a non-security reason — see TD-017.) Dependency refresh, 2026-06-12 |
✔ |
| TD-012 | rustls-pemfile unmaintained |
PEM parsing migrated to rustls-pki-types (PemObject: pem_slice_iter / from_pem_slice) in tls.rs and tls_generate.rs; the rustls-pemfile dependency is gone from the workspace. Dependency refresh, 2026-06-12 |
✔ |
| TD-013 | AMQP/Kafka connector tests | Root cause was NOT DNS: the AMQP receiver crashed on startup because docker exec rabbitmq-diagnostics (run as root in a tight wait_for_amqp_receiver loop) raced with RabbitMQ's cookie initialization, producing an EACCES error that killed the container. Fix: replace the exec-based wait for AMQP/Kafka with a docker inspect health poll (_wait_for_container_healthy), so the receiver is probed by the container's own healthcheck only. Kafka tests also had two unrelated test-side issues: consumer-group coordination raced with the first produce (fixed by switching to manual partition assign + seek_to_end), and the connectivity-failure test timed out because Arca's Kafka test() can take ~10s on unreachable hosts while the test HTTP timeout was also 10s (fixed by raising the conftest SigV4 default to 30s). |
✔ |
| TD-015 | Cluster inter-node TLS cert verification | Verified mutual TLS with an operator-distributed cluster CA ([cluster.tls] — REQUIRED when the cluster runs over HTTPS, no insecure fallback): inter-node reqwest clients trust the CA and present the node's CA-signed client identity; danger_accept_invalid_certs removed from membership.rs and client.rs. The listener requests client certificates (optional at the TLS layer — S3 clients share the port) and /cluster/v1/* refuses requests without a CA-verified one. Material minted by arca tls generate-cluster. HA hardening R4 (decision H12) |
✔ |
| TD-016 | Cluster control-plane reconcile was partial | The anti-entropy snapshot reconcile now covers EVERY control-plane family: grant attachments (user_grants, team_grants) and team memberships (per-row updated_at, migrations sqlite v22 / pg 0010, composite-key tombstones on detach/remove), bucket_config per-key LWW, bucket_tags as one set-level entity per bucket, server_config per-key LWW (node-local keys excluded at build AND on apply), plus in-progress multipart uploads and parts (D4). Child upserts are parent-filtered in plan_control_merge (a row whose parent resolved deleted is never adopted — replaces cascade tombstones and respects the PG join-table FKs). A returning node now fully self-heals on all families. HA hardening R5 (review D9/D4) |
✔ |
| TD-026 | bin/test tls could not create certs/ on Linux |
The self-contained TLS suites (bin/test tls, bin/perf-test --tls) no longer write into a host bind mount: make_test_certs in bin/lib/compose.sh generates a throwaway CA + server certificate into the named volume arca-tls-test-certs (ownership is real there on every host, as in run_tls_permissions), mounted through ARCA_CERTS_DIR, and removed afterwards; clients verify against its CA. The tls-init compose service, the only writer into certs/, is gone; bin/arca start --tls keeps its own local certificates in certs/local with arca tls ensure, run as the host user with group 65532. |
✔ |
| TD-027 | trivy fs misconfig: k8s console manifest |
deploy/kubernetes/arca-console.yaml pins the console image to the release tag (KSV-0013; the tag is a version location, listed in step 4 of RELEASING.md), runs with readOnlyRootFilesystem: true (KSV-0014, with an emptyDir for every path nginx writes to and an init container that seeds /usr/share/nginx/html and /etc/nginx/http.d), and now drops every capability without adding any back (KSV-0117, KSV-0022). That needed the console image to listen on unprivileged ports, 8080 for HTTP and 8443 for HTTPS (were 80 and 443), with no file capability on the nginx binary: a setcapped binary does not start under drop: ["ALL"]. The image's user is renumbered from the package's 100:101 to 10001:10001 (KSV-0020, KSV-0021) and USER is numeric. The Service keeps port 80 (targetPort: http), host ports are unchanged (9080, 9443), and the image's HEALTHCHECK, the Compose files, bin/test tls-permissions and the docs follow. bin/test tls-permissions gained a case that runs the console with --cap-drop ALL and checks its user. Verified with trivy fs: no misconfiguration or vulnerability finding is left. Two findings that were never part of this item are accepted in .trivyignore, each with its reason: KSV-0125 (ghcr.io, Arca's official registry, is not in the check's allow-list) and KSV-0013 on arca-cluster.yaml (latest is its documented evaluation default). The change is incompatible for anyone running the published console image with their own port mapping, reverse proxy or manifest (see the CHANGELOG entry). |
✔ |
| TD-028 | trivy fs misconfig: root user / missing HEALTHCHECK in test-only Dockerfiles |
docker/screenshots/Dockerfile now runs as the Playwright image's own pwuser (uid 1001; Chromium launches without --no-sandbox as that user, checked in the built image). The output directory is a bind mount, so on a Linux host it must be writable by uid 1001. docker/Dockerfile.perf, docker/Dockerfile.test, docker/s3-tests/Dockerfile and the screenshots image declare HEALTHCHECK NONE: they are short-lived batch runners with no service to probe, so any real probe would be invented; the instruction satisfies DS-0026 and states that explicitly. |
✔ |
| TD-029 | Compression + encryption read failed | Root cause on the read side: with a CompressingBlobStore above an EncryptingBlobStore, the encrypting layer encrypts the compressed frame of the object, but EncryptingBlobStore::get took the length of what it had encrypted from the sidecar's size, which is the object's uncompressed size. Ranged reads (the compressing layer reads its footer and chunks through ranged reads) were therefore clamped and their last chunk located in the wrong space: read chunks: early eof when the frame is shorter than the object, compressed footer count truncated (an overflow in debug builds) when it is longer. It now uses compression.compressed_size when the sidecar records compression (payload_size in encrypted_blob.rs). The write side was always correct, so objects already stored compressed and encrypted are read correctly by the fixed server; no repair is needed. Covered by five stacked-store unit tests in compressed_blob.rs and by a second, encrypted pass of bin/test compression (run_compression), which also gained incompressible-body, suffix-range, multipart and copy tests. |
✔ |
| TD-030 | Self-contained test modes destroyed the developer's server | Every self-contained bin/test mode and bin/perf-test's self-contained mode now call use_test_project <name> (bin/lib/compose.sh) first: compose_cmd passes -p arca-test-<name>, so containers, networks and the arca-data volume are the suite's own and its down -v removes only them; build_config writes config/.generated-test-<name>.toml, which docker-compose.config.yml mounts through ARCA_GENERATED_CONFIG; every host port a test service publishes (ARCA_HOST_PORT, ARCA_REPLICA_HOST_PORT, and the new OPENBAO_HOST_PORT, POSTGRES_HOST_PORT, WEBHOOK_RECEIVER_HOST_PORT) becomes an ephemeral loopback port, since the suites reach their services over the compose network and a fixed 9000 would collide with the running server (the TLS overlay's redundant fixed 9000:9000 is gone). .arca-env is written by bin/arca only. presigned/ssec and bin/perf-test without features, which start nothing, now run with --no-deps against the running server instead of recreating it. Covered by tests/scripts/test_isolation_test.sh (bin/test scripts, docker mocked) |
✔ |
| TD-031 | Encryption/KMS integration tests skipped under bin/test integration |
run_integration adds integration_feature_env (bin/lib/compose.sh) to the compose run options: from the features recorded in .arca-env, encryption sets ARCA_ENCRYPTION_ENABLED, kms sets ARCA_ENCRYPTION_ENABLED and ARCA_KMS_ENABLED, encryption-per-bucket / kms-per-bucket set ARCA_PER_BUCKET_ENCRYPTION (per-bucket and re-encryption suites; test_kms.py asserts global encryption, so not for a per-bucket server). Each mapping was run against a real server of that kind |
✔ |
| TD-034 | Request body integrity was never verified | PutObject and UploadPart now verify (through every blob stack: plain, SSE-S3/KMS, compressed, and SSE-C for PutObject) the body as AWS does, in handlers/integrity.rs with the body stream from body::verified_body_stream: Content-MD5 against the plaintext MD5 the blob store already returns as the ETag (mismatch BadDigest, malformed InvalidDigest); x-amz-checksum-{crc32,crc32c,crc64nvme,sha1,sha256} from a header or from an aws-chunked trailer named by x-amz-trailer (the decoder now parses trailers instead of stopping at the last chunk), computed while the body streams (mismatch BadDigest "The InvalidRequest); a hex x-amz-content-sha256 (mismatch XAmzContentSHA256Mismatch). Malformed declarations are refused before the body is read; a mismatch deletes the written blob before any sidecar or row exists. Trailer checksums (boto3 over HTTPS) are now recorded too, SHA1 is supported, UploadPart returns and stores its checksum, CreateMultipartUpload records x-amz-checksum-algorithm. FsBlobStore::put now removes its temp file when the body stream fails (it leaked one, e.g. on EntityTooLarge). Left out: chunk-signature verification (TD-045) and multipart object checksums (TD-046). Unit tests in integrity.rs / body.rs, integration suite test_integrity.py (41) |
✔ |
| TD-035 | UploadPartCopy ignored copy-source conditionals |
upload_part_copy now calls check_copy_source_conditionals (x-amz-copy-source-if-match, -if-none-match, -if-modified-since, -if-unmodified-since) right after resolving the source and before any source byte is read, as copy_object does; a failed condition answers 412 PreconditionFailed and stores no part. Tests: test_copy_source_conditions.py, a passing and a failing case for each header on both CopyObject and UploadPartCopy (the four failing UploadPartCopy cases failed before the fix) |
✔ |
| TD-036 | Cluster drift fingerprint ignored cluster_size |
config_fingerprint hashed the write quorum (n/2+1) but not cluster_size, so a resize between 2 and 3 or 4 and 5 went unflagged. Fixed with a second fingerprint, config_fingerprint_v2 (domain tag arca-cluster-cfg-v2), that adds cluster_size in quorum mode only (in available mode the setting has no effect, so a difference there is not drift). The ping carries both: v1 stays in config_fingerprint, so a node of the previous version keeps matching and a rolling upgrade shows no drift (a drifted peer is excluded from the write quorum, H7); a node compares v2 only with a peer that sends it, so cluster_size drift is caught between upgraded nodes. |
✔ |
| TD-037 | Virtual-hosted-style requests were routed wrong | Confirmed by a test, not just by reading the code: with server.domain set, bucket.domain/key was dispatched as bucket key (NoSuchBucket), and the bucket root as ListBuckets, because virtual_host_middleware was attached with Router::layer, which runs after routing. The rewrite now lives in NormalizeLayer (outside the router, NormalizeLayer::new(domain)), after saving the OriginalUri auth verifies and before the trailing-slash rule, so a virtual-hosted /dir/ stays a directory-marker key; the host comes from Host or the HTTP/2 URI authority. The pure rules are bucket_for_host and path_style_path in middleware/virtual_host.rs; the dead middleware and AppState.domain are gone. Tests: unit tests for the rules and for the layer, and test_virtual_host.py (put/get/head/delete, bucket root, query string, missing bucket) run by the new self-contained bin/test virtual-host; five of its six tests failed on the old code |
✔ |
| TD-038 | No fsync on the write path | Every blob and sidecar writer in arca-storage is now durable: the blob temp file is sync_alled before the rename (in the spawn_blocking worker of FsBlobStore::put; write_raw, the concat fallback and re-encryption likewise), newly created shard directories are fsynced up to the first existing ancestor (create_dir_all_durable), and write_sidecar writes a temp file in the same directory, fsyncs it, renames it and fsyncs the directory (write_file_atomic), so a sidecar is never partial. Invariant: a blob is durable once its sidecar has been written — the blob rename in put / write_raw / the concat fallback does not fsync the directory, because the sidecar lives in the same directory and its directory fsync covers both entries; every handler writes the sidecar before the metadata commit and fails the request (discarding the fresh blob, handlers/sidecar.rs) when that write fails, and UploadPartCopy now writes a part sidecar for every part. 3 fsyncs per small PUT; the in-place re-encryption rename, whose follow-up sidecar write is not durable, keeps rename_durable. SQLite commits with synchronous=FULL, so the metadata row survives a power loss too. The cost is significant on slow disks, so it is switchable: [storage] fsync (default true; false = no fsync and SQLite NORMAL, the old behaviour, still atomic) applies to the live server only, the offline tools are always durable. Audit entries are committed in batches (1024 entries or 1 s, at once on drain) so that audit logging does not add an fsync per request. Measured 2026-10-02..05 with warp, 1 KiB PUTs at concurrency 16, against no fsync: local NVMe -37%, gp3 EBS -78%, Hyper-V virtual disk -77%, LUKS HDD about -90%; 1 MiB -6..-20% on NVMe and concurrent gp3; 512 MiB 0-2%. The single-writer FULL ceiling on slow disks is tracked as TD-048. Unit tests cover the atomic temp + rename protocol in both modes and the synchronous level (fsync itself cannot be observed without a power cut) |
✔ |
| TD-039 | PostgreSQL CAS regression test | bin/test postgres now also runs test_conditional_writes.py against the PostgreSQL-backed server (36 tests with test_postgres.py), so a reordering of the CAS read relative to the transaction's first locking statement in pg/metadata.rs is caught. |
✔ |
| TD-041 | Blob ids from cluster peers were not validated | parse_blob_id guarded only the /cluster/v1/blob/{blob_id} path parameter: ids inside a received ObjectRecord, inside the composite parts of a received sidecar and inside anti-entropy manifest rows reached FsBlobStore::blob_path unchecked, which pushed the id onto the blobs path (traversal with .. or /) and sliced it by byte index (panic on short or multi-byte ids). Authenticated peers only (decision H12), so defence in depth. Fixed: BlobId::is_valid (exactly the lowercase hyphenated UUID form BlobId::new mints); blob_path is total and maps any other id to a fixed name inside the base directory; the raw operations (read_raw, write_raw, exists, read_sidecar, write_sidecar, delete_blob_file) reject it; receive_object and receive_blob answer 400 for an invalid record id or composite part id; anti-entropy skips a manifest row with an invalid id (so it cannot wedge the cursor) and refuses a peer sidecar naming an invalid part. Files under blobs/ that are not blob ids are no longer enumerated as blobs (they were GC candidates); arca fsck reports them (FOREIGN_FILE) and arca gc --delete-foreign-files removes them. |
✔ |
| TD-042 | Object owner was always root |
PutObject, CopyObject and CompleteMultipartUpload built the ObjectRecord with owner: "root" whatever the credential, and the same hard-coded value reached delete markers, buckets (create_bucket(name) had no owner parameter, so the column default applied), multipart uploads (ListParts and ListMultipartUploads printed root as Owner and Initiator), ListObjects V1 (no <Owner> at all) and the notification userIdentity.principalId (always empty). Fixed: requester_username (middleware/identity.rs) gives the authenticated user's name, recorded by the three object handlers, by delete_object_if (new marker_owner argument; lifecycle expiration keeps passing root), by create_bucket_owned (create_bucket stays as the root shortcut; admin import keeps the exported owner) and by CreateMultipartUpload (new initiator column, SQLite migration 27 / PostgreSQL 0015). ListObjects V1 now returns <Owner>; the event principalId is the requester's username. CopyObject gives the copier as owner, not the source's; CompleteMultipartUpload gives the user that completes it. Replicated rows keep the origin's owner. Objects, buckets and uploads written before the fix keep root: the writer was never recorded. Renaming a user does not rewrite owners. arca recover still writes root (sidecars carry no owner, see TD-032). |
✔ |
| TD-045 | aws-chunked chunk and trailer signatures were not verified | For STREAMING-AWS4-HMAC-SHA256-PAYLOAD (and -TRAILER) bodies the decoder stripped each chunk-signature= and the x-amz-trailer-signature without checking them, so only the headers were authenticated, and a body that declared no digest (mc over plain HTTP sends signed chunks and none) was verified by nothing: anyone able to alter the stream could change the stored data. Fixed: the auth middleware hands the request's signature chain (arca_auth::ChunkVerifier, holding the derived signing key and the seed signature, never the secret) to PutObject and UploadPart, and AwsChunkedDecoder verifies every chunk (AWS4-HMAC-SHA256-PAYLOAD) and the trailer (AWS4-HMAC-SHA256-TRAILER) while the body streams, in constant time, hashing each chunk incrementally (a running SHA-256, nothing is buffered beyond what the stream delivers). A chunk or trailer that does not verify, a chunk without a signature, a signed body cut anywhere (the signed final chunk is required, which also closes truncation at a chunk boundary), a trailer line in a -PAYLOAD body, and signed chunks under a presigned URL (no seed to chain from) all fail with 403 SignatureDoesNotMatch before the blob is committed, so nothing is stored and an existing object is left untouched. STREAMING-UNSIGNED-PAYLOAD-TRAILER (boto3 / AWS CLI) has no chunk signatures and keeps relying on the declared checksum (TD-034). Verified against the AWS example vectors (the chunked-upload example of the S3 documentation), an independent Python signer in the integration tests, and mc over plain HTTP (PutObject and multipart parts). Not covered: STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD* (SigV4A, unsupported by the auth layer anyway). |
✔ |
| TD-046 | Multipart uploads got no object checksum | CompleteMultipartUpload stored no checksum on the object and compared neither the part checksums of the request XML nor a whole-object x-amz-checksum-* header, and GetObjectAttributes could not report ObjectParts. Fixed: CreateMultipartUpload takes x-amz-checksum-algorithm and x-amz-checksum-type (COMPOSITE or FULL_OBJECT; CRC64NVME only has the full-object form, SHA1 and SHA256 only the composite one; contradictory or unknown values are 400 InvalidRequest) and records the type (multipart_uploads.checksum_type, SQLite migration 28 / PostgreSQL 0016). UploadPart of such an upload computes the algorithm's checksum when the client declares none, and refuses another algorithm's. Complete builds the object checksum from the stored part checksums (handlers/multipart_checksum.rs: composite = the algorithm over the concatenated raw part checksums, -N appended; full object = CRC combination over the part CRCs and sizes, checked against the Ceph s3-tests vectors), after plan_complete has validated x-amz-mp-object-size, x-amz-checksum-type, the whole-object header (400 BadDigest) and the part checksums of the XML (400 BadDigest), all before any blob is assembled. The checksum is returned by Complete, HEAD and GET (with x-amz-checksum-type) and GetObjectAttributes. Every object completed from parts keeps its layout in the nullable JSON column objects.parts, which GET/HEAD ?partNumber (206, x-amz-mp-parts-count, the part's checksum) and ObjectParts (paginated) read. The layout is read ON DEMAND (get_object_parts): the S3 read and write queries do not select the column, so a GET, a HEAD, a listing or a PUT costs the same whatever the number of parts (up to 10,000); only apply_remote_object and the replication manifest select it, to avoid losing it. CopyObject keeps a FULL_OBJECT checksum and drops a COMPOSITE one, and never copies the layout. The response of GetObjectAttributes also names the part count PartsCount (it was TotalPartsCount, which SDKs do not read). Performance, measured A/B against the previous commit on the project's perf suite (default and heavy loads) and on 128 MiB uploads: every scenario within 1% (small PUT/GET/HEAD/DELETE, listing, multipart with and without a checksum, HEAD and ranged GET of a multipart object). Limits: uploads without a checksum algorithm still get no object checksum (AWS now defaults to CRC64NVME FULL_OBJECT, see the divergences in the S3 reference); objects completed before the change have no layout (?partNumber answers 400 InvalidPart, ObjectParts reports only the count from the ETag); arca recover rebuilds objects from sidecars, which carry neither checksum nor layout (TD-032). |
✔ |
| TD-047 | Plain-HTTP listener did not set TCP_NODELAY |
Both listeners now share one accept loop (crates/arca-server/src/serve.rs), whose accept calls set_nodelay(true) on every accepted connection (first fixed in a plain_http_listener wrapper around axum::serve, since replaced). Keep-alive GETs no longer wait about 40 ms for the client's delayed ACK. Regression test accepted_connections_disable_nagle checks the option on an accepted socket (deterministic, no timing) |
✔ |
| TD-048 | SQLite commits were not grouped | With [storage] fsync = true every commit on the single writer connection is an fsync of the WAL, and each write was its own transaction, so small writes were capped at one per fsync latency of the database disk whatever the concurrency (about 200-225 PUT/s at concurrency 16-64 on a 3 ms disk, measured 2026-10-02..05). A group committer (crates/arca-storage/src/sqlite/group_commit.rs) now runs the request-path writes (object writes and deletes, DeleteObjects, multipart, object tags, replication journal and notification event inserts) that queue up while a commit is in flight in one transaction, each under its own savepoint, at most 64 per group and with no linger, answering each caller only after the COMMIT. A failed write rolls back its savepoint alone; when the group cannot commit (COMMIT error, or SQLite ending the transaction) every write of it that had succeeded gets the error. Writes run in submission order, so CAS, the unique is_latest index and the cluster seq order behave as before. The rarer writes keep calling the writer directly. Measured 2026-10-20 with an interleaved A/B, durable SQLite, reference host: PUT 1 KiB c16 924 to 1,509 obj/s, c64 888 to 2,293, PUT 64 KiB c16 805 to 1,139, DeleteObjects c16 4,522 to 8,750, c1 unchanged. Unit tests cover grouping, the group size bound, per-write failure and panic, a transaction ended by a write, a failed COMMIT, a closed writer, serial visibility inside a group, CAS and seq order, and that every converted write joins the group |
✔ |
| TD-049 | The server panicked at startup without CA certificates | The webhook and Elasticsearch connectors now build their HTTP client with build_http_client (crates/arca-server/src/connector/mod.rs), which returns an error instead of panicking and logs one startup warning per connector; the connector keeps the error and fails every delivery and connection test with HTTP client unavailable: <cause>, so a host without a CA bundle (e.g. the release binary in a FROM scratch image) still starts. Verified by starting the server with /etc/ssl/certs hidden; unit tests inject a failed client into both connectors |
✔ |
| TD-051 | Interrupted uploads left temp files that nothing reclaimed | A TempFileGuard (crates/arca-storage/src/fs/blob.rs) owns the temp path of put, write_raw and the concat fallback from the file's creation until the rename that publishes it, so an error return, a failed rename and a request future dropped on client disconnect all remove the temp file (put now creates the file on the async side, so the guard can never run before a writer creates it). The concat fallback also used to leak its temp file on any error, a missing part included. arca gc, the single-node GC worker and cluster anti-entropy GC reclaim temp files older than the grace period (RawBlobOps::list_reclaimable_temp_files / delete_temp_file, classified by fs::temp_file_kind), never the in-place rewrite recovery files *.rewrite.tmp / *.pending.tmp; arca fsck --repair removes them too. Tests: cancelled put and write_raw, failed concat, old vs young temps, recovery files kept |
✔ |
| TD-053 | An abort racing CompleteMultipartUpload deleted the completed object's data | MetadataStore::complete_multipart_object installs the object row (the same CAS, versioning and is_latest rules as put_object_if, shared through install_object) and consumes the upload and its parts in ONE transaction that fails with NoSuchUpload, writing nothing, when the upload is gone; the handler discards only the assembled blob and its sidecar then, never the parts, which are the abort's to delete. Whichever transaction commits first wins: an abort that wins leaves the Complete with no object row (so an object can never point at deleted parts), a Complete that wins leaves the abort with no parts to delete (delete_multipart_upload returns none). SQLite serializes on its writer connection; PostgreSQL takes the upload row with SELECT ... FOR UPDATE as the first statement of Complete and deletes the upload row before its parts in take_upload (parts are only read through the DELETE ... RETURNING), so under READ COMMITTED the second transaction finds nothing. A refused precondition leaves the upload and parts untouched, so the client can retry. The cluster decorator replicates it as the two messages peers already know (the object row, then the upload tombstone and MultipartDelete). Limit: only the local commit is atomic; an abort landing on ANOTHER node at the same moment is arbitrated by anti-entropy, as for TD-025. Tests: seven store tests (complete_multipart_object_*, abort_after_complete_finds_no_parts_to_delete), two cluster decorator tests, and the stress test test_multipart_abort_race.py (also run by bin/test postgres), which sweeps the abort across the life of a Complete and fails on the old code (InternalError reading the "completed" object) |
✔ |
| TD-054 | A version-id-marker deleted between two pages skipped the rest of its key | MetadataStore::list_object_versions_from takes a VersionPosition (last_modified, version id) as a value instead of looking the marker version up, so a page boundary stays valid when its version is deleted. The internal full listing (list_all_object_versions: re-encryption scan, arca fsck) pages with the position of the previous page's last row and can no longer skip a version. The S3 list_object_versions is a default trait method that resolves the marker through get_object_version and restarts from the beginning of the key-marker's key when the marker does not exist (never did, or deleted since the previous page, which is what a client purging a bucket page by page does: Ceph nuke_bucket, Terraform force_destroy, mc rm --versions): version ids are random UUIDs with no timestamp, so the position of a deleted marker cannot be reconstructed, and restarting the key can list one of its versions twice but never skips one, and never fails the client. The answer stays 200, as before TD-054. Tests: five store tests (deleted marker position resumes, no position starts after the key, null version, unknown and deleted marker restart the key), two S3 integration tests in test_versioning.py (also run by bin/test postgres) |
✔ |