Technical Debt

Workarounds, hardcoded values, and temporary fixes that pass S3 compatibility tests or satisfy client expectations but need proper implementation post-MVP.

Each entry has a unique ID referenced in the source code via // TECHDEBT(TD-XXX): comments. Run grep -r 'TECHDEBT' crates/ to find all markers.

---

Active Items

ID	Area	Workaround	Proper Fix	Tests Affected	Files
TD-007	Unimplemented ops	~33 bucket-level GET/PUT operations return 501 NotImplemented (ACLs, CORS, policies, etc.)	Implement each feature as needed post-MVP	All tests in skipped/unimplemented categories	bucket.rs
TD-009	~~Audit write contention~~	RESOLVED: Audit writes now batched via bounded mpsc channel (capacity 10,000) with dedicated writer task doing bulk inserts in a single transaction	—	—	middleware/audit.rs, state.rs
TD-010	SSE-C multipart	SSE-C headers on CreateMultipartUpload and UploadPart are rejected with InvalidArgument. SSE-C + multipart requires encrypting each part independently and tracking per-part nonce prefixes	Implement SSE-C for multipart: per-part encryption, nonce tracking, assembly of encrypted parts at complete time	test_ssec.py::TestSsecMultipartRejection	multipart.rs, object.rs
TD-017	time pinned to =0.3.47	time 0.3.48 adds From impls that clash (E0119 coherence error) with rcgen's blanket From conversions whenever time's parsing/formatting features are enabled elsewhere in the graph (they are, via transitive deps). 0.3.47 already contains the CVE-2026-25727 fix, so no security exposure — this is a freshness pin only	Unpin when rcgen or time resolves the conflict upstream (track rcgen releases past 0.14.8)	None	Cargo.toml
TD-018	Re-encryption skips multipart/composite	The encrypt / decrypt maintenance jobs (and the encrypt-existing / decrypt-existing CLI) skip multipart/composite objects: they are detected by their <32 hex>-<n> ETag and excluded from the candidate scan, so a store with multipart objects is only partially re-encrypted/decrypted. Related to TD-014 — composite objects live as a composite sidecar listing part blob ids, with no single on-disk blob to rewrite via copy-on-write. SSE-C objects are also skipped (the server never holds the customer key).	Teach re-encryption to walk a composite's part list, rewrite each part blob copy-on-write, and rebuild the composite sidecar with the new part ids and (de/en)cryption state — sharing the composite-walking logic added for TD-014. Add unit + integration coverage with mixed plain/multipart fixtures.	None at the runtime level; mixed-content stores re-encrypt only their single-blob objects	crates/arca-server/src/maintenance.rs (is_recrypt_candidate, is_multipart_etag), crates/arca-storage/src/recrypt.rs
TD-019	migrate-db online direction limited to postgres→sqlite	The online (maintenance-job) migrate-db can only run postgres → sqlite: Arca auto-detects the running backend from the config (PostgreSQL when [storage.postgres] is present), so a live SQLite instance has no PostgreSQL section to point the source at and cannot launch a sqlite→postgres job online. The sqlite→postgres direction is done with the offline CLI (arca migrate-db --to postgres), which can open both backends explicitly.	Either accept the target [storage.postgres] connection parameters in the job params so the online direction can target an external PostgreSQL without it being the running backend, or document the offline CLI as the supported sqlite→postgres path (current state).	None	crates/arca-server/src/maintenance.rs (process_migrate_db), crates/arca-server/src/migrate_db.rs
TD-020	Deferred integration tests: cold-CLI re-encryption + 3-node cluster round-trips	The offline-CLI re-encryption path (encrypt-existing / decrypt-existing against a stopped server) and the 3-node cluster behaviours of re-encryption (one-time full re-replication of rewritten blobs via anti-entropy) and migrate-topology (single→cluster→single round-trip across real nodes) are currently covered by unit tests and the online single-node integration suite only — there is no dedicated cold-CLI or cluster integration phase for them.	Add a cold-CLI integration phase (stop server, run the CLI, restart, verify) and a cluster integration phase (re-encrypt on the worker-leader, assert convergence on peers; migrate-topology round-trip).	None (covered by unit tests meanwhile)	tests/ (no cold-CLI/cluster phase yet), crates/arca-server/src/{recrypt_existing,migrate_topology}.rs
TD-021	Cluster re-encryption defers old-blob reclaim to GC	After a copy-on-write re-encryption swap, the old blob is deleted eagerly on a single node but left for the grace-bounded anti-entropy GC in a cluster, so out-of-band reclaim never races the blob-repair pulls peers issue while converging onto the re-encrypted row. Correctness is not affected (content and lock state now converge on independent LWW dimensions — content_updated_at for blob/algorithm/key vs lock_updated_at for retention/legal-hold — so a concurrent lock op can no longer revert blob_id; migrations sqlite v25 / pg 0013). The only residual is a transient extra copy of each re-encrypted object's old blob on every node until the next GC pass reclaims it.	Once the 3-node cluster re-encryption phase (TD-020) confirms it, consider reclaiming the old blob eagerly in the cluster path too (the new blob is already CAS-committed and propagates on its own dimension), or trigger a targeted GC sweep at job completion instead of waiting for the periodic pass.	None (unit tests cover the independent-merge convergence: apply_remote_reencryption_and_lock_change_merge_independently, apply_remote_reencrypted_row_converges_via_content_updated_at)	crates/arca-server/src/maintenance.rs (recrypt_one, RecryptCtx::clustered), crates/arca-core/src/types.rs (resolve_replicated)
TD-022	migrate-db / migrate-topology online jobs are not cancellable mid-copy	The migrate-db maintenance job copies every metadata table in one pass; unlike the re-encryption jobs it does not check the pause/cancel flag between tables, so a cancel request only takes effect once the whole copy finishes (or the job errors). The destination is left with a partial copy on error, which the operator must drop before retrying.	Check the job status between tables (and ideally between row batches) and abort cleanly, truncating or dropping the partially-populated destination tables on cancel. Surface a "not resumable, will restart from scratch" note in the console.	None	crates/arca-server/src/maintenance.rs (process_migrate_db)
TD-023	Offline migrate-db has no crash-safe checkpoint	The offline arca migrate-db CLI copies every table to the destination in a single process with no checkpoint; a crash or SIGKILL mid-run leaves the destination partially written and the operator must drop it before retrying. Row-count reconciliation at the end catches a torn copy, but only if the process survives to print it. (The migrate-topology CLI is not affected: its operations are idempotent/in-place — tombstone purge is safely re-runnable, VACUUM is a no-op on restart.)	Write a resume marker (last-completed table / cursor) to the destination or a sidecar file and skip already-copied tables on restart; or wrap the whole copy in a single destination transaction where the backend allows it.	None	crates/arca-server/src/migrate_db.rs, crates/arca-storage/src/migration.rs
TD-024	rustls-webpki 0.102.8 pinned transitively (vulnerable code paths unreachable)	rumqttc 0.25.1 (MQTT notification connector) hard-pins rustls-webpki = "^0.102.8", which carries 4 published advisories, tracked upstream as GHSA-82j2-j2ch-gfr8 (HIGH, DoS panic on a malformed CRL BIT STRING, fixed in 0.103.13), GHSA-pwjx-qhcg-rvj4 (MEDIUM, CRLs not treated as authoritative by Distribution Point, fixed in 0.103.10), GHSA-xgp8-3hg3-c2mh and GHSA-965h-392x-2mh5 (LOW, name constraints wrongly accepted for wildcard and URI names, fixed in 0.103.12); these are the same four issues previously recorded here by their RUSTSEC IDs (RUSTSEC-2026-0104/0049/0098/0099). The vulnerable code is not reachable from Arca. Cargo.lock carries two copies of the crate: 0.102.8, whose only dependent is rumqttc, and 0.103.13, which is already patched for all four and is what rustls 0.23.40, async-nats, rustls-connector and rustls-platform-verifier resolve to. rumqttc's entire use of its 0.102.8 copy is the WebPki(#[from] webpki::Error) variant in rumqttc/src/tls.rs; it never configures a CRL, so both CRL advisories are dead code, and MQTTS certificate path validation actually runs through tokio-rustls -> rustls 0.23.40 -> rustls-webpki 0.103.13. No fix is available upstream: 0.25.1 is the newest published rumqttc, bytebeamio/rumqtt master still declares rustls-webpki = "0.102.8", no patched 0.102.x release exists, and a [patch.crates-io] override to 0.103.13 would be reported as unused because it cannot satisfy ^0.102.8 (forcing it would need a rumqttc fork). Dependabot alerts #1 to #4 on dxc-technology/arca were dismissed as not_used on 2026-08-22, referencing this entry.	Track rumqttc releases that relax the rustls-webpki requirement and bump as soon as one lands; if upstream stays dormant, evaluate rumqttc's native-tls backend or a maintained fork. Re-assess immediately if the MQTT connector ever starts configuring CRLs, or if rumqttc begins calling into webpki for path validation, since either would make the advisories reachable.	None (MQTT connector integration tests pass; the vulnerable code paths are unreachable as analysed above)	Cargo.toml, Cargo.lock
TD-025	Cluster conditional writes (CAS)	Cluster-wide conditional-write atomicity is not provided: If-Match / If-None-Match / x-amz-if-match-* preconditions are evaluated authoritatively only against the serving node's own state (put_object_if/delete_object_if/delete_object_version_if commit locally, then fan out — there is no per-key authoritative node to arbitrate across peers). Concurrent conditional writes for the same key landing on two different nodes at once can both pass their own local CAS and both be accepted (200 OK); the row then converges by last-writer-wins, the same as any other conflicting write. Single-node CAS (the common case, and the whole point of the conditional-write-atomicity fix) is unaffected — see crates/arca-storage/src/{sqlite,pg}/metadata.rs. Mitigated operationally by pinning a given key's conditional-write traffic to one node (a sticky/session-aware load balancer — see documentation/docs/guide/ha.md).	Owner-node forwarding for conditional writes: the node that receives a conditional PUT/DELETE for a key it doesn't own forwards it to that key's designated owner node instead of committing locally, giving cluster mode the same per-key authoritative arbitration single-node CAS already has.	None (cluster CAS boundary is pinned by tests/integration/test_cluster.py::test_conditional_put_is_per_node_not_cluster_wide; single-node CAS is fully covered by tests/integration/test_conditional_writes.py)	crates/arca-server/src/cluster/cluster_meta.rs (put_object_if, delete_object_if, delete_object_version_if)
TD-014	recover / fsck composite blobs	Composite blobs (the result of the optimised CompleteMultipartUpload) live as a sidecar that lists the part blob ids — no on-disk file exists at the composite's blob path. recover::process_sidecar rejects them as "orphaned sidecar (blob file missing)" and aborts the rebuild. fsck reports the same sidecar as an orphaned_sidecars entry (the parts themselves are not orphaned because they each carry their own sidecar). Both tools therefore produce false positives on any data directory that has experienced multipart uploads after the composite-concat optimisation. Runtime S3 reads/writes are unaffected — the request path fully handles composites.	(1) recover: when a sidecar has composite: Some(parts), skip the blob-existence check, walk the part list, and verify each referenced part blob exists. Insert one ObjectRecord per composite that points at the composite blob_id (the part records are not surfaced to the metadata DB — they are a storage-layer detail). (2) fsck: treat composite sidecars as valid (no blob file expected); cross-check each referenced part exists; report a new dangling_part category when a composite references a missing part. Add unit tests for both tools using fixtures that mix composite and ordinary blobs.	None at the runtime level; recovery / fsck workflows on data with composites currently fail	crates/arca-server/src/recover.rs:206, crates/arca-server/src/fsck.rs (orphaned_sidecars collection, around line 112)
TD-026	bin/test tls cannot create certs/ on a real Linux host	The self-signed TLS flow creates the bind-mount directory on the host (mkdir -p certs in run_tls, mirrored by the "Self-signed certificates" quick start in the TLS guide) and then writes into it from the tls-init container, which runs as 65532:65532 like every Arca image. On Linux the directory belongs to the invoking user with mode 0755, so arca tls generate gets EACCES and the whole bin/test tls phase fails. The gap is invisible on macOS, where Docker Desktop fakes bind-mount ownership and the container sees itself as the owner, so the suite passes locally and would only break on a Linux developer machine or CI runner. Not reproducible from a Mac, and not fixable inside the script without sudo on the host.	Stop writing generated material into a host bind mount: run tls-init against a named volume (ownership is real there on every host — this is what run_tls_permissions already does) and docker cp the files out if the host needs them, or run the tls-init service as the invoking $(id -u):$(id -g) so the generated files belong to the caller (the file modes from arca tls generate are already correct either way). Update the quick start in documentation/docs/guide/tls.md to match whichever is chosen.	bin/test tls (7 tests) on Linux; passes on macOS	bin/test (run_tls, the mkdir -p certs step), docker/docker-compose.tls.yml (tls-init), documentation/docs/guide/tls.md

---

Legend

• Tests Affected: Ceph s3-tests that are impacted (either currently failing, or passing due to the workaround)
• Files: Source files containing the workaround (abbreviated — full paths under crates/)
• Items are moved to the Resolved table when properly implemented

How to Use

1. When adding a new workaround, assign the next TD-XXX ID
2. Add a // TECHDEBT(TD-XXX): brief description comment in the source code
3. Add a row to the Active Items table above
4. When implementing the proper fix, remove the code comment and move the row to the Resolved table below

---

Resolved Items

ID	Area	Resolution	Status
TD-001	Owner identity	Owner ID derived from credential's user. Buckets and objects store creator's username. Migration v8 adds owner columns and user_id on credentials. Phase 16	✔
TD-003	Versioning	Full object versioning: PutBucketVersioning/GetBucketVersioning, version IDs, delete markers, version-specific GET/HEAD/DELETE/COPY, ListObjectVersions with real data. Migration v9 adds versioning columns. Phase 17	✔
TD-004	Region	Region configurable in [server] TOML section or via Admin API (/admin/settings/region). Per-bucket region via bucket_config. HeadBucket and GetBucketLocation return effective region. Phase 18	✔
TD-005	Request ID	Error XML placeholder <RequestId> replaced by the request-ID middleware with the real x-amz-request-id value	✔
TD-006	Encryption config	PutBucketEncryption / GetBucketEncryption / DeleteBucketEncryption implemented with SSE-S3 (AES-256-GCM). Per-bucket config in bucket_config table, global default from [encryption] config section. Phase 13	✔
TD-002	Storage class	storage_class field added to ObjectRecord, DB migration v13, x-amz-storage-class header accepted on PutObject, returned in list and head responses. Phase 22	✔
TD-008	Content-Type source	Verified: Content-Type captured from CreateMultipartUpload matches AWS behavior. Clients set it at init time. Phase 22	✔
TD-015	Cluster inter-node TLS cert verification	Verified mutual TLS with an operator-distributed cluster CA ([cluster.tls] — REQUIRED when the cluster runs over HTTPS, no insecure fallback): inter-node reqwest clients trust the CA and present the node's CA-signed client identity; danger_accept_invalid_certs removed from membership.rs and client.rs. The listener requests client certificates (optional at the TLS layer — S3 clients share the port) and /cluster/v1/* refuses requests without a CA-verified one. Material minted by arca tls generate-cluster. HA hardening R4 (decision H12)	✔
TD-016	Cluster control-plane reconcile was partial	The anti-entropy snapshot reconcile now covers EVERY control-plane family: grant attachments (user_grants, team_grants) and team memberships (per-row updated_at, migrations sqlite v22 / pg 0010, composite-key tombstones on detach/remove), bucket_config per-key LWW, bucket_tags as one set-level entity per bucket, server_config per-key LWW (node-local keys excluded at build AND on apply), plus in-progress multipart uploads and parts (D4). Child upserts are parent-filtered in plan_control_merge (a row whose parent resolved deleted is never adopted — replaces cascade tombstones and respects the PG join-table FKs). A returning node now fully self-heals on all families. HA hardening R5 (review D9/D4)	✔
TD-011	time crate CVE	Docker builder moved from rust:1.85-alpine to rust:alpine (currently 1.96); time updated to 0.3.47, which contains the CVE-2026-25727 fix. The MSRV pins for home/serde_with/darling in the Dockerfile became unnecessary and were removed. (0.3.48+ stays excluded for a non-security reason — see TD-017.) Dependency refresh, 2026-06-12	✔
TD-012	rustls-pemfile unmaintained	PEM parsing migrated to rustls-pki-types (PemObject: pem_slice_iter / from_pem_slice) in tls.rs and tls_generate.rs; the rustls-pemfile dependency is gone from the workspace. Dependency refresh, 2026-06-12	✔
TD-013	AMQP/Kafka connector tests	Root cause was NOT DNS: the AMQP receiver crashed on startup because docker exec rabbitmq-diagnostics (run as root in a tight wait_for_amqp_receiver loop) raced with RabbitMQ's cookie initialization, producing an EACCES error that killed the container. Fix: replace the exec-based wait for AMQP/Kafka with a docker inspect health poll (_wait_for_container_healthy), so the receiver is probed by the container's own healthcheck only. Kafka tests also had two unrelated test-side issues: consumer-group coordination raced with the first produce (fixed by switching to manual partition assign + seek_to_end), and the connectivity-failure test timed out because Arca's Kafka test() can take ~10s on unreachable hosts while the test HTTP timeout was also 10s (fixed by raising the conftest SigV4 default to 30s).	✔