Blobs

A synced row is small: a few columns of text and numbers in a changeset. A photo attached to a todo is not. coven syncs those large files separately from the changesets that reference them. The host declares which rows carry a file and which columns locate it; coven derives the blob set itself and owns the encryption, the cloud layout, the upload, and the retry.

This page is the cloud lifecycle: how a blob is described, uploaded, pulled, and deleted across devices. The one device's copy (where a pulled blob lands, how it is kept and read offline) is the cache.

Examples: a todo_attachments row points at one photo file. When that row syncs, the photo has to reach every other device too.

Declaring which rows carry blobs

Three different paths must each work out which files a set of rows carries: an outgoing changeset, an incoming one, and a whole database after bootstrap. All three must derive the same set from the rows alone, and two of them run where the host is not involved at all. So blob-bearing-ness is a per-table declaration coven can evaluate anywhere, not a runtime callback. The host marks a synced table with carries_blob, passing a BlobDecl that names the columns locating each blob plus its namespace, encryption scope, provenance, and cache fill:

SyncedTable::new("todo_attachments", RowIdentity::IndependentUuid).carries_blob(
    BlobDecl::new("attachments", Provenance::UserProvided, CacheFill::CacheEager)
        // .with_id_column("file_id")              // defaults to the PK ("id")
        // .with_cloud_path_column("path")         // for a browsable home
        // .with_scope(BlobScope::Derived("attachments".into()))
        // .write_once()                           // this row is never repointed
)

The attachment rows are independently created, so their row ids are canonical UUIDv4 or UUIDv7 values. The row id remains the global logical identity whether or not the table carries a blob.

pub struct BlobDecl {
    pub id_column: String,                  // blob-id column; defaults to the PK ("id")
    pub namespace: String,                  // cloud namespace, e.g. "attachments"
    pub cloud_path_column: Option<String>,  // readable-key column for a browsable home
    pub scope: BlobScope,                   // Master | Derived(name)
    pub provenance: Provenance,             // UserProvided | HostProvided  (the Local story)
    pub fill: CacheFill,                    // CacheEager | CacheLazy       (the Remote story)
    pub replacement: BlobReplacement,       // Replaceable | WriteOnce      (the replacement story)
}

coven resolves the declaration's column names against the live schema once per cycle into BlobDecls, then derives every blob set it needs itself, reading the declared columns off a row:

• over an outgoing changeset's rows: what to upload;
• over an incoming changeset's rows: what to download (and, for a deleted row, whose local cache to drop);
• over the whole database after a snapshot bootstrap: the backfill. A bootstrapped device receives the catalog rows but not the per-row blobs (the snapshot is a whole-database image, and the incremental pull that follows starts past the changesets that carried them), so coven re-derives them from the live rows.

A row maps to the same blob whichever way it moves, so one declaration serves every path. coven reads the declared columns off the row to build a BlobRef, one blob's resolved reference:

pub struct BlobRef {
    pub namespace: String,          // cloud namespace, e.g. "attachments"
    pub id: String,                 // blob id, from the id column (the row's id by default)
    pub scope: BlobScope,           // Master | Derived(id)
    pub cloud_path: Option<String>, // readable path for a browsable home
    pub provenance: Provenance,     // UserProvided | HostProvided
    pub fill: CacheFill,            // CacheEager | CacheLazy
}

A pulled blob is Remote: its bytes land in coven's own cache (storage/cache/<namespace>/<ab>/<cd>/<locator_hash>); the host never names where a blob file lives.

cloud_path is consulted only by a browsable home; an opaque home (the default) ignores it, so leave the cloud_path_column unset unless the home is browsable. A browsable path must be relative, contain no empty, . or .. components, and avoid backslashes, drive prefixes, and the reserved .coven-versions component. The filename does not need to include the blob id.

Cache fill

CacheFill is the blob's Remote story: how a device gets the bytes once the blob is Remote, declared per blob and read the same way on every device:

• CacheEager: fetched into the cache on pull, on every device, part of "having the store". A todo's photo, an album's cover art.
• CacheLazy: skipped on pull; a device fetches it into the cache on first read instead of up front. Large blobs a device may never open, audio being the case it exists for.

The fill has to be a declared property, not a per-device choice: a device deciding during its own pull whether to fetch a blob can only read the blob's declared fill, never what another device chose locally. The cache is a Remote-only mechanism, and what a device does with a cached blob (keep it, evict it, pin it) is the cache's job, so CacheEager/CacheLazy/pin/budget describe a blob only while it is Remote, never while it is Local.

Provenance

Provenance is the blob's Local story: where the bytes live while the blob is Local. Orthogonal to the cache fill; a blob declares both:

• UserProvided: the user's own file at a path coven references but does not own. Bringing the blob back from Remote writes the bytes to a user file, so it needs a destination path.
• HostProvided: data the host hands coven, kept in coven's own local store at storage/local/<namespace>/<id>. Bringing it back from Remote restores it there, no path needed.

Encryption scope

The declaration's BlobScope selects the key the blob is encrypted under. The host names what a blob is scoped to, never the raw key bytes:

• Master encrypts with the store master key. Every member holds it, so every member can decrypt the blob. The common case, with no key management at all.
• Derived(scope_id) encrypts with a key coven derives from the master key, one distinct key per scope_id (see Chunked encryption for the derivation). Deterministic: the same scope_id yields the same key on push and on pull, which is what lets a puller re-derive it and decrypt. The corollary is that scope_id must be stable for the lifetime of the reference. When the scope is the row id, a primary-key change deletes the old identity and inserts a new row whose scope derives a different key.

How a blob moves out

Blobs reach the cloud through row publication or a gated root's make_remote transition. Provenance determines where local source bytes live; it does not decide whether a make-remote upload uses the queue.

With a Store write. The host supplies row changes and host-provided bytes through CovenHandle::write. Publication prepares and uploads the required objects before publishing their Store commit. An unchanged blob can reuse its previously accepted exact object without local bytes. If required content has no usable source, the write cannot publish.

Through make-remote. The host starts a gated root's transition with CovenHandle::make_remote:

handle.make_remote("todos", todo_id, todo_label, pin, refs).await?;

Coven atomically records the root's transition intent and one upload for every current blob-bearing row beneath it, including both provenances. Each upload retains the exact row version and plaintext size/hash, its source path, and the pin request. A user-provided source is the registered external file; a host-provided source is coven's local file. Enqueueing performs no upload and does not yet assign the final provider object.

The durable cloud_outbox contains upload and delete operations. The host observes it through coven's APIs rather than mutating it. Each upload progresses through:

1. Pending: the drain derives the locator and protection from the queued row and current upload authority, verifies the source while preparing its spool, and allocates an exact provider object.
2. Prepared: the exact stored reference and spool are recorded. Uploads and retries use that retained object and spool, rather than choosing another destination or rereading changed source bytes.
3. Created: provider creation succeeded and that result is recorded. The drain retires the spool, optionally pins the plaintext, and attempts to finalize the root's transition. A retry resumes these steps without uploading again.

The root stays Local until all its current uploads are Created. Coven then records the gate change and its Store write together, and marks the intent Publishing. The Created upload records remain as exact handoffs to that write. Its publication consumes those records and the intent atomically; a completed byte transfer alone does not finish the transition.

The drain admits uploads in queue order up to its concurrency limit. Failed entries retain their state and record the failure and attempt time; other eligible entries can continue. The retry delay is 30s · 2^(attempt_count - 1), capped at one hour. A fresh entry is eligible immediately. When the oldest root being advanced finishes its uploads, the drain stops admitting more work, settles its active attempts, and yields for publication.

Before Publishing, cancellation records Cancelling on the root's intent; it is not a third outbox operation. Cleanup removes each upload's exact object, spool, and cached copy before retiring its record. The last record and the cancellation intent are removed together. A cleanup failure retains the work for retry, and the root remains Local. Once the intent is Publishing, cancel_make_remote refuses the cancellation.

The pull side

Pull derives the blobs an incoming Store package references from the declarations and downloads required eager bytes before materializing the commit, so a row is never accepted before its required blobs are durable. A downloaded blob lands in the cache, at storage/cache/<namespace>/<ab>/<cd>/<locator_hash> under the store directory, decrypted under its scope. A download is skipped when the exact file is already present.

Only CacheEager blobs download here. A CacheLazy blob is skipped on pull and fetched on its first read_blob.

When the applied changeset deletes a blob-bearing row (a gate retract or a genuine delete), coven drops that blob from both cache folders on this device. A peer only drops its own local cache here; it never writes a cloud tombstone; that belongs to the deleting owner (see Deleting a blob).

The exact materialized position is the durable boundary. It advances in the same SQLite transaction as the package rows only after required blob work succeeds. If a download fails, coven leaves that device sequence and commit hash unmaterialized and reports asset_downloads_failed, so the commit and its blobs retry together.

Deleting a blob

A blob is shared cloud state that rows on every device may still reference. Deleting it the instant the deletion drains would strand a device that has not yet pulled the row removal: it would see a row pointing at nothing. So a delete is not immediate. The host requests blob replacement or row removal through CovenHandle::write, and coven enqueues the cloud delete with the row change.

The next cycle's drain_tombstones writes a signed tombstone (a durable, signed record that the blob was deleted, and when) and keeps the blob. The tombstone is signed because the bucket is untrusted: the at-rest cipher proves only confidentiality, not authorship, so the deletion is signed by its author like every other control object, and a later GC verifies the signature and that the author is a current write-capable member before acting on it.

The blob is held for the tombstone grace, the convergence window: 7 days by default (BLOB_TOMBSTONE_GRACE), host-configurable through CovenBuilder::blob_tombstone_grace (a zero-or-negative grace is refused at open). A device offline for less than the grace is never stranded: it comes back, pulls the row removal, and the blob is still there in the meantime. Once the grace passes, gc_tombstones on any device verifies the tombstone, authorizes the author against the membership chain, deletes the blob, then deletes the tombstone. An unreferenced-but-not-yet-deleted blob is correct state during the window, not garbage a later pass repairs.

Cloud layout

A cloud blob is identified by its BlobLocator. The locator records the namespace, blob id, uploader registration, plaintext length and hash, and protection facts. Its hash names an immutable version. An opaque home's semantic object key is:

{namespace}/opaque/{locator_hash}

The locator also names the audience, encryption scope, and key fingerprint. A change to these facts produces a different locator hash. The provider sees the namespace and hash in the key, plus encrypted bytes; the key contains no readable file name.

A StoredBlobRef binds that locator to the exact provider object, including its stored length and hash. The host obtains the row's installed reference through CovenHandle::row_blob_ref. The returned RowBlobRef exposes its committed object through stored(); a row whose blob has no committed object returns None. Diagnostics use that retained reference rather than reconstructing an object key from a blob id.

On disk, remote copies use the locator hash as well: storage/cache/{namespace}/{ab}/{cd}/{locator_hash} or storage/pinned/{namespace}/{ab}/{cd}/{locator_hash}. Here ab and cd are the first two byte-pairs of the locator hash. Local host-provided sources remain keyed by their logical blob id.

Browsable-home blob paths

A browsable home stores a blob's plaintext bytes under a readable path with an immutable version:

{namespace}/readable/{cloud_path}/.coven-versions/{locator_hash}

cloud_path is the value read from the declaration's cloud_path_column, such as Project Plan/diagram.png. The host supplies this path on every blob-bearing row; a missing path is an error. The readable prefix is part of the locator, so changing it produces a different version key. Anyone with provider access can see the names and read these blob bytes.

	Opaque home (default)	Browsable home
Config cloud_home.storage	opaque	browsable
Runtime scheme	BlobPathScheme::Hashed	BlobPathScheme::Plain
cloud_path_column	ignored	required
Cloud key	{namespace}/opaque/{locator_hash}	{namespace}/readable/{cloud_path}/.coven-versions/{locator_hash}
Blob bytes	encrypted	plaintext

A cloud object is never rewritten

The locator hash separates immutable versions in both path schemes. The stored reference records the exact object's length and hash, and downloads are checked against that reference and the locator's plaintext facts. An object existing at a path alone is not proof that it contains the expected bytes.

The blob declaration separately states whether a changeset update may repoint its row at another blob id. Version addressing supplies cloud-object identity for both replacement policies.

Replaceable (the default)

A row may point to a replacement blob while retaining its readable path, such as Live at Leeds/cover.jpg. The new locator includes the replacement's identity and content, so its exact object has a different version key. The earlier object is unchanged and remains readable until independently reclaimed.

Write-once

SyncedTable::new("release_files", RowIdentity::IndependentUuid).carries_blob(
    BlobDecl::new("audio", Provenance::UserProvided, CacheFill::CacheLazy)
        .with_cloud_path_column("cloud_path")
        .write_once()
)

A write-once row cannot change its blob-id column through a changeset update. The declaration resolver refuses that update. This row policy does not constrain the readable filename or replace the locator's exact object identity.

Exact versions

Different locators occupy different version keys, including when two devices choose the same readable path. The installed row's stored reference identifies which exact object it reads. Replacing a blob cannot overwrite the object named by an earlier reference; reclaiming the earlier object is a separate deletion operation.

Where a blob's bytes come from

For make-remote, coven reads the source selected by the row's provenance: its own local file for a host-provided blob, or the registered external file for a user-provided blob. Preparation verifies the queued plaintext size and hash and records a durable upload spool. A Prepared retry reads that spool, so later edits to the original file cannot change the reserved upload's bytes.

Store writes retain their own publication sources and exact prepared objects. They can reuse an accepted object or read its verified plaintext when local bytes are absent; readable-path changes still require a destination with the new path.

Observing transitions and uploads

Use CovenHandle::subscribe_cloud_outbox and CloudOutboxLiveQuery for durable queue and transition status. The query returns an initial committed snapshot and updates after outbox or intent changes, including after reopening the app. Render that status together with transient progress from BlobTransitionObserver; its Rustdoc defines the callback signatures and defaults.

Preparation and upload callbacks receive &RowBlobRef, identifying the queued row version and plaintext facts. Preparation reports plaintext bytes consumed; upload reports stored bytes sent to the provider (encrypted in an opaque home, plaintext in a browsable home). Progress is cumulative within each attempt and coalesced at a 300 ms cadence, with the final total forwarded when needed.

on_blob_uploaded follows successful provider creation and the durable Created record. Spool cleanup, pinning, or transition finalization can still fail after that notification. on_blob_upload_failed reports attempt failures; the retained journal determines what retry resumes. A Created retry skips preparation and the upload's start, progress, and success callbacks; remaining work can still report an attempt failure. Use the durable Publishing state, rather than the upload callback, to distinguish completed transfers from an unfinished Store publication.

on_blob_materialize_progress counts copied blobs, not bytes, during make-local. on_root_made_local follows installed local files and the database commit of their ownership, the gate change, and queued cloud deletions. These callbacks describe the current call; they are not persisted events replayed after restart. There is no on_root_made_remote callback.

should_skip_uploads supplies the absolute pause state. While paused, the drain starts no new uploads; the host can still enqueue work. Pausing an active drain suspends preparation and stops upload bodies from yielding further chunks, retaining the same preparation or provider operation. Resume continues that attempt. A pausing observer also implements wait_until_uploads_paused and wait_until_uploads_resumed, completing each when its absolute state is reached. The defaults never pause and leave both waits pending.