Thinking on how to make clay more git-like and make file sharing between desks easier

clay-design-notes.md

Clay Repo Design Notes

Goal

Extend Clay so it can support repo-style workflows while preserving Urbit's referential transparency where it matters.

The main additions discussed here are:

• repos containing multiple desks
• different desk classes with different validation rules
• repo-level snapshots for sync
• commit messages and signed provenance
• propagated revision labels/tags
• local mutable WORK and INDEX areas for test builds

Current Clay Model

Today, the globally referentially transparent path to a file is effectively:

• /[ship]/[desk]/[desk-revision]/[path]

Clay's build cache is already split into:

• a global cache of actual Ford results
• per-desk lightweight indexes/root sets

Important current properties:

• actual Ford results are shared globally when the resolved build graph is identical
• desk-local cache state is mostly just indexing/root metadata
• local desk commits eagerly validate changed files
• foreign desk sync stores data and validates lazily on later read/build

Proposed Repo Model

Introduce a repo namespace:

• /[ship]/[repo]/...

Within a repo, there are three desk classes.

%master

This is the canonical installable desk.

Commits to %master should keep current Clay-style requirements:

• mark validation succeeds
• agent build/state transition succeeds
• kernel compatibility requirements are met

Data Desks

These are non-self-hosting desks whose files are interpreted using marks from %master in the same repo.

They are not primarily about making Clay more Git-like. Their purpose is to:

• create a cleaner separation between source desks and data
• allow data to be shared across different %master desks without forcing those desks to share one source tree
• avoid mark breakage by making the mark context explicit and controlled

Commits to data desks should succeed only if:

• files validate against marks from that repo's %master

They are not required to be installable or self-hosting.

Branch Desks

These are branch desks used for normal development work.

They should:

• be self-hosting when built
• not be required to build successfully in order to exist or accept commits

This allows broken work-in-progress to exist naturally on branches.

Referential Transparency

Self-Hosting Desks

For %master and branch desks, a desk revision is enough to determine meaning:

• /ship/repo/desk/desk-revision/path

Data Desks

For data desks, desk revision alone is not enough for interpreted reads, because meaning depends on %master.

If the same data-desk revision is interpreted against different %master revisions, marked scries/builds could differ.

Therefore, interpreted access to a data desk must pin %master somehow.

Two equivalent ways to do that:

• include master-rev directly
• include a repo-rev that determines master-rev

We converged on repo-rev as the cleaner abstraction for repo-wide sync.

Repo Revisions

repo-rev is useful primarily as a coherent repo snapshot identity.

It should represent:

• a snapshot of %master
• the current state of all desks in the repo, including branch desks
• propagated label/tag state
• any other replicated repo metadata

It should support:

• atomic whole-repo sync
• whole-repo subscriptions
• referentially transparent interpretation of data desks

Important Constraint

repo-rev should advance for every committed repo change.

That includes:

• %master commits
• data-desk commits
• branch-desk commits
• propagated label/tag updates

The only mutable state excluded from repo-rev should be local-only workspace state such as WORK and INDEX.

Cache Semantics

Current Clay Behavior

Current Clay does not key the real global cache by desk name.

The real key is a recursive build identity:

• leak = [pour deps]

So global cache entries are shared across desk forks when:

• effective paths match
• file lobes match
• resolved transitive dependencies match

Per-desk cache metadata is separate, but heavy results are shared.

Implication for Repos

To preserve fork sharing in the new repo model:

• do not key the real global cache by nominal repo identity alone
• use repo/desk context only for resolution
• derive the real global cache key from resolved semantic inputs

That means:

• fast per-context lookup can be repo-qualified
• global sharing should still happen when resolved file/mark/dependency identity is equal

Data Desk Forked Into Another Repo

If a data desk is forked into another repo:

• raw content can still be shared
• interpreted/build results must be reconsidered under the new repo's %master

If the new %master is semantically identical to the old one:

• cache entries can still be reused

If it differs:

• affected validations/builds must be recomputed

This is analogous to current Clay invalidation when marks change.

Commit Metadata

Current Clay commits do not track provenance beyond parentage/history.

We discussed extending commits to include:

• commit message
• author
• committer
• signing key identity / revision
• crypto suite

Commit Messages

Commit messages should be part of the commit object, not side metadata.

That means:

• commit messages propagate naturally with commits
• changing a message creates a new commit object and new hash

This makes commit messages part of immutable commit identity.

Author and Committer

We want both:

• author: who originally wrote the change
• committer: who created/published this specific commit object

This is useful for collaborative merge workflows.

Recommended behavior:

• ordinary authored commits: author and committer may be the same
• merge commit: maintainer is committer

Signatures

Recommended minimal model:

• commit payload includes author, committer, key identity, crypto suite, message, etc.
• commit hash is derived from that intrinsic payload
• committer signature signs that payload

Optional later extension:

• also support author signatures

For a strict merge-based PR workflow, this is simple:

• author signs their own branch commits
• maintainer signs the merge commit

Immutable History

Desks are the branch identities in this design.

Because desk revisions are part of the globally referentially transparent namespace, desk history must remain fully immutable.

That means:

• old desk revisions must never change meaning
• accepted committed history must remain append-only
• merge and fast-forward are acceptable
• true history-rewriting operations are not

So this design explicitly excludes:

• rebase
• cherry-pick
• squash

Those operations only make full sense in a system with mutable branch refs distinct from the historical identity being addressed. In this design, that would conflict with immutable desk revisions.

Labels / Tags

Current Clay revision labels are local-only mappings from label to revision.

We discussed how propagated labels should work.

Correct Model

Labels/tags are best treated as repo metadata, not as part of commit identity.

They are conceptually:

• repo-level refs from name to revision

This keeps mutable naming metadata separate from immutable commit objects.

Referential Transparency

Mutable labels are not referentially transparent unless their state is versioned.

Therefore:

• label/tag state should be included in repo-level snapshot history
• repo-rev should determine both content state and label/tag state

Then:

• a read of a specific repo-rev sees the corresponding label map
• a subscription to repo updates advances both content and labels coherently

Pull Requests

Pull-request style workflows are a hosting/workflow layer, not a core Clay history primitive.

Likewise, Clay should probably only provide primitives:

• commit DAGs
• branches/desks
• merges
• repo snapshots
• labels/tags
• provenance metadata

Review/proposal workflows should live in userspace:

• Gall agent
• web UI
• merge proposal/review system

Working State and Staging

We considered using branch commits only for WIP, but that creates too much repo churn and poor UX.

So we converged on having local mutable workspace state after all.

Local Mutable Workspace State

Each workspace should have:

• HEAD / base committed snapshot
• WORK overlay
• INDEX overlay

For a first version, snapshot-style staging is enough.

That means path-level staging, not hunk-level staging.

Recommended Representation

For each workspace:

• work=(map path (unit page))
• staged=(map path (unit page))

Semantics:

• path absent from map: inherit from HEAD
• path present with ~: deleted
• path present with `page: overridden

WORK and INDEX are:

• local-only
• mutable
• not globally referentially transparent
• not synced across the network

They should be logically separate from normal Clay history storage, even if physically stored in the same vane state tree.

Why Not Use Branch Commits as a Working Tree?

Because then:

• every edit becomes a commit
• WIP branches would churn repo-rev and downstream subscriptions
• users lose normal dirty/staged workflow ergonomics

So local mutable WORK/INDEX is still worthwhile.

Building Against WORK / INDEX

We do want test builds from WORK and INDEX.

Therefore they need to be hooked into the real build system.

Recommended approach:

• do not build only against committed history
• allow the build system to operate against a virtual source view:
  • HEAD + WORK
  • HEAD + INDEX

This means:

• reusing the same Ford/mark build machinery
• parameterizing it over an overlay-aware file resolver
• not building a totally separate system

Caching for WORK / INDEX

These mutable overlays need distinct local-only cache domains.

They should not be treated as normal immutable committed-history cache roots.

Recommended split:

• existing global/shared cache for immutable committed history
• local workspace caches for builds against:
  • HEAD + WORK
  • HEAD + INDEX

Workspace builds should still reuse immutable committed subgraphs where possible.

Only overlay-affected build results need workspace-local cache entries.

Mark Fallbacks in Userspace

If marks are broken while editing, the UI should still be able to show common text files.

Recommended userspace fallback:

1. Try normal Clay marked read/build.
2. On failure, read raw page.
3. Decode common basic marks locally in userspace, such as:
   • %hoon
   • %txt
   • %json
   • maybe %md, %mime, etc.

This fallback is only for UX.

Authoritative semantics still come from Clay's mark system.

Workspace Coherency When Branches Move

WORK and INDEX overlays are defined relative to a base snapshot.

So if the underlying branch moves, those overlays can become invalid or ambiguous.

Recommended policy:

• each workspace tracks its pinned base
• if the branch changes and overlaps paths touched in WORK or INDEX, block auto-advance
• require the user to resolve, discard, commit, or stash

For a first version, simple blocking is enough.

This is especially manageable if:

• users do most local work on branches that are not auto-advanced under them

Stash

A stash can be added later if needed.

It would be local-only and store:

• base snapshot
• WORK overlay
• INDEX overlay
• optional message/date

Applying a stash would amount to replaying those overlays onto a new base, effectively a local three-way merge.

This is feasible, but not required for the initial design.

Simplifications / Non-Goals for Now

These were discussed and are intentionally not core priorities right now:

• .gitignore-style ignored files
• reflog-style recovery
• submodules
• line-by-line blame
• authoritative rename tracking

Renames

Core semantics can stay:

• delete + add

UI may infer probable renames heuristically, but that should not be authoritative.

Summary of Main Design Decisions

• Introduce repos containing %master, data desks, and branch desks.
• %master keeps current strict Clay-style validity requirements.
• Data desks validate against %master in the same repo.
• Branch desks are self-hosting when built, but commits need not be buildable.
• Use repo-rev as the coherent synced repo snapshot.
• Preserve global build-cache sharing by keying global cache on resolved semantic inputs, not nominal repo identity.
• Add commit messages to commit objects.
• Add author and committer provenance fields.
• Add committer signatures to commits.
• Treat propagated labels/tags as repo metadata included in repo snapshot history, not as part of commit identity.
• Keep desk history fully immutable; allow merges/fast-forwards but not rebase, cherry-pick, or squash.
• Keep PR/review workflows in userspace.
• Add local mutable WORK and INDEX overlays.
• Support test builds against HEAD + WORK and HEAD + INDEX.
• Give WORK/INDEX their own local cache domains.
• Block workspace advancement when upstream branch movement conflicts with local overlays.

Main Open Questions

• Exact shape of the new commit object and signature envelope.
• Exact representation of repo-rev and what metadata it covers.
• How much of the workspace/build/cache machinery belongs in Clay core vs helper abstractions around Ford.
• Whether author signatures should be supported in addition to committer signatures.
• Exact label/tag conflict policy if labels are mutable and replicated.