Archivematica - Remove activeTransfers from transfer submission (V2)

remove-active-transfers-transfer-submission-v2.md

Proposal: Remove activeTransfers from transfer submission

Date: 2026-07-13

Foundation: PR #2313 moved transfer-source retrieval for auto-approved package creation into the normal workflow and is present on qa/1.x.

Summary

Archivematica currently starts several kinds of transfer by moving content into activeTransfers and waiting for MCPServer to notice it. This makes the filesystem location serve both as content storage and as the command to begin processing. It also couples submission APIs to internal paths and creates an unclear boundary between "the content was copied" and "the Transfer is ready."

This proposal makes transfer submission explicit. MCPServer will create the Transfer first, arrange for its content to be copied into a non-watched submittedTransfers location, and start the appropriate workflow directly. The filesystem will continue holding transfer content, but the appearance of a path will no longer create or initiate a supported submission.

This builds directly on PR #2313. That change moved retrieval for auto-approved API transfers out of MCPServer bootstrap workers and into normal workflow tasks performed by MCPClient. It made slow, queued, and failed retrieval visible in the workflow and reduced the chance that a slow Storage Service copy could occupy MCPServer workers. It intentionally left non-auto-approved submission on the watched-directory path. This proposal applies the same explicit handoff to the remaining supported submission paths and removes that architectural split.

Existing public APIs remain available. Their request fields, response shapes, status values, approval outcomes, accession information, and metadata associations are preserved. In particular, the existing Automation Tools v1 sequence remains supported without client code changes: start a transfer, modify it with pre-transfer hooks, find it through /unapproved, approve it by directory and type, and poll its status.

Some incidental filesystem behavior will change. Returned paths and approval directory values will use submittedTransfers and include the Transfer UUID. Direct drops into activeTransfers, collision-padding names, and watcher-based restart rediscovery will no longer be supported submission behavior.

This is a focused submission change, not a replacement of Archivematica's shared filesystem, workflow engine, or queues. Durable retries and restart reconstruction remain possible future improvements.

Motivation

Archivematica needs a shared filesystem to hold transfer content. The problem is not that files live on a filesystem; it is that the arrival of files currently also acts as the command to create or start a Transfer.

Submission intent should be explicit. A caller should ask Archivematica to create a Transfer, and Archivematica should then arrange storage and processing for that known Transfer. The filesystem should hold its content without also being the message that processing should begin.

Foundation and follow-up to PR #2313

Issue #1803 documented a specific operational gap: Storage Service retrieval happened before the first normal workflow Job, so operators could not reliably see whether retrieval was queued, running, failed, or complete. Slow or blocked storage I/O could make a Transfer appear idle while tying up an MCPServer bootstrap worker.

PR #2313 addressed that gap for auto-approved package creation. MCPServer now plans retrieval and schedules it; MCPClient performs the Storage Service copy as normal workflow work. The same change records early failures and represents a Transfer waiting for its first Job instead of reporting it as unknown.

That work was also connected to concrete production reports:

• Issue #1790 showed that a Storage Service or startup failure could leave an API Transfer permanently in UNKNOWN; PR #2313 was considered to address that failure path.
• Issue #1791 described MCPServer stalling under sustained parallel API submission, requiring manual restarts and leaving newly submitted Transfers unprocessed. Moving retrieval out of MCPServer was expected to reduce that pressure, but the issue called for further load testing.
• Issue #1807 described MCPClient workers that could remain alive without processing Gearman work. PR #2313 changes the load and failure path through those components, but it was not established as a complete fix for worker readiness.

PR #2313 deliberately retained the active-transfer watched-directory path for non-auto-approved packages. As a result, Archivematica currently has a stronger, observable retrieval model for one API mode while other supported submissions still depend on filesystem discovery. This proposal is an important follow-up: it reuses that workflow foundation for both package approval modes, the legacy API, SWORD, and transfer-target reingest, then ends filesystem discovery as a supported submission mechanism.

This proposal should not be read as claiming to solve every MCPServer, MCPClient, or Gearman stall described in the related issues. Its contribution is narrower: all supported submissions receive an explicit Transfer identity and a defined handoff into processing; transfer-source retrieval and early failures are represented in the workflow instead of relying on an indirect watcher event.

One path carries too many meanings

An activeTransfers path currently represents several things at once:

• the submitted content;
• a request to start processing;
• the transfer type;
• part of the Transfer's identity;
• the workflow starting point.

As a result, directory names, parent directories, archive detection, trailing separators, and collision padding become part of submission behavior. A storage layout has effectively become an implicit API and data model.

The handoff is not explicit

Moving content into activeTransfers and starting its workflow are separate events. A successful move does not acknowledge that MCPServer has observed the path, selected the workflow, or created the approval Job.

The Automation Tools v1 client demonstrates this boundary. After start_transfer returns, it sleeps and retries while waiting for the same Transfer to appear in /unapproved. The client is compensating for a hidden asynchronous gap between "the files were copied" and "the Transfer is ready for approval."

An explicit submission command provides a clearer contract: the system knows which Transfer is being created, and the legacy endpoint can wait for a defined readiness condition before returning success.

Filesystem observation depends on the environment

The watched-directory implementation supports two different mechanisms:

• polling, which repeatedly scans every watched directory and can delay discovery until the next interval;
• inotify, which is efficient on Linux but may not work as expected with some shared or network filesystems, including NFS.

The current mechanism has worked for many deployments, so this proposal does not assume that watchers always lose events. The concern is that submission coordination and timing depend on operating-system and filesystem behavior and do not have an explicit acknowledgement boundary.

The filesystem is an unofficial submission API

The active-transfer watcher can create a Transfer when it observes a path with no matching database row. Any actor able to write into the directory can therefore initiate processing without using a supported submission endpoint.

That makes it harder to apply request validation, identify the initiating caller, reject unsupported input early, and distinguish deliberate submissions from stray or abandoned content.

Public behavior depends on internal paths

The legacy /unapproved endpoint derives the transfer type and approval selector by parsing a workflow Job path. /approve reverses the process and reconstructs a watched path from the caller's directory and type.

This spreads one public operation across Dashboard copying code, MCPServer watchers, workflow watched-directory definitions, filesystem naming rules, and approval-path parsing. A change to an internal directory convention can affect API behavior in another component.

Archivematica has two submission models

Auto-approved package creation already creates a Transfer explicitly and schedules transfer-source retrieval as workflow work. Non-auto-approved package creation, the legacy API, SWORD, and transfer-target reingest still use the active-transfer watcher.

Maintaining both models requires contributors to understand two different ways that a Transfer can come into existence and begin processing. Converging on explicit creation gives supported submission paths the same ownership and starting model.

Partial failures are difficult to interpret

Database creation, content copying, the final move, watcher observation, and workflow scheduling are separate events. When one step fails, operators and developers may need to correlate a database row, a filesystem path, watcher logs, and Jobs to determine how far submission progressed.

This proposal does not make the queues durable or provide exactly-once execution. It does make the Transfer UUID the correlation point, defines when the materialized path becomes authoritative, and records failures against a known Transfer instead of relying on later filesystem discovery.

Scope and trade-off

This is not a general removal of shared storage or watched directories from Archivematica. Existing workflows continue using the shared filesystem, and unrelated watched-directory transitions remain unchanged.

The active-transfer watcher also provides some incidental restart rediscovery and supports direct filesystem drops. This proposal deliberately stops relying on those behaviors for submission. Durable dispatch, restart reconstruction, and exactly-once retry can be addressed separately if their operational value justifies the additional model and complexity.

Submission mental model

A submission has three distinct parts:

Part	Meaning
Request	A caller asks Archivematica to create a Transfer.
Content	The files that belong to that Transfer.
Workflow	The processing Archivematica performs on those files.

The current design often uses one filesystem event to connect all three. A caller places content in activeTransfers; MCPServer notices the new path, infers which Transfer and workflow it represents, and starts processing.

The proposed design connects them explicitly. MCPServer receives a submission request, creates the Transfer, arranges for its content to be materialized, and starts the selected workflow. The filesystem holds the content but does not act as the submission message.

Current legacy API example

Automation Tools is a useful example because it uses the complete v1 submission and approval sequence:

Automation Tools
      |
      | 1. POST /api/transfer/start_transfer/ and wait
      v
Dashboard
      |
      | 2. Copy the selected content
      v
activeTransfers/<type>/<name>
      |
      | 3. A filesystem watcher notices the new path
      v
MCPServer
      |
      | 4. Infer the workflow from the watched directory
      | 5. Create an approval Job
      v
Transfer waits for approval

After start_transfer returns a path, Automation Tools modifies the submitted content with any configured pre-transfer hooks. It then calls /unapproved, finds the path basename, and sends that directory and type to /approve.

The copy in step 2 is part of the synchronous HTTP request: the caller waits for it. Steps 3 through 5 happen independently after the filesystem changes. This means the copy can be complete before the watcher has made the Transfer visible through /unapproved.

Proposed legacy API example

The caller sees the same API sequence, but the internal handoffs are explicit:

Automation Tools
      |
      | 1. POST /api/transfer/start_transfer/ and wait
      v
Dashboard
      |
      | 2. Ask MCPServer to create a Transfer
      v
MCPServer creates the Transfer and returns its UUID internally
      |
      | 3. Queue transfer-source retrieval
      v
MCPClient and Storage Service copy the content
      |
      | 4. Move it to submittedTransfers/<name>-<uuid>
      | 5. Record the actual path
      | 6. Start the approval workflow directly
      v
Transfer waits for approval
      |
      | 7. Dashboard returns the writable path
      v
Automation Tools runs hooks, calls /unapproved, and calls /approve

The retrieval and workflow work are asynchronous inside Archivematica: they can wait in queues and run in other processes. The legacy start_transfer request remains synchronous from the caller's perspective. Dashboard keeps waiting until both of these statements are true:

1. the content exists at the returned path;
2. the Transfer is visible through /unapproved and ready for a decision.

/unapproved is a synchronous, read-only snapshot. It does not discover files or start work. It reports approval Jobs that MCPServer has already created. /approve records the caller's decision synchronously; after that response, the accepted or rejected workflow continues asynchronously.

Concrete Automation Tools round trip

For example, Automation Tools submits a standard directory named records. The response remains:

{
  "message": "Copy successful.",
  "path": "/var/archivematica/sharedDirectory/submittedTransfers/records-<uuid>/"
}

Automation Tools can add or modify files at that path before approval. Its next /unapproved request includes:

{
  "directory": "records-<uuid>",
  "type": "standard",
  "uuid": "<uuid>"
}

The client derives directory from the returned path and sends the same value and type to /approve. It does not need to know that Archivematica stopped using activeTransfers, and it does not need to adopt UUID approval.

The externally important boundary is therefore:

start_transfer succeeds
        means
content is writable + the Transfer is ready to appear in /unapproved

The main change is who initiates the workflow:

Question	Current design	Proposed design
What submits a Transfer?	Content appearing in a watched path.	An explicit MCPServer command.
Where is submitted content held?	activeTransfers/<type>/.	submittedTransfers/.
How is the workflow selected?	From the watched directory.	From the validated request type.
Legacy return point	Copy done; watcher may lag.	Copy and approval ready.
/unapproved source	Parsed workflow path.	Transfer metadata.
Automation Tools	Existing API sequence.	Same sequence; no stock client changes.

Decision

The database and MCPServer command establish that a Transfer exists. The filesystem stores its content; the appearance of a path does not create a Transfer or act as the authoritative submission command.

MCPServer owns these submission steps:

1. validate the request and select the transfer workflow starting point;
2. create the Transfer and its compatibility metadata;
3. schedule transfer-source retrieval, or accept a confined materialized source;
4. publish the actual materialized path after the final move succeeds;
5. schedule the existing approval or accepted workflow directly.

MCPClient continues performing Storage Service retrieval and workflow file operations. Dashboard remains an HTTP compatibility layer and does not copy transfer-source content itself.

Goals

• Remove activeTransfers from every supported submission path in this delivery.
• Stop creating Transfers from unknown filesystem drops.
• Make MCPServer the authority that creates a Transfer and schedules its first workflow.
• Use the existing retrieval workflow for both package API approval modes.
• Preserve request fields, response shapes, public status labels, accession and metadata associations, and accept/reject outcomes.
• Keep legacy directory/type approval working when callers pass back the value returned by /api/transfer/unapproved.
• Keep the supported Automation Tools v1 client working without source changes, including its start, pre-transfer hook, unapproved lookup, directory/type approval, and status sequence.
• Add UUID approval as the preferred selector for new integrations.
• Keep the implementation incremental and avoid introducing a second durable workflow runtime.

Compatibility boundary

Behaviors preserved

• POST /api/v2beta/package/ accepts its existing payload and returns the Transfer UUID.
• POST /api/transfer/start_transfer/ accepts its existing form fields, creates one Transfer per selected path, preserves name suffixing for later paths, and returns the existing JSON keys.
• A successful start_transfer response identifies content that exists, is writable by the Automation Tools process in the supported shared-filesystem deployment, and is already visible through /api/transfer/unapproved.
• GET /api/transfer/unapproved keeps its current response shape.
• For a Transfer created by start_transfer, the directory returned by /unapproved equals the basename of the returned start_transfer.path after removing a trailing separator.
• POST /api/transfer/approve continues accepting directory and type and returns the approved Transfer UUID.
• Transfer status APIs continue returning PROCESSING, USER_INPUT, COMPLETE, and FAILED according to the existing public model.
• Non-auto-approved submissions appear in /unapproved only after their content has been materialized successfully.
• Rejection operates on materialized content and follows the existing rejected transfer workflow.
• Transfer-target reingest retains its request and response shapes and creates a Transfer whose workflow type is Archivematica AIP.
• SWORD deposits still become standard Transfers and are approved automatically by the Storage Service integration.

Behaviors allowed to change

• Physical paths move from activeTransfers to submittedTransfers.
• New approval directory values include the Transfer UUID and should be treated as opaque round-trip selectors.
• Archive UUID placement changes so the complete archive extension remains at the end of the physical filename.
• Filesystem _2, _3, and similar collision padding is not preserved for new submissions; the Transfer UUID provides uniqueness.
• start_transfer.path returns the actual new materialized location, not an activeTransfers path.
• start_transfer can spend additional time waiting for capacity in PackageQueue before copying begins.
• Detailed workflow Job IDs, descriptions, ordering, and timestamps may change.
• MCPServer restart no longer rediscovers an unapproved Transfer from a watched path. Interrupted submissions fail conservatively.
• Direct filesystem drops into activeTransfers no longer submit Transfers.
• New and old Archivematica/Storage Service component versions are not expected to provide a supported mixed-version submission topology.
• Names previously accepted only because path operations failed later may now be rejected during request validation.
• Deployment-specific pre-transfer hooks that parse the old activeTransfers/<type>/<name> layout or require a UUID-free basename may need adjustment. The stock Automation Tools client treats the returned path dynamically and does not depend on that layout.

These changes must be documented in release notes. They are intentional compatibility boundaries, not implementation accidents.

Non-goals

• Do not make PackageQueue or Gearman durable.
• Do not reconstruct interrupted jobs or approval decisions after restart.
• Do not provide exactly-once submission or retry semantics for SWORD, reingest, or the legacy APIs.
• Do not add a general local-filesystem submission endpoint.
• Do not remove unrelated watched directories from the workflow runtime.
• Do not redesign normal Transfer completion or failure workflows.
• Do not automatically delete every abandoned temporary or submitted path.
• Do not retarget internal failed-compliance links unless required for the submission cutover.
• Do not preserve unsupported mixed MCPServer versions during rolling restart.
• Do not preserve undocumented path-layout assumptions in deployment-specific Automation Tools hooks beyond the absolute writable path and transfer-type arguments.

Current state

Package API with auto approval

POST /api/v2beta/package/ with auto_approve=True already creates a Transfer through MCPServer and schedules transfer-source retrieval as workflow work. The retrieval task currently materializes directly into currentlyProcessing and continues at an optimized post-move link.

Package API without auto approval

auto_approve=False still copies through shared tmp, moves into an active-transfer watched directory, and relies on the watcher to create the approval decision.

Legacy start_transfer

Dashboard copies each selected transfer-source path and moves it into a type-specific directory under activeTransfers. It creates the Transfer before the move and returns only after copying all selected paths.

Approval APIs

/api/transfer/unapproved parses Job.directory as an active-transfer path. /api/transfer/approve reconstructs that path from directory and type before asking MCPServer to decide the matching Job.

Storage Service SWORD

Storage Service moves a finalized deposit into activeTransfers/standardTransfer, polls /unapproved by deposit name, and approves it.

Transfer-target reingest

The transfer-target reingest endpoint moves content from shared tmp into activeTransfers/standardTransfer, creates an Archivematica AIP Transfer, and relies on the standard transfer watcher.

Direct drops and internal recovery

The active-transfer watcher creates a Transfer when no matching database row exists. This makes the directory an unofficial filesystem API.

Some failed standard and DSpace compliance paths also move known Transfers back into activeTransfers. Those are internal recovery operations rather than new submission and can remain temporarily if the watcher is restricted to exact known Transfer locations.

Submission storage

MCPServer creates the shared root during startup:

/var/archivematica/sharedDirectory/submittedTransfers/

It is not a watched directory. Operators and integrations must not place content there directly.

Create the root and submitted children with the same shared group and writable permission contract used by the supported Archivematica deployment. An Automation Tools process that could modify the old returned activeTransfers path through the standard shared-directory mount must also be able to modify a returned submittedTransfers path.

New destinations contain the Transfer UUID:

directory: %sharedPath%submittedTransfers/<validated-name>-<uuid>/
archive:   %sharedPath%submittedTransfers/<validated-stem>-<uuid><extension>

Examples:

records-3c35c11e-d34a-43c2-a3b7-105f68f7c5d0/
deposit-3c35c11e-d34a-43c2-a3b7-105f68f7c5d0.tar.gz

Do not pad a UUID destination on collision. An existing destination for the same UUID is a conflict or an abandoned path requiring operator attention; it must not be overwritten.

submittedTransfers is pre-processing storage. The accepted standard workflow moves content from there into currentlyProcessing. Archive workflows perform their existing extraction or move as their first accepted step. Rejection moves content through the existing rejected-transfer workflow.

Minimal Transfer metadata

Add nullable fields to Transfer:

submission_type
submission_display_name
submission_approval_directory
submission_materialized_at

Null fields identify legacy and pre-upgrade Transfers.

• submission_type stores the legacy API value such as standard, zipfile, or zipped bag. Do not overload Transfer.type, which normal workflows set to values such as Standard, Dspace, TRIM, or Archivematica AIP.
• submission_display_name is the stable logical name shown by status APIs.
• submission_approval_directory is the opaque UUID-bearing selector returned by /unapproved and accepted by /approve.
• submission_materialized_at is set only after the final destination exists. It lets the synchronous legacy endpoint distinguish accepted submission from completed copying without introducing a general lifecycle state machine.

Use these values for new submissions:

Surface	Display name	Approval directory
v2beta directory	Validated request name	<name>-<uuid>
v2beta archive	Archive basename	<stem>-<uuid><extension>
legacy start directory	Suffixed target name	<target>-<uuid>
legacy start archive	Archive basename	<stem>-<uuid><extension>
SWORD	Validated deposit name	<name>-<uuid>
transfer reingest	Validated normalized AIP name	<name>-<uuid>

The approval directory is the submitted physical basename without a trailing separator. It is intentionally UUID-bearing for every new surface.

Do not add submission-state, retry-attempt, planned-path, dispatch, approval-key, or recovery columns in this change. The existing Transfer status, Job status, workflow unit variables, and actual currentlocation remain authoritative for their existing purposes.

Keep Transfer.currentlocation empty until content has been materialized. In one database transaction after the final move, set:

currentlocation = <actual %sharedPath%submittedTransfers path>
submission_materialized_at = <current time>

The retrieval Job receives its copied and final paths through explicit workflow context. It must not require a planned nonexistent value in currentlocation. Status APIs use submission_display_name while no physical path or Job exists.

Transfer type starting points

Make the two workflow entry points for every supported transfer type explicit:

• approval chain and decision link;
• accepted chain and its first link.

The mapping covers:

• standard;
• zipfile;
• unzipped bag;
• zipped bag;
• dspace;
• maildir;
• TRIM;
• dataverse.

The temporary mapping can retain the legacy watched-directory path for pre-upgrade lookup and tests, but new submission code must never use it as a destination.

Continue using the existing retrieval chain:

Retrieve transfer source
Continue after transfer source retrieval

Set linkAfterTransferSourceRetrieval to:

auto_approve=True  -> accepted chain first link
auto_approve=False -> approval decision link

Starting at the accepted chain's first link is intentional. Retrieval now ends in submittedTransfers, so the established move or extraction step is required and the current post-move optimization should be removed.

Transfer-source submission flow

For either package API approval mode:

1. validate name, transfer type, source selection, user, and processing config;
2. create the Transfer with public PROCESSING status and minimal submission metadata;
3. create an MCP-controlled temporary retrieval directory;
4. plan the exact UUID destination under submittedTransfers;
5. schedule the retrieval chain with explicit source, copied, and final paths;
6. MCPClient asks Storage Service to copy into the temporary directory;
7. MCPClient renames the copied result to the exact submitted destination;
8. publish currentlocation and submission_materialized_at together;
9. continue to the accepted first link or approval decision.

If retrieval fails, the retrieval workflow records a failed Job and marks the Transfer FAILED. It must not create an approval prompt.

If the final move succeeds but database publication fails, log the exact orphan path and mark the Transfer failed when possible. Automatic compensation and replay are outside this proposal.

If scheduling the continuation fails after materialization, mark the Transfer failed and leave its submitted content for documented operator cleanup. Do not attempt durable redispatch.

Name and path safety

Any label used as a physical path component must be validated before creating a Transfer or moving content. This applies to package names, start_transfer names, archive basenames, SWORD deposit names, and reingest names.

A valid component must:

• be non-empty;
• not be . or ..;
• not contain NUL, /, or \\;
• not be absolute;
• represent exactly one component;
• fit the destination filesystem's component-length limit after adding the UUID and complete archive extension.

Reject invalid input rather than silently rewriting it. Keep the validated Unicode label as the display name.

For SWORD and reingest, accept only an opaque source kind and identifier. Resolve the source under a fixed configured root, require it to be strictly contained by that root, and reject missing paths and symlinks that escape it. Callers cannot provide a destination.

Check UUID destinations with lexists semantics and never overwrite them. Transfer-source selection confinement continues to rely on the existing Storage Service location association and copy API.

v2beta package API

Keep:

POST /api/v2beta/package/

and its response:

{"id": "<transfer-uuid>"}

Both auto_approve values use MCPServer retrieval into submittedTransfers. They differ only in the link selected after retrieval.

New submissions remain visible as PROCESSING before their first Job row, using the existing processing-configuration timestamp fallback already present on qa/1.x. The status response uses submission_display_name instead of deriving the name from an empty or UUID-bearing path.

Legacy start_transfer API

Keep the endpoint and form fields:

POST /api/transfer/start_transfer/

Dashboard should no longer copy files or move content into a watcher. For each selected path, in order:

1. preserve the current target naming (name, name_2, name_3, ...);
2. call MCPClient.create_package(..., auto_approve=False);
3. receive the Transfer UUID from the normal short Gearman RPC;
4. poll that Transfer, treating public FAILED as an error before checking submission readiness;
5. wait for both a non-null submission_materialized_at and the corresponding known approval Job to be committed as AWAITING_DECISION;
6. only then submit the next selected path.

Return the existing success response after all components are materialized and ready for approval:

{
  "message": "Copy successful.",
  "path": "<last actual materialized path>"
}

The returned path is below submittedTransfers and includes the Transfer UUID. Directory paths preserve the trailing separator; archive paths do not add one. It must exist, remain at that location, and remain writable while the Transfer awaits approval. Changes made there by pre-transfer hooks must be the content seen by the accepted or rejected workflow.

For each returned Transfer:

unapproved.directory == basename(start_transfer.path without trailing slash)

The approval workflow must not move or otherwise process the submitted content before the explicit approval decision.

The request now waits for PackageQueue delay, copy and move time, and the short continuation into the approval decision. Do not hold the packageCreate RPC open for that duration. Do not return "Copy successful." before both materialization and approval readiness. Use the same approval-readiness query in start_transfer and /unapproved so their views cannot drift. This closes the race where an Automation Tools client receives a path but cannot find its Transfer in /unapproved during its finite approval retry window.

If the client disconnects, the accepted Transfer continues. Because the legacy request has no idempotency key, retrying an uncertain response can create another Transfer. Clients should inspect transfer status before retrying.

The existing partial-success behavior remains: an earlier selected path may have materialized when a later path fails.

Unapproved and approve APIs

GET /api/transfer/unapproved keeps its response shape but stops parsing active-transfer paths.

For new submissions:

• find awaiting Jobs by known transfer approval link IDs;
• read type from Transfer.submission_type;
• read directory from Transfer.submission_approval_directory;
• return the Job's Transfer UUID.

Do not return a submission before its materialization metadata is committed and its actual path exists. A successful legacy start_transfer response must imply that the same Transfer is immediately eligible for this result set.

The approval directory is derived from the UUID-bearing submitted basename and is unique without collision padding. Callers must treat it as opaque.

Add UUID approval:

POST /api/transfer/approve

uuid=<transfer-uuid>

Retain directory/type approval:

POST /api/transfer/approve

directory=<value returned by unapproved>
type=<legacy-transfer-type>

Resolution order is:

1. UUID, when supplied;
2. exact new submission metadata match;
3. exact pre-upgrade watched-path fallback.

Only match Jobs that are awaiting a known transfer-approval decision. Do not approve an arbitrary Job or choose ambiguously.

Automation Tools v1 compatibility

The supported Automation Tools v1 client uses this sequence without knowing the Transfer UUID in advance:

1. call start_transfer and read its path;
2. run pre-transfer hooks against that absolute path;
3. derive the selector from the path basename;
4. find the same selector and its type in /unapproved;
5. submit that directory and type to /approve;
6. poll the existing Transfer and ingest status APIs.

Preserve this sequence for the unmodified supported Automation Tools client and its pinned amclient==1.6.0. It is safe for the physical basename to include the Transfer UUID because the stock client derives and round-trips the returned value instead of constructing it.

The compatibility guarantee assumes the normal deployment in which Automation Tools can access /var/archivematica/sharedDirectory at the same absolute path. Deployments that mount only the old activeTransfers subtree must expose the new submittedTransfers root. Custom hooks that parse the old parent-directory layout or a UUID-free basename are outside the stock-client guarantee and must be called out in the release notes.

Materialized submission helper

Add one MCPServer helper for content already present in shared storage:

submit_materialized_transfer(
    source_kind,
    source_identifier,
    transfer_uuid,
    display_name,
    submission_type,
    ...existing Transfer metadata,
)

The helper accepts a closed source-kind mapping:

sword    -> %sharedPath%tmp/transferSubmissionIntake/sword/<deposit-uuid>
reingest -> %sharedPath%tmp/<validated-reingest-name>

It does not accept an arbitrary path. It validates the source, creates the Transfer, moves content to the exact UUID destination, publishes the actual location and materialization timestamp, and schedules the approval decision.

The helper does not promise idempotent retry. A duplicate Transfer UUID or existing destination is an explicit conflict. It never overwrites content or silently creates a second destination for the same UUID.

Storage Service SWORD

Storage Service stages a finalized deposit at:

%sharedPath%tmp/transferSubmissionIntake/sword/<deposit-uuid>/

It calls a dedicated authenticated materialized-package endpoint with:

• source kind sword;
• deposit UUID as source identifier and requested Transfer UUID;
• validated public transfer name.

The endpoint accepts no filesystem path. Dashboard forwards the closed source identity to MCPServer.

After receiving the Transfer UUID, Storage Service polls /api/transfer/unapproved until that UUID appears and approves it using UUID approval. It no longer correlates by deposit name.

There is no fallback that moves the deposit into activeTransfers. The Archivematica release must pin a Storage Service revision that implements this endpoint contract. Unsupported old/new component pairings fail visibly rather than silently returning to filesystem submission.

On an uncertain integration retry, a duplicate UUID can return a conflict. The deposit remains in its intake or submitted location for status inspection and operator handling. Automatic retry reconciliation is future work.

Transfer-target reingest

The existing transfer-target reingest endpoint keeps its public request and response shapes.

Before destructive database cleanup or filesystem movement:

• validate name as one component;
• resolve the source strictly beneath %sharedPath%tmp;
• plan the exact UUID destination and reject an existing path.

Then perform the existing SIP, Job, rights, and Dublin Core cleanup and invoke submit_materialized_transfer with the initial Transfer.type=Archivematica AIP. The helper creates the new Transfer, moves the payload, and schedules the standard transfer approval decision directly.

The response remains:

{
  "message": "Approval successful.",
  "reingest_uuid": "<uuid>"
}

The historical message does not mean the transfer bypasses its normal approval decision.

Reingest retry remains non-idempotent unless its caller supplies and retains a stable UUID in a future contract. A repeated request can conflict or create a new operation, as it can today.

Direct activeTransfers drops

At final cutover, the active-transfer watcher must not call get_or_create_from_db_by_path() for unknown paths.

For an active-transfer watched directory:

1. convert the observed path to its exact database form;
2. look up an existing Transfer whose currentlocation is exactly that value;
3. schedule the internal recovery chain only when that exact row exists;
4. otherwise log a warning and leave the unknown content untouched.

A UUID-looking filename is not sufficient evidence of ownership.

This guarded behavior keeps known failed-compliance recovery working while ending direct filesystem submission immediately. Release notes must tell users to use the package or legacy transfer APIs.

After failed-compliance links are retargeted, a future change can remove the remaining active-transfer watcher definitions and directories entirely.

Failure and restart behavior

Submission uses the existing MCPServer failure model rather than adding a durable lifecycle.

• Validation failure creates no Transfer and moves no content.
• Executor or initial scheduling rejection marks a created Transfer failed.
• Retrieval failure creates a visible failed Job and marks the Transfer failed.
• Final move failure marks the Transfer failed and leaves enough logging for temporary-path cleanup.
• Database publication failure after a move logs the exact orphan destination and attempts to mark the Transfer failed.
• Continuation scheduling failure after materialization marks the Transfer failed and leaves content for operator cleanup.
• Rejection follows the existing rejected-transfer workflow.

On MCPServer restart, retain existing conservative cleanup:

• queued or retrieving PROCESSING Transfers become FAILED;
• materialized Transfers waiting for approval become FAILED when their in-memory decision is lost;
• Transfers whose accepted workflow was running become FAILED as today;
• awaiting-decision Job rows are not reconstructed.

The old active-transfer watcher previously rediscovered some waiting Transfers. That recovery was an incidental consequence of filesystem submission and is not preserved. Content left under submittedTransfers after failure is never treated as a new submission.

Cleanup and operations

• Successful accepted workflows move or extract content out of submittedTransfers.
• Rejection moves content into the rejected-transfer location.
• Retrieval failure should remove its temporary copy on a best-effort basis.
• Failed materialized content and UUID destination conflicts are logged for operator inspection.
• Initially, do not automatically delete unreferenced submitted content.
• Document how operators correlate submitted paths with Transfer UUIDs and remove confirmed abandoned content.
• Monitor shared-directory capacity because unapproved and failed content can remain for an extended period.

Age-based orphan cleanup can be introduced later after operational experience defines a safe retention policy.

Upgrade and deployment

• Stop all old MCPServer processes before starting the new version.
• Deploy the Archivematica and Storage Service revisions from the supported release pairing.
• Drain active submissions before upgrade where practical.
• Keep exact-path approval fallback for pre-upgrade awaiting Jobs.
• Do not automatically migrate arbitrary content already in activeTransfers.
• Unknown active-transfer content remains for operator inspection after cutover.
• Application rollback after new submissions requires draining or manually handling content under submittedTransfers.

The cutover release intentionally ends direct active-transfer filesystem submission. It does not require a prior compatibility release because the replacement public APIs already exist, but the removal must be prominent in release and upgrade notes.

Delivery plan

Use an expand-migrate-contract sequence within the same delivery.

PR 1: Add explicit package submission

• add submittedTransfers and the SWORD intake root;
• add the four nullable Transfer fields;
• make approval and accepted starting points explicit;
• validate names and plan exact UUID destinations;
• route both package API approval modes through retrieval;
• continue at the accepted first link or approval decision;
• update status, unapproved, and approve readers for new metadata;
• add UUID approval while keeping pre-upgrade path fallback.

PR 2: Migrate start_transfer

• replace Dashboard-side copying with package creation;
• preserve sequential multi-path behavior and response keys;
• wait for materialization and the matching awaiting-approval Job after the short RPC;
• preserve the Automation Tools writable-path and directory/type round trip;
• document new paths and PackageQueue delay.

PR 3: Add confined materialized submission

• add submit_materialized_transfer;
• migrate transfer-target reingest;
• add the authenticated path-free SWORD endpoint;
• cover source containment and no-overwrite behavior.

PR 4: Migrate Storage Service SWORD

• stage deposits in the UUID intake root;
• call the new endpoint;
• wait and approve by returned Transfer UUID;
• remove SWORD writes to activeTransfers;
• do not add an old-pipeline filesystem fallback.

PR 5: Complete the cutover

• update the Storage Service submodule revision;
• guard the active-transfer watcher against unknown paths;
• verify repository-wide that supported writers no longer use activeTransfers;
• retain only named internal failed-compliance compatibility writes;
• add release notes and upgrade guidance.

PRs 1 through 4 add and migrate the replacement while the old watcher remains available to callers not yet migrated in the development branch. PR 5 is the release cutover and must not merge until every supported writer is migrated.

Testing strategy

MCPServer and workflow

• Both package approval modes retrieve to submittedTransfers.
• Every supported transfer type has valid approval and accepted starting links.
• Auto approval enters the accepted chain at its first link.
• Non-auto approval stops at the approval decision after materialization.
• currentlocation remains empty before materialization.
• currentlocation and submission_materialized_at are published together.
• Retrieval, move, publication, and continuation failures mark the Transfer failed without creating an approval prompt.
• Standard accepted, rejected, archive, bag, DSpace, maildir, TRIM, and Dataverse continuations operate from the submitted location.
• Restart marks queued, retrieving, awaiting-approval, and continuing Transfers failed according to the documented policy.
• Submitted content is not rediscovered as a new Transfer after restart.

Dashboard APIs

• v2beta request and response shapes remain unchanged.
• start_transfer submits every selected path in order and waits for both materialization and approval readiness.
• Multi-path suffixing and partial success remain.
• start_transfer.path is the last actual UUID-bearing path.
• The returned path exists, remains writable while awaiting approval, and contains changes made before approval when processing continues.
• Queue delay does not produce early "Copy successful." responses.
• A delay between materialization and creation of the awaiting-approval Job does not produce an early success response.
• Status APIs use submission_display_name before and after Jobs exist.
• /unapproved returns the existing shape and opaque UUID-bearing directory.
• The new /unapproved result is committed before start_transfer returns, and its directory equals the basename of start_transfer.path.
• UUID approval works.
• Directory/type approval round-trips the new value.
• Pre-upgrade watched-path approval fallback works.
• Duplicate display names remain independently approvable because selectors include different UUIDs.

Automation Tools

• Run a Compose integration test with the supported Automation Tools v1 client unchanged and its pinned amclient==1.6.0.
• Exercise its real sequence: start_transfer, a pre-transfer hook, /unapproved, directory/type /approve, and status polling.
• Have the hook write a marker or metadata file through the returned absolute path and verify that the accepted workflow processes that change.
• Cover a standard directory and an archive transfer so trailing-separator and archive-basename behavior are both exercised.
• Delay approval-Job creation beyond the client's finite retry window and verify that start_transfer absorbs the delay instead of returning early.
• Do not patch Automation Tools to understand UUIDs or use the new UUID approval parameter; compatibility is demonstrated by its existing basename round trip.

Path safety

• Empty, dot, traversal, separator, absolute, NUL, and overlong components are rejected before Transfer creation or movement.
• Escaping symlinks and sources outside fixed local roots are rejected.
• Broken symlink and existing UUID destinations are conflicts.
• Destinations are never padded or overwritten.
• Complete archive extensions are preserved after UUID insertion.

SWORD and reingest

• SWORD uses the new endpoint and never writes to activeTransfers.
• SWORD correlates and approves by returned Transfer UUID.
• Missing intake content and duplicate UUID conflicts fail visibly.
• No HTTP error causes a fallback into activeTransfers.
• Reingest validates its source before destructive cleanup.
• Transfer-target reingest moves to submittedTransfers, retains Archivematica AIP, and enters standard transfer approval.
• Existing endpoint response shapes remain.

Cutover

• Unknown direct drops do not create Transfers.
• UUID-looking unknown paths are also rejected.
• Exact known failed-compliance paths still run while compatibility remains.
• Repository-wide search finds no supported submission writer targeting activeTransfers.
• Remaining activeTransfers writes are enumerated internal recovery cases.

Run the relevant Compose targets for each pull request, including:

make -C hack test-dashboard
make -C hack test-mcp-server
make -C hack test-mcp-client
make -C hack test-storage-service
make -C hack test-storage-service-integration
make -C hack test-linting

Known limitations and follow-up opportunities

Durable dispatch and restart recovery

MCPServer still keeps PackageQueue and approval decisions in memory. A future MCPServer-wide design could persist dispatch attempts or reconstruct waiting decisions. That work should address Transfers, SIPs, and DIPs consistently rather than creating a durable subsystem only for submission.

Integration retry reconciliation

SWORD and reingest can later gain a persisted operation identity and explicit retry contract. This proposal detects UUID conflicts but does not decide whether to resume, return, or retry every partial filesystem/database state.

Transfer-target reingest also retains the existing lack of one transaction across destructive cleanup and filesystem movement. A failure after database cleanup can require operator repair. Solving that belongs with a future idempotent reingest operation contract, not with the active-transfer cutover.

Submitted-content retention

Operational experience should define an age-based cleanup policy for failed or unreferenced content. Automatic deletion is deferred to avoid removing content that an operator may need to inspect or recover.

Remaining active-transfer recovery links

Failed-compliance workflows may temporarily retain exact-known-path use of the active-transfer watcher. Retargeting those links will allow removal of the active-transfer watcher definitions and physical directories.

Versioned asynchronous legacy API

A future versioned endpoint could make materialization explicitly asynchronous, return operation identifiers for multi-path requests, and avoid holding an HTTP request through PackageQueue delay. The existing endpoint retains synchronous copy semantics in this delivery.

Submission lifecycle observability

If operators later need to distinguish queued, copying, materialized, waiting, and continuing states independently of Jobs, add a focused lifecycle model at that time. The current design deliberately uses only the one materialization timestamp required by the legacy synchronous contract.

Release notes and documentation

Release notes must state:

• supported transfer submission no longer uses watchedDirectories/activeTransfers;
• direct filesystem drops there are no longer supported;
• public submission APIs remain available;
• unapproved directory values and returned physical paths can now contain the Transfer UUID and use the new root;
• interrupted unapproved submissions are no longer rediscovered after MCPServer restart and instead fail conservatively;
• Archivematica and Storage Service must use the supported paired versions;
• operators may see submittedTransfers and must not submit content there manually;
• the supported Automation Tools v1 client remains compatible without code changes when it can access the full shared-directory mount;
• custom pre-transfer hooks that parse the old parent path or a UUID-free basename may require changes, and mounts limited to the old activeTransfers subtree must expose submittedTransfers;
• old active-transfer content should be drained or inspected before upgrade.

Developer and operator documentation must explain:

• the distinction between submitted and currently-processing content;
• UUID approval and opaque directory/type compatibility approval;
• the synchronous start_transfer wait and PackageQueue delay;
• manual inspection and cleanup of failed submitted content;
• the guarded purpose of any remaining active-transfer watcher.

Acceptance criteria

• No supported package API, legacy start, SWORD, or transfer-target reingest submission writes to activeTransfers after cutover.
• Both v2beta package approval modes use the same retrieval path into submittedTransfers.
• MCPServer creates the Transfer and schedules the workflow explicitly.
• Only four nullable submission compatibility/materialization fields are added to Transfer; no submission state machine or durable dispatch model is added.
• currentlocation represents actual materialized content, not a planned path.
• Public request and response shapes and status labels remain supported.
• New approval directory values are opaque and UUID-bearing; UUID and directory/type approval both work.
• Duplicate display names do not create ambiguous approval selectors.
• start_transfer waits for materialization and returns the actual new path.
• start_transfer does not return success until the matching Transfer is also observable through /unapproved as awaiting approval.
• The returned legacy path remains writable until approval, and its basename is the exact directory/type approval selector returned by /unapproved.
• The supported Automation Tools v1 client and amclient==1.6.0 complete start, pre-transfer modification, unapproved lookup, directory/type approval, and status polling without source changes.
• SWORD uses the path-free endpoint and approves by returned Transfer UUID, without an activeTransfers fallback.
• Transfer-target reingest uses confined materialized submission and preserves its public response and Archivematica AIP workflow type.
• Unsafe path components, escaping sources, and existing UUID destinations are rejected without overwrite.
• Unknown direct drops do not create Transfers.
• Remaining activeTransfers behavior is limited to enumerated exact-known internal recovery paths.
• Interrupted submissions fail visibly on restart and are not reconstructed.
• Release and upgrade notes describe the intentional path, naming, restart, and component-pairing changes.