Date: 2026-06-29
Branch context:
dev/issue-1803-workflow-transfer-retrieval
Remove watchedDirectories/activeTransfers/* from transfer submission.
PR#2313 already moves the main auto-approved package-create path toward workflow-native transfer-source retrieval. The next step should complete that direction for the remaining submission paths:
POST /api/v2beta/package/withauto_approve=False;- legacy
POST /api/transfer/start_transfer/; - Storage Service SWORD deposit submission;
- transfer-target reingest;
- direct filesystem drops into
watchedDirectories/activeTransfers/*.
The central change is to introduce an MCP-controlled submission staging root:
%sharedPath%submittedTransfers/
API-created and integration-created transfer content should be retrieved or moved into this root first. MCPServer should then schedule normal workflow jobs directly, without depending on a filesystem watcher to notice a directory.
The old public API contracts must remain compatible. Internal filesystem paths
can change, but clients that call start_transfer, unapproved, and approve
should keep the same request and response semantics.
- Stop using
watchedDirectories/activeTransfers/*as a transfer submission mechanism. - Preserve public API request and response semantics as much as possible,
especially for legacy
/api/transfer/start_transfer/,/api/transfer/unapproved, and/api/transfer/approve. - Make
auto_approve=Trueandauto_approve=Falseshare the same transfer-source retrieval mechanism. - Keep retrieval visible as normal workflow work, building on the current branch.
- Keep unapproved transfer behavior: a transfer should appear for user approval only after its source content has been materialized successfully.
- Include Storage Service SWORD and transfer reingest in the migration plan.
- Remove direct
activeTransfersfilesystem drops as a supported start path. - Keep the implementation incremental enough to review and test.
- Do not remove every watched directory in Archivematica in this change. Other workflow transitions still use watched directories.
- Do not redesign MCPServer persistence, PackageQueue durability, Gearman, or the broader workflow runtime in this change.
- Do not change public API status labels such as
PROCESSING,USER_INPUT,COMPLETE, andFAILED. - Do not require clients to switch from directory-based approval to UUID-based approval immediately.
- Do not support mixed old/new MCPServer versions as a valid rolling-deploy topology. The new path reduces accidental watcher interference, but it does not make mixed runtime versions safe.
- Do not change Storage Service copy semantics unless needed to preserve current transfer submission behavior.
POST /api/v2beta/package/ calls Dashboard's MCPClient.create_package(),
which calls MCPServer's packageCreate RPC, which calls
MCPServer.server.packages.create_package().
In the current branch, auto_approve=True no longer copies the transfer into an
active transfer watched directory. MCPServer creates a Transfer, schedules the
new retrieval workflow, and the MCPClient retrieval task copies from Storage
Service before the workflow continues.
This is the right direction, but the current implementation retrieves directly
into currentlyProcessing and continues at an optimized post-move link for some
transfer types.
auto_approve=False still uses the old path:
create Transfer row
copy selected source content into sharedDirectory/tmp
move content into watchedDirectories/activeTransfers/<type>
watcher notices the path
watched-dir chain creates an approval decision job
user approves or rejects
This keeps non-auto-approved transfers tied to watched directories.
POST /api/transfer/start_transfer/ is still implemented in Dashboard by
filesystem_ajax.views.start_transfer().
It:
- accepts form data, including multiple
paths[]; - copies from transfer sources in the Dashboard process;
- creates one transfer per selected path;
- suffixes the transfer name for subsequent components;
- moves the copied payload into
watchedDirectories/activeTransfers/<type>; - returns
{"message": "Copy successful.", "path": <destination>}.
Many customers rely on this endpoint, so its public behavior is more important than its current implementation.
GET /api/transfer/unapproved currently finds awaiting approval jobs and parses
their Job.directory value as if it is always under:
%sharedPath%watchedDirectories/activeTransfers/<type-dir>/<name>
POST /api/transfer/approve receives directory and type, reconstructs that
watched-directory path, and asks MCPServer to approve the matching job.
These endpoints are public API. The path parsing is internal detail and can change, but request and response semantics should remain compatible.
MCPServer still starts transfers when files or directories appear under
watchedDirectories/activeTransfers/*. If a matching Transfer row does not
exist, watched_dir_handler() creates one from the path.
That behavior effectively makes activeTransfers an unofficial filesystem API.
It is brittle and should be removed as part of this work.
The Storage Service SWORD helper currently moves a deposit directly into:
watchedDirectories/activeTransfers/standardTransfer/
It then polls /api/transfer/unapproved and calls /api/transfer/approve.
This must be migrated before activeTransfers submission is removed.
The transfer-target reingest endpoint moves content from shared tmp into:
watchedDirectories/activeTransfers/standardTransfer/
and creates a Transfer with type Archivematica AIP.
This is transfer submission and should be included in the same migration.
Add a shared-directory root:
%sharedPath%submittedTransfers/
Physical default:
/var/archivematica/sharedDirectory/submittedTransfers/
MCPServer should create this directory during shared-directory initialization. It should be an internal workflow submission area, not a watched directory.
Submitted transfer paths should include the transfer UUID:
%sharedPath%submittedTransfers/<transfer-name>-<transfer-uuid>/
For archive-file submissions, preserve existing archive behavior but still place
the retrieved source under submittedTransfers. The exact physical name can be
the copied archive filename with collision padding, but the transfer row and API
directory name should remain stable and should include the transfer UUID where
the workflow expects that.
auto_approve should no longer choose between retrieval and watched-directory
copy. It should choose the post-retrieval continuation.
auto_approve=True:
create Transfer row
schedule retrieval to submittedTransfers
after retrieval, continue at accepted transfer-type chain
auto_approve=False:
create Transfer row
schedule retrieval to submittedTransfers
after retrieval, continue at approval decision link
user approves or rejects
The public semantic remains:
- auto-approved transfers begin processing without user input;
- non-auto-approved transfers appear as unapproved only after content is ready;
- retrieval failures fail visibly and do not create an approval prompt for missing content.
The current branch's auto-approved path can skip the old "Move to processing
directory" link because retrieval already places content in currentlyProcessing.
With submittedTransfers, that optimization should stop.
After retrieval, auto-approved transfers should continue at the accepted transfer-type chain's first link. For standard transfers, that means the existing chain still performs:
Move to processing directory
Set transfer type: Standard
...
This preserves the historic workflow shape. submittedTransfers replaces
activeTransfers as the pre-processing location; currentlyProcessing remains
the place for admitted workflow processing.
For zipped transfer types whose accepted chain begins with extraction into
currentlyProcessing, the same principle applies: retrieval materializes the
submitted archive, and the accepted chain performs the established extraction or
move into processing.
For auto_approve=False, retrieval should happen before the approval decision is
shown.
This intentionally preserves today's visible behavior: if a transfer is listed as unapproved, its content has already been copied into Archivematica-controlled shared storage.
Do not create approval prompts for unresolved Storage Service source selections. That alternative is more abstract but changes legacy semantics: users could approve or reject a source reference that might fail to retrieve later. It also complicates the old v1 API's "Copy successful" behavior.
Replace or extend PACKAGE_TYPE_STARTING_POINTS.
Today each transfer type effectively has two important links:
- the watched-directory chain's approval decision link;
- the accepted transfer chain or link used after approval.
Make those explicit.
Suggested shape:
TransferTypeStart = namedtuple(
"TransferTypeStart",
[
"legacy_watched_dir",
"approval_chain",
"approval_link",
"accepted_chain",
"accepted_link",
"legacy_api_type",
],
)Or use a small dataclass if the codebase accepts it in this module.
The mapping should cover:
standard;zipfile;unzipped bag;zipped bag;dspace;maildir;TRIM;dataverse.
legacy_watched_dir can remain temporarily for compatibility and tests, but new
submission should not move content there.
The current helper plan_transfer_source_paths() plans copies into
sharedDirectory/tmp. Keep that temporary copy step.
Add a final destination concept:
submitted_transfer_root = %sharedPath%submittedTransfers/
submitted_transfer_path = submitted_transfer_root + <name>-<uuid>
The retrieval task should:
- copy from the Storage Service transfer source into temporary shared storage;
- move the copied result into
submittedTransfers; - update
Transfer.currentlocationonly after the final submitted path exists; - return success so the workflow can continue to the approval or accepted link.
The current retrieve_transfer_source() helper already performs most of this.
It should accept the final internal directory as an argument. The workflow
context can override %processingDirectory% for the retrieval task, or the
retrieval link can use a new explicit context variable such as
%transferSourceFinalDirectory%.
Prefer an explicit variable over overloading %processingDirectory% if it keeps
the workflow easier to read:
"%transferSourceSubmittedDirectory%"
Keep the current retrieval chain shape:
Retrieve transfer source
Continue after transfer source retrieval
The second link pulls a unit variable, currently
linkAfterTransferSourceRetrieval, and jumps to that link.
Set that variable based on auto_approve:
auto_approve=True -> accepted chain first link
auto_approve=False -> approval decision link
This makes both modes share the retrieval implementation and differ only by workflow continuation.
For both auto-approve modes, set Transfer.status to PROCESSING when the
package is accepted by MCPServer.
For auto_approve=False, this means "Archivematica is handling this submitted
transfer and it may be waiting for user input." This is consistent with existing
behavior, where the transfer is active once the watched-directory chain creates
the approval job.
When the approval decision job is awaiting user input, public status should be
USER_INPUT, as it is today.
If retrieval fails before any approval job exists, mark the transfer failed and show a visible failed retrieval job when possible.
Keep request and response shape:
POST /api/v2beta/package/Response remains:
{"id": "<transfer-uuid>"}Implementation remains:
Dashboard API -> MCPClient.create_package -> MCPServer packageCreate
MCPServer changes:
- always create the transfer row before retrieval;
- always schedule retrieval for transfer-source paths;
- choose continuation by
auto_approve; - do not move into
activeTransfers; - do not require watched-directory detection.
Keep the endpoint and form fields:
POST /api/transfer/start_transfer/Current important semantics:
- accepts
name; - accepts
type; - accepts
accession; - accepts
access_system_id; - accepts one or more base64-encoded
paths[]; - accepts
row_ids[]; - creates one transfer per path;
- suffixes transfer names for paths after the first;
- returns a success response after the copy/submission request has been accepted.
New implementation should route through MCPServer package creation instead of Dashboard-side copy plus watched-directory move.
Suggested flow:
for each selected path:
choose target name:
first path -> name
later paths -> name_2, name_3, ...
call MCPClient.create_package(
name=target_name,
type_=type,
accession=accession,
access_system_id=access_system_id,
path=decoded_path,
metadata_set_id=row_id,
auto_approve=False,
)
return legacy success payload
The response should remain compatible:
{
"message": "Copy successful.",
"path": "<compatibility path>"
}The safest compatibility path is the last submitted transfer's current path once
known. If the response is returned before retrieval has completed, return a path
that represents the submitted transfer request, not an activeTransfers path.
This needs one implementation decision during development:
- keep
start_transfersynchronous enough to return after retrieval has created the submitted path; or - return immediately after MCPServer accepts the request and make
pathcompatibility-only.
Given existing semantics say "Copy successful", preserving synchronous behavior is better if feasible. If it is too slow or brittle, document that the endpoint now means "submission accepted" while keeping the response shape.
Keep response shape:
{
"message": "Fetched unapproved transfers successfully.",
"results": [
{
"type": "standard",
"directory": "my-transfer",
"uuid": "<transfer-uuid>"
}
]
}Stop deriving type and directory by parsing active-transfer paths.
Instead:
- find awaiting decision jobs whose
microservicechainlinkis one of the known transfer approval decision links; - derive
typefrom the approval link or from a unit variable set at submission; - derive
directoryfromTransfer.currentlocation, stripped of the UUID suffix where that matches existing API behavior; - return
uuidfrom the job'ssipuuid.
This preserves the public response while removing the active-transfer path dependency.
Consider setting a submission variable:
submittedTransferType = <legacy api type>
submittedTransferName = <legacy directory/name value>
That avoids fragile reverse inference and avoids misusing Transfer.type, which
workflow scripts currently set later to values such as Standard, Dspace,
TRIM, or Archivematica AIP.
Keep current request compatibility:
POST /api/transfer/approve
directory=<legacy-directory>
type=<legacy-transfer-type>Add UUID-based approval as the preferred internal path:
POST /api/transfer/approve
uuid=<transfer-uuid>Do not require clients to use UUID immediately.
Resolution order:
- If
uuidis provided, approve the awaiting transfer approval job for that transfer. - Otherwise, use
(directory, type)to find a matching awaiting transfer approval job through the compatibility metadata. - As a temporary fallback, support the old exact watched-directory path lookup for jobs that were already created before the upgrade.
The response remains:
{
"message": "Approval successful.",
"uuid": "<transfer-uuid>"
}MCPServer's RPC should evolve from approveTransferByPath toward a selector
that can approve by UUID or by legacy path/directory. The old method name can
remain as a facade to avoid touching every caller at once.
Direct filesystem drops into watchedDirectories/activeTransfers/* should stop
being a supported submission API.
Recommended behavior:
- Remove active-transfer watched directories from the loaded workflow watcher list, or mark them disabled for external starts.
- If an active-transfer watcher must remain temporarily for upgrade cleanup, do
not create new
Transferrows for unknown paths. - Log a clear warning when unknown content is found there.
- Document the removal and point users to API submission.
This proposal intentionally allows breaking direct watched-directory submission. It was never a proper public API and has become a major source of complexity.
SWORD must stop moving deposits into activeTransfers.
Preferred target:
Storage Service -> Archivematica package API
The SWORD helper should submit the deposit through an API path that creates a non-auto-approved transfer. There are two possible implementations:
- Submit through
/api/v2beta/package/if the deposit can be represented as a Storage Service transfer-source path. - Add or reuse an internal transfer submission endpoint that accepts content
already present in shared storage and asks MCPServer to register it under
submittedTransfers.
The first option is cleaner when the deposit lives in a Storage Service location that the pipeline can copy from. The second option may be necessary if SWORD has already materialized the deposit in pipeline-local shared storage and only needs Archivematica to take ownership.
In either case, the old SWORD polling behavior can remain externally similar:
submit deposit
poll /api/transfer/unapproved
approve via /api/transfer/approve
The difference is that the unapproved transfer is backed by an explicit MCPServer submission, not by a watcher discovering a directory.
Transfer-target reingest should stop moving the reingest payload into
activeTransfers/standardTransfer.
New flow:
delete old SIP/job/right/dublin core rows as today
prepare reingest Transfer row with type Archivematica AIP
move payload from tmp into submittedTransfers/<name>-<uuid>
schedule standard transfer approval or accepted workflow directly
The existing endpoint response should remain:
{
"message": "Approval successful.",
"reingest_uuid": "<uuid>"
}This path is slightly different from transfer-source retrieval because the source
is already in the shared directory. It should use a MCPServer helper that starts
from an already materialized local path, moves it to submittedTransfers, and
schedules the appropriate workflow entry directly.
Do not route this through Storage Service copy unless there is a clear benefit.
New submissions should not write to activeTransfers.
Existing in-flight transfers that were already in activeTransfers before the
upgrade are the hard case. Reasonable compatibility handling:
- keep approval-by-old-path fallback for existing waiting jobs;
- keep startup cleanup behavior for in-progress old transfers;
- do not try to migrate arbitrary files left in
activeTransfersautomatically; - document that operators should drain or clear
activeTransfersbefore upgrade where possible.
Using submittedTransfers avoids accidental discovery by an old watched-dir
thread, but it does not make mixed versions safe.
An old MCPServer process could still:
- consume Gearman RPC jobs;
- run startup cleanup;
- schedule old workflow jobs;
- disagree about transfer state.
The release/upgrade note should require stopping old MCPServer processes before
starting the new version. submittedTransfers is a risk reduction, not a full
rolling-upgrade compatibility mechanism.
This change creates a new class of materialized-but-not-admitted transfer content.
Cleanup rules:
- successful approval moves or extracts content from
submittedTransfersintocurrentlyProcessing; - rejection moves content from
submittedTransfersinto the rejected directory using the existing reject workflow; - retrieval failure cleans up temporary retrieval staging and marks the transfer failed;
- startup cleanup marks in-progress transfers failed as today, but should also
handle orphaned
submittedTransfersdirectories; - manual cleanup documentation should mention
submittedTransfers.
There should be a focused follow-up for old temporary retrieval directories if the current branch's TODO remains.
- Add
submittedTransfersto shared-directory creation. - Add settings/constants for the submitted transfer root.
- Refactor transfer type starting-point metadata to include:
- approval decision link;
- accepted chain first link;
- legacy API type;
- optional old watched directory only for fallback.
- Add tests that each supported transfer type has valid workflow links.
- Make
create_package()always schedule retrieval for transfer-source paths. - Change retrieval final destination to
submittedTransfers. - Set
linkAfterTransferSourceRetrievalto:- accepted chain first link for
auto_approve=True; - approval decision link for
auto_approve=False.
- accepted chain first link for
- Persist submission compatibility variables such as
submittedTransferTypeandsubmittedTransferName. - Remove
_start_package_transfer()from new package submission. - Keep old helpers only if tests or old compatibility fallbacks still need them.
- Update
/api/transfer/unapprovedto use approval links and submission metadata instead of active-transfer path parsing. - Update
/api/transfer/approveto approve by UUID when provided. - Keep directory/type approval as compatibility fallback.
- Keep exact old watched-path matching for jobs created before upgrade.
- Add tests for old and new approval lookup paths.
- Change
/api/transfer/start_transfer/to call MCP package creation for each submitted path. - Preserve name suffixing for multiple paths.
- Preserve accession, access system ID, metadata row, transfer type, and response shape.
- Decide during implementation whether the endpoint waits for retrieval or returns after submission acceptance. Prefer preserving "Copy successful" if feasible.
- Change Storage Service SWORD submission to use API/MCPServer transfer
submission instead of direct filesystem move into
activeTransfers. - Preserve its poll-unapproved-and-approve behavior where possible.
- Add integration tests or contract tests around SWORD deposit submission.
- Replace direct move into
activeTransfers/standardTransferwith move intosubmittedTransfers. - Schedule the transfer workflow directly.
- Preserve
Archivematica AIPtransfer type behavior. - Preserve endpoint response shape and existing cleanup behavior.
- Stop watching active-transfer directories for new external starts, or reject
unknown active-transfer paths in
watched_dir_handler. - Keep narrow fallback only for pre-existing jobs during upgrade.
- Update tests that currently assert unknown active-transfer drops create transfers.
- Add release notes explaining that direct active-transfer filesystem submission is no longer supported.
auto_approve=Trueschedules retrieval tosubmittedTransfers.auto_approve=Truecontinues at the accepted chain's first link.auto_approve=Falseschedules retrieval tosubmittedTransfers.auto_approve=Falsecontinues at the approval decision link.- Retrieval failure marks the transfer failed before approval exists.
- Retrieval failure does not leave an approval decision job.
- All supported transfer types have valid approval and accepted links.
- Auto-approved standard transfer retrieves, moves into processing, and continues through normal workflow links.
- Non-auto-approved standard transfer retrieves, creates an approval job, and stops for user input.
- Approving the non-auto-approved transfer continues into the accepted workflow.
- Rejecting the non-auto-approved transfer runs reject cleanup from
submittedTransfers. - Zipped, bag, DSpace, maildir, TRIM, and Dataverse transfer types have at least focused continuation tests or parametrized link tests.
/api/v2beta/package/preservesauto_approve=False./api/transfer/start_transfer/decodes paths and submits one package per path.- Multiple
paths[]preserve legacy name suffixing. /api/transfer/unapprovedreturns the same JSON shape without active-transfer paths./api/transfer/approveworks by UUID./api/transfer/approveworks by legacydirectoryandtype./api/transfer/approvestill handles pre-upgrade watched-path jobs.- Status endpoints return
PROCESSING,USER_INPUT, andFAILEDconsistently during retrieval, approval wait, and retrieval failure.
- SWORD no longer moves deposits into
activeTransfers. - SWORD-created transfers appear in
/api/transfer/unapproved. - SWORD approval continues the transfer.
- Transfer-target reingest moves into
submittedTransfers. - It creates a
Transferwith typeArchivematica AIP. - It schedules or reaches the expected approval/processing workflow.
- It preserves endpoint response shape.
- Existing duplicate-name behavior is replaced with equivalent collision
detection under
submittedTransfers.
- Unknown direct drops into
activeTransfersno longer create new transfers. - The rejection or warning behavior is explicit and logged.
- Old pre-existing approval jobs can still be approved through compatibility fallback.
The hardest compatibility risk is not request parsing; it is subtle client
expectations around directory and path fields.
Clients may:
- pass
directoryfrom/api/transfer/unapprovedinto/api/transfer/approve; - pass
directoryfrom transfer status into/api/transfer/approve; - compare
start_transfer's returnedpathto filesystem locations; - assume the unapproved directory does not include a UUID suffix.
The compatibility layer must preserve these practical behaviors even though the filesystem backend changes.
Archive transfer behavior is historically different from directory behavior: archive selections copy a file, while directory selections copy contents. The new submitted path logic must preserve these semantics.
Avoid ad hoc path parsing. Reuse and extend
archivematicaCommon.transfer_source_retrieval so package API and legacy API
paths stay consistent.
Do not overload Transfer.type to mean the submitted transfer type.
Workflow scripts set Transfer.type later to values such as Standard,
Dspace, TRIM, or Archivematica AIP. The API uses legacy type values such
as standard, zipfile, and zipped bag.
Use explicit submission metadata or link mappings for API compatibility.
The current branch already tightens failure handling for retrieval before normal jobs exist. Moving all submission through retrieval increases reliance on that startup cleanup path.
The proposal should be implemented with tests for:
- queued retrieval before first job row exists;
- executing retrieval with a task row;
- submitted-but-unapproved transfer after retrieval succeeds;
- orphaned submitted content after MCPServer shutdown.
Removing direct filesystem starts can break users who built private automation around watched directories.
That break is acceptable for this proposal because activeTransfers was never a
proper public API. It still needs release notes and an upgrade path: use
/api/v2beta/package/ or /api/transfer/start_transfer/.
This change removes activeTransfers as a submission surface, not every watched
directory from the workflow engine.
Some workflow links still move content into watched directories for later processing. Removing those is a broader workflow-runtime project.
The new submittedTransfers root reduces accidental pickup by old watcher
threads, but it does not allow safe mixed old/new MCPServer deployments.
Document that old MCPServer processes must be stopped before upgrade.
Release notes should say:
- Transfer submission no longer uses
watchedDirectories/activeTransfers. - Direct filesystem submission into
activeTransfersis no longer supported. - Public transfer submission APIs remain supported.
- Integrations should use
/api/v2beta/package/or/api/transfer/start_transfer/. - Operators may see a new
submittedTransfersdirectory in the shared directory. - Operators should not place content manually in
submittedTransfers. - Before upgrade, drain or clean old
activeTransferscontent where possible.
Developer documentation should explain:
submittedTransfersis pre-processing submission storage;currentlyProcessingremains admitted workflow processing storage;- approval jobs no longer imply an
activeTransferspath; - approval lookup should use UUID where possible.
- No new API-driven transfer submission writes into
watchedDirectories/activeTransfers/*. auto_approve=Truepackage creation still processes transfers automatically.auto_approve=Falsepackage creation retrieves content and waits for approval.- Legacy
/api/transfer/start_transfer/still works for existing clients. /api/transfer/unapprovedreturns the same public shape as before./api/transfer/approvestill works with legacydirectoryandtype.- UUID-based approval is available for newer callers.
- Storage Service SWORD no longer writes deposits into
activeTransfers. - Transfer-target reingest no longer writes into
activeTransfers. - Unknown direct drops into
activeTransfersno longer create transfers. - Tests cover success, approval, rejection, retrieval failure, and old fallback approval paths.