Archivematica Issue 1803 - Workflow-native transfer-source retrieval plan

issue-1803-workflow-native-transfer-source-retrieval-plan.md

Issue 1803 workflow-native transfer-source retrieval plan

Issue: archivematica/Issues#1803

Problem

Transfer-source retrieval is currently outside the normal transfer workflow in the MCPServer package-create path. MCPServer creates a Transfer row, copies the selected content from Storage Service into the shared directory, moves the content into the processing area, and only then schedules the first normal workflow job.

That leaves a visibility gap. If Storage Service, rsync, or shared-storage I/O blocks during retrieval, operators may not see a normal workflow job that explains what the transfer is doing.

The accepted direction is to move retrieval into workflow.json, not to add a synthetic pre-workflow job.

Goals And Non-Goals

Goals

The first implementation should make transfer-source retrieval visible as a normal workflow job for the main package-create path.

Concretely, it should:

• move the Storage Service retrieval step out of MCPServer pre-workflow startup;
• represent retrieval with a real workflow.json link;
• execute retrieval as a normal MCPClient task;
• keep the existing package API request and response shape;
• keep legacy status strings compatible;
• preserve watched-directory transfer behavior;
• fail visibly when Storage Service copy or local filesystem work fails.

Non-Goals For The First Implementation

The first implementation should not try to redesign every transfer-entry path.

Specifically, it should not:

• add a dedicated retrieval scheduler or concurrency cap;
• persist queued retrieval jobs before they start running;
• add durable retrieval resume state;
• change direct watched-directory starts;
• change transfer-target reingest;
• move the legacy /api/transfer/start_transfer/ copy path;
• make auto_approve=False package creation workflow-native;
• depend on Storage Service async endpoints.

These are valid follow-ups, but bundling them into the first change would make the first PR much harder to review and test.

Design Decisions

Retrieval is a normal workflow link

Add a real transfer-source retrieval link to workflow.json.

The retrieval step should appear as a normal transfer job once it starts executing. It should have a real microservicechainlink, use the existing job and task tables, and be visible through the same dashboard/API paths as other workflow jobs.

Retrieval runs on MCPClient

The retrieval link should execute as a normal MCPClient task, not as a new MCPServer-local workflow job.

This matches the existing workflow model more closely and avoids introducing a new MCPServer job execution type. MCPClient already has Storage Service settings and several client scripts call archivematicaCommon.storageService, so this is an incremental extension of an existing pattern.

Use the existing PackageQueue

Do not introduce a dedicated retrieval scheduler, executor, or concurrency cap in the first implementation.

Retrieval should use the existing PackageQueue and therefore the existing CONCURRENT_PACKAGES and WORKER_THREADS controls. Long retrievals may occupy normal workflow capacity. That is an accepted first-pass tradeoff; if load tests show fairness or throughput problems, a retrieval-specific cap can be proposed later.

Do not solve queued visibility yet

The first implementation does not need to show "retrieval queued" before the job starts running.

Current MCPServer job rows are generally created when job.run() starts. If a retrieval job is waiting behind CONCURRENT_PACKAGES, it may still be invisible in the job table and dashboard until it is released to the active job queue. Full queued visibility requires a more general PackageQueue/job persistence change and should remain a later enhancement.

Keep bootstrap data in memory

Do not add a durable retrieval bootstrap model in the first implementation.

MCPServer can build the first workflow job with in-memory context containing the transfer-source path, temporary path, destination path, transfer type, and selected post-retrieval workflow link.

If MCPServer restarts while retrieval is queued or running, the first implementation does not attempt to resume the retrieval. Existing startup cleanup should fail executing jobs and in-progress transfers. This needs to be verified and tightened where necessary.

Preserve external status semantics

Do not add new top-level API statuses such as QUEUED, RETRIEVING, or WAITING_FOR_WORKFLOW.

Legacy clients expect transfer status to remain:

retrieval queued/running/done but workflow still processing -> PROCESSING
operator decision point                                    -> USER_INPUT
transfer handed off to SIP/backlog                         -> COMPLETE
retrieval or workflow failure                              -> FAILED

First-Pass Scope

The first pass should target the MCPServer package-create path used by /api/v2beta/package:

Dashboard API
  -> MCPClient.create_package()
  -> MCPServer packageCreate RPC
  -> create_package()
  -> PackageQueue
  -> workflow retrieval job
  -> MCPClient retrieval script
  -> existing type-specific transfer workflow

It should not attempt to change every transfer-start mechanism in one batch.

That boundary is important because Archivematica has several transfer-entry mechanisms with different contracts:

• /api/v2beta/package asks MCPServer to create a package from a Storage Service transfer source;
• /api/transfer/start_transfer/ copies from transfer sources in the Dashboard process and then uses watched directories;
• direct watched-directory drops already have content on disk;
• /api/transfer/approve approves content already in activeTransfers;
• reingest routes packages through shared-directory and watched-directory paths.

The first implementation should fix the main MCPServer package-create path without changing the others by accident.

Impact By Area

MCPServer

create_package() changes from "copy before workflow" to "schedule retrieval as the first workflow job".

Current auto-approve path:

create Transfer row
create tmpdir
copy from Storage Service in MCPServer
move to currentlyProcessing
construct Transfer package
schedule first existing workflow link

Proposed auto-approve path:

create Transfer row
mark Transfer processing
create retrieval bootstrap context in memory
construct Transfer package
schedule API-only retrieval chain/link
retrieval task copies/moves content
retrieval task updates Transfer.currentlocation
next workflow link is the existing type-specific start link

The important MCPServer work is:

• stop doing the Storage Service copy inside create_package() for the targeted auto-approve path;
• construct a Transfer package before content has been materialized;
• pass in-memory retrieval context into the retrieval job;
• preserve accession, access system ID, metadata-set row, processing configuration, and active agent setup;
• keep Transfer.status in PROCESSING while retrieval/workflow is active;
• ensure startup cleanup fails in-progress transfers if MCPServer restarts before retrieval has completed.

The existing PackageQueue remains the scheduler. No new retrieval pool is introduced in the first pass.

MCPClient

Add a new retrieval client script.

The script should:

• parse the source location_uuid:path or default transfer-source path;
• use the existing Storage Service API client settings;
• request a copy from the transfer source location to the currently processing location;
• preserve current directory/file/archive behavior;
• move or materialize the transfer into the final processing path;
• update Transfer.currentlocation after the content is in place;
• return non-zero on Storage Service or filesystem failures;
• include useful error text in task output.

The script should call the synchronous Storage Service copy endpoint first. The workflow job itself is already the asynchronous boundary from the operator's point of view.

Operationally, this means MCPClient workers need:

• network access to Storage Service;
• Storage Service credentials/settings;
• access to the shared directory;
• enough task timeout/headroom for long retrievals.

Those are already common MCPClient assumptions for storage and indexing scripts, but the release notes should call out that transfer-source retrieval now runs from MCPClient rather than MCPServer.

Workflow Data

Add an API-only retrieval entry path.

The safer shape is a retrieval chain used only by MCPServer package-create requests from transfer sources. Watched-directory starts should continue to use their existing watched-directory chains because their content is already present on disk.

The retrieval link should jump to the existing type-specific starting link after success. The existing PACKAGE_TYPE_STARTING_POINTS mapping can remain the source of truth for the post-retrieval chain/link target.

The workflow must preserve transfer type behavior for:

• standard;
• zipfile;
• unzipped bag;
• zipped bag;
• dspace;
• maildir;
• TRIM;
• dataverse.

Dashboard Monitor And RPC

Once the retrieval job starts, it will have a normal Jobs row and should show up naturally in:

• MCPServer getUnitsStatuses;
• MCPServer getUnitStatus;
• dashboard monitor rows;
• job detail views.

Known first-pass limitation: if retrieval is waiting behind CONCURRENT_PACKAGES, no job row may exist yet. In that window the dashboard may still not show the retrieval job. That is the deferred queued-visibility work.

This needs explicit implementation testing. Creating the Transfer row is not necessarily enough for dashboard/API visibility because the monitor and status paths are largely job-driven. getUnitsStatuses currently builds its transfer list from Jobs joined to Transfers, and the legacy transfer status endpoint fetches the Transfer row but then expects get_unit_status() to find at least one job. During development, test the state where many transfers have been submitted and some are waiting behind CONCURRENT_PACKAGES before their retrieval job has started. If those transfers are invisible or status polling is fragile, add a follow-up change before considering #1803 fully addressed.

Legacy V1 API

/api/transfer/status/<uuid>/ should keep the same response shape and status strings.

When the retrieval job is running, the status should be PROCESSING and the current microservice should be the retrieval job. When retrieval succeeds, later workflow jobs take over as usual. Retrieval completion alone must not return COMPLETE.

Known first-pass limitation: the legacy status endpoint currently expects at least one Jobs row. If a transfer exists but its retrieval job is still queued and no job row has been created, this endpoint can still fail to compute status. That is part of the deferred queued-visibility/status-hardening work, but it must be tested before release because customers may poll the transfer UUID immediately after package creation.

/api/transfer/start_transfer/ is different. It currently copies from transfer sources in the Dashboard process, moves the result into activeTransfers, and then relies on the watched-directory machinery. The first #1803 workflow-native MCPServer change should not silently change this older API. If customers still use it with large transfer-source copies, it remains a separate follow-up.

/api/transfer/approve is also unchanged. It approves a transfer that already exists in activeTransfers, so it should continue to enter the existing approval workflow.

V2beta API

/api/v2beta/package is the primary target.

The request and response shape should remain unchanged:

POST /api/v2beta/package
-> 202 {"id": "<transfer uuid>"}

The difference is internal:

• MCPServer returns after scheduling workflow retrieval, not after doing a pre-workflow copy;
• the retrieval job appears in /api/v2beta/jobs/<uuid> once it starts;
• /api/v2beta/task/<task uuid> can expose the MCPClient retrieval task like other client-script tasks.

No new status strings or response fields are required in the first pass.

Storage Service Async Endpoints

Do not use Storage Service async endpoints in the first implementation.

Using /api/v2/location/<uuid>/async/ would create nested async behavior:

Archivematica workflow job
  -> MCPClient task
  -> Storage Service async task
  -> polling

That would bring back the Storage Service AsyncManager/gevent failure modes without solving #1803's main visibility problem. The workflow job is the operator-visible async boundary. The Storage Service request can remain synchronous and bounded by existing client timeouts and rsync behavior.

Future Storage Service work should focus on making copy operations more reliable and better bounded, not on depending more heavily on the current in-process async endpoints.

Watched Directories

Direct watched-directory starts should not use the new retrieval link.

These starts are already content-present:

operator/process drops content into watchedDirectories/activeTransfers/...
watcher sees path
MCPServer creates or loads Transfer by path
existing watched-directory chain starts

Adding retrieval to those chains would be wrong because there is no Storage Service transfer-source copy to perform. It would also risk changing long-lived drop-folder workflows and local integrations.

The new retrieval link should therefore be an API-created transfer-source entry path, not a universal first transfer link.

Non-Auto-Approve Starts

auto_approve=False should remain out of scope for the first implementation.

The reason is that it does not continue directly into the accepted-transfer workflow. Today it copies the selected transfer-source contents and moves them into the appropriate activeTransfers watched directory. After that, the watched-directory observer starts the existing approval workflow and the operator can approve or reject the transfer.

So auto_approve=True is a straight workflow continuation:

retrieval workflow job
  -> copy from transfer source
  -> move into currentlyProcessing
  -> continue to existing accepted-transfer workflow

auto_approve=False is a workflow-to-filesystem-to-workflow handoff:

retrieval workflow job
  -> copy from transfer source
  -> move into activeTransfers
  -> end retrieval chain
  -> watched-directory approval workflow starts

That is doable, but it introduces different failure windows:

• retrieval can complete before the watcher creates the approval job;
• a restart can happen between moving into activeTransfers and watcher detection;
• the dashboard may briefly show only a completed retrieval job;
• duplicate Transfer row/path handling needs care because the Transfer row may already exist before the watcher sees the path.

For the first pass, leave auto_approve=False on the old path and document that #1803 initially targets auto-approved package creation. A later PR can make non-auto-approve retrieval workflow-native once the workflow/watched-directory handoff behavior is designed and tested.

Reingest

Transfer-target reingest uses the shared-directory and watched-directory paths. It should remain unchanged in the first pass.

This work is about Storage Service transfer-source retrieval for new package creation, not AIP reingest routing.

Important Design Details To Lock During Implementation

Path semantics before retrieval completes

Workflow normally starts after the transfer content exists in the processing directory. This change starts workflow before that is true.

The retrieval job must therefore be the only job that can run before content is materialized. The next link must not be scheduled until retrieval has:

1. copied from Storage Service successfully;
2. moved/materialized the package in the processing area;
3. updated Transfer.currentlocation;
4. exited successfully.

Later jobs should keep seeing the normal meanings of:

%SIPDirectory%
%transferDirectory%
%relativeLocation%

The implementation needs a small path-planning component that preserves current behavior for directories, archive files, and transfer-name collision handling.

Error handling

storageService.copy_files() currently returns (None, error) on request failures. The retrieval script must convert that into a failed task, not a successful task with an error message.

This matters because Storage Service rsync timeout fixes should surface as workflow failures that operators can see.

Restart behavior

First pass does not resume queued or running retrieval after MCPServer restart.

Expected behavior:

• executing retrieval job rows are marked failed by startup cleanup;
• processing transfers with no recoverable workflow state are marked failed;
• temporary retrieval directories are best-effort cleaned where safe;
• operators get an explicit failed transfer rather than an indefinitely processing transfer.

This is containment, not durable orchestration.

Metrics

Avoid a large metrics design in the first pass.

Existing MCPServer package/job/task gauges and common Storage Service API timing metrics should already show more than today once retrieval is a workflow task. If needed, add retrieval-specific metrics later after the execution model has settled.

Proposed PR Sequence

PR 1: Extract transfer-source retrieval planning

Goal: Create tested code that computes source, temporary, and final retrieval paths without changing behavior.

Scope:

• extract or centralize duplicated transfer-source path logic;
• preserve default transfer-source location behavior from MCPServer;
• preserve directory vs archive handling;
• preserve UTF-8 path handling;
• unit-test path planning and Storage Service copy request payload generation.

This PR should be behavior-preserving.

PR 2: Add the MCPClient retrieval script

Goal: Make retrieval executable as a normal MCPClient task.

Scope:

• add a client script for transfer-source retrieval;
• call the synchronous Storage Service copy endpoint;
• move/materialize content into the processing area;
• update Transfer.currentlocation;
• fail clearly on Storage Service or filesystem errors;
• unit-test success and failure behavior with mocked Storage Service calls.

This PR can be tested by invoking the script directly before workflow wiring is complete.

PR 3: Add workflow retrieval link and script registration

Goal: Represent retrieval in workflow.json.

Scope:

• add the retrieval link and chain data;
• make the link execute the new MCPClient script;
• define success and failure routing;
• keep watched-directory chains unchanged;
• add workflow validation tests.

The link should be API-entry only, not inserted into every watched-directory transfer chain.

PR 4: Schedule retrieval from packageCreate

Goal: Switch the targeted MCPServer package-create path from pre-workflow copy to workflow-native retrieval.

Scope:

• update create_package() for the auto-approved transfer-source path;
• create/mark the Transfer as processing before scheduling;
• build a Transfer package before content exists;
• pass retrieval bootstrap values through in-memory job context;
• schedule the retrieval chain/link in PackageQueue;
• continue to the existing type-specific starting link after retrieval.

This is the main behavior-changing PR.

PR 5: Harden status and failure behavior

Goal: Ensure API and dashboard behavior is stable around retrieval failure and restart.

Scope:

• verify /api/transfer/status/<uuid>/ reports PROCESSING while retrieval is running;
• verify retrieval failure becomes visible as FAILED;
• verify retrieval success alone does not produce COMPLETE;
• verify submitted transfers waiting behind CONCURRENT_PACKAGES are either visible enough for supported API/dashboard workflows, or explicitly identify the missing queued-visibility work that must be completed before release;
• verify /api/transfer/status/<uuid>/ behaves predictably when called immediately after package creation and before the retrieval job has started;
• verify startup cleanup fails interrupted retrievals/transfers;
• add regression tests around Storage Service exceptions.

This PR may be merged with PR 4 if the patch is still small, but it should be reviewable as its own acceptance boundary.

PR 6: Documentation and operational guidance

Goal: Document the changed execution location and tuning knobs.

Scope:

• explain that transfer-source retrieval for the package API now runs as an MCPClient workflow task;
• call out MCPClient Storage Service connectivity and credentials;
• document relevant knobs:
  • ARCHIVEMATICA_MCPSERVER_MCPSERVER_CONCURRENT_PACKAGES;
  • ARCHIVEMATICA_MCPSERVER_MCPSERVER_WORKER_THREADS;
  • ARCHIVEMATICA_MCPCLIENT_MCPCLIENT_STORAGE_SERVICE_CLIENT_TIMEOUT;
  • ARCHIVEMATICA_MCPCLIENT_MCPCLIENT_STORAGE_SERVICE_CLIENT_QUICK_TIMEOUT;
  • Storage Service rsync timeout settings once available;
• note that queued retrieval visibility is not included yet.

Deferred Follow-Ups

Queued retrieval visibility

Persist selected workflow jobs when they are scheduled, not only when they start executing, so the dashboard can show retrieval waiting behind CONCURRENT_PACKAGES.

This should be a general PackageQueue/job visibility improvement, not a special retrieval-only hack.

Durable retrieval bootstrap state

Persist the retrieval bootstrap state so queued/running retrieval can be recovered or explicitly reconciled after restart.

This is the path toward true restart resilience.

Non-auto-approve workflow-native retrieval

Decide whether auto_approve=False package creation should also use workflow retrieval before handing off to watched-directory approval.

This needs its own design because it deliberately crosses from workflow execution back into watched-directory detection.

Legacy start-transfer API

Decide whether /api/transfer/start_transfer/ should be migrated to the package-create workflow-native path, deprecated, or given a separate visibility fix.

It currently performs transfer-source copy work in the Dashboard process and is not fixed by the first #1803 implementation.

Retrieval-specific back pressure

If load testing shows that long retrievals unfairly occupy normal workflow capacity, add a retrieval-specific concurrency cap later.

This should be driven by observed production behavior after the workflow-native implementation exists.

Storage Service async/background jobs

Do not build the first #1803 implementation around the current Storage Service async endpoints.

A longer-term Storage Service direction could replace in-process async/gevent work with durable background jobs, but that is larger than this MCPServer workflow visibility change.

Acceptance Criteria For The First Implementation

• Starting an auto-approved transfer through /api/v2beta/package no longer performs the transfer-source copy in MCPServer before workflow starts.
• Once retrieval starts, operators see a normal workflow job for transfer-source retrieval.
• Retrieval is represented by a real workflow link and microservicechainlink.
• Retrieval runs as an MCPClient task.
• Retrieval success updates Transfer.currentlocation before later transfer workflow jobs run.
• Retrieval failure marks the workflow job failed and results in a failed transfer, not an indefinitely processing transfer.
• Legacy status strings remain compatible.
• Watched-directory direct starts are unchanged.
• Storage Service async endpoints are not required.