Transfer Service Schemes for 4.0.0

TransferServices-4.0.0.md

PacBio Transfer Schemes Notes for 4.0.0

Supported Transfer Schemes:

• Rsync Server (aka "rsync")
• Rsync+SSH (aka "srs")

The new 4.0.0 data model (specifically relativePath) allows users to create Transfer Schemes for grouping of projects to manage under one root location. For example, user or group "alpha" can create Transfer Scheme "rsync-alpha" that writes to "/pbi/collections/groups/alpha". This Transfer Scheme can be completely separately managed from group "beta" that writes to "/pbi/collections/groups/beta".

Rsync Server

Transfer Scheme Config

• "id" the root key will be the transfer scheme id. This must only contain A-z, 0-9 and "- or "_" (example “pbi-collections-test”)
• "name" Display name of the transfer scheme
• "description" Description of the transfer scheme
• "rsyncModule" Rsync module name (example: “rsyncModuleName”)
• "destPath" Rsync Module absolute path of the rsyncModule (example: "/pbi/collections" for rsyncModule "pbi_collections")
• "relativePath": Relative path to root rsync module dir. Must not contain a leading slash (Example: "1234/my-runs" or null if there is no relative path). If the relative path is not null, the path must exist.
• "user" rsync user (example: "my-user")
• "password" rsync server password to rsync server (example: "my-password")
• "host" Host (example: "mp-rsync", see below for known issue)
• "port" Port of rsync server (example: 873, see belown for known issues)

Known Issues

• port is not used, to use a non-standard port use ":" when defining the host to set a custom port. Example, "my-host:877". The "port" key will be ignored.

Rsync Transfer Service Data Model and Service Endpoint Example

Get Rsync transfer resource by id "alpha" <HOST>:<PORT>:smrt-transfer/schemes/rsync/alpha

Example:

http get http://<HOST>:8090/smrt-transfer/schemes/rsync/rsync-pbi-collections

From Transfer Services:

{
    "description": "Example Testing for Poca/pocb", 
    "destPath": "/pbi/collections", 
    "host": "mp-rsync", 
    "id": "rsync-pbi-collections", 
    "name": "PBI Collections Transfer test using Rsync Server", 
    "password": "my-password", 
    "port": 873, 
    "relativePath": null, 
    "rsyncModule": "pbi_collections", 
    "user": "oixfr"
}

ICS + PA Resolving of SubreadSet Path

ICS will set add TransferResource to the OutputOptions in the SubreadSet XML to communicate the custom subdirectories to write the SubreadSet XML and other Primary Analysis output to.

Example from SubreadSet XML:

<pbmeta:OutputOptions>
    <pbmeta:ResultsFolder>3120306/r54054_20161222_154928/4_A01/</pbmeta:ResultsFolder>
    <pbmeta:CopyFiles>
        <pbmeta:CollectionFileCopy>Trace</pbmeta:CollectionFileCopy>
        <pbmeta:CollectionFileCopy>Fasta</pbmeta:CollectionFileCopy>
        <pbmeta:CollectionFileCopy>Bam</pbmeta:CollectionFileCopy>
        <pbmeta:CollectionFileCopy>Baz</pbmeta:CollectionFileCopy>
        <pbmeta:CollectionFileCopy>StatsH5</pbmeta:CollectionFileCopy>
    </pbmeta:CopyFiles>
    <pbmeta:Readout>Pulses</pbmeta:Readout>
    <pbmeta:MetricsVerbosity>High</pbmeta:MetricsVerbosity>
    <pbmeta:TransferResource>
        <pbmeta:Id>rsync-pbi-collections</pbmeta:Id>
        <pbmeta:TransferScheme>RSYNC</pbmeta:TransferScheme>
        <pbmeta:Name>PBI Collections Transfer test using Rsync Server</pbmeta:Name>
        <pbmeta:Description>Example Rsync Server Transfer Scheme</pbmeta:Description>
        <pbmeta:DestPath>/pbi/collections</pbmeta:DestPath>
        <pbmeta:RelativePath>312</pbmeta:RelativePath>
    </pbmeta:TransferResource>
</pbmeta:OutputOptions>

The ResultsFolder will be written as subdirectory relative to the relative path defined in TransferResource

Specifically,

 <pbmeta:ResultsFolder>3120306/r54054_20161222_154928/4_A01/</pbmeta:ResultsFolder>

From the rsync-pbi-collections Rsync transfer scheme, the resolved SubreadSet XML path will be resolved to

/pbi/collections/312/3120306/r54054_20161222_154928/4_A01/<movie-context-id>.subreadset.xml

This path must be accessible from the SMRT Link instance to import the SubreadSet file into SMRT Link from ICS.

Rsync+SSH

Transfer Scheme Config

• "id" the root key will be the transfer scheme id. This must only contain A-z, 0-9 and "- or "_" (example “pbi-collections-test”)
• "name" Display name of the transfer scheme
• "description" Description of the transfer scheme
• "destPath" Root directory for output (example: "/pbi/collections", or "/pbi/collections/312")
• "relativePath": Relative path to the destPath (Example: "1234/my-runs" or null if there is no relative path). If the relative path is not null, the path must exist. see below of comments)
• "user" Rsync SSH user (example: "my-user")
• "sshKey" Path to SSH Key used via ssh -i (example: “/path/to/key”)
• "host" Host (example: "mp-rsync")
• "port" Port of rsync server (example: 22)

Note, there are two possible ways to configure the output. It's recommended that setting the "relativePath" to null and only use destPath

{
    destPath:"/alpha/beta/gamma",
    relativePath: null

}

Or

{
    destPath:"/alpha/beta",
    relativePath: "gamma"

}

Rsync+SSH Transfer Service Data Model and Service Endpoint Example

Get Rsync transfer resource by id "alpha" <HOST>:<PORT>/smrt-transfer/schemes/srs/alpha

Example:

http get http://<HOST>:8090/smrt-transfer/schemes/srs/srs-pbi-collections-test

From Transfer Services:

{
    "description": "Example Rsync+SSH for nightly tests", 
    "destPath": "/pbi/collections/xfer-test", 
    "host": "mp-rsync", 
    "id": "srs-pbi-collections-test", 
    "name": "PBI Collections Xfer test using Rsync+SSH", 
    "port": 22, 
    "relativePath": null, 
    "sshKey": "/home/pbi/.ssh/id_rsa", 
    "user": "limswriter"
}

ICS + PA Resolving of SubreadSet Path

This follows the same model as the Rsync Server. See above for details.