DRS batch request PoC using GA4GH starter kit

Intro

This gist provides a working, proof-of-concept example of DRS object batch requests. This prototype feature of the DRS specification was developed in the GA4GH starter kit DRS implementation.

The goal is to have a working example for this prototype feature that can be tried out, commented on, and refined as the group works towards a specification for DRS batch requests. Refinements will be incorporated back into the starter kit implementation until a standardized approach is reached.

Setup

The following steps will allow you to perform DRS batch requests on your local machine using the starter kit. The setup only requires docker to be installed locally, and for ports 4500 and 4501 to be free.

Pull the docker image:

docker pull ga4gh/ga4gh-starter-kit-drs:2021-06-14-SNAPSHOT

Run the container:

docker run --rm --name drs-batch-request-test -p 4500:4500 -p 4501:4501 ga4gh/ga4gh-starter-kit-drs:2021-06-14-SNAPSHOT

The DRS service should now be running at http://localhost:4500. In its default configuration, the starter kit DRS service serves a small test set DrsObjects.

Using the GET endpoint for single objects

To start, let's request information about a single DrsObject using the standard GET /objects/{object_id} endpoint. Issue the following GET request: http://localhost:4500/ga4gh/drs/v1/objects/b8cd0667-2c33-4c9f-967b-161b905932c9

The server should respond with the following JSON payload:

{
    "id": "b8cd0667-2c33-4c9f-967b-161b905932c9",
    "description": "Open dataset of 384 phenopackets",
    "created_time": "2021-03-12T20:00:00Z",
    "name": "phenopackets.test.dataset",
    "size": 143601,
    "updated_time": "2021-03-13T12:30:45Z",
    "version": "1.0.0",
    "self_uri": "drs://localhost:4500/b8cd0667-2c33-4c9f-967b-161b905932c9",
    "contents": [
        {
            "name": "phenopackets.mundhofir.family",
            "drs_uri": [
                "drs://localhost:4500/1af5cdcf-898c-4dbc-944e-1ac95e82c0ea"
            ],
            "id": "1af5cdcf-898c-4dbc-944e-1ac95e82c0ea"
        },
        {
            "name": "phenopackets.zhang.family",
            "drs_uri": [
                "drs://localhost:4500/355a74bd-6571-4d4a-8602-a9989936717f"
            ],
            "id": "355a74bd-6571-4d4a-8602-a9989936717f"
        },
        {
            "name": "phenopackets.cao.family",
            "drs_uri": [
                "drs://localhost:4500/a1dd4ae2-8d26-43b0-a199-342b64c7dff6"
            ],
            "id": "a1dd4ae2-8d26-43b0-a199-342b64c7dff6"
        },
        {
            "name": "phenopackets.lalani.family",
            "drs_uri": [
                "drs://localhost:4500/c69a3d6c-4a28-4b7c-b215-0782f8d62429"
            ],
            "id": "c69a3d6c-4a28-4b7c-b215-0782f8d62429"
        }
    ]
}

Using the POST endpoint for multiple objects (batch request)

Now, we will issue the prototype request for retrieving multiple DrsObjects in a single request. Currently, this is a POST request to the /objects endpoint.

Construct and issue the following POST request:

URL: http://localhost:4500/ga4gh/drs/v1/objects
Headers:
- Content-type: application/json
Body:

{
    "selection": [
        "b8cd0667-2c33-4c9f-967b-161b905932c9",
        "00000000-0000-0000-0000-000000000000"
    ]
}

The server should respond with the following payload:

{
    "summary": {
        "requested": 2,
        "loaded": 1,
        "unloaded": 1
    },
    "drs_objects": {
        "b8cd0667-2c33-4c9f-967b-161b905932c9": {
            "id": "b8cd0667-2c33-4c9f-967b-161b905932c9",
            "description": "Open dataset of 384 phenopackets",
            "created_time": "2021-03-12T20:00:00Z",
            "name": "phenopackets.test.dataset",
            "size": 143601,
            "updated_time": "2021-03-13T12:30:45Z",
            "version": "1.0.0",
            "self_uri": "drs://localhost:4500/b8cd0667-2c33-4c9f-967b-161b905932c9",
            "contents": [
                {
                    "name": "phenopackets.mundhofir.family",
                    "drs_uri": [
                        "drs://localhost:4500/1af5cdcf-898c-4dbc-944e-1ac95e82c0ea"
                    ],
                    "id": "1af5cdcf-898c-4dbc-944e-1ac95e82c0ea"
                },
                {
                    "name": "phenopackets.zhang.family",
                    "drs_uri": [
                        "drs://localhost:4500/355a74bd-6571-4d4a-8602-a9989936717f"
                    ],
                    "id": "355a74bd-6571-4d4a-8602-a9989936717f"
                },
                {
                    "name": "phenopackets.cao.family",
                    "drs_uri": [
                        "drs://localhost:4500/a1dd4ae2-8d26-43b0-a199-342b64c7dff6"
                    ],
                    "id": "a1dd4ae2-8d26-43b0-a199-342b64c7dff6"
                },
                {
                    "name": "phenopackets.lalani.family",
                    "drs_uri": [
                        "drs://localhost:4500/c69a3d6c-4a28-4b7c-b215-0782f8d62429"
                    ],
                    "id": "c69a3d6c-4a28-4b7c-b215-0782f8d62429"
                }
            ]
        }
    },
    "unloaded_drs_objects": {
        "00000000-0000-0000-0000-000000000000": 404
    }
}

Explanation of the batch request endpoint

The above POST request example makes a request for multiple DrsObjects using a provided selection in the request body. The selection is simply an array of multiple requested DRS IDs. In the example, 2 ids were provided. The first id maps to a valid DrsObject in the database, the second id is invalid, in that no DrsObject with the id exists.

The response body is a complex object with 3 main attributes: summary, drs_objects, and unloaded_drs_objects:

summary simply indicates how many objects were requested in the selection, how many of those were successfully returned, and how many were not returned.
drs_objects is a JSON object/dictionary in which the key is the requested DRS id, and the value is the DrsObject for that id. Each successfully loaded DrsObject is in this dictionary.
unloaded_drs_objects is a JSON object/dictionary for requested objects that could not be loaded. The key is the requested DRS id, and the value is a simple explanation of why a DrsObject could not be loaded (in this case, the explanation is an HTTP code, indicating the requested entity was not found). Each requested DRS id for which a DrsObject was not loaded is in this dictionary.

Batch request endpoint: additional examples

Any combination of valid and invalid DRS ids can be submitted to the batch request endpoint

Additional example 1:

URL: http://localhost:4500/ga4gh/drs/v1/objects
Headers:
- Content-type: application/json
Body:

{
    "selection": [
        "1a570e4e-2489-4218-9333-f65549495872",
        "4d83ba3f-a476-4c7c-868f-3d1fcf77fe29",
        "924901d5-6d31-4c33-b443-7931eadfac4b",
        "0f8abce3-e161-4bdf-981f-86257d505d69",
        "00000000-0000-0000-0000-000000000000",
        "11111111-1111-1111-1111-111111111111"
    ]
}

Additional IDs

The following IDs can be provided to the test DRS service (either GET or POST) to retrieve a valid DrsObject. Please experiment with the endpoint using various combinations of the following ids, as well as non-valid id:

b8cd0667-2c33-4c9f-967b-161b905932c9
a1dd4ae2-8d26-43b0-a199-342b64c7dff6
1a570e4e-2489-4218-9333-f65549495872
4d83ba3f-a476-4c7c-868f-3d1fcf77fe29
924901d5-6d31-4c33-b443-7931eadfac4b
0f8abce3-e161-4bdf-981f-86257d505d69
c69a3d6c-4a28-4b7c-b215-0782f8d62429
456e9ee0-5b60-4f38-82b5-83ba5d338038
1af6b862-7fc8-411a-86c5-d8e280e5b77a
c37b37fd-0450-432d-b6aa-9ffdece35ad0
0bb9d297-2710-48f6-ab4d-80d5eb0c9eaa
a3bb76d7-76ae-414b-81f7-97e663b02206
1af5cdcf-898c-4dbc-944e-1ac95e82c0ea
2506f0e1-29e4-4132-9b37-f7452dc8a89b
c00c264a-8f17-471f-8ded-1a1f10e965ac
355a74bd-6571-4d4a-8602-a9989936717f
697907bf-d5bd-433e-aac2-1747f1faf366
3a45fab2-f350-445d-a137-4b1f946bf184
ac53ca59-46f3-4f61-b1e7-bb75a49bfbfe
1275f896-4c8f-47e1-99a1-873a6b2ef5fb
8f40acc0-0c54-45c5-8c85-a6f5fb32a1a7
41898242-62a9-4129-9a2c-5a4e8f5f0afb
6b994f18-6189-4233-bb83-139686490d68

jb-adams/drs_batch_request_poc.md