The Data Repository Service (DRS) API provides a generic interface to data repositories so data consumers, including workflow systems, can access data objects in a single, standard way regardless of where they are stored and how they are managed. The primary functionality of DRS is to map a logical ID to a means for physically retrieving the data represented by the ID. The sections below describe the characteristics of those IDs, the types of data supported, how they can be pointed to using URIs, and how clients can use these URIs to ultimately make successful DRS API requests. This document also describes the DRS API in detail and provides information on the specific endpoints, request formats, and responses. This specification is intended for developers of DRS-compatible services and of clients that will call these DRS services.

The key words MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL in this document are to be interpreted as described in RFC 2119.

Each implementation of DRS can choose its own id scheme, as long as it follows these guidelines:

• DRS IDs are strings made up of uppercase and lowercase letters, decimal digits, hyphen, period, underscore and tilde [A-Za-z0-9.-_~]. See RFC 3986 § 2.3.
• DRS IDs can contain other characters, but they MUST be encoded into valid DRS IDs whenever they are used in API calls. This is because non-encoded IDs may interfere with the interpretation of the objects/{id}/access endpoint. To overcome this limitation use percent-encoding of the ID, see RFC 3986 § 2.4
• One DRS ID MUST always return the same object data (or, in the case of a collection, the same set of objects). This constraint aids with reproducibility.
• DRS implementations MAY have more than one ID that maps to the same object.
• DRS version 1.x does NOT support semantics around multiple versions of an object. (For example, there’s no notion of “get latest version” or “list all versions”.) Individual implementations MAY choose an ID scheme that includes version hints.

For convenience, including when passing content references to a WES server, we define a URI scheme for DRS-accessible content. This section documents the syntax of DRS URIs, and the rules clients follow for translating a DRS URI into a URL that they use for making the DRS API calls described in this spec.

There are two styles of DRS URIs, Hostname-based and Compact Identifier-based, both using the drs:// URI scheme. DRS servers may choose either style when exposing references to their content;. DRS clients MUST support resolving both styles.

Tip:

See Appendix: Background Notes on DRS URIs for more information on our design motivations for DRS URIs.

Hostname-based DRS URIs

Hostname-based DRS URIs are simpler than compact identifier-based URIs. They contain the DRS server name and the DRS ID only and can be converted directly into a fetchable URL based on a simple rule. They take the form:

drs://<hostname>/<id>

DRS URIs of this form mean "you can fetch the content with DRS id <id> from the DRS server at <hostname>". For example, here are the client resolution steps if the URI is:

drs://drs.example.org/314159

1. The client parses the string to extract the hostname of “drs.example.org” and the id of “314159”.
2. The client makes a GET request to the DRS server, using the standard DRS URL syntax:

GET https://drs.example.org/ga4gh/drs/v1/objects/314159

The protocol is always https and the port is always the standard 443 SSL port. It is invalid to include a different port in a DRS hostname-based URI.

Tip:

See the Appendix: Hostname-Based URIs for information on how hostname-based DRS URI resolution to URLs is likely to change in the future, when the DRS v2 major release happens.

Compact Identifier-based DRS URIs

Compact Identifier-based DRS URIs use resolver registry services (specifically, identifiers.org and n2t.net (Name-To-Thing)) to provide a layer of indirection between the DRS URI and the DRS server name — the actual DNS name of the DRS server is not present in the URI. This approach is based on the Joint Declaration of Data Citation Principles as detailed by Wimalaratne et al (2018).

For more information, see the document More Background on Compact Identifiers.

Compact Identifiers take the form:

drs://[provider_code/]namespace:accession

Together, provider code and the namespace are referred to as the prefix. The provider code is optional and is used by identifiers.org/n2t.net for compact identifier resolver mirrors. Both the provider_code and namespace disallow spaces or punctuation, only lowercase alphanumerical characters, underscores and dots are allowed (e.g. [A-Za-z0-9._]).

Tip:

See the Appendix: Compact Identifier-Based URIs for more background on Compact Identifiers and resolver registry services like identifiers.org/n2t.net (aka meta-resolvers), how to register prefixes, possible caching strategies, and security considerations.

For DRS Servers

If your DRS implementation will issue DRS URIs based on your own compact identifiers, you MUST first register a new prefix with identifiers.org (which is automatically mirrored to n2t.net). You will also need to include a provider resolver resource in this registration which links the prefix to your DRS server, so that DRS clients can get sufficient information to make a successful DRS GET request. For clarity, we recommend you choose a namespace beginning with drs.

For DRS Clients

A DRS client parses the DRS URI compact identifier components to extract the prefix and the accession, and then uses meta-resolver APIs to locate the actual DRS server. For example, here are the client resolution steps if the URI is:

drs://drs.42:314159

1. The client parses the string to extract the prefix of drs.42 and the accession of 314159, using the first occurrence of a colon (":") character after the initial drs:// as a delimiter. (The colon character is not allowed in a Hostname-based DRS URI, making it easy to tell them apart.)

2. The client makes API calls to a meta-resolver to look up the URL pattern for the namespace. (See Calling Meta-Resolver APIs for Compact Identifier-Based DRS URIs for details.) The URL pattern is a string containing a {$id} parameter, such as:

https://drs.myexample.org/ga4gh/drs/v1/objects/{$id}

3. The client generates a DRS URL from the URL template by replacing {$id} with the accession it extracted in step 1. It then makes a GET request to the DRS server:

GET https://drs.myexample.org/ga4gh/drs/v1/objects/314159

4. The client follows any HTTP redirects returned in step 3, in case the resolver goes through an extra layer of redirection.

For performance reasons, DRS clients SHOULD cache the URL pattern returned in step 2, with a suggested 24 hour cache life.

Choosing a URI Style

DRS servers can choose to issue either hostname-based or compact identifier-based DRS URIs, and can be confident that compliant DRS clients will support both. DRS clients must be able to accommodate both URI types. Tradeoffs that DRS server builders, and third parties who need to cite DRS objects in datasets, workflows or elsewhere, may want to consider include:

Table 1: Choosing a URI Style

DRS's job is data access, period. Therefore, the DRS API supports a simple flat content model -- every DrsObject, like a file, represents a single opaque blob of bytes. DRS has no understanding of the meaning of objects and only provides simple domain-agnostic metadata. Understanding the semantics of specific object types is the responsibility of the applications that use DRS to fetch those objects (e.g. samtools for BAM files, DICOM viewers for DICOM objects).

Atomic Objects

DRS can be used to access individual objects of all kinds, simple or complex, large or small, stored in type-specific formats (e.g. BAM files, VCF files, CSV files). At the API level these are all the same; at the application level, DRS clients and servers are expected to agree on object semantics using non-DRS mechanisms, including but not limited to the GA4GH Data Connect API.

Compound Objects

DRS can also be used to access compound objects, consisting of two or more atomic objects related to each other in a well-specified way. See the Appendix: Compound Objects for suggested best practices for working with compound objects.

[DEPRECATED] Bundles

Previous versions of the DRS API spec included support for a bundle content type, which was a folder-like collection of other DRS objects (either blobs or bundles), represented by a DrsObject with a contents array. As of v1.3, bundles have been deprecated in favor of the best practices documented in the Appendix: Compound Objects. A future version of the API spec may remove bundle support entirely and/or replace bundles with a scalable approach based on the needs of our driver projects.

DRS v1 is a read-only API. We expect that each implementation will define its own mechanisms and interfaces (graphical and/or programmatic) for adding and updating data.

The DRS API specification is written in OpenAPI and embodies a RESTful service philosophy. It uses JSON in requests and responses and standard HTTPS on port 443 for information transport. Optionally, it supports authentication and authorization using the GA4GH Passport standard.

The DRS implementation is responsible for defining and enforcing an authorization policy that determines which users are allowed to make which requests. GA4GH recommends that DRS implementations use an OAuth 2.0 bearer token or a GA4GH Passport, although they can choose other mechanisms if appropriate.

The DRS API allows implementers to support a variety of different content access policies, depending on what AccessMethod records they return. Implementers have a choice to make the GET /objects/{object_id} and GET /objects/{object_id}/access/{access_id} calls open or requiring a Basic, Bearer, or Passport token (Passport requiring a POST). The following describes the various access approaches following a successful GET/POST /objects/{object_id} request in order to them obtain access to the bytes for a given object ID/access ID:

• public content:
  • server provides an access_url with a url and no headers
  • caller fetches the object bytes without providing any auth info
• private content that requires the caller to have out-of-band auth knowledge (e.g. service account credentials):
  • server provides an access_url with a url and no headers
  • caller fetches the object bytes, passing the auth info they obtained out-of-band
• private content that requires the caller to pass an Authorization token:
  • server provides an access_url with a url and headers
  • caller fetches the object bytes, passing auth info via the specified header(s)
• private content that uses an expensive-to-generate auth mechanism (e.g. a signed URL):
  • server provides an access_id
  • caller passes the access_id to the /access endpoint
  • server provides an access_url with the generated mechanism (e.g. a signed URL in the url field)
  • caller fetches the object bytes from the url (passing auth info from the specified headers, if any)

In the approaches above GA4GH Passports are not mentioned and that is on purpose. A DRS server may return a Bearer token or other platform-specific token in a header in response to a valid Bearer token or GA4GH Passport (Option 3 above). But it is not the responsibility of a DRS server to return a Passport, that is the responsibility of a Passport Broker and outside the scope of DRS.

DRS implementers should ensure their solutions restrict access to targets as much as possible, detect attempts to exploit through log monitoring, and they are prepared to take action if an exploit in their DRS implementation is detected.

Discovery

The APIs to fetch DrsObjects and AccessURLs may require authorization. The authorization mode may vary between DRS objects hosted by a service. The authorization mode may vary between the APIs to fetch a DrsObject and an associated AccessURL. Implementers should indicate how to authenticate to fetch a DrsObject by implementing the OptionsOjbect API. Implementers should indicate how to authenticate to fetch an AccessURL within a DrsObject.

Modes

BasicAuth

A valid authorization token must be passed in the 'Authorization' header, e.g. "Basic ${token_string}"

BearerAuth

A valid authorization token must be passed in the 'Authorization' header, e.g. "Bearer ${token_string}"

PassportAuth

A valid authorization GA4GH Passport token must be passed in the body of a POST request

Data sharing requires portable data, consistent with the FAIR data principles (findable, accessible, interoperable, reusable). Today’s researchers and clinicians are surrounded by potentially useful data, but often need bespoke tools and processes to work with each dataset. Today’s data publishers don’t have a reliable way to make their data useful to all (and only) the people they choose. And today’s data controllers are tasked with implementing standard controls of non-standard mechanisms for data access.

Figure 1: there’s an ocean of data, with many different tools to drink from it, but no guarantee that any tool will work with any subset of the data

We need a standard way for data producers to make their data available to data consumers, that supports the control needs of the former and the access needs of the latter. And we need it to be interoperable, so anyone who builds access tools and systems can be confident they’ll work with all the data out there, and anyone who publishes data can be confident it will work with all the tools out there.

Figure 2: by defining a standard Data Repository API, and adapting tools to use it, every data publisher can now make their data useful to every data consumer

We envision a world where: there are many many data consumers, working in research and in care, who can use the tools of their choice to access any and all data that they have permission to see there are many data access tools and platforms, supporting discovery, visualization, analysis, and collaboration there are many data repositories, each with their own policies and characteristics, which can be accessed by a variety of tools there are many data publishing tools and platforms, supporting a variety of data lifecycles and formats there are many many data producers, generating data of all types, who can use the tools of their choice to make their data as widely available as is appropriate

Figure 3: a standard Data Repository API enables an ecosystem of data producers and consumers

This spec defines a standard Data Repository Service (DRS) API (“the yellow box”), to enable that ecosystem of data producers and consumers. Our goal is that the only thing data consumers need to know about a data repo is "here’s the DRS endpoint to access it", and the only thing data publishers need to know to tap into the world of consumption tools is "here’s how to tell it where my DRS endpoint lives".

The world’s biomedical data is controlled by groups with very different policies and restrictions on where their data lives and how it can be accessed. A primary purpose of DRS is to support unified access to disparate and distributed data. (As opposed to the alternative centralized model of "let’s just bring all the data into one single data repository”, which would be technically easier but is no more realistic than “let’s just bring all the websites into one single web host”.)

In a DRS-enabled world, tool builders don’t have to worry about where the data their tools operate on lives — they can count on DRS to give them access. And tool users only need to know which DRS server is managing the data they need, and whether they have permission to access it; they don’t have to worry about how to physically get access to, or (worse) make a copy of the data. For example, if I have appropriate permissions, I can run a pooled analysis where I run a single tool across data managed by different DRS servers, potentially in different locations.

The DRS API supports access to data objects, with each DrsObject representing a single opaque blob of bytes. Much content (e.g. VCF files) is well represented as a single atomic DrsObject. Some content, however (e.g. DICOM images) is best represented as a compound object consisting of a structured collection of atomic DrsObjects. In both cases, DRS isn't aware of the semantics of the objects it serves -- understanding those semantics is the responsibility of the applications that call DRS.

Common examples of compound objects in biomedicine include:

• BAM+BAI genomic reads, with a small index (the BAI object) to large data (the BAM object), each object using a well-defined file format.
• DICOM images, with a contents object pointing to one or more raw image objects, each containing pixels from different aspects of a single logical biomedical image (e.g. different z-coordinates)
• studies, with a single table of contents listing multiple objects of various types that were generated together and are meant to be processed together

As with atomic objects, DRS applications and servers are expected to agree on the semantics of compound objects using non-DRS mechanisms. The recommended best practice for representing a particular compound object type is:

1. Define a manifest file syntax, which contains the DRS IDs of the constituent atomic objects, plus type-specific information about the relationship between those constituents.
   • Manifest file syntax isn't prescribed by the spec, but we expect they will often be JSON files.
   • For example, for a BAM+BAI pair the manifest file could contain two key-value pairs mapping the type of each constituent file to its DRS ID.
2. Make manifest objects and their constituent objects available using standard DRS mechanisms -- each object is referenced via its own DRS ID, just like any other atomic object.
   • For example, for a BAM+BAI pair, there would be three DRS IDs -- one for the manifest, one for the BAM, and one for the BAI.
3. Document the expected client logic for processing compound objects of interest. This logic typically consists of using standard DRS mechanisms to fetch the manifest, parsing its syntax, extracting the DRS IDs of constituent objects, and using standard DRS mechanisms to fetch the constituents as needed.
   • In some cases the application will always want to fetch all of the constituents; in other cases it may want to initially fetch a subset, and only fetch the others on demand. For example, a DICOM image viewer may only want to fetch the layers that are being rendered.

DRS URIs are aligned with the FAIR data principles and the Joint Declaration of Data Citation Principles — both hostname-based and compact identifier-based URIs provide globally unique, machine-resolvable, persistent identifiers for data.

• We require all URIs to begin with drs:// as a signal to humans and systems consuming these URIs that the response they will ultimately receive, after transforming the URI to a fetchable URL, will be a DRS JSON packet. This signal differentiates DRS URIs from the wide variety of other entities (HTML documents, PDFs, ontology notes, etc.) that can be represented by compact identifiers.
• We support hostname-based URIs because of their simplicity and efficiency for server and client implementers.
• We support compact identifier-based URIs, and the meta-resolver services of identifiers.org and n2t.net (Name-to-Thing), because of the wide adoption of compact identifiers in the research community. as detailed by Wimalaratne et al (2018) in "Uniform resolution of compact identifiers for biomedical data."

Note: Identifiers.org/n2t.net API Changes

The examples below show the current API interactions with n2t.net and identifiers.org which may change over time. Please refer to the documentation from each site for the most up-to-date information. We will make best efforts to keep the DRS specification current but DRS clients MUST maintain their ability to use either the identifiers.org or n2t.net APIs to resolve compact identifier-based DRS URIs.

See the documentation on the n2t.net and identifiers.org meta-resolvers for adding your own compact identifier type and registering your DRS server as a resolver. You can register new prefixes (or mirrors by adding resource provider codes) for free using a simple online form. For more information see More Background on Compact Identifiers.

Clients resolving Compact Identifier-based URIs need to convert a prefix (e.g. “drs.42”) into a URL pattern. They can do so by calling either the identifiers.org or the n2t.net API, since the two meta-resolvers keep their mapping databases in sync.

Calling the identifiers.org API as a Client

It takes two API calls to get the URL pattern.

1. The client makes a GET request to identifiers.org to find information about the prefix:

GET https://registry.api.identifiers.org/restApi/namespaces/search/findByPrefix?prefix=drs.42

This request returns a JSON structure including various URLs containing an embedded namespace id, such as:

"namespace" : {
  "href":"https://registry.api.identifiers.org/restApi/namespaces/1234"
}

2. The client extracts the namespace id (in this example 1234), and uses it to make a second GET request to identifiers.org to find information about the namespace:

GET https://registry.api.identifiers.org/restApi/resources/search/findAllByNamespaceId?id=1234

This request returns a JSON structure including an urlPattern field, whose value is a URL pattern containing a ${id} parameter, such as:

"urlPattern" : "https://drs.myexample.org/ga4gh/drs/v1/objects/{$id}"

Calling the n2t.net API as a Client

It takes one API call to get the URL pattern.

The client makes a GET request to n2t.net to find information about the namespace. (Note the trailing colon.)

GET https://n2t.net/drs.42:

This request returns a text structure including a redirect field, whose value is a URL pattern containing an $id parameter, such as:

redirect: https://drs.myexample.org/ga4gh/drs/v1/objects/$id

Identifiers.org/n2t.net compact identifier resolver records do not change frequently. This reality is useful for caching resolver records and their URL patterns for performance reasons. Builders of systems that use compact identifier-based DRS URIs should cache prefix resolver records from identifiers.org/n2t.net and occasionally refresh the records (such as every 24 hours). This approach will reduce the burden on these community services since we anticipate many DRS URIs will be regularly resolved in workflow systems. Alternatively, system builders may decide to directly mirror the registries themselves, instructions are provided on the identifiers.org/n2t.net websites.

As mentioned earlier, identifiers.org/n2t.net performs some basic verification of new prefixes and provider code mirror registrations on their sites. However, builders of systems that consume and resolve DRS URIs may have certain security compliance requirements and regulations that prohibit relying on an external site for resolving compact identifiers. In this case, systems under these security and compliance constraints may wish to whitelist certain compact identifier resolvers and/or vet records from identifiers.org/n2t.net before enabling in their systems.

The compact identifier format used by identifiers.org/n2t.net does not percent-encode reserved URI characters but, instead, relies on the first ":" character to separate prefix from accession. Since these accessions can contain any characters, and characters like "/" will interfere with DRS API calls, you must percent encode the accessions extracted from DRS compact identifier-based URIs when using as DRS IDs in subsequent DRS GET requests. An easy way for a DRS client to handle this is to get the initial DRS object JSON response from whatever redirects the compact identifier resolves to, then look for the self_uri in the JSON, which will give you the correctly percent-encoded DRS ID for subsequent DRS API calls such as the access method.

For additional examples, see the document More Background on Compact Identifiers.

In hostname-based DRS URIs, the ID is always percent-encoded to ensure special characters do not interfere with subsequent DRS endpoint calls. As such, ":" is not allowed in the URI and is a convenient way of differentiating from a compact identifier-based DRS URI. Also, if a given DRS service implementation uses compact identifier accessions as their DRS IDs, they must be percent encoded before using them as DRS IDs in hostname-based DRS URIs and subsequent GET requests to a DRS service endpoint.

The GA4GH Service Registry API specification allows information about GA4GH-compliant web services, including DRS services, to be aggregated into registries and made available via a standard API. The following considerations should be followed when registering DRS services within a service registry.

• The DRS service attributes returned by /service-info (i.e. id, name, description, etc.) should have the same values as the registry entry for that service.
• The value of the type object's artifact property should be drs (i.e. the same as it appears in service-info)
• Each entry in a Service Registry must have a url, indicating the base URL to the web service. For DRS services, the registered url must include everything up to the standardized /ga4gh/drs/v1 path. Clients should be able to assume that:
  • Adding /ga4gh/drs/v1/objects/{object_id} to the registered url will hit the DrsObject endpoint
  • Adding /ga4gh/drs/v1/service-info to the registered url will hit the Service Info endpoint

Example listing of a DRS API registration from a service registry's /services endpoint:

[
    {
        "id": "com.example.drs",
        "name": "Example DRS API",
        "type": {
            "group": "org.ga4gh",
            "artifact": "drs",
            "version": "1.4.0"
        },
        "description": "The Data Repository Service (DRS) API ...",
        "organization": {
            "id": "com.example",
            "name": "Example Company"
        },
        "contactUrl": "mailto:support@example.com",
        "documentationUrl": "https://docs.example.com/docs/drs",
        "createdAt": "2021-08-09T00:00:00Z",
        "updatedAt": "2021-08-09T12:30:00Z",
        "environment": "production",
        "version": "1.13.4",
        "url": "https://drs-service.example.com"
    }
]