The Data Repository Service (DRS) API provides a generic interface to data repositories so data consumers, including workflow systems, can access data objects in a single, standard way regardless of where they are stored and how they are managed. The primary functionality of DRS is to map a logical ID to a means for physically retrieving the data represented by the ID. The sections below describe the characteristics of those IDs, the types of data supported, how they can be pointed to using URIs, and how clients can use these URIs to ultimately make successful DRS API requests. This document also describes the DRS API in detail and provides information on the specific endpoints, request formats, and responses. This specification is intended for developers of DRS-compatible services and of clients that will call these DRS services.

The key words MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL in this document are to be interpreted as described in RFC 2119.

Each implementation of DRS can choose its own id scheme, as long as it follows these guidelines:

• DRS IDs are strings made up of uppercase and lowercase letters, decimal digits, hyphen, period, underscore and tilde [A-Za-z0-9.-_~]. See RFC 3986 § 2.3.
• DRS IDs can contain other characters, but they MUST be encoded into valid DRS IDs whenever they are used in API calls. This is because non-encoded IDs may interfere with the interpretation of the objects/{id}/access endpoint. To overcome this limitation use percent-encoding of the ID, see RFC 3986 § 2.4
• One DRS ID MUST always return the same object data (or, in the case of a collection, the same set of objects). This constraint aids with reproducibility.
• DRS implementations MAY have more than one ID that maps to the same object.
• DRS version 1.x does NOT support semantics around multiple versions of an object. (For example, there’s no notion of “get latest version” or “list all versions”.) Individual implementations MAY choose an ID scheme that includes version hints.

For convenience, including when passing content references to a WES server, we define a URI scheme for DRS-accessible content. This section documents the syntax of DRS URIs, and the rules clients follow for translating a DRS URI into a URL that they use for making the DRS API calls described in this spec.

There are two styles of DRS URIs, Hostname-based and Compact Identifier-based, both using the drs:// URI scheme. DRS servers may choose either style when exposing references to their content;. DRS clients MUST support resolving both styles.

Tip:

See Appendix: Background Notes on DRS URIs for more information on our design motivations for DRS URIs.

Hostname-based DRS URIs

Hostname-based DRS URIs are simpler than compact identifier-based URIs. They contain the DRS server name and the DRS ID only and can be converted directly into a fetchable URL based on a simple rule. They take the form:

drs://<hostname>/<id>

DRS URIs of this form mean "you can fetch the content with DRS id <id> from the DRS server at <hostname>". For example, here are the client resolution steps if the URI is:

drs://drs.example.org/314159

1. The client parses the string to extract the hostname of “drs.example.org” and the id of “314159”.
2. The client makes a GET request to the DRS server, using the standard DRS URL syntax:

GET https://drs.example.org/ga4gh/drs/v1/objects/314159

The protocol is always https and the port is always the standard 443 SSL port. It is invalid to include a different port in a DRS hostname-based URI.

Tip:

See the Appendix: Hostname-Based URIs for information on how hostname-based DRS URI resolution to URLs is likely to change in the future, when the DRS v2 major release happens.

Compact Identifier-based DRS URIs

Compact Identifier-based DRS URIs use resolver registry services (specifically, identifiers.org and n2t.net (Name-To-Thing)) to provide a layer of indirection between the DRS URI and the DRS server name — the actual DNS name of the DRS server is not present in the URI. This approach is based on the Joint Declaration of Data Citation Principles as detailed by Wimalaratne et al (2018).

For more information, see the document More Background on Compact Identifiers.

Compact Identifiers take the form:

drs://[provider_code/]namespace:accession

Together, provider code and the namespace are referred to as the prefix. The provider code is optional and is used by identifiers.org/n2t.net for compact identifier resolver mirrors. Both the provider_code and namespace disallow spaces or punctuation, only lowercase alphanumerical characters, underscores and dots are allowed (e.g. [A-Za-z0-9._]).

Tip:

See the Appendix: Compact Identifier-Based URIs for more background on Compact Identifiers and resolver registry services like identifiers.org/n2t.net (aka meta-resolvers), how to register prefixes, possible caching strategies, and security considerations.

For DRS Servers

If your DRS implementation will issue DRS URIs based on your own compact identifiers, you MUST first register a new prefix with identifiers.org (which is automatically mirrored to n2t.net). You will also need to include a provider resolver resource in this registration which links the prefix to your DRS server, so that DRS clients can get sufficient information to make a successful DRS GET request. For clarity, we recommend you choose a namespace beginning with drs.

For DRS Clients

A DRS client parses the DRS URI compact identifier components to extract the prefix and the accession, and then uses meta-resolver APIs to locate the actual DRS server. For example, here are the client resolution steps if the URI is:

drs://drs.42:314159

1. The client parses the string to extract the prefix of drs.42 and the accession of 314159, using the first occurrence of a colon (":") character after the initial drs:// as a delimiter. (The colon character is not allowed in a Hostname-based DRS URI, making it easy to tell them apart.)

2. The client makes API calls to a meta-resolver to look up the URL pattern for the namespace. (See Calling Meta-Resolver APIs for Compact Identifier-Based DRS URIs for details.) The URL pattern is a string containing a {$id} parameter, such as:

https://drs.myexample.org/ga4gh/drs/v1/objects/{$id}

3. The client generates a DRS URL from the URL template by replacing {$id} with the accession it extracted in step 1. It then makes a GET request to the DRS server:

GET https://drs.myexample.org/ga4gh/drs/v1/objects/314159

4. The client follows any HTTP redirects returned in step 3, in case the resolver goes through an extra layer of redirection.

For performance reasons, DRS clients SHOULD cache the URL pattern returned in step 2, with a suggested 24 hour cache life.

Choosing a URI Style

DRS servers can choose to issue either hostname-based or compact identifier-based DRS URIs, and can be confident that compliant DRS clients will support both. DRS clients must be able to accommodate both URI types. Tradeoffs that DRS server builders, and third parties who need to cite DRS objects in datasets, workflows or elsewhere, may want to consider include:

Table 1: Choosing a URI Style

DRS's job is data access, period. Therefore, the DRS API supports a simple flat content model -- every DrsObject, like a file, represents a single opaque blob of bytes. DRS has no understanding of the meaning of objects and only provides simple domain-agnostic metadata. Understanding the semantics of specific object types is the responsibility of the applications that use DRS to fetch those objects (e.g. samtools for BAM files, DICOM viewers for DICOM objects).

Atomic Objects

DRS can be used to access individual objects of all kinds, simple or complex, large or small, stored in type-specific formats (e.g. BAM files, VCF files, CSV files). At the API level these are all the same; at the application level, DRS clients and servers are expected to agree on object semantics using non-DRS mechanisms, including but not limited to the GA4GH Data Connect API.

Compound Objects

DRS can also be used to access compound objects, consisting of two or more atomic objects related to each other in a well-specified way. See the Appendix: Compound Objects for suggested best practices for working with compound objects.

[DEPRECATED] Bundles

Previous versions of the DRS API spec included support for a bundle content type, which was a folder-like collection of other DRS objects (either blobs or bundles), represented by a DrsObject with a contents array. As of v1.3, bundles have been deprecated in favor of the best practices documented in the Appendix: Compound Objects. A future version of the API spec may remove bundle support entirely and/or replace bundles with a scalable approach based on the needs of our driver projects.

DRS v1 is a read-only API. We expect that each implementation will define its own mechanisms and interfaces (graphical and/or programmatic) for adding and updating data.

The DRS API specification is written in OpenAPI and embodies a RESTful service philosophy. It uses JSON in requests and responses and standard HTTPS on port 443 for information transport. Optionally, it supports authentication and authorization using the GA4GH Passport standard.

The DRS implementation is responsible for defining and enforcing an authorization policy that determines which users are allowed to make which requests. GA4GH recommends that DRS implementations use an OAuth 2.0 bearer token or a GA4GH Passport, although they can choose other mechanisms if appropriate.

The DRS API allows implementers to support a variety of different content access policies, depending on what AccessMethod records they return. Implementers have a choice to make the GET /objects/{object_id} and GET /objects/{object_id}/access/{access_id} calls open or requiring a Basic, Bearer, or Passport token (Passport requiring a POST). The following describes the various access approaches following a successful GET/POST /objects/{object_id} request in order to them obtain access to the bytes for a given object ID/access ID:

• public content:
  • server provides an access_url with a url and no headers
  • caller fetches the object bytes without providing any auth info
• private content that requires the caller to have out-of-band auth knowledge (e.g. service account credentials):
  • server provides an access_url with a url and no headers
  • caller fetches the object bytes, passing the auth info they obtained out-of-band
• private content that requires the caller to pass an Authorization token:
  • server provides an access_url with a url and headers
  • caller fetches the object bytes, passing auth info via the specified header(s)
• private content that uses an expensive-to-generate auth mechanism (e.g. a signed URL):
  • server provides an access_id
  • caller passes the access_id to the /access endpoint
  • server provides an access_url with the generated mechanism (e.g. a signed URL in the url field)
  • caller fetches the object bytes from the url (passing auth info from the specified headers, if any)

In the approaches above GA4GH Passports are not mentioned and that is on purpose. A DRS server may return a Bearer token or other platform-specific token in a header in response to a valid Bearer token or GA4GH Passport (Option 3 above). But it is not the responsibility of a DRS server to return a Passport, that is the responsibility of a Passport Broker and outside the scope of DRS.

DRS implementers should ensure their solutions restrict access to targets as much as possible, detect attempts to exploit through log monitoring, and they are prepared to take action if an exploit in their DRS implementation is detected.

Discovery

The APIs to fetch DrsObjects and AccessURLs may require authorization. The authorization mode may vary between DRS objects hosted by a service. The authorization mode may vary between the APIs to fetch a DrsObject and an associated AccessURL. Implementers should indicate how to authenticate to fetch a DrsObject by implementing the OptionsOjbect API. Implementers should indicate how to authenticate to fetch an AccessURL within a DrsObject.

Modes

BasicAuth

A valid authorization token must be passed in the 'Authorization' header, e.g. "Basic ${token_string}"

BearerAuth

A valid authorization token must be passed in the 'Authorization' header, e.g. "Bearer ${token_string}"

PassportAuth

A valid authorization GA4GH Passport token must be passed in the body of a POST request

Optional Functionality: Upload operations are optional extensions to the DRS API. Not all DRS servers are required to implement upload functionality. Clients should check for the availability of upload endpoints before attempting to use them.

The DRS API provides optional upload functionality that allows clients to obtain upload methods and temporary credentials for storing files before they are registered as DRS objects. This capability extends the traditional read-only nature of DRS to support data ingestion workflows.

The upload feature is designed as a negotiation mechanism between client and server to identify mutually convenient storage services and access patterns. The DRS specification defines the negotiation protocol and credential exchange, but the details of how data is physically uploaded to the underlying storage systems are outside the scope of the DRS specification and depend on the specific storage service protocols (S3, HTTPS, etc.). DRS does not reinvent existing storage protocols, but rather facilitates their discovery and authorization.

Upload operations in DRS follow a three-phase approach:

1. Upload Request Phase: Clients request upload methods by providing file metadata
2. File Upload Phase: Clients use the returned upload methods to store files in the underlying storage system
3. DRS Object Registration Phase: Clients register the uploaded files as DRS objects using the POST /objects endpoint

This design separates the concerns of obtaining upload credentials from the actual file transfer and subsequent DRS object registration, providing flexibility in how files are uploaded while maintaining security through temporary, scoped credentials.

Upload operations are entirely optional. Servers implement uploads when they support data ingestion, collaborative workflows, or staging areas. Read-only servers, mirrors, or security-constrained environments typically do not implement uploads.

Clients should implement proper discovery mechanisms to determine upload support:

Service Info Discovery (Recommended)

The most reliable way to discover upload support is through the /service-info endpoint:

{
  "drs": {
    "uploadSupported": true,
    "supportedUploadMethods": ["s3", "https", "gs"],
    "maxUploadSize": 5368709120,
    "maxUploadRequestLength": 50,
    "validateUploadChecksums": true,
    "validateUploadFileSizes": false
  }
}

Service Info Fields:

• uploadSupported: Boolean indicating if upload operations are available
• supportedUploadMethods: Array of upload method types the server supports
• maxUploadSize: Maximum file size in bytes (optional)
• maxUploadRequestLength: Maximum files per upload request (optional)
• validateUploadChecksums: Boolean indicating if server validates uploaded file checksums (optional, defaults to false)
• validateUploadFileSizes: Boolean indicating if server validates uploaded file sizes (optional, defaults to false)

Client Best Practices

1. Always check /service-info first for uploadSupported, supportedUploadMethods, and validation flags
2. Respect server limits (maxUploadSize, maxUploadRequestLength)
3. Handle missing upload support gracefully with alternative workflows
4. Prepare for validation behavior based on validateUploadChecksums/validateUploadFileSizes flags

The /uploadrequest endpoint accepts POST requests containing file metadata and returns available upload methods with temporary credentials.

Request Structure

Upload requests must include:

• File metadata: Name, size, MIME type, and checksums for each file
• Authentication: Optional GA4GH Passport tokens for authorization
• File descriptions: Optional human-readable descriptions and aliases

Note: Servers do not validate the provided MIME type against the actual file content. Clients are responsible for providing accurate MIME type information.

Upload Method Selection: Clients must select one or more upload methods from the server response to upload their files. The selected upload methods determine the storage locations, and clients must include corresponding access methods in the DRS object registration that point to these same storage locations.

Response Structure

Upload responses provide:

• DRS object metadata: Pre-assigned IDs and URIs for the files
• Upload methods: Available storage protocols (S3, HTTPS, GCS, etc.)
• Temporary credentials: Time-limited access tokens or keys
• Upload URLs: Specific endpoints for file transfer

• https: Presigned POST URLs for simple HTTP uploads
• s3: Direct S3 upload with temporary AWS credentials (supports multipart uploads)
• gs: Google Cloud Storage upload with OAuth2 tokens
• ftp/sftp: Traditional file transfer protocols
• globus: High-performance transfer service for large files

Note: While servers advertise their supported upload methods in service-info, the actual upload response may include only a subset of these methods. Servers may choose specific upload methods based on file characteristics such as size, type, or internal policies.

Upload operations support GA4GH Passports (embedded in request body), Basic authentication, and Bearer tokens for flexible authorization.

All upload requests must include checksums to ensure data integrity:

Required Checksums

• At least one checksum per file is mandatory for client requests
• Supported algorithms include SHA-256, MD5, and IANA-registered hash types
• Multiple checksums per file are supported for enhanced verification

Server Validation Options

Servers MAY validate checksums and/or file sizes (advertised via validateUploadChecksums/validateUploadFileSizes flags) but are not required to do so. Validation increases security but adds computational overhead.

Validation Process

1. Client calculates checksums before requesting upload methods
2. Server MAY validate checksums and file sizes during or after upload, but is not required to do so
3. If server performs validation and detects mismatches, it SHOULD reject the upload or registration
4. Servers that do not perform validation rely on client-provided metadata for DRS object registration
5. Successful upload (with or without server validation) enables DRS object registration

Upload operations are designed to integrate seamlessly with the broader DRS object lifecycle:

Pre-Upload Phase

1. Client prepares files and calculates metadata
2. Client requests upload methods via /uploadrequest
3. Server responds with upload options and pre-assigned DRS identifiers

Upload Phase

1. Client selects one or more appropriate upload methods from the server response
2. Client uploads files using the selected methods' credentials and URLs
3. Storage system receives and stores files at the locations corresponding to the selected upload methods

Post-Upload Phase

1. Files are available in storage with pre-assigned DRS identifiers
2. DRS Object Registration: Clients use the POST /objects endpoint to register uploaded files as DRS objects
3. DRS objects can be queried using standard DRS endpoints
4. Access methods are automatically configured based on storage location

After successfully uploading files using the /uploadrequest endpoint, clients must register the uploaded files as DRS objects to make them accessible through the DRS API. This can be accomplished using either:

• Single Object Registration: POST /objects/{object_id} for registering one object at a time
• Bulk Object Registration: POST /objects for registering multiple objects at once

Registration Process

1. Prepare DRS Object Metadata: Create fully formed DRS object structures with:

   • The object ID returned from the upload request
   • Complete metadata (name, size, checksums, MIME type, timestamps)
   • Access methods that correspond to the upload methods used during file upload
   • Optional descriptions and aliases

2. Choose Registration Method:

   • Single Object: POST to /objects/{object_id} with object field containing one DRS object
   • Bulk Objects: POST to /objects with objects array containing multiple DRS objects

3. Submit Registration Request: Include GA4GH Passport tokens for authorization (if required)

4. Confirm Registration: Server validates and registers the DRS objects, making them available for subsequent queries

Single Object Registration

The POST /objects/{object_id} endpoint operates in two modes:

• Register Mode: When the request body contains an object field with a fully formed DRS object, the endpoint registers the provided object
• Retrieve Mode: When the object field is missing, the endpoint behaves as a GET request with passport authentication

Bulk Object Registration

The POST /objects endpoint operates in two modes:

• Register Mode: When the request body contains an objects array with fully formed DRS objects, the endpoint registers the provided objects
• Retrieve Mode: When the request body contains bulk_object_ids array, the endpoint behaves as a bulk retrieval operation returning metadata for the specified objects

Registration Requirements

For successful DRS object registration after upload:

• Complete Metadata: All required DRS object fields must be present and valid
• Valid Access Methods: Access methods must point to the actual uploaded file locations
• Checksum Consistency: Checksums in registration request SHOULD match those provided during upload request
• Authorization: Appropriate GA4GH Passport tokens must be provided if required by the server
• Bulk Efficiency: Multiple objects can be registered in a single request for better performance

Note: Servers MAY verify that registered object metadata matches uploaded file characteristics, but are not required to do so.

Example Registration Workflows

Single Object Registration:

1. Upload file via /uploadrequest
   → Receive object ID: "drs_obj_123"

2. File is uploaded to storage location
   → File available at provided upload URL

3. Register DRS object via POST /objects/drs_obj_123
   → Request body contains object field with fully formed DRS object
   → Server validates and registers the object

4. Query registered object via GET /objects/drs_obj_123
   → Standard DRS object retrieval now works

Bulk Object Registration:

1. Upload files via /uploadrequest
   → Receive object IDs: "drs_obj_123", "drs_obj_456"

2. Files are uploaded to storage locations
   → Files available at provided upload URLs

3. Register DRS objects via POST /objects
   → Request body contains objects array with fully formed DRS objects
   → Server validates and registers all objects in bulk

4. Query registered objects via GET /objects/drs_obj_123
   → Standard DRS object retrieval now works for all registered objects

Upload operations include comprehensive error handling:

Client Errors (4xx)

• 400 Bad Request: Invalid file metadata or malformed requests
• 401 Unauthorized: Missing or invalid authentication credentials
• 403 Forbidden: Insufficient permissions for upload operations

Server Errors (5xx)

• 500 Internal Server Error: Storage system unavailable or configuration issues
• 503 Service Unavailable: Temporary capacity limitations or maintenance

Upload-Specific Errors

• Checksum validation failures (only if server performs validation)
• File size mismatches (only if server performs validation)
• File size limit exceeded (server-imposed limits)
• Storage quota exceeded
• Upload timeout or connection failures

Clients: Calculate checksums, handle multiple upload methods, implement retry logic, check service-info for validation behavior.

Servers: Use short-lived credentials, provide multiple upload methods when possible, implement consistent validation (if any), use rate limiting.

Security: Time-limited credentials, single-use URLs, proper logging, input validation.

Simple File Upload

1. Check service info via GET /service-info to confirm upload support and available methods
2. Calculate SHA-256 checksum of local file
3. Send upload request with file metadata to /uploadrequest
4. Receive HTTPS upload URL, headers, and pre-assigned DRS object ID
5. POST file to upload URL using multipart form data
6. Verify upload success
7. Register DRS object via POST /objects/{object_id} with fully formed DRS object metadata
8. Confirm DRS object is accessible via standard DRS endpoints

Large File Upload with S3

1. Check service info to confirm S3 upload support and size limits
2. Prepare large genomic dataset file
3. Request upload methods specifying file size via /uploadrequest
4. Receive S3 upload method with temporary AWS credentials and pre-assigned DRS object ID
5. Use AWS SDK to perform multipart upload
6. Monitor upload progress and handle retries
7. Confirm upload completion
8. Register DRS object via POST /objects/{object_id} with complete metadata and S3 access methods
9. Verify DRS object availability through standard DRS queries

Authenticated Upload with Passports

1. Check service info to confirm upload support and understand available methods
2. Obtain GA4GH Passport tokens for dataset access
3. Include passports in upload request body to /uploadrequest
4. Server validates passport visas and permissions
5. Receive authorized upload methods and pre-assigned DRS object IDs
6. Upload files to authorized storage locations
7. Register DRS objects via POST /objects with the same passport tokens (bulk registration)
8. Server validates passports again during registration
9. DRS objects become available with appropriate access controls based on passport authorization

Bulk Upload and Registration

1. Check service info to confirm upload support and bulk limits
2. Request upload methods for multiple files via /uploadrequest
3. Receive upload methods and pre-assigned DRS object IDs for all files
4. Upload all files using their respective upload methods
5. Register all DRS objects via POST /objects with objects array containing all uploaded files
6. All objects become available simultaneously through standard DRS endpoints

Optional DRS functionality for removing objects and, optionally, data from the underlying storage service. Servers remain fully compliant without implementing delete endpoints.

• Optional: Delete support is completely optional
• Safe by Default: Preserves storage data unless explicitly requested
• Backward Compatible: No impact on existing DRS functionality
• Flexible Auth: Supports GA4GH Passports and Bearer tokens

Check /service-info for delete capabilities:

{
  "drs": {
    "deleteSupported": true,
    "maxBulkDeleteLength": 100,
    "deleteStorageDataSupported": true
  }
}

• deleteSupported: Whether server implements delete endpoints
• maxBulkDeleteLength: Maximum objects per bulk delete request
• deleteStorageDataSupported: Whether server can attempt to delete underlying storage files

Why POST instead of DELETE? GA4GH Passports require request bodies, which DELETE methods don't reliably support across all HTTP infrastructure. POST ensures broad compatibility.

Single Object Delete: POST /objects/{object_id}/delete

curl -X POST "https://drs.example.org/objects/drs_object_123456/delete" \
  -H "Content-Type: application/json" \
  -d '{"passports": ["..."], "delete_storage_data": false}'
# Response: 204 No Content (indicates metadata deletion success only)

Note: HTTP responses indicate metadata deletion status only. Storage deletion (delete_storage_data: true) is a best effort attempt with no guarantee of success.

Bulk Object Delete: POST /objects/delete

curl -X POST "https://drs.example.org/objects/delete" \
  -H "Content-Type: application/json" \
  -d '{
    "bulk_object_ids": ["obj_1", "obj_2", "obj_3"],
    "passports": ["..."],
    "delete_storage_data": false
  }'
# Response: 204 No Content (all metadata deleted) or 207 Multi-Status (mixed metadata results)

GA4GH Passports (in request body):

{"passports": ["eyJhbGci..."], "delete_storage_data": false}

Bearer Tokens (in headers):

curl -H "Authorization: Bearer token" -d '{"delete_storage_data": false}' ...

delete_storage_data: false (default): Removes metadata only, preserves storage files delete_storage_data: true: Removes metadata AND attempts to delete storage files (requires server support, not guaranteed)

Why no native updates? DRS promotes immutability, data safety, and clear audit trails through explicit operations. This approach also makes implementation simpler by reusing existing endpoints.

Safe Update Process:

1. Delete metadata only: POST /objects/{id}/delete with delete_storage_data: false
2. Re-register object: POST /objects/{id} with updated metadata

# Delete metadata (preserves storage)
curl -X POST ".../objects/obj_123/delete" -d '{"delete_storage_data": false}'
# Re-register with updates
curl -X POST ".../objects/obj_123" -d '{"name": "updated.txt", ...}'

• 400: Unsupported storage deletion
• 403: Insufficient permissions
• 404: Object not found or delete not supported
• 413: Bulk request too large

Metadata Update:

curl ".../service-info"  # Check capabilities
curl -X POST ".../objects/obj_123/delete" -d '{"delete_storage_data": false}'
curl -X POST ".../objects/obj_123" -d '{"name": "updated.vcf", ...}'

Complete Removal:

curl -X POST ".../objects/obj_456/delete" -H "Authorization: Bearer token" \
  -d '{"delete_storage_data": true}'

Bulk Delete:

curl -X POST ".../objects/delete" -d '{
  "bulk_object_ids": ["obj_1", "obj_2"],
  "passports": ["..."],
  "delete_storage_data": false
}'

Clients: Check service-info, default to safe deletion, handle errors, respect limits, confirm destructive operations

Servers: Advertise capabilities, validate permissions, implement limits, audit logging, rate limiting

• Authentication: Validate GA4GH Passports and Bearer tokens
• HTTPS Required: Protect credentials in transit
• Rate Limiting: Prevent abuse of delete endpoints
• Input Validation: Sanitize all request parameters

Delete functionality is designed to be 100% backward compatible:

• No Impact on Existing Endpoints: All existing DRS endpoints remain unchanged
• Optional Implementation: Servers can ignore delete functionality entirely
• Graceful Degradation: Clients receive 404 responses when delete is not supported
• Safe Defaults: New fields in service-info have safe default values

Data sharing requires portable data, consistent with the FAIR data principles (findable, accessible, interoperable, reusable). Today’s researchers and clinicians are surrounded by potentially useful data, but often need bespoke tools and processes to work with each dataset. Today’s data publishers don’t have a reliable way to make their data useful to all (and only) the people they choose. And today’s data controllers are tasked with implementing standard controls of non-standard mechanisms for data access.

Figure 1: there’s an ocean of data, with many different tools to drink from it, but no guarantee that any tool will work with any subset of the data

We need a standard way for data producers to make their data available to data consumers, that supports the control needs of the former and the access needs of the latter. And we need it to be interoperable, so anyone who builds access tools and systems can be confident they’ll work with all the data out there, and anyone who publishes data can be confident it will work with all the tools out there.

Figure 2: by defining a standard Data Repository API, and adapting tools to use it, every data publisher can now make their data useful to every data consumer

We envision a world where: there are many many data consumers, working in research and in care, who can use the tools of their choice to access any and all data that they have permission to see there are many data access tools and platforms, supporting discovery, visualization, analysis, and collaboration there are many data repositories, each with their own policies and characteristics, which can be accessed by a variety of tools there are many data publishing tools and platforms, supporting a variety of data lifecycles and formats there are many many data producers, generating data of all types, who can use the tools of their choice to make their data as widely available as is appropriate

Figure 3: a standard Data Repository API enables an ecosystem of data producers and consumers

This spec defines a standard Data Repository Service (DRS) API (“the yellow box”), to enable that ecosystem of data producers and consumers. Our goal is that the only thing data consumers need to know about a data repo is "here’s the DRS endpoint to access it", and the only thing data publishers need to know to tap into the world of consumption tools is "here’s how to tell it where my DRS endpoint lives".

The world’s biomedical data is controlled by groups with very different policies and restrictions on where their data lives and how it can be accessed. A primary purpose of DRS is to support unified access to disparate and distributed data. (As opposed to the alternative centralized model of "let’s just bring all the data into one single data repository”, which would be technically easier but is no more realistic than “let’s just bring all the websites into one single web host”.)

In a DRS-enabled world, tool builders don’t have to worry about where the data their tools operate on lives — they can count on DRS to give them access. And tool users only need to know which DRS server is managing the data they need, and whether they have permission to access it; they don’t have to worry about how to physically get access to, or (worse) make a copy of the data. For example, if I have appropriate permissions, I can run a pooled analysis where I run a single tool across data managed by different DRS servers, potentially in different locations.

The DRS API supports access to data objects, with each DrsObject representing a single opaque blob of bytes. Much content (e.g. VCF files) is well represented as a single atomic DrsObject. Some content, however (e.g. DICOM images) is best represented as a compound object consisting of a structured collection of atomic DrsObjects. In both cases, DRS isn't aware of the semantics of the objects it serves -- understanding those semantics is the responsibility of the applications that call DRS.

Common examples of compound objects in biomedicine include:

• BAM+BAI genomic reads, with a small index (the BAI object) to large data (the BAM object), each object using a well-defined file format.
• DICOM images, with a contents object pointing to one or more raw image objects, each containing pixels from different aspects of a single logical biomedical image (e.g. different z-coordinates)
• studies, with a single table of contents listing multiple objects of various types that were generated together and are meant to be processed together

As with atomic objects, DRS applications and servers are expected to agree on the semantics of compound objects using non-DRS mechanisms. The recommended best practice for representing a particular compound object type is:

1. Define a manifest file syntax, which contains the DRS IDs of the constituent atomic objects, plus type-specific information about the relationship between those constituents.
   • Manifest file syntax isn't prescribed by the spec, but we expect they will often be JSON files.
   • For example, for a BAM+BAI pair the manifest file could contain two key-value pairs mapping the type of each constituent file to its DRS ID.
2. Make manifest objects and their constituent objects available using standard DRS mechanisms -- each object is referenced via its own DRS ID, just like any other atomic object.
   • For example, for a BAM+BAI pair, there would be three DRS IDs -- one for the manifest, one for the BAM, and one for the BAI.
3. Document the expected client logic for processing compound objects of interest. This logic typically consists of using standard DRS mechanisms to fetch the manifest, parsing its syntax, extracting the DRS IDs of constituent objects, and using standard DRS mechanisms to fetch the constituents as needed.
   • In some cases the application will always want to fetch all of the constituents; in other cases it may want to initially fetch a subset, and only fetch the others on demand. For example, a DICOM image viewer may only want to fetch the layers that are being rendered.

DRS URIs are aligned with the FAIR data principles and the Joint Declaration of Data Citation Principles — both hostname-based and compact identifier-based URIs provide globally unique, machine-resolvable, persistent identifiers for data.

• We require all URIs to begin with drs:// as a signal to humans and systems consuming these URIs that the response they will ultimately receive, after transforming the URI to a fetchable URL, will be a DRS JSON packet. This signal differentiates DRS URIs from the wide variety of other entities (HTML documents, PDFs, ontology notes, etc.) that can be represented by compact identifiers.
• We support hostname-based URIs because of their simplicity and efficiency for server and client implementers.
• We support compact identifier-based URIs, and the meta-resolver services of identifiers.org and n2t.net (Name-to-Thing), because of the wide adoption of compact identifiers in the research community. as detailed by Wimalaratne et al (2018) in "Uniform resolution of compact identifiers for biomedical data."