logo
API docs by Redocly

Task Execution Service (1.1.0)

Download OpenAPI specification:Download

License: Apache 2.0

Executive Summary

The Task Execution Service (TES) API is a standardized schema and API for describing and executing batch execution tasks. A task defines a set of input files, a set of containers and commands to run, a set of output files and some other logging and metadata.

TES servers accept task documents and execute them asynchronously on available compute resources. A TES server could be built on top of a traditional HPC queuing system, such as Grid Engine, Slurm or cloud style compute systems such as AWS Batch or Kubernetes.

Introduction

This document describes the TES API and provides details on the specific endpoints, request formats, and responses. It is intended to provide key information for developers of TES-compatible services as well as clients that will call these TES services. Use cases include:

  • Deploying existing workflow engines on new infrastructure. Workflow engines such as CWL-Tes and Cromwell have extentions for using TES. This will allow a system engineer to deploy them onto a new infrastructure using a job scheduling system not previously supported by the engine.

  • Developing a custom workflow management system. This API provides a common interface to asynchronous batch processing capabilities. A developer can write new tools against this interface and expect them to work using a variety of backend solutions that all support the same specification.

Standards

The TES API specification is written in OpenAPI and embodies a RESTful service philosophy. It uses JSON in requests and responses and standard HTTP/HTTPS for information transport. HTTPS should be used rather than plain HTTP except for testing or internal-only purposes.

Authentication and Authorization

Is is envisaged that most TES API instances will require users to authenticate to use the endpoints. However, the decision if authentication is required should be taken by TES API implementers.

If authentication is required, we recommend that TES implementations use an OAuth2 bearer token, although they can choose other mechanisms if appropriate.

Checking that a user is authorized to submit TES requests is a responsibility of TES implementations.

CORS

If TES API implementation is to be used by another website or domain it must implement Cross Origin Resource Sharing (CORS). Please refer to https://w3id.org/ga4gh/product-approval-support/cors for more information about GA4GH’s recommendations and how to implement CORS.

TaskService

GetServiceInfo

Provides information about the service, this structure is based on the standardized GA4GH service info structure. In addition, this endpoint will also provide information about customized storage endpoints offered by the TES server.

Authorizations:
BearerAuthBasicAuth

Responses

Response samples

Content type
application/json
{
  • "id": "org.ga4gh.myservice",
  • "name": "My project",
  • "type": {
    },
  • "description": "This service provides...",
  • "organization": {},
  • "contactUrl": "mailto:support@example.com",
  • "documentationUrl": "https://docs.myservice.example.com",
  • "createdAt": "2019-06-04T12:58:19Z",
  • "updatedAt": "2019-06-04T12:58:19Z",
  • "environment": "test",
  • "version": "1.0.0",
  • "storage": [
    ],
  • "tesResources_backend_parameters": [
    ]
}

ListTasks

List tasks tracked by the TES server. This includes queued, active and completed tasks. How long completed tasks are stored by the system may be dependent on the underlying implementation.

Authorizations:
BearerAuthBasicAuth
query Parameters
name_prefix
string

OPTIONAL. Filter the list to include tasks where the name matches this prefix. If unspecified, no task name filtering is done.

state
string (tesState)
Default: "UNKNOWN"
Enum: "UNKNOWN" "QUEUED" "INITIALIZING" "RUNNING" "PAUSED" "COMPLETE" "EXECUTOR_ERROR" "SYSTEM_ERROR" "CANCELED" "PREEMPTED" "CANCELING"
Example: state=COMPLETE

OPTIONAL. Filter tasks by state. If unspecified, no task state filtering is done.

tag_key
Array of strings

OPTIONAL. Provide key tag to filter. The field tag_key is an array of key values, and will be zipped with an optional tag_value array. So the query:

  ?tag_key=foo1&tag_value=bar1&tag_key=foo2&tag_value=bar2

Should be constructed into the structure { "foo1" : "bar1", "foo2" : "bar2"}

  ?tag_key=foo1

Should be constructed into the structure {"foo1" : ""}

If the tag_value is empty, it will be treated as matching any possible value. If a tag value is provided, both the tag's key and value must be exact matches for a task to be returned. Filter Tags Match?

{"foo": "bar"} {"foo": "bar"} Yes {"foo": "bar"} {"foo": "bat"} No {"foo": ""} {"foo": ""} Yes {"foo": "bar", "baz": "bat"} {"foo": "bar", "baz": "bat"} Yes {"foo": "bar"} {"foo": "bar", "baz": "bat"} Yes {"foo": "bar", "baz": "bat"} {"foo": "bar"} No {"foo": ""} {"foo": "bar"} Yes {"foo": ""} {} No

tag_value
Array of strings

OPTIONAL. The companion value field for tag_key

page_size
integer <int32>

Optional number of tasks to return in one page. Must be less than 2048. Defaults to 256.

page_token
string

OPTIONAL. Page token is used to retrieve the next page of results. If unspecified, returns the first page of results. The value can be found in the next_page_token field of the last returned result of ListTasks

view
string
Default: "MINIMAL"
Enum: "MINIMAL" "BASIC" "FULL"

OPTIONAL. Affects the fields included in the returned Task messages.

MINIMAL: Task message will include ONLY the fields:

  • tesTask.Id
  • tesTask.State

BASIC: Task message will include all fields EXCEPT:

  • tesTask.ExecutorLog.stdout
  • tesTask.ExecutorLog.stderr
  • tesInput.content
  • tesTaskLog.system_logs

FULL: Task message includes all fields.

Responses

Response samples

Content type
application/json
{
  • "tasks": [
    ],
  • "next_page_token": "string"
}

CreateTask

Create a new task. The user provides a Task document, which the server uses as a basis and adds additional fields.

Authorizations:
BearerAuthBasicAuth
Request Body schema: application/json
name
string

User-provided task name.

description
string

Optional user-provided description of task for documentation purposes.

Array of objects (tesInput)

Input files that will be used by the task. Inputs will be downloaded and mounted into the executor container as defined by the task request document.

Array of objects (tesOutput)

Output files. Outputs will be uploaded from the executor container to long-term storage.

object (tesResources)

Resources describes the resources requested by a task.

required
Array of objects (tesExecutor)

An array of executors to be run. Each of the executors will run one at a time sequentially. Each executor is a different command that will be run, and each can utilize a different docker image. But each of the executors will see the same mapped inputs and volumes that are declared in the parent CreateTask message.

Execution stops on the first error.

volumes
Array of strings

Volumes are directories which may be used to share data between Executors. Volumes are initialized as empty directories by the system when the task starts and are mounted at the same path in each Executor.

For example, given a volume defined at /vol/A, executor 1 may write a file to /vol/A/exec1.out.txt, then executor 2 may read from that file.

(Essentially, this translates to a docker run -v flag where the container path is the same for each executor).

object

A key-value map of arbitrary tags. These can be used to store meta-data and annotations about a task. Example:

{
  "tags" : {
      "WORKFLOW_ID" : "cwl-01234",
      "PROJECT_GROUP" : "alice-lab"
  }
}

Responses

Request samples

Content type
application/json
{
  • "name": "string",
  • "description": "string",
  • "inputs": [
    ],
  • "outputs": [
    ],
  • "resources": {
    },
  • "executors": [
    ],
  • "volumes": [
    ],
  • "tags": {
    }
}

Response samples

Content type
application/json
{
  • "id": "string"
}

GetTask

Get a single task, based on providing the exact task ID string.

Authorizations:
BearerAuthBasicAuth
path Parameters
id
required
string

ID of task to retrieve.

query Parameters
view
string
Default: "MINIMAL"
Enum: "MINIMAL" "BASIC" "FULL"

OPTIONAL. Affects the fields included in the returned Task messages.

MINIMAL: Task message will include ONLY the fields:

  • tesTask.Id
  • tesTask.State

BASIC: Task message will include all fields EXCEPT:

  • tesTask.ExecutorLog.stdout
  • tesTask.ExecutorLog.stderr
  • tesInput.content
  • tesTaskLog.system_logs

FULL: Task message includes all fields.

Responses

Response samples

Content type
application/json
{
  • "id": "job-0012345",
  • "state": "COMPLETE",
  • "name": "string",
  • "description": "string",
  • "inputs": [
    ],
  • "outputs": [
    ],
  • "resources": {
    },
  • "executors": [
    ],
  • "volumes": [
    ],
  • "tags": {
    },
  • "logs": [
    ],
  • "creation_time": "2020-10-02T10:00:00-05:00"
}

CancelTask

Cancel a task based on providing an exact task ID.

Authorizations:
BearerAuthBasicAuth
path Parameters
id
required
string

ID of task to be canceled.

Responses

Response samples

Content type
application/json
{ }