diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..2f506c0 --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +.RData +.Rhistory diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 0000000..26e41af --- /dev/null +++ b/.gitmodules @@ -0,0 +1,9 @@ +[submodule "common-api-metadata"] + path = common-api-metadata + url = git@github.com:federated-data-sharing/common-api-metadata.git +[submodule "common-api-selection"] + path = common-api-selection + url = git@github.com:federated-data-sharing/common-api-selection.git +[submodule "common-api-tasks"] + path = common-api-tasks + url = git@github.com:federated-data-sharing/common-api-tasks.git diff --git a/README.md b/README.md index 6be2758..0ad6352 100755 --- a/README.md +++ b/README.md @@ -2,46 +2,76 @@ ## Introduction -This repository contains OpenAPI definitions for the Common API for Federated Data Sharing. The API was original developed to facilitate collaboration and trusted data sharing networks between trusted research environments and data repositories. +This repository contains OpenAPI definitions for the Common API for Federated Data Sharing. The Common API was developed to facilitate collaboration and trusted data sharing networks between trusted research environments and data providers. -The code is licensed under the [Mozilla Public License 2.0](https://www.mozilla.org/en-US/MPL/2.0/) see [LICENSE](./LICENSE). As more organisations are joining the effort, a new governance process will be established. In the meantime, please contact [Aridhia Informatics](https://www.aridhia.com/contact-our-team/) for more information. +## Documentation -## API overview +- [API Overview](./doc/API_Overview.md) +- [User Guide](./doc/User_Guide.md) +- [Origins](./doc/Origins.md) +- A separate repository provides [Worked examples](https://github.com/federated-data-sharing/common-api-examples) +- *Coming Soon* A reference implementation is being developed to facilitate the technical choices for data providers. -The federated data sharing API provides a set of endpoints required that provide a 'common' API to organisations wishing to participate in data sharing or federated analysis. There are three sections to the API: +## Summary benefits -- Metadata -- Selection -- Federated compute + + + + + +
-Note that the federated compute API section is being reviewed and will be added shortly. +**For Data providers** -The table below illustrates how different sections of the API could be opened up to support levels of sharing between a hub and a client (such as a user in a trusted Workspace). +A data provider may be an existing data repository or platform, or groups managing research data at their institutions. They have complex and varying data governance constraints and technical capabiliities which means that contributing data to research projects or more data sharing in a network may be difficult. -| Mode | Metadata | Selection & Filtering of record-level data | Federated compute on record level data. | -|:---------|:-----------------------------|:----------------------------------------------------|:-------------------------------------------------------| -| Level 0 | Can be queried and retrieved | Can be queried remotely and transferred to a client | Federation not required, computation happens at client | -| Level 1 | Can be queried and retrieved | Can be queried remotely and transferred to a client | Federation not required, computation happens at client | -| Level 2 | Can be queried and retrieved | Not permitted | Containerised computations can be executed remotely with
selection query input, approved results returned | +The Common API approach allows data providers to choose how they join a collaboration network. -Features: +- Level 0: transferring data directly for hosting to a trusted research environment (TRE) +- Level 1: providing remote access to data +- Level 2: providing remote computation on data held at source -- The API is defined in an [Open API specification](api/common_api.yml) -- API endpoints should be authenticated using OAuth tokens (out of band for this version) -- Selections are defined in [GraphQL](https://graphql.org/) as an abstraction over querying +Level 0 is provided by a Trusted Research Environment (TRE), while data providers must implement Level 1 or Level 2 using their own infrastructure. -Details of each endpoint: +Data providers are often in multiple collaborations at the same time. Investment in a Level 1 and Level 2 implementation can be repurposed for more than one network. + 		+**For Data users** + +A researcher or group of researchers working with multiple data sources have to navigate varying access mechanisms and APIs. By working in a network with data providers that implement the Common API, they can use their favourite tools to query, compute and analyse data in a consistent and efficient way. + +The Common API allows users to: + +- Find data and detailed metadata about available data sources +- Define selections and filters on data +- Retrieve record level data (Level 1) or compute over record level data using containerised scripts (Level 2) + +Currently the API is geared at users within a research team who can program. We expect in time that graphical user interfaces will be built or adapted that take advantage of the standard and reach a wider audience more directly. +
+ +## Partners + +The Common API is an open source co-development between a number of partner organisations + +[![ADDI logo](./doc/addi-logo.png "ADDI logo")](https://www.alzheimersdata.org/) +     +[![ICODA Research logo](./doc/icoda-research-logo.png "ICODA Research Logo")](https://www.icoda-research.org) +     +[![HDR UK logo](./doc/hdruk-logo.png "HDR UK Logo")](https://www.hdruk.ac.uk) + +[![Aridhia DRE logo](./doc/aridhia-dre-logo.png "Aridhia DRE Logo")](https://www.aridhia.com) + +## Acknowledgments + +The Common API gratefully builds on work from standardisation communities: + +- [World Wide Web Consortium (W3C)](https://www.w3.org/) +- [GraphQL Foundation](https://foundation.graphql.org/) +- [Global Alliance for Genomics and Health (GA4GH)](https://www.ga4gh.org/) +- [IETF](https://www.ietf.org/) OAuth Working Group - see https://oauth.net/2/ + +## Contributing + +The code is licensed under the [Mozilla Public License 2.0](https://www.mozilla.org/en-US/MPL/2.0/) see [LICENSE](./LICENSE). + +The project was [originally](./doc/Origins.md) part of an international collaboration on sharing data in clinical research. We now welcome contributions from a wider community. As more organisations are joining the effort, a new governance process will be established. In the meantime, please contact the [maintainers of the repository](mailto:info@fds-api.org). -|Endpoint |HTTP |Summary | -|:----------------------------------------------------|:------|:----------------------------------------------------------| -|`/datasets` |`GET` |Get a list of available datasets. Shows the list of all datasets available for querying. | -|`/datasets/{datasetid}` |`GET` |Get Catalogue entry (metadata) and Dictionaries (field descriptions) for dataset. Returns the catalogue metadata and a list of field descriptions for a specified dataset (by dataset ID). | -|`/datasets/{datasetid}/catalogue` |`GET` |Get Catalogue entry (metadata) for dataset. Returns the catalogue metadata for a specified dataset (by dataset ID). | -|`/datasets/{datasetid}/dictionaries` |`GET` |Get Dictionaries (field descriptions) for dataset. Returns a list of field descriptions for each table within a specified dataset (by dataset ID). | -|`/datasets/{datasetid}/dictionaries/{tableid}` |`GET` |Get a single dataset Dictionary for a specified table. Returns a set field descriptions for the specified table (by table ID) within a specified dataset (by dataset ID). | -|`/selection/validate` |`POST` |Validate a given selection query. With a simple GraphQL query, check whether the query is valid and corresponds to real fields at this location. | -|`/selection/beacon` |`POST` |Get a Beacon (T/F) for a specified data selection. With a simple Graph QL query, check which locations contain data relevant to a specific query. | -|`/selection/select` |`POST` |Perform a selection operation on a dataset. With a simple Graph QL query, returns the full selection of data in a JSON or .csv format. | -|`/selection/preview` |`POST` |Preview the results of a selection operation on a dataset. With a simple Graph QL query, returns a small sample of the selection in a JSON or .csv format. | -|`/selection/profile` |`POST` |Get a profile of a selection operation on a dataset. Returns a set of metrics for the given selection operation. | -|`/health_check` |`GET` |Get a health check of the service. | diff --git a/VERSION b/VERSION new file mode 100644 index 0000000..1c6f7de --- /dev/null +++ b/VERSION @@ -0,0 +1 @@ +1.1.0-alpha diff --git a/api/common_api.yaml b/api/common_api.yaml deleted file mode 100644 index d2a19b4..0000000 --- a/api/common_api.yaml +++ /dev/null @@ -1,626 +0,0 @@ -openapi: 3.0.0 - -info: - title: Common API for Federated Data Sharing - description: | - A set of APIs to support different levels of data sharing in a federated network. - The API has three parts Metadata, Selection and Compute. A site must implement the Metadata API, - and it can either implement the Selection API or the Compute API. In the case that the Compute API - is implemented, the Selection API must be implemented as input to compute tasks. Currently, the Compute API endpoints are omitted as they are under review. - version: "1.1.0" - -externalDocs: - description: Common API for Federated Data Sharing Github repository - url: https://github.com/federated-data-sharing/common-api - - -paths: - /datasets: - get: - summary: Get a list of available datasets. - description: Shows the list of all datasets available for querying. - responses: - "200": - description: OK - content: - application/json: - schema: - $ref: "#/components/schemas/dataset_list" - "401": - description: Unauthorized (401) - - tags: - - Metadata API - - - /datasets/{datasetid}: - get: - summary: Get Catalogue entry (metadata) and Dictionaries (field descriptions) for dataset. - description: Returns the catalogue metadata and a list of field descriptions for a specified dataset (by dataset ID). - - parameters: - - name: datasetid - in: path - description: Dataset ID - required: true - style: simple - explode: false - schema: - type: string - example: - "hospital_a_patient_results" - - responses: - "200": - description: OK - content: - application/json: - schema: - $ref: "#/components/schemas/dataset_information" - "401": - description: Unauthorized (401) - - tags: - - Metadata API - - - /datasets/{datasetid}/catalogue: - get: - summary: Get Catalogue entry (metadata) for dataset. - description: Returns the catalogue metadata for a specified dataset (by dataset ID). - - parameters: - - name: datasetid - in: path - description: Dataset ID - required: true - style: simple - explode: false - schema: - type: string - example: - "hospital_a_patient_results" - - responses: - "200": - description: OK - content: - application/json: - schema: - $ref: "#/components/schemas/DCAT_metadata" - "401": - description: Unauthorized (401) - "404": - description: Not Found (404) - Dataset not found. - - tags: - - Metadata API - - - /datasets/{datasetid}/dictionaries: - get: - summary: Get Dictionaries (field descriptions) for dataset. - description: Returns a list of field descriptions for each table within a specified dataset (by dataset ID). - - parameters: - - name: datasetid - in: path - description: Dataset ID - required: true - style: simple - explode: false - schema: - type: string - example: - "hospital_a_patient_results" - - responses: - "200": - description: OK - content: - application/json: - schema: - $ref: "#/components/schemas/data_dictionary_list" - "401": - description: Unauthorized (401) - "404": - description: Not Found (404) - Dataset not found. - - tags: - - Metadata API - - - /datasets/{datasetid}/dictionaries/{tableid}: - get: - summary: Get a single dataset Dictionary for a specified table. - description: Returns a set field descriptions for the specified table (by table ID) within a specified dataset (by dataset ID). - - parameters: - - name: datasetid - in: path - description: Dataset ID - required: true - style: simple - explode: false - schema: - type: string - example: - "hospital_a_patient_results" - - name: tableid - in: path - description: Table ID - required: true - style: simple - explode: false - schema: - type: string - example: - "table_1" - - responses: - "200": - description: OK - content: - application/json: - schema: - $ref: "#/components/schemas/data_dictionary" - "401": - description: Unauthorized (401) - "404": - description: Not Found (404) - Dataset not found. - - tags: - - Metadata API - - - /selection/validate: - post: - summary: Validate a given selection query. - description: With a simple GraphQL query, check whether the query is valid and corresponds to real fields at this location. - - requestBody: - description: GraphQL Query - content: - text/plain: - schema: - type: string - example: "{hospital_b_patient_results {table_1 {sex, age}}" - required: true - - responses: - 200: - description: OK - content: - application/json: - schema: - $ref: "#/components/schemas/selection_success" - 400: - description: Bad Request (400) - The server cannot process the request. Please check your request body. - 401: - description: Unauthorized (401) - 404: - description: Not Found (404) - Dataset not found. - - tags: - - Selection API - - - /selection/beacon: - post: - summary: Get a Beacon (T/F) for a specified data selection. - description: With a simple Graph QL query, check which locations contain data relevant to a specific query. - - requestBody: - description: GraphQL Query - content: - text/plain: - schema: - type: string - example: "{hospital_b_patient_results {table_1 {sex, age}}" - required: true - - responses: - 200: - description: OK - content: - application/json: - schema: - $ref: "#/components/schemas/selection_success" - 400: - description: Bad Request (400) - The server cannot process the request. Please check your request body. - 401: - description: Unauthorized (401) - 404: - description: Not Found (404) - Dataset not found. - - tags: - - Selection API - - - /selection/select: - post: - summary: Perform a selection operation on a dataset. - description: With a simple Graph QL query, returns the full selection of data in a JSON or .csv format. - - requestBody: - description: GraphQL Query - content: - text/plain: - schema: - type: string - example: "{hospital_b_patient_results {table_1 {sex, age}}" - required: true - - responses: - 200: - description: OK - content: - application/json: - schema: - type: object - application/gzip: - schema: - type: object - x-content-type: application/gzip - 400: - description: Bad Request (400) - The server cannot process the request. Please check your request body. - 401: - description: Unauthorized (401) - 404: - description: Not Found (404) - Dataset not found. - - tags: - - Selection API - - - /selection/preview: - post: - summary: Preview the results of a selection operation on a dataset. - description: With a simple Graph QL query, returns a small sample of the selection in a JSON or .csv format. - - requestBody: - description: GraphQL Query - content: - text/plain: - schema: - type: string - example: "{hospital_b_patient_results {table_1 {sex, age}}" - required: true - - responses: - 200: - description: OK - content: - application/json: - schema: - type: object - application/gzip: - schema: - type: object - x-content-type: application/gzip - 400: - description: Bad Request (400) - The server cannot process the request. Please check your request body. - 401: - description: Unauthorized (401) - 404: - description: Not Found (404) - Dataset not found. - - tags: - - Selection API - - - /selection/profile: - post: - summary: Get a profile of a selection operation on a dataset. - description: Returns a set of metrics for the given selection operation. - - requestBody: - description: GraphQL Query - content: - text/plain: - schema: - type: string - example: "{hospital_b_patient_results {table_1 {sex, age}}" - required: true - - responses: - 200: - description: OK - content: - application/json: - schema: - type: object - 400: - description: Bad Request (400) - The server cannot process the request. Please check your request body. - 401: - description: Unauthorized (401) - 404: - description: Not Found (404) - Dataset not found. - - tags: - - Selection API - - - /health_check: - get: - summary: Get a health check of the service. - responses: - 200: - description: OK - content: - application/json: - schema: - $ref: "#/components/schemas/health_check" - - tags: - - Health Check - - -components: - schemas: - dataset_list: - description: A list of dataset summaries. - required: - - datasets - type: object - properties: - datasets: - type: array - items: - $ref: "#/components/schemas/dataset_summary" - - example: - datasets: - - id: "hospital_a_patient_results" - dataset_name: "Hospital A Patient Results" - author: "John Smith" - - id: "medical_trial_b_results" - dataset_name: "Medical Trial B Results" - author: "Jane Jones" - - dataset_summary: - description: A summary of a dataset, including the dataset ID, the dataset name, and the author of the dataset. - required: - - id - type: object - properties: - id: - type: string - dataset_name: - type: string - author: - type: string - - example: - id: "hospital_a_patient_results" - dataset_name: "Hospital A Patient Results" - author: "John Smith" - - dataset_information: - description: A combined object of metadata from the catalogue entry and the dictionary field descriptions. - required: - - catalogue - - dictionaries - type: object - properties: - catalogue: - $ref: "#/components/schemas/DCAT_metadata" - dictionaries: - type: array - items: - $ref: "#/components/schemas/data_dictionary" - - DCAT_metadata: - description: A catalogue of the dataset metadata defined to the DCAT specifications. - required: - - id - type: object - properties: - id: - type: string - title: - type: string - description: - type: string - creator: - type: string - contactPoint: - type: string - publisher: - $ref: "#/components/schemas/DCAT_metadata_publisher" - license: - type: string - versionInfo: - type: string - additionalProperties: - type: object - - example: - id: "medical_trial_b_results" - title: "Medical Trail B Results" - description: "A description of the example medical trial and the data contained in this example." - creator: "Jane Jones" - contactPoint: "jane.jones@example.com" - publisher: - name: "Example Medical Trial Org" - url: "www.examplemedicaltrial.org" - license: "https://creativecommons.org/licenses/by/3.0/" - versionInfo: "1.0" - additionalProperties: {additional_tags: ["tag1", "tag2", "tag3"], extra_info: "Some additional information."} - - - DCAT_metadata_publisher: - description: An object containing the publisher name and url for the DCAT catalogue. - required: - - name - - url - type: object - properties: - name: - type: string - url: - type: string - - example: - name: "Example Medical Trial Org" - url: "www.examplemedicaltrial.org" - - data_dictionary_list: - description: A list of dataset dictionaries for a particular dataset. - type: object - properties: - dictionaries: - type: array - items: - $ref: "#/components/schemas/data_dictionary" - - example: - dictionaries: - - id: "table_1" - fields: - - name: "sex" - label: "Sex" - type: "text" - description: "Sex of patient, factor with levels (F or M)" - constraints: "SEX" - - name: "age_brc" - label: "Age" - type: "text" - description: "Age group, of the forms 18-25 (B1), 25-50 (B2), 50+ (B3)." - constraints: "AGE" - lookups: { - AGE: [{name: "B1", description: "Bracket 1 - 18-25"}, - {name: "B2", description: "Bracket 2 - 25-50"}, - {name: "B3", description: "Bracket 3 - 50+"}], - SEX: [{name: "F", description: "female"}, {name: "M", description: "male"}]} - - id: "table_2" - fields: - - name: "sex" - label: "Sex" - type: "text" - description: "Sex of patient, factor with levels (F or M)" - constraints: "SEX" - - name: "age_brc" - label: "Age" - type: "text" - description: "Age group, of the forms 18-25 (B1), 25-50 (B2), 50+ (B3)." - constraints: "AGE" - lookups: {AGE: [{name: "B1", description: "Bracket 1 - 18-25"}, - {name: "B2", description: "Bracket 2 - 25-50"}, - {name: "B3", description: "Bracket 3 - 50+"}], - SEX: [{name: "F", description: "female"}, {name: "M", description: "male"}]} - - data_dictionary: - description: An object containing the list of fields and lookups for a particular dataset table. - required: - - id - type: object - properties: - id: - type: string - fields: - type: array - items: - $ref: "#/components/schemas/data_dictionary_field" - lookups: - type: object - additionalProperties: - $ref: "#/components/schemas/data_dictionary_lookup" - - example: - id: "table_1" - fields: - - name: "sex" - label: "Sex" - type: "text" - description: "Sex of patient, factor with levels (F or M)" - constraints: "SEX" - - name: "age_brc" - label: "Age" - type: "text" - description: "Age group, of the forms 18-25 (B1), 25-50 (B2), 50+ (B3)." - lookups: {AGE: [{name: "B1", description: "Bracket 1 - 18-25"}, - {name: "B2", description: "Bracket 2 - 25-50"}, - {name: "B3", description: "Bracket 3 - 50+"}], - SEX: [{name: "F", description: "female"}, {name: "M", description: "male"}]} - - data_dictionary_field: - description: A description of a field entry in the dataset table. - required: - - name - - label - - type - type: object - properties: - name: - type: string - label: - type: string - type: - type: string - description: - type: string - constraints: - type: string - - example: - name: "sex" - label: "Sex" - type: "text" - description: "Sex of patient, factor with levels (F or M)" - constraints: "SEX" - - data_dictionary_lookup: - description: A list of lookups for field entries in the dataset dictionary. The items can be arbitrarily named but follow a defined structure. - type: array - items: - $ref: "#/components/schemas/data_dictionary_lookup_inner" - - data_dictionary_lookup_inner: - description: A lookup reference for field entries in the dataset dictionary. - required: - - name - - description - type: object - properties: - name: - type: string - description: - type: string - - example: - name: "F" - description: "female" - - selection_success: - description: A object containing a boolean response for whether an action was successful or not. - type: object - properties: - success: - type: boolean - example: - success: true - - health_check: - description: A response - type: object - properties: - version: - type: string - health_check: - type: boolean - example: - version: "1.1.0" - health_check: true - - - securitySchemes: - oAuth2Implicit: - type: oauth2 - flows: - implicit: - authorizationUrl: https://example.org/api/authorize - scopes: - api://example_id/read_api: allows reading resources - x-tokenInfoFunc: common_api_server.controllers.authorization_controller.check_oAuth2Implicit - x-scopeValidateFunc: common_api_server.controllers.authorization_controller.validate_scope_oAuth2Implicit diff --git a/api/task_execution.openapi.yaml b/api/task_execution.openapi.yaml deleted file mode 100644 index 357fd07..0000000 --- a/api/task_execution.openapi.yaml +++ /dev/null @@ -1,654 +0,0 @@ -openapi: 3.0.0 -info: - title: Task Execution Service - version: "0.4.0-fds-patch" - contact: - name: Susheel Varma - email: susheel.varma@hdruk.ac.uk - license: - name: MIT - description: GA4GH Task Execution Service (with FDS extenstions) -tags: - - Task Service - - name: TaskService -paths: - /tasks: - get: - summary: List Tasks - operationId: ListTasks - responses: - '200': - description: '' - content: - application/json: - schema: - $ref: '#/components/schemas/tesListTasksResponse' - parameters: - - name: name_prefix - description: |- - OPTIONAL. Filter the list to include tasks where the name matches this prefix. - If unspecified, no task name filtering is done. - in: query - required: false - schema: - type: string - - name: page_size - description: |- - OPTIONAL. Number of tasks to return in one page. - Must be less than 2048. Defaults to 256. - in: query - required: false - schema: - type: integer - format: int64 - - name: page_token - description: |- - OPTIONAL. Page token is used to retrieve the next page of results. - If unspecified, returns the first page of results. - See ListTasksResponse.next_page_token - in: query - required: false - schema: - type: string - - name: view - description: |- - OPTIONAL. Affects the fields included in the returned Task messages. - See TaskView below. - - - MINIMAL: Task message will include ONLY the fields: - Task.Id - Task.State - - BASIC: Task message will include all fields EXCEPT: - Task.ExecutorLog.stdout - Task.ExecutorLog.stderr - Input.content - TaskLog.system_logs - - FULL: Task message includes all fields. - in: query - required: false - schema: - type: string - enum: - - MINIMAL - - BASIC - - FULL - default: MINIMAL - description: 'List tasks.TaskView is requested as such: "v1/tasks?view=BASIC"' - tags: - - TaskService - post: - summary: Create a new task. - operationId: CreateTask - responses: - '200': - description: '' - content: - application/json: - schema: - $ref: '#/components/schemas/tesCreateTaskResponse' - requestBody: - content: - application/json: - schema: - $ref: '#/components/schemas/tesTask' - required: true - tags: - - TaskService - description: Create a new task. - /tasks/service-info: - get: - summary: Service Info - operationId: GetServiceInfo - responses: - '200': - description: '' - content: - application/json: - schema: - $ref: '#/components/schemas/tesServiceInfo' - tags: - - TaskService - description: 'GetServiceInfo provides information about the service,such as storage details, resource availability, and other documentation.' - '/tasks/{id}': - get: - summary: Get a task. - operationId: GetTask - responses: - '200': - description: '' - content: - application/json: - schema: - $ref: '#/components/schemas/tesTask' - parameters: - - name: id - in: path - required: true - schema: - type: string - - name: view - description: |- - OPTIONAL. Affects the fields included in the returned Task messages. - See TaskView below. - - - MINIMAL: Task message will include ONLY the fields: - Task.Id - Task.State - - BASIC: Task message will include all fields EXCEPT: - Task.ExecutorLog.stdout - Task.ExecutorLog.stderr - Input.content - TaskLog.system_logs - - FULL: Task message includes all fields. - in: query - required: false - schema: - type: string - enum: - - MINIMAL - - BASIC - - FULL - default: MINIMAL - tags: - - TaskService - description: 'Get a task. TaskView is requested as such: "v1/tasks/{id}?view=FULL"' - '/tasks/{id}/cancel': - post: - summary: Cancel a task. - operationId: CancelTask - responses: - '200': - description: '' - content: - application/json: - schema: - $ref: '#/components/schemas/tesCancelTaskResponse' - tags: - - TaskService - description: Cancel a task. - parameters: - - schema: - type: string - name: id - in: path - required: true - description: Task ID - /tasks/validate: - post: - summary: Validate a Task - tags: [] - responses: - '200': - description: OK - content: - application/json: - schema: - $ref: '#/components/schemas/tesValidateTaskResponse' - '400': - description: Bad Request - content: - application/json: - schema: - $ref: '#/components/schemas/tesCancelTaskResponse' - operationId: ValidateTask - requestBody: - content: - application/json: - schema: - type: object - properties: {} - application/xml: - schema: - $ref: '#/components/schemas/tesTask' - description: Validate a Task -servers: - - url: /ga4gh/tes/v1 -components: - schemas: - tesCancelTaskResponse: - type: object - description: CancelTaskResponse describes a response from the CancelTask endpoint. - readOnly: true - tesCreateTaskResponse: - type: object - properties: - id: - type: string - description: Task identifier assigned by the server. - description: CreateTaskResponse describes a response from the CreateTask endpoint. - readOnly: true - required: - - id - tesExecutor: - type: object - properties: - image: - type: string - description: |- - Name of the container image, for example: - ubuntu - quay.io/aptible/ubuntu - gcr.io/my-org/my-image - etc... - command: - type: array - items: - type: string - description: |- - A sequence of program arguments to execute, where the first argument - is the program to execute (i.e. argv). - workdir: - type: string - description: |- - The working directory that the command will be executed in. - Defaults to the directory set by the container image. - stdin: - type: string - description: |- - Path inside the container to a file which will be piped - to the executor's stdin. Must be an absolute path. - stdout: - type: string - description: |- - Path inside the container to a file where the executor's - stdout will be written to. Must be an absolute path. - stderr: - type: string - description: |- - Path inside the container to a file where the executor's - stderr will be written to. Must be an absolute path. - env: - type: object - additionalProperties: - type: string - description: Enviromental variables to set within the container. - description: 'Executor describes a command to be executed, and its environment.' - required: - - image - - command - tesExecutorLog: - type: object - properties: - start_time: - type: string - description: 'Time the executor started, in RFC 3339 format.' - end_time: - type: string - description: 'Time the executor ended, in RFC 3339 format.' - stdout: - type: string - description: |- - Stdout content. - - This is meant for convenience. No guarantees are made about the content. - Implementations may chose different approaches: only the head, only the tail, - a URL reference only, etc. - - In order to capture the full stdout users should set Executor.stdout - to a container file path, and use Task.outputs to upload that file - to permanent storage. - stderr: - type: string - description: |- - Stderr content. - - This is meant for convenience. No guarantees are made about the content. - Implementations may chose different approaches: only the head, only the tail, - a URL reference only, etc. - - In order to capture the full stderr users should set Executor.stderr - to a container file path, and use Task.outputs to upload that file - to permanent storage. - exit_code: - type: integer - format: int32 - description: Exit code. - description: ExecutorLog describes logging information related to an Executor. - required: - - exit_code - readOnly: true - tesFileType: - type: string - enum: - - FILE - - DIRECTORY - default: FILE - tesInput: - type: object - properties: - name: - type: string - description: - type: string - url: - type: string - description: |- - REQUIRED, unless "content" is set. - - URL in long term storage, for example: - s3://my-object-store/file1 - gs://my-bucket/file2 - file:///path/to/my/file - /path/to/my/file - etc... - path: - type: string - description: |- - Path of the file inside the container. - Must be an absolute path. - type: - $ref: '#/components/schemas/tesFileType' - content: - type: string - description: |- - File content literal. - Implementations should support a minimum of 128 KiB in this field and may define its own maximum. - UTF-8 encoded - - If content is not empty, "url" must be ignored. - description: Input describes Task input files. - required: - - type - - path - tesListTasksResponse: - type: object - properties: - tasks: - type: array - items: - $ref: '#/components/schemas/tesTask' - description: List of tasks. - next_page_token: - type: string - description: |- - Token used to return the next page of results. - See TaskListRequest.next_page_token - description: ListTasksResponse describes a response from the ListTasks endpoint. - required: - - tasks - readOnly: true - tesOutput: - type: object - properties: - name: - type: string - description: - type: string - url: - type: string - description: |- - URL in long term storage, for example: - s3://my-object-store/file1 - gs://my-bucket/file2 - file:///path/to/my/file - /path/to/my/file - etc... - path: - type: string - description: |- - Path of the file inside the container. - Must be an absolute path. - type: - $ref: '#/components/schemas/tesFileType' - description: Output describes Task output files. - required: - - url - - path - - type - tesOutputFileLog: - type: object - properties: - url: - type: string - description: 'URL of the file in storage, e.g. s3://bucket/file.txt' - path: - type: string - description: Path of the file inside the container. Must be an absolute path. - size_bytes: - type: string - format: int64 - description: Size of the file in bytes. - description: |- - OutputFileLog describes a single output file. This describes - file details after the task has completed successfully, - for logging purposes. - readOnly: true - required: - - url - - path - - size_bytes - tesResources: - type: object - properties: - cpu_cores: - type: integer - format: int64 - description: Requested number of CPUs - preemptible: - type: boolean - format: boolean - description: Is the task allowed to run on preemptible compute instances (e.g. AWS Spot)? - ram_gb: - type: number - format: double - description: Requested RAM required in gigabytes (GB) - disk_gb: - type: number - format: double - description: Requested disk size in gigabytes (GB) - zones: - type: array - items: - type: string - description: Request that the task be run in these compute zones. - description: Resources describes the resources requested by a task. - tesServiceInfo: - type: object - description: |- - ServiceInfo describes information about the service, - such as storage details, resource availability, - and other documentation. - readOnly: true - properties: - name: - type: string - description: 'Returns the name of the service, e.g. "ohsu-compbio-funnel".' - doc: - type: string - description: 'Returns a documentation string, e.g. "Hey, we''re OHSU Comp. Bio!".' - storage: - type: array - description: |- - Lists some, but not necessarily all, storage locations supported by the service. - - Must be in a valid URL format. - e.g. - file:///path/to/local/funnel-storage - s3://ohsu-compbio-funnel/storage - etc. - items: - type: string - registries: - type: - - string - - array - items: - type: object - properties: - registry_id: - type: string - registry_name: - type: string - registry_host: - type: string - registry_url: - type: string - tesState: - type: string - enum: - - UNKNOWN - - QUEUED - - INITIALIZING - - RUNNING - - PAUSED - - COMPLETE - - EXECUTOR_ERROR - - SYSTEM_ERROR - - CANCELED - default: UNKNOWN - description: |- - Task states. - - - UNKNOWN: The state of the task is unknown. - - This provides a safe default for messages where this field is missing, - for example, so that a missing field does not accidentally imply that - the state is QUEUED. - - QUEUED: The task is queued. - - INITIALIZING: The task has been assigned to a worker and is currently preparing to run. - For example, the worker may be turning on, downloading input files, etc. - - RUNNING: The task is running. Input files are downloaded and the first Executor - has been started. - - PAUSED: The task is paused. - - An implementation may have the ability to pause a task, but this is not required. - - COMPLETE: The task has completed running. Executors have exited without error - and output files have been successfully uploaded. - - EXECUTOR_ERROR: The task encountered an error in one of the Executor processes. Generally, - this means that an Executor exited with a non-zero exit code. - - SYSTEM_ERROR: The task was stopped due to a system error, but not from an Executor, - for example an upload failed due to network issues, the worker's ran out - of disk space, etc. - - CANCELED: The task was canceled by the user. - readOnly: true - tesTask: - type: object - properties: - id: - type: string - description: Task identifier assigned by the server. - readOnly: true - state: - $ref: '#/components/schemas/tesState' - name: - type: string - description: - type: string - inputs: - type: array - items: - $ref: '#/components/schemas/tesInput' - description: |- - Input files. - Inputs will be downloaded and mounted into the executor container. - outputs: - type: array - items: - $ref: '#/components/schemas/tesOutput' - description: |- - Output files. - Outputs will be uploaded from the executor container to long-term storage. - resources: - $ref: '#/components/schemas/tesResources' - executors: - type: array - items: - $ref: '#/components/schemas/tesExecutor' - description: |- - A list of executors to be run, sequentially. Execution stops - on the first error. - volumes: - type: array - items: - type: string - description: |- - Volumes are directories which may be used to share data between - Executors. Volumes are initialized as empty directories by the - system when the task starts and are mounted at the same path - in each Executor. - - For example, given a volume defined at "/vol/A", - executor 1 may write a file to "/vol/A/exec1.out.txt", then - executor 2 may read from that file. - - (Essentially, this translates to a `docker run -v` flag where - the container path is the same for each executor). - tags: - type: object - additionalProperties: - type: string - description: A key-value map of arbitrary tags. - logs: - type: array - items: - $ref: '#/components/schemas/tesTaskLog' - description: |- - Task logging information. - Normally, this will contain only one entry, but in the case where - a task fails and is retried, an entry will be appended to this list. - readOnly: true - creation_time: - type: string - description: |- - Date + time the task was created, in RFC 3339 format. - This is set by the system, not the client. - readOnly: true - description: Task describes an instance of a task. - required: - - executors - tesTaskLog: - type: object - properties: - logs: - type: array - items: - $ref: '#/components/schemas/tesExecutorLog' - description: Logs for each executor - metadata: - type: object - additionalProperties: - type: string - description: Arbitrary logging metadata included by the implementation. - start_time: - type: string - description: 'When the task started, in RFC 3339 format.' - end_time: - type: string - description: 'When the task ended, in RFC 3339 format.' - outputs: - type: array - items: - $ref: '#/components/schemas/tesOutputFileLog' - description: |- - Information about all output files. Directory outputs are - flattened into separate items. - system_logs: - type: array - items: - type: string - description: |- - System logs are any logs the system decides are relevant, - which are not tied directly to an Executor process. - Content is implementation specific: format, size, etc. - - System logs may be collected here to provide convenient access. - - For example, the system may include the name of the host - where the task is executing, an error message that caused - a SYSTEM_ERROR state (e.g. disk is full), etc. - - System logs are only included in the FULL task view. - description: TaskLog describes logging information related to a Task. - required: - - logs - - outputs - readOnly: true - tesValidateTaskResponse: - title: tesValidateTaskResponse - type: object - properties: - success: - type: string - enum: - - 'true' - - 'false' diff --git a/common-api-metadata b/common-api-metadata new file mode 160000 index 0000000..55e3309 --- /dev/null +++ b/common-api-metadata @@ -0,0 +1 @@ +Subproject commit 55e33092a97779e11d90fedb5736a86917517472 diff --git a/common-api-selection b/common-api-selection new file mode 160000 index 0000000..f0c6207 --- /dev/null +++ b/common-api-selection @@ -0,0 +1 @@ +Subproject commit f0c6207dcc7b05f97143957454029e4af07d3f03 diff --git a/common-api-tasks b/common-api-tasks new file mode 160000 index 0000000..021cb73 --- /dev/null +++ b/common-api-tasks @@ -0,0 +1 @@ +Subproject commit 021cb7382470b7ae1a42e00c2d01a7b0484336e9 diff --git a/doc/API_Overview.md b/doc/API_Overview.md new file mode 100644 index 0000000..e345e47 --- /dev/null +++ b/doc/API_Overview.md @@ -0,0 +1,73 @@ +# API overview + +> Back to the main [README](../README.md) + +## Overview + +The federated data sharing Common API establishes an open standard for data platforms to participate in a open and closed data sharing networks. It speciies a set of endpoints required that provide a 'common' API to organisations wishing to participate in data sharing or federated analysis. Data sharing agreements are diverse and we need to remove barriers for data sharing amongst data controllers. This approach is intended to: + +- Clarity and transparency of the model in a complex ecosystem +- Accelerate availability of data for research +- Devolve the decision-making and governance to the appropriate level. +- Encourage convergence of existing (proprietary or niche) efforts +- Encourage an ecosystem of tools & syndication + +By adopting the API, a data provider and their network can implement “connector” layer once, join multiple networks. Our approach asks data controllers to self-select at what ‘level‘ they can join the network, mainly dependent on what they are permitted to do with data in their custody: + +| Mode | Metadata | Selection & Filtering of record-level data | Federated compute on record level data. | +|:--------------|:-----------------------------|:----------------------------------------------------|:-------------------------------------------------------| +| Level 0 | Can be queried and retrieved | Can be queried remotely and transferred to a client | Federation not required, computation happens at client | +| Level 1 | Can be queried and retrieved | Can be queried remotely and transferred to a client | Federation not required, computation happens at client | +| Level 2 | Can be queried and retrieved | Not permitted | Containerised computations can be executed remotely with
selection query input, approved results returned | + +## Open Standards + +Rather than reinventing the wheel, the Common API **adopts and adapts** existing standards efforts + +- The API is defined Open API specifications. +- API endpoints should be authenticated using OAuth2 (will be mandated in future versions) +- Descriptive metadata is defined in a variant of the [W3C DCAT](https://www.w3.org/TR/vocab-dcat-2/) standard and a simple data dictionary model. +- Selections are defined in [GraphQL](https://graphql.org/) as an abstraction over querying, selection and filtering +- Federated computations are defined in a variant of the [GA4GH Task Execution Service (TES) API](http://ga4gh.github.io/task-execution-schemas/) + +> Note: Field-level metadata (data dictionaries) are defined in a simple, pragmatic data model - existing partners aim to define or adopt a more robust community standard. + +## API modularity + +There are three sections to the API: + +| Section | Repository | Level 0 | Level 1 | Level 2 | +|:------------------|:--------------------------------------------------------------------------------------|---------|---------|---------| +|Metadata |[common-api-metadata](https://github.com/federated-data-sharing/common-api-metadata) | Yes | Yes | Yes | +|Selection |[common-api-selection](https://github.com/federated-data-sharing/common-api-selection) | N/A | Yes | Yes ** | +|Federated compute |[common-api-tasks](https://github.com/federated-data-sharing/common-api-tasks) | N/A | N/A | Yes | + +> \*\* Level 2 sites must implement the selection API "behind the scenes" to provide compute tasks with the selection required. + +For maximum flexibility each section of the Common API is defined in separate git submodules and repositories. In this way, sites can implement combinations as required or desirable in their particular setting.The table below illustrates how different sections of the API could be opened up to support levels of sharing between a hub and a client (such as a user in a trusted Workspace). + +## Endpoints + +Details of each endpoint: + +|Endpoint |HTTP | Payload | Result | Summary | +|:----------------------------------------------------|:------|:------------|:----------|:----------------------------------------------------------| +|`/datasets` |`GET` | N/A | JSON | Get a list of available datasets. Shows the list of all datasets available for querying. | +|`/datasets/{datasetid}` |`GET` | N/A | JSON | Get Catalogue entry (metadata) and Dictionaries (field descriptions) for dataset. Returns the catalogue metadata and a list of field descriptions for a specified dataset (by dataset ID). | +|`/datasets/{datasetid}/catalogue` |`GET` | N/A | DCAT JSON | Get Catalogue entry (metadata) for dataset. Returns the catalogue metadata for a specified dataset (by dataset ID). | +|`/datasets/{datasetid}/dictionaries` |`GET` | N/A | Dictionary JSON | Get Dictionaries (field descriptions) for dataset. Returns a list of field descriptions for each table within a specified dataset (by dataset ID). | +|`/datasets/{datasetid}/dictionaries/{tableid}` |`GET` | N/A | Dictionary JSON | Get a single dataset Dictionary for a specified table. Returns a set field descriptions for the specified table (by table ID) within a specified dataset (by dataset ID). | +|`/selection/validate` |`POST` | GraphQL | JSON | Validate a given selection query. With a simple GraphQL query, check whether the query is valid and corresponds to real fields at this location. | +|`/selection/beacon` |`POST` | GraphQL | JSON | Get a Beacon (T/F) for a specified data selection. With a simple Graph QL query, check which locations contain data relevant to a specific query. | +|`/selection/select` |`POST` | GraphQL | JSON - data selected | Perform a selection operation on a dataset. With a simple Graph QL query, returns the full selection of data in a JSON or .csv format. | +|`/selection/preview` |`POST` | GraphQL | JSON - data preview | Preview the results of a selection operation on a dataset. With a simple Graph QL query, returns a small sample of the selection in a JSON or .csv format. | +|`/selection/profile` |`POST` | GraphQL | JSON - summary | Get a profile of a selection operation on a dataset. Returns a set of metrics for the given selection operation. | +|`/tasks/service-info` |`GET` | N/A | JSON | Get service information about the service,such as storage details, resource availability, and other documentation| +|`/tasks` |`GET` | N/A | JSON | Get a list of of tasks for the current user| +|`/tasks` |`POST` | Task spec | JSON - with task ID | Create a new task using a task specification (links a selection query and containerised computation task)| +|`/tasks/validate` |`POST` | Task spec | JSON |Validate a task specification| +|`/tasks/{task_id}` |`GET` | N/A | JSON - task details | Get task details including status. If available, includes a link to the output of the task| +|`/tasks/{task_id}/cancel` |`POST` | N/A | JSON - task status |Cancel a task| +|`/health_check` |`GET` | N/A | JSON | Get a health check of the service. | + +> Note that the following endpoints are experimental at version 1.1: `/selection/beacon`, `/selection/preview` and `/selection/profile` - they are expected to be firmed up in later versions. diff --git a/doc/FDS_Strawman_Architecture_Sketch.png b/doc/FDS_Strawman_Architecture_Sketch.png new file mode 100755 index 0000000..b803305 Binary files /dev/null and b/doc/FDS_Strawman_Architecture_Sketch.png differ diff --git a/doc/Origins.md b/doc/Origins.md new file mode 100755 index 0000000..c689325 --- /dev/null +++ b/doc/Origins.md @@ -0,0 +1,45 @@ +# Origins - Federated Data Sharing Common API + +> Back to the main [README](../README.md) + +## Introduction + +This API originated in a strawman implementation of a federated data sharing API for an international collaboration on data sharing and federated compute. It began with support for a project focused on Alzheimer's and other dementias, [launched in November 2020](https://www.alzheimersdata.org/news/addi-press-release) as the [Alzheimers Disease Data Initiative (ADDI)](https://www.alzheimersdata.org). The objective here was to be able to analyse data in a consistent way, even if some of the of interest could not travel from a data repository. The successful [pilot](https://www.alzheimersdata.org/ad-workbench/pilot-phase) brought together leading groups in the AD research community including [Dementias Platform UK](https://www.dementiasplatform.uk/) and their databank at [UKSerp, Swansea University](https://serp.ac.uk/serp-uk/), [Critical Path Institute](https://c-path.org/) and [GAAIN](http://www.gaain.org/) working with technical partners including [Aridhia Informatics](https://www.aridhia.com). + +Since the initial pilot, the API has also been picked up for use by the [International COVID-19 Data Alliance](https://icoda-research.org/) as it builds out it's initiative and is under also investigation for usage by [Health Data Research UK](https://www.hdruk.ac.uk/) as both look to increase data usage and sharing in a controlled fashion, supporting the [UK ONS five safes model](https://blog.ons.gov.uk/2017/01/27/the-five-safes-data-privacy-at-ons/). We expect the API to evolve rapidly throughout 2021 as it's usage scales across these initiatives and we warmly welcome new collaborators & participants. + +This document summarises the approach that led to the Common API. + +> Please [get in touch](mailto:info@fds-api.org) if you would like to learn more and participate in this effort. + +## Approach + +Imagine a narrative for a research user who needs to access data from multiple data repositories, each with their own systems and processes. What is their user experience flow, how is that supported by underlying APIs provided by data platforms? Could that situation be improved through a Common API? + +For the implementation it should not be too important what their study is about - it might impact researcher accreditation and the type of UX we might want to provide for selection & filtering but not for the generic, Common API. + +The ecosystem is diverse. Different participating data platforms can support different levels of data sharing. Ultimately the default level of sharing is a property of the *dataset* not the data platform or the hub. + +### Levels of Sharing + +The system envisages a network of nodes (data nodes, data provider platforms/data platforms) that participate in data sharing at different levels. Architecturally above the nodes sites another network of nodes (hubs) that form a collaboraiton network. In the simplest model, a single hub is a gateway to one or more data nodes. In the real world, multiple hubs are required to deal with legal boundaries for data hosting of healthcare data. + +This results in different levels of sharing that coexist. + +- Level 0: Data hosted in hubs, can be found by metadata, and can be shared directly then linked and analysed at the user's hub (may need to be de-identified) +- Level 1: Data hosted in data nodes, and can be selected from, then linked and analysed in the user's hub +- Level 2: Data hosted in data nodes, and can be computed on at source then derived data is shared and analysed in the user's hub + +## Overview/Architecture + +With this project, an organisation can deploy a prototype instance of the Common API as a Docker container and participate as a 'node' in a network of federated data sharing. Below is the original sketch architecture, in the context of adding federated data sharing to Aridhia's Workspace as part of a collaboration project: + +![Strawman Architecture Sketch](FDS_Strawman_Architecture_Sketch.png) + +A client system communicates with nodes using secure web connections and can implement whatever user workflow makes sense for user case.Endpoints are only provided over HTTPS. Calls to bare HTTP are redirect to HTTPS by the node. Nodes must use valid certificates. + +Users or more generally, clients, are authorised using the OAuth2 system - it's up to the implementation what identity providers it accepts. Authentication and Authorisation is not role-based in the current prototype so all API end points are accessible. In some cases, the node may return a 'Not Implemented' error (HTTP code 501) if the end point is not supported. + +Nodes can store data as they wish. In the prototype implementation, files are used. Either the embedded 'data' folder is used or an external folder can be mounted on the container, allowing a separate files ystem to be used. Nodes can implement their own storage layer by adapting the prototype server or completely re-implementing the server using technologies of choice. + +The API supports finding data, selecting and computing on data. All nodes must support the catalogue and dictionary APIs ('/dataset'), and can support a subset of the selection end points ('/selection'). For example, nodes could support the beacon or sample select endpoints, but not the endpoints that would return data (e.g. '/selection/select'). In this case they should (but are not obliged to) implement the federated compute API. Specifically, if a node does *not* support the select endpoints and they support the federated compute endpoints, the user can assume that the selection API is provided to specification in a container context. This means that the same selection would work whether the user was selecting the data remotely and computing over it remotely, or they submitted a compute job to be run locally near the data. \ No newline at end of file diff --git a/doc/User_Guide.md b/doc/User_Guide.md new file mode 100755 index 0000000..216e2bb --- /dev/null +++ b/doc/User_Guide.md @@ -0,0 +1,30 @@ +# User Guide + +> Back to the main [README](../README.md) + +## Contents + +- Introduction - this file +- [Background_Concepts](./User_Guide_Background.md) +- Understanding [Analysis plans](./User_Guide_Analysis_Plans.md) in a federated setting +- [Containerising scripts](./User_Guide_Containerising_Tasks.md) +- [Key Payloads](./User_Guide_Key_Payloads.md) +- [Command Line](./User_Guide_CLI.md) - e.g. with `curl` +- [Python](./User_Guide_Python.md) +- [R](./User_Guide_R.md) + +A separate repository provides [Worked examples](https://github.com/federated-data-sharing/common-api-examples) + +## Introduction + +The main purpose of the Federated Data Sharing Common API is to support analysis of multiple data sets while allowing a data owner (custodian, controller) to control how data is exposed to the analysis. A set of APIs allow users to query metadata, select data and perform computation on data held remotely. This is intended to support a cycle where you can: + +- get field-level type and content metadata for remote data +- define a selection (pick fields) or filter (pick rows) you want to analyse +- execute some computation on the selection + +In Level 1 federated data sharing, you may have access to select row level data whereas in Level 2 you will only be able to specify your selection and computation task and have both executed remotely. This means that your analysis plan needs to adapt to the protocol. + +## Next steps + +Understand the [Background concepts](./User_Guide_Background.md) to enable you to use the API effectively. \ No newline at end of file diff --git a/doc/User_Guide_Analysis_Plans.md b/doc/User_Guide_Analysis_Plans.md new file mode 100644 index 0000000..509b299 --- /dev/null +++ b/doc/User_Guide_Analysis_Plans.md @@ -0,0 +1,70 @@ +# Analysis plans in a federated setting + +> Back to the [main page](./User_Guide.md) + +## Getting started + +As a scientist, statistician or data scientist, you may be used to developing scripts working directly with data - your `R` or `python` scripts can load the data directly, or maybe you're using a spreadsheet package or a statistical tool which wraps up the analysis plan for you. If you're using federated analysis this may not be an option for you - the data is held remotely. Your interaction with the data will be via the programming interface ("API") or a tool that uses it. Whereas you would normally expect a high degree of iteration (or "trial and error") when working with local data, you need to plan for a different form of iteration. + +## Trade-offs + +Most existing clinical research and statistics as well as the supporting libraries, assumes data is adjacent to scripts. You will be using a federated model because data providers cannot share data directly with you. As a result, in a federated setting, your analysis plan will need to be adapted in a few ways: + +- Adopt a “scatter/gather” approach across partitioned, separate data silos +- Package up some of the analysis into Docker containers and queries +- Develop scripts to orchestrate the whole thing for reproducibility + +Data platforms that implement the Common API commit to helping reduce the friction of remote access in a number of ways intended to help you as a researcher: + +- Implementing a *standard* API, which means you don't have to keep learning new ways of getting metadata or processing their data +- Providing detailed and up-to-date metadata on data they share through the API, giving you enough detail to adapt your analysis to what data is really there +- Letting you run *your* code on the data through the use of [docker containers](https://www.docker.com/resources/what-container) +- Supporting iterative sessions - where you can run your analysis as often as needed to answer your research questions (within some fair usage limits) + +In return, you should revisit your analysis plan and structure it to the remote arrangement. You may want to consider: + +- whether you will be interacting with one or more than one remote site (also known as a "node") +- whether you will be interacting with a mix of nodes at different levels of sharing (also known as "Level 0", "Level 1" or "Level 2" nodes) +- setting a number of stages or phases for your analysis in order to get early feedback on the process and build trust in the data and your connection with the remote sites. +- whether integrating data from multiple sources is important for your analysis algorithm (for example to develop a machine learning model) + +> Over time, we expect community efforts to provide standard modules and distributed versions of algorithms. See [Worked Examples](https://github.com/federated-data-sharing/common-api-examples) for some starting points. + +## What can you run in a federated model? + +Examples of staging or phasing analysis might reflect a standard research or data science life cycle: + +- exploratory analysis: systematically validating, summarising or exploring the remote data early on in order to define the main statistical analysis +- quality checks, outlier analysis +- sampling +- visualisation +- data engineering or integration (or at least creating a standard data frame for analysis) +- statistical tests +- modelling data +- validating models + +A modular and composable approach to your analysis code will allow you to iterate at each stage. This should reduce frustrations with not having direct access to data. It provides some benefit later when trying to package up your work as reproducible research. + +## Defining a selection + +All remote sites must provide field level metadata for data you have access to. This allows you to define a selection query. In a simple example this could define: + +- what table you want to read from +- what fields within the table you want to analyse + +Selections are defined using a standard called [GraphQL](https://graphql.org/). With this you can express a selection in relatively simple terms. + +For example, we can imagine a table called `virtual_cohort` with fields that include `family_history` (of a disease like Alzheimer's), `age` (in years) and `apoe` (a biomarker for the disease). If you just wanted to to analyse `age` and `apoe` the selection would look like this: +``` +{ + virtual_cohort { + age + apoe + } +} +``` +More details of how define GraphQL queries can be found on the [community pages](https://graphql.org/learn/) - bear in mind for this use case, we only need to know about selection *queries* and can ignore schemas and mutations. + +## Next steps + +Understand [Containerising scripts](./User_Guide_Containerising_Tasks.md) \ No newline at end of file diff --git a/doc/User_Guide_Background.md b/doc/User_Guide_Background.md new file mode 100644 index 0000000..afc300d --- /dev/null +++ b/doc/User_Guide_Background.md @@ -0,0 +1,58 @@ +# Background Concepts + +> Back to the [main page](./User_Guide.md) + +## History + +Federated Analysis and data sharing is a strategy to network data platforms and users. See the [Origins](Origins.md) for some of the background to this specific design. + +## Terminology + +- site - a data repository implementing the API +- client - a user's programme interacting with the API +- container - a Docker container encapsulating the analysis or computation require + +## High-level workflow + +The Common API provides a standard interface for data sharing protocols to connect between a user or client programme and a site implementing the API. + +Once you have obtained credentials from a site, you interact using the protocol of the Common API. The protocol is very lightweight and clients (users) are free to call the API in any sequence but we expect the following typical sequence of events: + +1. Discovering data by inspecting **metadata** +2. Defining field selections and filters to **select** +3. Selecting data directly for centralised analysis or using federated **tasks** (computation) to process a selection +4. **Combining results** from analysing data at source to produce a final report (e.g. a chart). + +## Understanding Federated sites + +Sites are also free to implement the API as they wish but there are some conventions expected. We expect sites to implement in one of two modes: + +- Level 1: users can connect to the API and select data which can be downloaded directly. This may be suitably de-identified: + + - metadata API - implemented, externally accessible + - selection API - implemented, externally accessible + - task API - not required + +- Level 2: since data cannot be shared directly, the selection API is not available externally, but is still used to define the selection to be analysed using the task API: + + - metadata API - implemented, externally accessible + - selection API - implemented, only available within task protocol + - task API - implemented, externally accessible + +For more details, see the [API Overview](./API_Overview.md) + +> Note that in most cases you should expect the output of your computation to be "quarantined" and reviewed for disclosure risk or similar criteria by the data provider. + +## Accessing the API + +The API is a RESTful standard web-based programming interface, and the user can select whatever client language or tool that they want. We have tested using `curl`, `python` and `R` as well as graphical clients like Postman. We assume the user is familiar with programmatic access to [Web API](https://en.wikipedia.org/wiki/Web_API) endpoints. + +We assume the user has been provided credentials to obtain a bearer token for API requests. The method for providing a token is not currently part of the specification but in a typical [OAuth](https://en.wikipedia.org/wiki/OAuth) model, a user is provided a client ID and client secret (password). Using those credentials, they call an API endpoint and obtain a token. This token is added as a header on subsequent calls. The token is intended to provide authentication AND authorisation. Sites are free to change the output of API calls based on what the individual user is authorised. + +Examples below are provided in `R`, `python` and `curl` in a Linux environment or similar. + +> Note that for simplicity and portability, code should be developed in Linux compatible environments. + +## Next Steps + +Understanding the [Key Payloads](./User_Guide_Key_Payloads.md) for API calls. \ No newline at end of file diff --git a/doc/User_Guide_CLI.md b/doc/User_Guide_CLI.md new file mode 100644 index 0000000..36da7c5 --- /dev/null +++ b/doc/User_Guide_CLI.md @@ -0,0 +1,89 @@ +# Command line - using curl and jq + +> Back to the [main page](./User_Guide.md) + +## Getting started + +Using `curl` and `jq` on the command line is a low level way to interact with the API in shell scripts. + +We assume the API is accessible at an endpoint `FDS_ENDPOINT`. For example, if you run the reference implementation, this will be: + +```sh +FDS_ENDPOINT="https://localhost:8443/federated-data-sharing/1.1.0" +``` + +We use `curl_opts` to set some useful options. For example in the reference implementation the certificate (for `https`) is self-signed so should not be checked. Set this option: + +```sh +curl_opts="-k" +``` + +To get a token, use the endpoint provided by the site. For example, the following is an example of how to retrieve a token + +```sh +token=`curl $curl_opts --user "$API_USER:$API_PASS"\ + -d grant_type=client_credentials -X POST\ + "$FDS_ENDPOINT/auth/connect/token" | jq -r '.access_token'` +``` + +## Data Discovery + +From there get a list of datasets: +```sh +curl $curl_opts -H "Authorization: Bearer $token"\ + -H "Accept: application/json"\ + "$FDS_ENDPOINT/datasets" | jq +``` + +To pick the first dataset: +```sh +dataset_id=`curl $curl_opts -X GET -H "Accept: application/json"\ + "$FDS_ENDPOINT/datasets"\ + | jq -r '.datasets[] | .id' | head -1` +echo $dataset_id +``` + +You can then get the catalogue for that dataset... +```sh +curl $curl_opts -X GET -H "Accept: application/json"\ + "$FDS_ENDPOINT/datasets/$dataset_id/catalogue" | jq +``` + +... and then get the dictionary for that dataset - note that there may multiple 'tables' within the dataset, with individual dictionaries. +```sh +curl $curl_opts -X GET -H "Accept: application/json"\ + "$FDS_ENDPOINT/datasets/$dataset_id/dictionaries"\ + | jq -r '.dictionaries[].id' +``` + +## Data selection + +Selection currently uses [GraphQL](https://graphql.org/) as a format for defining selections and filters on data. An example is provided in [src/examples/example-query.graphql](src/examples/example-query.graphql). This is intended as an abstraction from underlying query mechanisms such as SQL. + +The API is intended to be incremental: a user can define a query based on the data discovery stage. This query can be validated, then executed to different levels. The API may be implemented to support different levels or none (if Federated compute is the only approved method for selection and analysis). + +To validate a query such as the example query provided, we can post the JSON body: +```sh +curl $curl_opts -X POST\ + -H "Authorization: Bearer $token" -H "Accept: application/json"\ + -H "Content-Type: text/plain" --data @src/examples/example-query.graphql\ + "$FDS_ENDPOINT/selection/validate" | jq +``` + +And then select using that same query: +```sh +curl $curl_opts -X POST\ + -H "Authorization: Bearer $token" -H "Accept: application/json"\ + -H "Content-Type: text/plain" --data @src/examples/example-query.graphql\ + "$FDS_ENDPOINT/selection/select" | jq +``` +The other endpoints `/selection/beacon`, `/selection/preview`, `/selection/profile` work in the same way. + +### Tasks using federated computation + +> TODO - pending updating the task API + +## Next steps + +- Back to the [main page](./User_Guide.md) +- Look at some [Worked examples](https://github.com/federated-data-sharing/common-api-examples) diff --git a/doc/User_Guide_Containerising_Tasks.md b/doc/User_Guide_Containerising_Tasks.md new file mode 100644 index 0000000..742770c --- /dev/null +++ b/doc/User_Guide_Containerising_Tasks.md @@ -0,0 +1,40 @@ +# Containerising a script as a federated compute task + +> Back to the [main page](./User_Guide.md) + +## Getting started + +In order to execute a remote computation task, your analysis code must be packaged up in a docker container. + +> For simplicity we use the word "script" for this code as this is common data science but your container could be any program that can be run in a docker container, including complex programmes with library or package dependencies all encapsulated within the container. + +## Workflow + +In order to containerise the script, it is recommended to go through the following steps: + +1. Run the script on dummy or synthetic data on the command line on your local machine +2. Package up the script in a container and run using local docker installation, via command line on your local machine +3. Run the containerised script on one or more remote site via Federated Data Sharing API + +![Developing a containerised script](./sketch_process.jpg) + +A set of conventions set out how you should expect inputs to your containers or outputs of your computation to be handled. In the base, simple case your script should *read* one or more input CSV files corresponding to your selection query from a specified folder (`/mnt/input`) which may be read-only. Your script is then able to *write* one or more files to a specified folder (`/mnt/ouput`). + +![Reading and writing from the container](./sketch_docker.jpg) + +Federated analysis can be run on more than one node or site. When running a federated task in that more complex scenario, the same container might be used across multiple sites, or a selection filter might process only some of the data available at a given site. In this case it is important to consider a two stage process: + +- distribute the code to each site and obtain intermediate results +- combine data in a post-processing step + +The 'post-processing' step is what produced the final output you are aiming for: + +![Overview](./sketch_full.jpg) + +## Next steps + +Try the API using: + +- [Command Line](./User_Guide_CLI.md) tools - e.g. with `curl` +- [Python](./User_Guide_Python.md) +- [R](./User_Guide_R.md) diff --git a/doc/User_Guide_Key_Payloads.md b/doc/User_Guide_Key_Payloads.md new file mode 100644 index 0000000..acc6745 --- /dev/null +++ b/doc/User_Guide_Key_Payloads.md @@ -0,0 +1,80 @@ +## Key payloads + +> Back to the [User Guide](./User_Guide.md) + +The metadata API is a read-only API to discover and navigate what data might be available at a site. The API uses specific payloads to define tasks or selections. These can be constructed programmatically or in files passed to commands and libraries. + +Selections are currently defined in [GraphQL](https://graphql.org/) and posted with `Content-type: plain-text`. This was chosen to abstact from specific query languages like SQL or RDF and to leverage a wider range of underlying data management technologies. A selection query is defined using GraphQL [queries](https://graphql.org/learn/queries/). Broadly speaking for a query selection of fields `field1`, `field2` and `field3` fom the table `table_name` the GraphQL would look like: + +``` +{ + table_name { + field1 + field2 + field3 + } +} +``` + +Tasks are currently specified in a variant of the GA4GH [Task Execution Service](https://github.com/ga4gh/task-execution-schemas). We expect closer alignment by Version 1.0. A task specification is a JSON object, with a selection query embedded: + +> TODO: This payload specification is being reviewed at the time of writing. + +```json +{ + "name": "MD5 example", + "description": "Task which runs md5sum on the input file.", + "tags": { + "custom-tag": "tag-value" + }, + "inputs": [ + { + "content": "{table_name { field1 field2 field3 }}" + } + ], + "outputs" : [ + { + "url" : "/path/to/output_file", + "path" : "/mnt/output/" + } + ], + "resources" : { + "cpuCores": 1, + "ramGb": 1.0, + "diskGb": 100.0, + "preemptible": false + }, + "executors" : [ + { + "image" : "container_registry:image-aname:version_no", + "command" : ["entrypoint", "/container/input"], + "stdout" : "/mnt/logs/stdout", + "stderr" : "/mnt/logs/stderr", + "workdir": "/tmp" + } + ] +} +``` + +The structure of the JSON is: + +| Property | Specification | +|:--------------------------|:------------------------------------------------------------------| +| name | A short name for the task; does not need to be unique | +| description | A short description of the what the task is | +| inputs/content | The GraphQL selection query | +| outputs/path | Key outputs +| executors/image | The URL of a container image (in an approved registry) | +| resources | Not enforced now but estimates the compute resources for the task | + +A task can assume that inputs are provided in the `/mnt/input` folder attached to their container, whereas outputs can be written to `/mnt/output`. Logs may be delivered to `/mnt/logs`. + +## Next Step + +Understanding [Analysis plans](./User_Guide_Analysis_Plans.md) in a federated setting + +Try the API using: + +- [Command Line](./User_Guide_CLI.md) tools - e.g. with `curl` +- [Python](./User_Guide_Python.md) +- [R](./User_Guide_R.md) diff --git a/doc/User_Guide_Python.md b/doc/User_Guide_Python.md new file mode 100644 index 0000000..7704fef --- /dev/null +++ b/doc/User_Guide_Python.md @@ -0,0 +1,74 @@ +# Using Python + +> Back to the [main page](./User_Guide.md) + +Examples below follow the same pattern as those using [`curl`](./User_Guide_CLI.md). The `requests` library is recommended. Use Python shell to run the commands below, Jupyter Notebook or create and execute a `.py` file. + +## Data Discovery + +Import following two libraries, define the endpoints - authentication is not needed for metadata discovery on local docker deployment +```python +import requests +import json +import os + +FDS_ENDPOINT="https://localhost:8443/federated-data-sharing/1.1.0" +``` + +> In development, insecure SSL connections warnings can safely be ignored. Change `verify=False` to `True` as required. + +Make a call to the /dataset/list endpoint, then print the result dictionary: +```python +r = requests.get(f'{FDS_ENDPOINT}/datasets', verify=False) +dataset_list = r.json() +print(json.dumps(dataset_list, indent=4, sort_keys=True)) +``` + +Choose 1st dataset for further exploration: +```python +dataset_1 = dataset_list['datasets'][0]['id'] +``` + +Request first dataset catalogue by making a get call to /catalogue endpoint: +```python +r = requests.get(f'{FDS_ENDPOINT}/datasets/{dataset_1}/catalogue', verify=False) +catalogue = r.json() +print(json.dumps(catalogue, indent=4, sort_keys=True)) +``` + +Request first dataset dictionary by making a get call to /dictionary endpoint: +```python +r = requests.get(f'{FDS_ENDPOINT}/datasets/{dataset_1}/dictionaries', verify=False) +dictionaries = r.json() +print(json.dumps(dictionaries, indent=4, sort_keys=True)) +``` +## Data Selection + +Set the payload (Assuming you are in the top level folder of this project): +```python +file = open('./src/examples/example-query.graphql', 'r') +query = file.read() +``` +Verify if the request is valid: +```python +headers = {'Content-type': 'plain/text', 'Accept': 'application/json' } +r = requests.post(f'{FDS_ENDPOINT}/selection/validate', data = query, headers=headers, verify=False) +valid = r.json() == 'True' +``` +The response should be the simple JSON token `"True"` or `"False"`. + +Then if the request is valid, the selection can be made: +```python +headers = {'Content-type': 'plain/text', 'Accept': 'application/json' } +r = requests.post(f'{FDS_ENDPOINT}/selection/select', data = query, headers=headers, verify=False) +data = r.json() +``` + +## Tasks using federated computation + +> TODO - pending updating the task API + +## Next steps + +- Back to the [main page](./User_Guide.md) +- Look at some [Worked examples](https://github.com/federated-data-sharing/common-api-examples) diff --git a/doc/User_Guide_R.md b/doc/User_Guide_R.md new file mode 100644 index 0000000..016ce7b --- /dev/null +++ b/doc/User_Guide_R.md @@ -0,0 +1,104 @@ +# Using R + +> Back to the [main page](./User_Guide.md) + +## Getting Started + +For R interacting with an API, the [`httr`](https://httr.r-lib.org/) library is recommended. This can be done interactively, or in an `.R` script. The examples below are at an R console prompt (`>`). The [tidyverse libraries](https://www.tidyverse.org/) and related libraries for processing JSON are recommended and used in the examples below. + +For these examples, R 3.6.1 was used, with the following dependencies installed: + +- [tidyverse](https://www.tidyverse.org/) including purrr, stringr and readr +- [jsonlite](https://www.rdocumentation.org/packages/jsonlite) +- [httr](https://httr.r-lib.org/) + +Set up the library dependency and define the end point. +```R +library(tidyverse) +library(jsonlite) +library(httr) +library(purrr) +library(stringr) +library(readr) +FDS_ENDPOINT <- "https://localhost:8443/federated-data-sharing/1.1.0" +``` +If your implementation has a self-signed certificate, temporarily disable SSL warnings with: +```R +httr::set_config(httr::config(ssl_verifypeer=0L, ssl_verifyhost=0L)) +``` + +## Access tokens + +If the end point requires an acess token (note the reference implementation does not) obtain an access token. This example may vary by endpoint but with OAuth2 normally this involves the following parameters that are provided by the endpoint provider. + +- grant_type +- client_id +- client_secret + +> Handling JSON in R can be awkward so the examples below take a case-by-case approach to parsing responses from JSON to useful data structures for use in R. + +```R +> TOKEN_ENDPOINT <- 'AS PROVIDED' +> GRANT_TYPE <- 'AS PROVIDED' +> CLIENT_ID <- 'AS PROVIDED' +> CLIENT_SECRET <- 'AS PROVIDED' +> r <- POST(TOKEN_ENDPOINT, + body=list(grant_type=GRANT_TYPE), + authenticate(CLIENT_ID, CLIENT_SECRET) +> response <- content(r, 'parsed') +> access_token <- response$access_token +``` + +> Note: The access token may need to be refreshed from time to time. + +In what follows, if you have a token, add the following to `GET` and `POST` calls: +```R +add_headers(Authorization=paste('Bearer', access_token, sep=" ")) +``` +## Data Discovery + +To get the dataset list using an `httr` `GET()` call: +```R +r <- GET(paste0(FDS_ENDPOINT, '/datasets')) +resp <- content(r, 'parsed') +dataset_list <- map(resp$datasets, 'id') +``` + +To obtain the dictionary for a dataset `dataset_id`: +```R +r <- GET(paste0(FDS_ENDPOINT, '/datasets/', dataset_id, '/dictionaries')) +resp <- content(r, 'text', encoding='UTF-8') +dictionaries <- fromJSON(resp, flatten=TRUE) +``` + +## Data selection + +To validate a query: +```R +r <- POST(paste0(FDS_ENDPOINT, '/selection/validate'), + body=upload_file('src/examples/example-query.graphql'), + encode='raw') +resp <- content(r, 'parsed') +validation <- resp$success +``` + +> Note: 'raw' implies plain text + +To submit the query: +```R +r <- POST(paste0(FDS_ENDPOINT, '/selection/select'), + body=upload_file('src/examples/example-query.graphql'), + add_headers('Accept'='application/json'), + encode='raw') +resp <- content(r) +``` +In this worked example, the data is accessible via `resp$data$synthetic_alzheimers_profile` + +## Tasks using federated computation + +> TODO - pending updating the task API + +## Next steps + +- Back to the [main page](./User_Guide.md) +- Look at some [Worked examples](https://github.com/federated-data-sharing/common-api-examples) diff --git a/doc/addi-logo.png b/doc/addi-logo.png new file mode 100755 index 0000000..b10fe18 Binary files /dev/null and b/doc/addi-logo.png differ diff --git a/doc/aridhia-dre-logo.png b/doc/aridhia-dre-logo.png new file mode 100755 index 0000000..6733476 Binary files /dev/null and b/doc/aridhia-dre-logo.png differ diff --git a/doc/hdruk-logo.png b/doc/hdruk-logo.png new file mode 100755 index 0000000..662d48d Binary files /dev/null and b/doc/hdruk-logo.png differ diff --git a/doc/icoda-research-logo.png b/doc/icoda-research-logo.png new file mode 100755 index 0000000..aa8b48e Binary files /dev/null and b/doc/icoda-research-logo.png differ diff --git a/doc/sketch_docker.jpg b/doc/sketch_docker.jpg new file mode 100755 index 0000000..73adc04 Binary files /dev/null and b/doc/sketch_docker.jpg differ diff --git a/doc/sketch_full.jpg b/doc/sketch_full.jpg new file mode 100755 index 0000000..fcdb70f Binary files /dev/null and b/doc/sketch_full.jpg differ diff --git a/doc/sketch_process.jpg b/doc/sketch_process.jpg new file mode 100755 index 0000000..ae6c4b3 Binary files /dev/null and b/doc/sketch_process.jpg differ diff --git a/src/examples/example-query.graphql b/src/examples/example-query.graphql new file mode 100755 index 0000000..08f6664 --- /dev/null +++ b/src/examples/example-query.graphql @@ -0,0 +1,9 @@ +{ + synthetic_alzheimers_profile { + sex + age_at_inclusion + cdr_global_score_baseline + abeta_1_42_year_1 + hippocampal_volume_year_1 + } +} \ No newline at end of file