DELTA API - Getting Started

This tutorial steps through using the Delta API to submit form data using the interactive Swagger documentation available at https://api.delta.stage.communities.gov.uk/.

Requirements

To follow this guide you will need:

The name and code of an organisation on Delta to submit data for
- The organisation will need to be added to the api-simple-testing dataset in Delta by an admin
- Below we'll use "Adur District Council", with code "E07000223"
An account on Delta Staging
- With access to the organisation, the "Data Provider" role and the "API Simple Testing" Access Group
A client id and secret to authenticate to the API

Please raise a request on the DLUHC Service Desk if you're missing any of the above.

Authenticate to the API

Before we can use any of the API endpoints we'll need to authenticate and get a bearer token.

I'm using delta.user@example.com/Password123 as the Delta username and password, and client-1234/client-secret-1234 as the client id and secret. You'll need to replace these with your own credentials.

Visit https://api.delta.stage.communities.gov.uk/ and click on "Authorize"
Fill in your Delta username and password, and your client id and secret. In your Delta username replace @ with !
Click Authorize on the modal, then close it

The access tokens are valid for an hour, after that you will need to re-authenticate.

When submitting or validating data via the Delta API, you act as a user (by providing username and password during auth), connecting via a trusted service (by providing client id and secret during auth), and on behalf of a particular organisation (by specifying the organisation when submitting data).

Equivalent curl command

curl -X POST 'https://auth.delta.stage.communities.gov.uk/realms/delta/protocol/openid-connect/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'grant_type=password' \
--data-urlencode 'client_id=client-1234' \
--data-urlencode 'client_secret=client-secret-1234' \
--data-urlencode 'username=delta.user!example.com' \
--data-urlencode 'password=Password123'

Extract the access_token field from the returned JSON.

List collections

To make sure our access token is working, and our Delta account is set up correctly, let's make a request to get information about the instances associated with the dataset tag we'll be using: "api-simple-testing".

"instance" and "dataset" are Delta terms. In short, individual form submissions ("records") are submitted to an instance, which is part of a dataset.
A dataset can have multiple instances (one per quarter, for example), and multiple datasets can share the same dataset tag.

Find the request "GET /rest-api/datasets/{dataset-tag}/collections" in Swagger, under "List dataset tags and collections"
Click on the request to expand it, then on "Try it out"
Fill in the dataset-tag input with "api-simple-testing" (without quotes)
Press "Execute"

You should see a response like the following:

<?xml version="1.0" encoding="UTF-8"?>
<instances estimate="1">
  <instance has-user-provided-data="false" xmlns="http://www.gov.uk/dclg/delta/instance">
    <instance-id>api-simple-testing-1</instance-id>
    <instance-description>api-simple-testing-1</instance-description>
    <dataset-id>api-simple-testing</dataset-id>
    <form-name>api-simple-testing</form-name>
    <form-version>1</form-version>
    <status>Signed-off</status>
    <sequence>2</sequence>
    <instance-start-date>2023-01-01T00:00:00Z</instance-start-date>
    <instance-end-date>2025-01-01T23:59:59Z</instance-end-date>
    <availability-date>2023-01-01T00:00:00Z</availability-date>
    <submission-deadline-date>2025-01-01T23:59:00Z</submission-deadline-date>
    <availability-notification-days notification-date="2022-12-31T00:00:00Z">0</availability-notification-days>
    <submission-deadline-notification-days notification-date="2024-12-29T00:00:00Z">3</submission-deadline-notification-days>
    <collection-type>data-collection</collection-type>
    <allow-bulk-import>false</allow-bulk-import>
    <allow-xml-bulk-import>true</allow-xml-bulk-import>
    <is-record-level-data>true</is-record-level-data>
    <allow-bulk-delete>true</allow-bulk-delete>
    <enable-notifications>false</enable-notifications>
    <allow-non-financial-data-certifier-to-certify>false</allow-non-financial-data-certifier-to-certify>
    <allow-certifier-provided-data-to-certify>false</allow-certifier-provided-data-to-certify>
    <created-date>2023-08-15T15:07:12.226976+01:00</created-date>
    <last-modified-date>2023-08-15T15:07:39+01:00</last-modified-date>
  </instance>
</instances>

This is the instance we'll be submitting data to.

If you get an empty response (<instances estimate="0"/>) then check your Delta account has the "API Simple Testing" Access Group on the My account page on Delta.

Equivalent curl command

curl -X 'GET' 'https://api.delta.stage.communities.gov.uk/rest-api/datasets/api-simple-testing/collections' \
  -H 'Authorization: Bearer access_token_here

Submitting data with the API

Constructing the request body

Form data record

The API accepts form data records in an XML format. For the test dataset we're using the form in Delta looks like this

API testing form

and an XML version of the record would be

<form xmlns="http://www.gov.uk/dclg/delta">
  <section-1>
    <grid-1>
      <submission-reference>E07000223_1</submission-reference>
      <favourite-number>7</favourite-number>
    </grid-1>
  </section-1>
</form>

Datasets that support API submission always include a unique identifier in the record, the value of which must start with the organisation code. In our case that is "submission-reference".

When using the API to submit real data you will be given an XML schema file which describes the expected structure of the form data. For our example form the schema file is:

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema elementFormDefault="qualified" targetNamespace="http://www.gov.uk/dclg/delta"
  version="5.0" xmlns:xs="http://www.w3.org/2001/XMLSchema"
  xmlns:delta="http://www.gov.uk/dclg/delta">
  <xs:annotation>
    <xs:documentation>XSD for api-simple-testing</xs:documentation>
  </xs:annotation>
  <xs:element name="form"><xs:complexType><xs:sequence>
    <xs:element name="section-1"><xs:complexType><xs:sequence>
      <xs:element name="grid-1"><xs:complexType><xs:sequence>
        <xs:element name="submission-reference" minOccurs="0" type="xs:string" />
        <xs:element name="favourite-number" minOccurs="0" type="xs:string" />
      </xs:sequence></xs:complexType></xs:element>
    </xs:sequence></xs:complexType></xs:element>
  </xs:sequence></xs:complexType></xs:element>
</xs:schema>

Example submission body

An upload to the API includes:

One or more form entries
Information about the submitting organisation

<?xml version="1.0" encoding="UTF-8"?>
<forms xmlns="http://www.gov.uk/dclg/delta">
  <form xmlns="http://www.gov.uk/dclg/delta">
    <section-1>
      <grid-1>
        <submission-reference>E07000223_1</submission-reference>
        <favourite-number>7</favourite-number>
      </grid-1>
    </section-1>
  </form>
  <metadata>
    <organisation-id>E07000223</organisation-id>
    <organisation-name>Adur District Council</organisation-name>
  </metadata>
</forms>

Validating form data

The API allows you to validate a form submission without saving it in Delta.

Open the POST /rest-api/datasets/api-simple-testing/data/validate-only request, under "Validate without submitting"
Paste the Example submission from above in the post body
Fill in the dataset-tag input with "api-simple-testing" (without quotes)

You should get a response like the following:

<?xml version="1.0" encoding="UTF-8"?>
<validation-messages estimate="1" xmlns="http://www.gov.uk/dclg/delta/validation/message/result">
  <validation-message type="form-data" xmlns="http://www.gov.uk/dclg/delta/validation/message">
    <submission-id/>
    <id>E07000223_1</id>
    <dataset-tag>api-simple-testing</dataset-tag>
    <dataset-id>api-simple-testing</dataset-id>
    <instance-id>api-simple-testing-1</instance-id>
    <organisation-id>E07000223</organisation-id>
    <source-uri>/datasets/api-simple-testing/api-simple-testing-1/data/E07000223_1.xml</source-uri>
    <created-date>2023-08-15T15:46:29.938709Z</created-date>
    <position>1</position>
    <messages count="1">
      <message type="information">Document validated successfully</message>
    </messages>
    <elapsed-time>PT0.009938S</elapsed-time>
    <last-modified-date>2023-08-15T15:46:29.948663Z</last-modified-date>
  </validation-message>
</validation-messages>

Submitting a malformed request will give validation messages, for example, changing favourite-number to `-10`` gives

<?xml version="1.0" encoding="UTF-8"?>
<validation-messages estimate="1" xmlns="http://www.gov.uk/dclg/delta/validation/message/result">
  <validation-message type="form-data" xmlns="http://www.gov.uk/dclg/delta/validation/message">
    <submission-id/>
    <id>E07000223_1</id>
    <dataset-tag>api-simple-testing</dataset-tag>
    <dataset-id>api-simple-testing</dataset-id>
    <instance-id>api-simple-testing-1</instance-id>
    <organisation-id>E07000223</organisation-id>
    <source-uri>/datasets/api-simple-testing/api-simple-testing-1/data/E07000223_1.xml</source-uri>
    <created-date>2023-08-15T15:51:46.158803Z</created-date>
    <position>1</position>
    <messages count="2">
      <message type="error" summary="favourite-number - Favourite Number must be a positive number">favourite-number - Favourite Number must be a positive number</message>
    </messages>
    <elapsed-time>PT0.013784S</elapsed-time>
    <last-modified-date>2023-08-15T15:51:46.172606Z</last-modified-date>
  </validation-message>
</validation-messages>

Equivalent curl command

curl -X 'POST' 'https://api.delta.stage.communities.gov.uk/rest-api/datasets/api-simple-testing/data/validate-only' \
  -H 'authorization: Bearer auth_token_here' \
  -H 'Content-Type: application/xml' \
  -d '<?xml version="1.0" encoding="UTF-8"?>
<forms xmlns="http://www.gov.uk/dclg/delta">
    <form>
        <section-1>
            <grid-1>
                <submission-reference>E07000223_1</submission-reference>
                <favourite-number>7</favourite-number>
            </grid-1>
        </section-1>
    </form>
    <metadata>
        <organisation-id>E07000223</organisation-id>
        <organisation-name>Adur District Council</organisation-name>
    </metadata>
</forms>'

Submitting form data

This endpoint is used to save records to Delta.

Select the POST /rest-api/datasets/{dataset-tag}/data endpoint in Swagger to try out
Fill in the post body with the same Example submission body from above
Enter "api-simple-testing" for the dataset tag and press Execute

Unlike the validation endpoint, submissions are processed asynchronously. You should receive a 201 response with a path to the submission in the response body, for example

/rest-api/datasets/api-simple-testing/submissions/0ec6e2490495543f4a1f966d79908c55

Include multiple <form> elements in the request body to send multiple records in one submission.

By default the API will automatically select an open instance to submit data to, but you can override this with the instance-id query parameter.

Equivalent curl command

curl -X 'POST' 'https://api.delta.stage.communities.gov.uk/rest-api/datasets/api-simple-testing/data' \
  -H 'authorization: Bearer auth_token_here' \
  -H 'Content-Type: application/xml' \
  -d '<?xml version="1.0" encoding="UTF-8"?>
<forms xmlns="http://www.gov.uk/dclg/delta">
  <form xmlns="http://www.gov.uk/dclg/delta">
    <section-1>
      <grid-1>
        <submission-reference>E07000223_1</submission-reference>
        <favourite-number>7</favourite-number>
      </grid-1>
    </section-1>
  </form>
  <metadata>
    <organisation-id>E07000223</organisation-id>
    <organisation-name>Adur District Council</organisation-name>
  </metadata>
</forms>'

Checking the status of a submission

You can use the path returned from the submission request above to check on the status of the submission. You will also get an email once the submission has finished processing.

Open the GET /rest-api/datasets/{dataset-tag}/submissions/{submission-id} endpoint in Swagger to try out
Fill in "api-simple-testing" as the dataset tag, and use the returned submission id from the previous request ("0ec6e2490495543f4a1f966d79908c55" in our case)

Executing the request will give a response like the following:

<?xml version="1.0" encoding="UTF-8"?>
<submission xmlns="http://www.gov.uk/dclg/delta/submission/form-data">
  <submission-id>0ec6e2490495543f4a1f966d79908c55</submission-id>
  <submission-type>form-data</submission-type>
  <dataset-tag>api-simple-testing</dataset-tag>
  <dataset-id>api-simple-testing</dataset-id>
  <instance-id>api-simple-testing-1</instance-id>
  <submission-file-uri>/datasets/api-simple-testing/api-simple-testing-1/submissions/0ec6e2490495543f4a1f966d79908c55/forms.xml</submission-file-uri>
  <submitted-by-user-id>ben.ramchandani@softwire.com</submitted-by-user-id>
  <submitted-by-user-org-id>E07000223</submitted-by-user-org-id>
  <created-date>2023-08-16T09:29:54.13251Z</created-date>
  <last-modified-date uri="/datasets/api-simple-testing/api-simple-testing-1/data/E07000223_1.xml">2023-08-16T09:30:03.237478Z</last-modified-date>
  <status>ingested</status>
  <stats total-records="1" records-no-errors-no-warnings="1" records-failed="0" records-with-errors="0" records-with-warnings="0" records-with-duplicates="0" records-skipped="0"/>
  <validation-messages count="0"/>
  </validation-messages>
  <validation-elapsed-time>PT0.018155S</validation-elapsed-time>
  <elapsed-time>PT9.086813S</elapsed-time>
</submission>

In particular:

The status will be "ingested" once processing has finished
The attributes on the "stats" element tell you how many records were processed successfully
Records with errors will be in the "Saved" state in Delta, records with no errors will be "Submitted", even if they have warnings

Equivalent curl command

curl -X 'GET' \
  'https://api.delta.stage.communities.gov.uk/rest-api/datasets/api-simple-testing/submissions/0ec6e2490495543f4a1f966d79908c55' \
  -H 'authorization: Bearer auth_token_here'

Wrapping up

We have successfully submitted data to Delta through the API and it will be visible in the Data Store in Delta.

This covers the primary use case for the API, though it also has endpoints to get, edit and delete existing form data records and whole submissions.