Writing your first CWL Workflow document

Goal

Building on the CommandLineTool created on the previous step, the first Workflow will include two processing steps:

a fan-out processing pattern to use gdal_translate to clip the bands B04, B03 and B02
a fan-in processing pattern using OTB's image stacking CLI to create an RGB composite

At this stage, there's a translate.cwl CWL document and a file params.yml.

CWL Workflow skeleton

Start creating the CWL workflow document with it's initial structure:

class: 
label:  
doc:  
id: 

requirements:

inputs:

outputs:

steps:

  node_:

    run: 

    in:


    out:

cwlVersion: v1.0

Set the CWL class to Workflow:

class: Workflow
label:  
doc:  
id: 

requirements:

inputs:

outputs:

steps:

  node_:

    run: 

    in:


    out:

cwlVersion: v1.0

Workflow information

Set the Workflow information (label, doc) and id:

class: Workflow
label: Sentinel-2 RGB creation
doc:  This workflow creates an RGB composite
id: main

requirements:

inputs:

outputs:

steps:

  node_:

    run: 

    in:


    out:

cwlVersion: v1.0

Workflow requirements

Set the Workflow requirements to support the fan-out pattern. Setting the ScatterFeatureRequirement requirement tells the CWL runner to spawn several processes (these may run in parallel).

class: Workflow
label: Sentinel-2 RGB creation
doc:  This workflow creates an RGB composite
id: main

requirements:
- class: ScatterFeatureRequirement

inputs:

outputs:

steps:

  node_:

    run: 

    in:


    out:

cwlVersion: v1.0

Workflow inputs

Define the inputs: - the list of geotifs - the area of interest - the EPSG code used to express the area of interest coordinates

The geotiffs are a list of files, this workflow parameter type is now File[]:

class: Workflow
label: Sentinel-2 RGB creation
doc:  This workflow creates an RGB composite
id: main

requirements:
- class: ScatterFeatureRequirement

inputs:

  geotiffs:
    doc: list of geotifs
    type: File[]

  aoi: 
    doc: area of interest as a bounding box
    type: string

  epsg:
    doc: EPSG code 
    type: string
    default: "EPSG:4326"

outputs:

steps:

  node_:

    run: 

    in:


    out:

cwlVersion: v1.0

Workflow processing step

Update the first processing step to set the sub-workflow translate.cwl to be run:

class: Workflow
label: Sentinel-2 RGB creation
doc:  This workflow creates an RGB composite
id: main

requirements:
- class: ScatterFeatureRequirement

inputs:

  geotiffs:
    doc: list of geotifs
    type: File[]

  aoi: 
    doc: area of interest as a bounding box
    type: string

  epsg:
    doc: EPSG code 
    type: string
    default: "EPSG:4326"

outputs:

steps:

  node_:

    run: translate.cwl

    in:

    out:

cwlVersion: v1.0

Workflow input mapping

Now, set the step inputs mapping where:

translate.cwl input geotiff is mapped to worklow input geotiffs
translate.cwl input aoi is mapped to worklow input aoi
translate.cwl input epsg is mapped to worklow input epsg

class: Workflow
label: Sentinel-2 RGB creation
doc:  This workflow creates an RGB composite
id: main

requirements:
- class: ScatterFeatureRequirement

inputs:

  geotiffs:
    doc: list of geotifs
    type: File[]

  aoi: 
    doc: area of interest as a bounding box
    type: string

  epsg:
    doc: EPSG code 
    type: string
    default: "EPSG:4326"

outputs:

steps:

  node_:

    run: translate.cwl

    in:

      geotiff: geotiffs  
      aoi: aoi
      epsg: epsg

    out:

cwlVersion: v1.0

Workflow step output

Set the node_translate output, i.e. the clipped_tif generated by translate.cwl:

class: Workflow
label: Sentinel-2 RGB creation
doc:  This workflow creates an RGB composite
id: main

requirements:
- class: ScatterFeatureRequirement

inputs:

  geotiffs:
    doc: list of geotifs
    type: File[]

  aoi: 
    doc: area of interest as a bounding box
    type: string

  epsg:
    doc: EPSG code 
    type: string
    default: "EPSG:4326"

outputs:

steps:

  node_translate:

    run: translate.cwl

    in:

      geotiff: geotiffs  
      aoi: aoi
      epsg: epsg

    out:
    - clipped_tif

cwlVersion: v1.0

Scatter (fan-out)

Now set the fan-out configuration by setting the scatter method and input (the geotiffs):

class: Workflow
label: Sentinel-2 RGB creation
doc:  This workflow creates an RGB composite
id: main

requirements:
- class: ScatterFeatureRequirement

inputs:

  geotiffs:
    doc: list of geotifs
    type: File[]

  aoi: 
    doc: area of interest as a bounding box
    type: string

  epsg:
    doc: EPSG code 
    type: string
    default: "EPSG:4326"

outputs:

steps:

  node_translate:

    run: translate.cwl

    in:

      geotiff: geotiffs  
      aoi: aoi
      epsg: epsg

    out:
    - clipped_tif

    scatter: geotiffs
    scatterMethod: dotproduct

cwlVersion: v1.0

Adding another step

The next step requires an additional CWL CommandLineTool. It's a file called concatenate.cwl and it contains:

class: CommandLineTool

requirements:
  InlineJavascriptRequirement: {}
  DockerRequirement: 
    dockerPull: terradue/otb-7.2.0

baseCommand: otbcli_ConcatenateImages
arguments: 
- -out
- composite.tif

inputs:

  tifs:
    type: File[]
    inputBinding:
      position: 5
      prefix: -il
      separate: true

outputs:

  composite:
    outputBinding:
      glob: "composite.tif"
    type: File

stderr: stderr
stdout: stdout

cwlVersion: v1.0

Add the additional step to concatenate the clipped geotiffs.

class: Workflow
label: Sentinel-2 RGB creation
doc:  This workflow creates an RGB composite
id: main

requirements:
- class: ScatterFeatureRequirement

inputs:

  geotiffs:
    doc: list of geotifs
    type: File[]

  aoi: 
    doc: area of interest as a bounding box
    type: string

  epsg:
    doc: EPSG code 
    type: string
    default: "EPSG:4326"

outputs:

steps:

  node_translate:

    run: translate.cwl

    in:

      geotiff: geotiffs  
      aoi: aoi
      epsg: epsg

    out:
    - clipped_tif

    scatter: geotiffs
    scatterMethod: dotproduct

  node_concatenate:

    run: concatenate.cwl

    in: 
      tifs:
        source: node_translate/clipped_tif

    out:
    - composite


cwlVersion: v1.0

Mapping the inputs from the previous step

The inputs come from the previous step using the mapping:

class: Workflow
label: Sentinel-2 RGB creation
doc:  This workflow creates an RGB composite
id: main

requirements:
- class: ScatterFeatureRequirement

inputs:

  geotiffs:
    doc: list of geotifs
    type: File[]

  aoi: 
    doc: area of interest as a bounding box
    type: string

  epsg:
    doc: EPSG code 
    type: string
    default: "EPSG:4326"

outputs:

steps:

  node_translate:

    run: translate.cwl

    in:

      geotiff: geotiffs  
      aoi: aoi
      epsg: epsg

    out:
    - clipped_tif

    scatter: geotiffs
    scatterMethod: dotproduct

  node_concatenate:

    run: concatenate.cwl

    in: 
      tifs:
        source: node_translate/clipped_tif

    out:
    - composite


cwlVersion: v1.0

The workflow only misses the Workflow output block, a mapping to the node_concatenate outputs:

class: Workflow
label: Sentinel-2 RGB creation
doc:  This workflow creates an RGB composite
id: main

requirements:
- class: ScatterFeatureRequirement

inputs:

  geotiffs:
    doc: list of geotifs
    type: File[]

  aoi: 
    doc: area of interest as a bounding box
    type: string

  epsg:
    doc: EPSG code 
    type: string
    default: "EPSG:4326"

outputs:
  rgb:
    outputSource:
    - node_concatenate/composite
    type: File

steps:

  node_translate:

    run: translate.cwl

    in:

      geotiff: geotiffs  
      aoi: aoi
      epsg: epsg

    out:
    - clipped_tif

    scatter: geotiffs
    scatterMethod: dotproduct

  node_concatenate:

    run: concatenate.cwl

    in: 
      tifs:
        source: node_translate/clipped_tif

    out:
    - composite


cwlVersion: v1.0

Test the CWL workflow

Download the three geotiff with:

curl -o B04.tif https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B04.tif
curl -o B03.tif https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B03.tif
curl -o B02.tif https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B02.tif

Create a file called composite-params.yml with:

bbox: "136.659,-35.96,136.923,-35.791"
geotiffs: 
- { "class": "File", "path": "B04.tif" }
- { "class": "File", "path": "B03.tif" }
- { "class": "File", "path": "B02.tif" }
epsg: "EPSG:4326"

Run the workflow with:

cwltool --parallel composite.cwl composite-params.yml

This creates a file called composite.tif.