Skip to content

How to create a file at runtime

Goal

Create a file at runtime.

Sometimes you need to create a file on the fly from input parameters, such as tools which expect to read their input configuration from a file rather than the command line parameters, or need a small wrapper shell script.

To generate such files we can use the InitialWorkDirRequirement.

Recipe

The CWL document below shows a small wrapper shell script called run.sh that gets created at runtime.

The basecommand is an array combining /bin/sh and the created file run.sh and it will execute the command /bin/sh run.sh. This will run the file we create in the shell.

InitialWorkDirRequirement requires a listing. As the listing is a YAML array we need a - on the first line of each element of the array, in this case we have just one element. entryname: can have any value, but it must match what was specified in the baseCommand.

The final part is entry:, this is followed by |- which is YAML quoting syntax, and means that you are using a multiline string (without it we would need to write the whole script on one line).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
cwlVersion: v1.0

$graph:
- class: Workflow

  id: main

  requirements: 
  - class: MultipleInputFeatureRequirement
  - class: ScatterFeatureRequirement

  inputs:

    red: 
      type: string
    green: 
      type: string
    blue: 
      type: string

  outputs:

    preview: 
      outputSource: node_gdal/preview
      type: File[]

  steps:

    node_gdal:

      in: 
        band: [red, green, blue]

      out: 
      - preview

      run:
        "#gdal"

      scatter: band
      scatterMethod: dotproduct 


- class: CommandLineTool

  id: gdal

  requirements:
    InlineJavascriptRequirement: {}
    EnvVarRequirement:
      envDef:
        PROJ_LIB: /srv/conda/envs/notebook/share/proj
    InitialWorkDirRequirement:
      listing:
        - entryname: run.sh
          entry: |-
            gdal_translate \
              -of PNG \
              -ot Byte \
              -srcwin 1000 1000 500 500 \
              $( inputs.band ) \
              $( inputs.band.split("/").slice(-1)[0].replace(".tif", ".png") )
  hints:
    DockerRequirement: 
      dockerPull: docker.io/osgeo/gdal          

  baseCommand: ['/bin/sh', 'run.sh']

  arguments: []

  inputs:

    band: 
      type: string

  outputs:

    preview: 
      type: File
      outputBinding:
        glob: "*.png"

Execution

Use the parameters:

1
2
3
red: '/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/10/T/FK/2021/7/S2B_10TFK_20210713_0_L2A/B04.tif'
green: '/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/10/T/FK/2021/7/S2B_10TFK_20210713_0_L2A/B03.tif'
blue: '/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/10/T/FK/2021/7/S2B_10TFK_20210713_0_L2A/B02.tif'

The execution shows:

INFO /srv/conda/bin/cwltool 3.1.20210803132435
INFO Resolved 'create-file-at-runtime.cwl' to 'file:///home/fbrito/work/guide/docs/how-to/cwl-how-to/create-file-at-runtime.cwl'
INFO [workflow ] start
INFO [workflow ] starting step node_gdal
INFO [step node_gdal] start
INFO ['docker', 'pull', 'docker.io/osgeo/gdal']
Using default tag: latest
latest: Pulling from osgeo/gdal
Digest: sha256:fd9cbc42d2854783451a2503d58d34f7893f42650afb07dbc91eb78a628a610d
Status: Image is up to date for osgeo/gdal:latest
docker.io/osgeo/gdal:latest
INFO [job node_gdal] /tmp/50h8z4pj$ docker \
    run \
    -i \
    --mount=type=bind,source=/tmp/50h8z4pj,target=/ziURtg \
    --mount=type=bind,source=/tmp/rqajh51c,target=/tmp \
    --workdir=/ziURtg \
    --read-only=true \
    --user=1000:1000 \
    --rm \
    --cidfile=/tmp/_igcz1hh/20210908155350-669810.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/ziURtg \
    --env=AWS_NO_SIGN_REQUEST=YES \
    --env=AWS_REGION=us-west-2 \
    docker.io/osgeo/gdal \
    /bin/sh \
    run.sh
Input file size is 8121, 8201
0...10...20...30...40...50...60...70...80...90...100 - done.
INFO [job node_gdal] Max memory used: 0MiB
INFO [job node_gdal] completed success
INFO [step node_gdal] start
INFO [job node_gdal_2] /tmp/egb7trj8$ docker \
    run \
    -i \
    --mount=type=bind,source=/tmp/egb7trj8,target=/ziURtg \
    --mount=type=bind,source=/tmp/jdzoi6gz,target=/tmp \
    --workdir=/ziURtg \
    --read-only=true \
    --user=1000:1000 \
    --rm \
    --cidfile=/tmp/xltuldrb/20210908155358-070807.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/ziURtg \
    --env=AWS_NO_SIGN_REQUEST=YES \
    --env=AWS_REGION=us-west-2 \
    docker.io/osgeo/gdal \
    /bin/sh \
    run.sh
Input file size is 8121, 8201
0...10...20...30...40...50...60...70...80...90...100 - done.
INFO [job node_gdal_2] Max memory used: 12MiB
INFO [job node_gdal_2] completed success
INFO [step node_gdal] start
INFO [job node_gdal_3] /tmp/san4kbve$ docker \
    run \
    -i \
    --mount=type=bind,source=/tmp/san4kbve,target=/ziURtg \
    --mount=type=bind,source=/tmp/rxdhcudk,target=/tmp \
    --workdir=/ziURtg \
    --read-only=true \
    --user=1000:1000 \
    --rm \
    --cidfile=/tmp/izgoa3x3/20210908155401-198012.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/ziURtg \
    --env=AWS_NO_SIGN_REQUEST=YES \
    --env=AWS_REGION=us-west-2 \
    docker.io/osgeo/gdal \
    /bin/sh \
    run.sh
Input file size is 8121, 8201
0...10...20...30...40...50...60...70...80...90...100 - done.
INFO [job node_gdal_3] Max memory used: 19MiB
INFO [job node_gdal_3] completed success
INFO [step node_gdal] completed success
INFO [workflow ] completed success
{
    "preview": [
        {
            "location": "file:///home/fbrito/work/guide/docs/how-to/cwl-how-to/LC08_L1GT_019022_20170802_20170803_01_RT_B4.png",
            "basename": "LC08_L1GT_019022_20170802_20170803_01_RT_B4.png",
            "class": "File",
            "checksum": "sha1$481bffe667120d2c41f5b9346c17db1541e56a28",
            "size": 322,
            "path": "/home/fbrito/work/guide/docs/how-to/cwl-how-to/LC08_L1GT_019022_20170802_20170803_01_RT_B4.png"
        },
        {
            "location": "file:///home/fbrito/work/guide/docs/how-to/cwl-how-to/LC08_L1GT_019022_20170802_20170803_01_RT_B3.png",
            "basename": "LC08_L1GT_019022_20170802_20170803_01_RT_B3.png",
            "class": "File",
            "checksum": "sha1$481bffe667120d2c41f5b9346c17db1541e56a28",
            "size": 322,
            "path": "/home/fbrito/work/guide/docs/how-to/cwl-how-to/LC08_L1GT_019022_20170802_20170803_01_RT_B3.png"
        },
        {
            "location": "file:///home/fbrito/work/guide/docs/how-to/cwl-how-to/LC08_L1GT_019022_20170802_20170803_01_RT_B2.png",
            "basename": "LC08_L1GT_019022_20170802_20170803_01_RT_B2.png",
            "class": "File",
            "checksum": "sha1$481bffe667120d2c41f5b9346c17db1541e56a28",
            "size": 322,
            "path": "/home/fbrito/work/guide/docs/how-to/cwl-how-to/LC08_L1GT_019022_20170802_20170803_01_RT_B2.png"
        }
    ]
}
INFO Final process status is success

Reference