Skip to content

SNAP 101

Graph Processing Framework

The SNAP architecture provides a flexible Graph Processing Framework (GPF) allowing the creation of processing graphs for batch processing and customized processing chains.

A graph is a set of nodes connected by edges. In this case, the nodes are the processing steps called operators. The edges will show the direction in which the data is being passed between nodes; therefore it will be a directed graph. A graph can have no loops or cycles, so it will be a Directed Acyclic Graph (DAG).

The sources of the graph will be the data product readers, and the sinks can be either a product writer or an image displayed. An operator can have one or more image sources and other parameters that define the operation. Two or more operators may be connected together so that the first operator becomes an image source to the next operator. By linking one operator to another, an imaging graph or processing chain can be created

The graph processor will not introduce any intermediate files unless a writer is optionally added anywhere in the sequence.

Graphs offer the following advantages:

  • no intermediate files written, no I/O overhead
  • reusability of processing chains
  • simple and comprehensive operator configuration
  • reusability of operator configurations

SNAP EO Data Processors are implemented as GPF operators and can be invoked using the GPF Graph Processing Tool gpt which can be found in the bin directory of a SNAP installation.

The following command will dump the gpt print out a short description of what the tool is for and describes the arguments and options of the tool. A list of available operators is displayed according to the toolboxes installed.

docker run --rm docker.io/snap-gpt gpt -h

The partial output:

Usage:
  gpt <op>|<graph-file> [options] [<source-file-1> <source-file-2> ...]

Description:
  This tool is used to execute SNAP raster data operators in batch-mode. The
  operators can be used stand-alone or combined as a directed acyclic graph
  (DAG). Processing graphs are represented using XML. More info about
  processing graphs, the operator API, and the graph XML format can be found
  in the SNAP documentation.

Arguments:
  <op>               Name of an operator. See below for the list of <op>s.
  <graph-file>       Operator graph file (XML format).
  <source-file-i>    The <i>th source product file. The actual number of source
                     file arguments is specified by <op>. May be optional for
                     operators which use the -S option.

Options:
  -h                 Displays command usage. If <op> is given, the specific
                     operator usage is displayed.
  -e                 Displays more detailed error messages. Displays a stack
                     trace, if an exception occurs.
  -t <file>          The target file. Default value is 'target.dim'.
  -f <format>        Output file format, e.g. 'GeoTIFF', 'HDF5',
                     'BEAM-DIMAP'. If not specified, format will be derived
                     from the target filename extension, if any, otherwise the
                     default format is 'BEAM-DIMAP'. Ony used, if the graph
                     in <graph-file> does not specify its own 'Write'
                     operator.
  -p <file>          A (Java Properties) file containing processing parameters
                     in the form <name>=<value> or a XML file containing a
                     parameter DOM for the operator. Entries in this file are
                     overwritten by the -P<name>=<value> command-line option
                     (see below). The following variables are substituted in
                     the parameters file:
                         ${system.<java-sys-property>}
                         ${operatorName} (given by the <op> argument)
                         ${graphFile} (given by the <graph-file> argument)
                         ${targetFile} (pull path given by the -t option)
                         ${targetDir} (derived from -t option)
                         ${targetName} (derived from -t option)
                         ${targetBaseName} (derived from -t option)
                         ${targetFormat} (given by the -f option)
  -c <cache-size>    Sets the tile cache size in bytes. Value can be suffixed
                     with 'K', 'M' and 'G'. Must be less than maximum
                     available heap space. If equal to or less than zero, tile
                     caching will be completely disabled. The default tile
                     cache size is '1,073,741,824M'.
  -q <parallelism>   Sets the maximum parallelism used for the computation,
                     i.e. the maximum number of parallel (native) threads.
                     The default parallelism is '16'.
  -x                 Clears the internal tile cache after writing a complete
                     row of tiles to the target product file. This option may
                     be useful if you run into memory problems.
  -S<source>=<file>  Defines a source product. <source> is specified by the
                     operator or the graph. In an XML graph, all occurrences of
                     ${<source>} will be replaced with references to a source
                     product located at <file>.
  -P<name>=<value>   Defines a processing parameter, <name> is specific for the
                     used operator or graph. In an XML graph, all occurrences
                     of ${<name>} will be replaced with <value>. Overwrites
                     parameter values specified by the '-p' option.
  -D<name>=<value>   Defines a system property for this invocation.
  -v <dir>           A directory containing any number of Velocity templates.
                     Each template generates a text output file along with the
                     target product. This feature has been added to support a
                     flexible generation of metadata files.
                     See http://velocity.apache.org/ and option -m.
  -m <file>          A (Java Properties) file containing (constant) metadata
                     in the form <name>=<value> or any XML file. Its primary 
                     usage is to provide an additional context to be used
                     from within the Velocity templates. See option -v.
  --diag             Displays version and diagnostic information.
Operators:
  Aatsr.SST                             Computes sea surface temperature (SST) from (A)ATSR products.
  AATSR.Ungrid                          Ungrids (A)ATSR L1B products and extracts geolocation and pixel field of view data.
  AdaptiveThresholding                  Detect ships using Constant False Alarm Rate detector.
  AddElevation                          Creates a DEM band
...
  Warp                                  Create Warp Function And Get Co-registrated Images
  WdviOp                                Weighted Difference Vegetation Index retrieves the Isovegetation lines parallel to soil line. Soil line has an arbitrary slope and passes through origin
  Wind-Field-Estimation                 Estimate wind speed and direction
  Write                                 Writes a data product to a file.

The gpt can process individual operators or a graph of connected operators.

Type:

docker run --rm docker.io/snap-gpt gpt <operator-name> –h

to get usage information of an operator provided via <operator-name>.

The usage text of an operator also displays an XML template clipping of the operators configuration when used in a graph.

Example:

docker run --rm docker.io/snap-gpt gpt Calibration –h

This outputs:

Usage:
  gpt Calibration [options] 

Description:
  Calibration of products


Source Options:
  -Ssource=<file>    Sets source 'source' to <filepath>.
                     This is a mandatory source.

Parameter Options:
  -PauxFile=<string>                                    The auxiliary file
                                                        Value must be one of 'Latest Auxiliary File', 'Product Auxiliary File', 'External Auxiliary File'.
                                                        Default value is 'Latest Auxiliary File'.
  -PcreateBetaBand=<boolean>                            Create beta0 virtual band
                                                        Default value is 'false'.
  -PcreateGammaBand=<boolean>                           Create gamma0 virtual band
                                                        Default value is 'false'.
  -PexternalAuxFile=<file>                              The antenna elevation pattern gain auxiliary data file.
  -PoutputBetaBand=<boolean>                            Output beta0 band
                                                        Default value is 'false'.
  -PoutputGammaBand=<boolean>                           Output gamma0 band
                                                        Default value is 'false'.
  -PoutputImageInComplex=<boolean>                      Output image in complex
                                                        Default value is 'false'.
  -PoutputImageScaleInDb=<boolean>                      Output image scale
                                                        Default value is 'false'.
  -PoutputSigmaBand=<boolean>                           Output sigma0 band
                                                        Default value is 'true'.
  -PselectedPolarisations=<string,string,string,...>    The list of polarisations
  -PsourceBands=<string,string,string,...>              The list of source bands.

Graph XML Format:
  <graph id="someGraphId">
    <version>1.0</version>
    <node id="someNodeId">
      <operator>Calibration</operator>
      <sources>
        <source>${source}</source>
      </sources>
      <parameters>
        <sourceBands>string,string,string,...</sourceBands>
        <auxFile>string</auxFile>
        <externalAuxFile>file</externalAuxFile>
        <outputImageInComplex>boolean</outputImageInComplex>
        <outputImageScaleInDb>boolean</outputImageScaleInDb>
        <createGammaBand>boolean</createGammaBand>
        <createBetaBand>boolean</createBetaBand>
        <selectedPolarisations>string,string,string,...</selectedPolarisations>
        <outputSigmaBand>boolean</outputSigmaBand>
        <outputGammaBand>boolean</outputGammaBand>
        <outputBetaBand>boolean</outputBetaBand>
      </parameters>
    </node>
  </graph>

Calling GPT with a Graph

Rather than calling each operator and specifying all its parameters, it is more convenient to pass the required settings in an XML-encoded graph file.

To run gpt on a graph file type:

gpt <GraphFile.xml> [options] [<source-file-1> <source-file-2> ...]

Creating a Graph File

The basic format of a graph XML file is:

 <graph id="someGraphId">
 <version>1.0</version>
 <node id="someNodeId">
 <operator>OperatorName</operator>
 <sources>
<sourceProducts>${sourceProducts}</sourceProducts>
 </sources>
 <parameters>
 ....
 </parameters>
 </node>
 </graph>

Insert variables in the form ${variableName} in place of a parameter value. variableName is then replaced with a value at the command line.

For example, if a parameter for a file included the variable for ${myFilename}

<parameters>
 <file>${myFilename}</file>
</parameters>

gpt is then invoked with:

gpt mygraph.xml –PmyFilename=pathToMyFile

Batch processing

SNAP users often resort to scripts to batch process their SNAP graphs. Below two examples of such scripts:

For all envisat products in folder c:\ASAR run gpt Calibration and produce the output in the folder c:\output

for /r "c:\ASAR" %%X in (*.N1) do (gpt Calibration "%%X" -t "C:\output\%%~nX.dim")

A set of input Sentinel-2 products shall be processed with the Resample processor.

#!/bin/bash
# enable next line for debugging purpose
# set -x 

############################################
# User Configuration
############################################

# adapt this path to your needs
export PATH=~/progs/snap/bin:$PATH
gptPath="gpt"

############################################
# Command line handling
############################################

# first parameter is a path to the graph xml
graphXmlPath="$1"

# second parameter is a path to a parameter file
parameterFilePath="$2"

# use third parameter for path to source products
sourceDirectory="$3"

# use fourth parameter for path to target products
targetDirectory="$4"

# the fifth parameter is a file prefix for the target product name, typically indicating the type of processing
targetFilePrefix="$5"


############################################
# Helper functions
############################################
removeExtension() {
    file="$1"
    echo "$(echo "$file" | sed -r 's/\.[^\.]*$//')"
}


############################################
# Main processing
############################################

# Create the target directory
mkdir -p "${targetDirectory}"

# the d option limits the elemeents to loop over to directories. Remove it, if you want to use files.
for F in $(ls -1d "${sourceDirectory}"/S2*.SAFE); do
  sourceFile="$(realpath "$F")"
  targetFile="${targetDirectory}/${targetFilePrefix}_$(removeExtension "$(basename ${F})").dim"
  ${gptPath} ${graphXmlPath} -e -p ${parameterFilePath} -t ${targetFile} ${sourceFile}
done

While these are valid approaches, these scripts are not portable and hardly shareable.

Jump to next section to learn how CWL can be used to process SNAP graphs.