Advanced JSONPath and Golang templates

This tutorial describes the advanced mechanisms available in Torq to retrieve and process the data provided by event triggers and workflow steps. 

JSONPath is the notation used in Torq to access the data, and, using the advanced syntax explained below, it can also process complex data (such as a collection of objects, for example) to provide just the relevant part. Golang templates syntax can be used for more advanced data manipulation.

All examples below refer to this collection of data:

{
    "hosts" : [
      {
        "name":"host1",
        "interfaces" : [
          {
            "name": "Interface 1 of Host 1",
            "address": "10.10.10.1",
            "weight": 10
          },
          {
            "name": "Interface 2 of Host 1",
            "address": "10.10.10.2",
            "weight": 20
          }
        ]
      },
      {
        "name":"host2",
        "interfaces" : [
          {
            "name": "Interface 1 of Host 2",
            "address": "20.20.20.1",
            "weight": 10
          },
          {
            "name": "Interface 2 of Host 2",
            "address": "20.20.20.2",
            "weight": 20
          },
          {
            "name": "Interface 3 of Host 2",
            "address": "20.20.20.3",
            "weight": 30
          }
        ]
      }
    ]
  }

Handling arrays and collections

The workflow context can contain collections of several variables, for example, lists of users, devices, vulnerabilities, and more. In order to process the data stored in these, Golang templates offer three operators:

• len: returns the length (number of items) of the array provided as an argument.
  • For the collection above, {{ len $.hosts }} will return 2.
• index: returns a single item from a map or an array. The provided index can either be a numerical location of an item in an array or a named value inside a map. The index function can be used more than once in a single template in order to address a field of an object inside a collection.
  • {{ (index (index $.hosts 0).interfaces 0).name }} will return the name of the first interface of the first host in the collection. For the collection above, the result will be Interface 1 of Host 1.
• []: returns either a single item or a range of items. {{ $.hosts[0] }} returns the same as {{ index $.hosts 0 }}. It can be used across fields {{ $.hosts[0].name }} and inside collections {{ $.hosts[0].interfaces[0].name }}.
  • For the collection above, {{ $.hosts[0].interfaces[0].name }} will return: Interface 1 of Host 1.

Accessing and slicing arrays and collections

Torq supports the following JSONPath expressions for performing operations on arrays/collections. Reminder: Indexes start at 0.

Apply filters to arrays and collections

• Filtering complete objects: the expression $.hosts[1].interfaces[?(@.weight > 10)] will return a list of interfaces where the value of the weight key exceeds 10.
• Retrieving specific fields: the expression $.hosts[1].interfaces[?(@.weight > 10)].name will return a list of names rather than a list of complete interface objects. For the example above, $.hosts[1].interfaces[?(@.weight > 10)][0] will return Interface 2 of Host 2.
• Matching by JavaScript Regular Expression: the expression $.hosts[1].interfaces[?(@.name=~ /(Interface [23])/)] will return a list of interfaces that contains interfaces 2 and 3.
• Matching by multiple expressions: can be performed by combining expressions using && (logical AND) and || (logical OR) operators. For example, the expression $.hosts[1].interfaces[?(@.weight > 20 && @.name=~ /(Interface [23])/)] will return a list containing only Interface 3 of Host 2.

Comparisons

Comparison operators are useful either when defining a conditional execution of a step or when a step argument value depends on a specific condition. Each of the below is a boolean comparison operator:

• eq - Returns the boolean truth of arg1 == arg2
• ne - Returns the boolean truth of arg1 != arg2
• lt - Returns the boolean truth of arg1 < arg2
• gt - Returns the boolean truth of arg1 > arg2
• ge - Returns the boolean truth of arg1 >= arg2
• ieq - Returns the case-insensitive string comparison of arg1 and arg2 (==)
• ine - Returns the negation of a case-insensitive string comparison of arg1 and arg2 (!=)
• isNull - Returns true if the operand arg1 is Null (the actual word Null, not an empty argument)
• empty - Returns true when a JSON path is not defined or present in a step output

For example, if using these comparisons with the collection provided above:

{{ eq (len $.hosts) 2 }} and {{ ge (len $.hosts) 0 }} will return true, whereas {{ isNull $.hosts }} or {{ lt (len $.hosts[0].interfaces) 1}} will return false.

Logical operators 

Logical operators allow executing boolean logic using execution context fields/variables as arguments. Here are examples of such cases:

• and: returns the boolean AND of its arguments by returning the first empty argument or the last argument, that is, "and x y" behaves as "if x then y else x". All the arguments are evaluated.
  • For example: {{ and (eq $.event.Scope_Action "Firing") (eq $.event.Scope_Env "Production") }}
• or: returns the boolean OR of its arguments by returning the first non-empty argument or the last argument, that is, "or x y" behaves as "if x then x else y". All the arguments are evaluated.
  • For example: {{ or (eq $.event.Scope_Action "Firing") (eq $.event.Scope_Action "Warning") }}
• not: returns the boolean negation of its single argument. 
  • For example: {{ not (eq $.event.Scope_Action "Firing") }}

Useful template functions

The table below summarizes a number of useful functions that can be used inside Torq as templates. All of these capabilities are also available as full no-code steps. The usage of template functions is an advanced low-code capability that can make the workflow more concise.

Advanced templates

Torq supports advanced templates that generate output (text, markdown, script) that is dynamic and gets filled by the data coming from the workflow context.

The advanced syntax described in the Golang Template package allows iterating over collections of data/objects using the range function.
For example, you can create an advanced text template that would mention the number of hosts and list their names:

We currently have {{ len $.hosts }} hosts and their names are:
{{ range $index, $host:= $.hosts }}
{{ $host.name }}
{{ end }}

The above template would produce the following text:

We currently have 2 hosts and their names are:
host1
host2

The range template presented above performs the following actions:

1. Creates text lines for each member of $.hosts (will not create any if it's empty).
2. Defines two temporary variables that can be referenced inside the template (note that they don't have an extra dot between the $sign and the variable name):
   • $index will contain the index of the object in the map/array.
   • $host will contain the actual object.
3. For each host, prints its name field.

Use $index to improve the template verbosity

You can modify the template to create a more verbose output:

We currently have {{ len $.hosts }} hosts and their names are:
{{ range $index, $host:= $.hosts }}
The name of host {{$index}} is {{ $host.name }}
{{ end }}

We currently have 2 hosts and their names are:
The name of host 0 is host1
The name of hsot 1 is host2

As clearly seen from the example, the index is zero-based.

Nested range constructs

The principle of nesting range constructs can be applied to arrays within arrays (such as, for example, our interfaces, that are, in turn, located inside hosts):

We currently have {{ len $.hosts }} hosts and their names are:
{{ range $index, $host:= $.hosts }}
----------------------------------------------
The name of host {{$index}} is {{ $host.name }}
It has {{ len $host.interfaces }} interfaces:
{{ range $interfaceidx, $interface:= $host.interfaces }}
Interface number {{ $interfaceidx }} is named "{{$interface.name}}" and has an IP address of {{$interface.address}}
{{end}}
----------------------------------------------
{{ end }}

The above template will generate the following dynamic text:

We currently have 2 hosts and their names are:
----------------------------------------------
The name of host 0 is host1
It has 2 interfaces:
Interface number 0 is named "Interface 1 of Host 1" and has an IP address of 10.10.10.1
Interface number 1 is named "Interface 2 of Host 1" and has an IP address of 10.10.10.2
----------------------------------------------
----------------------------------------------
The name of host 1 is host2
It has 3 interfaces:
Interface number 0 is named "Interface 1 of Host 2" and has an IP address of 20.20.20.1
Interface number 1 is named "Interface 2 of Host 2" and has an IP address of 20.20.20.2
Interface number 2 is named "Interface 3 of Host 2" and has an IP address of 20.20.20.3
-----------------------------------------------

The construct described above can be combined with functions and variables described in the previous sections to achieve additional data manipulation. For example, the variable containing an index of the range can be used to retrieve a matching member of a different array:

{{index $.some_other_step.list_of_responses $idx}}

In this example, the index function is used to retrieve a member from an array stored in $.some_other_step.list_of_responses, and the $idx variable contains the index (0,1,2, ...) of the retrieved member.