Use loops

A loop consists of a sequence of actions that are iteratively performed. The type of loop chosen dictates whether these actions repeat until a specific condition is met, across all members of a given dataset, or for a defined number of iterations. 

Here are a few examples of use cases where loops are necessary, categorized by the loop type. More detailed examples are available in the How to use loops section below.

Loop in

• Loop over a dataset and perform an action on each dataset item. For example, enrich each IP address in a list.

Loop range

• Loop n times (or less) until a certain action takes place.

Loop until break

• Loop over a dataset until an object matches certain criteria.
• Loop to retrieve multiple pages of results (pagination).

Key concepts

These are the terms and concepts you need to know before using loops.

Loop operator

The step you add to the designer to implement loop logic in a workflow.

Mode

Specifies whether to execute the loop iterations sequentially or in parallel.

Action

These are the steps you will execute for each iteration in the loop.

Collect loop results

Torq has a dedicated operator that collects the results of a loop's iterations. The step includes two parameters.

• Input: The information to collect for a single loop iteration. The result can be a string, number, JSON object, or array representing an individual entry of a collection of results.
• Flatten results: If Yes is selected and the input is an array, merge that array into the array of collected loop values. Otherwise, it will add the input as a single item in the collected loop results.

Terminate a loop

The Break operator is used within the loop to terminate the loop based on any specific condition before the loop comes to its predefined end. This means that while looping over an array or looping n times, if a certain condition is met, terminate the loop.

How to use a loop: examples

1. In: Loop on a dataset (Basic)
2. In: Loop on a dataset: loop within a loop (Advanced)
3. Range: polling
4. Until Break: loop for pagination

Loop on a dataset (Basic)

In this example, the workflow uses the In type of loop to repeat an action on each member of an array. Follow the video tutorial or the step-by-step tutorial below.

This workflow is triggered by a scheduled task that pulls threat intel data from Recorded Future daily. You can substitute Recorded Future with your threat intel service.

Pull IP addresses from threat intel vendor

When triggered, the workflow pulls IP addresses from the threat intel vendor, Recorded Future. Below is the output of the Search IP Addresses step.

Extract IP addresses

Since the result of the Search IP Addresses step contains a bunch of additional information, you should use an extraction utility to extract all IPv4 addresses from the result. To extract the IP addresses, input the JSON path that points to $.search_ip_addresses.api_object.data.

The output of the extraction step provides you with a clean list of all IPv4 addresses from the event.

De-duplicate IP addresses

However, there are multiple entries for the same IP address. Passing the results of the extraction step, {{ $.extract_all_ipv4_addresses.results }} to the Unique IPs step de-duplicates the IP addresses.

The output provides you with a list of unique IP addresses from the event, meaning each IP address appears in the list only once.

Loop over unique IP addresses

The list of unique IP addresses contains 10 items. To loop (perform a series of actions) on each of those IP addresses, the input for the Loop operator is the result of the Unique IPs step {{ $.unique_ips.result }}.

Enrich each IP address with VirusTotal

The action performed on each IP address is IP enrichment using VirusTotal. You can substitute VirusTotal with your IP enrichment service.

Each IP address in the list is sent to VirusTotal. Every time an IP address is enriched is called an iteration. Each iteration results in a separate output entry in the execution log. Therefore, there are 10 entries in the execution log. This is easy to spot because they'll generally have the same or very close timestamps.

Loop on a dataset: loop within a loop (Advanced)

Implementing a loop within a loop will require you to manipulate the execution options of the Loop operator so you can access your data and act on it.

This example will show how you can further examine the IP information and ‘double-check’ your findings to make sure the information you have is correct. You can cross-check the IP addresses that came back benign with another platform, and check if the IP address is associated with any problematic domains.

Get information on the domain associated with the benign IP

The IP address that came back as not malicious can be run through an additional engine, in this case, AbuseIPDB. In the IP field, you can use $.value as that contains the IP address from the initial loop, which came back as not malicious at first glance.

AbuseIPDB provides the domain information for the IP address, which can be passed on to VirusTotal for a domain lookup.

Loop over the DNS records of a domain

VirusTotal returns all of the information about the domain, including the DNS records for the domain. You can loop over those records to check for a Type A.

It’s important to note that you can change the loop name and Execution Options to help identify each loop and make it easy to use each loop's Key and Value in context.

• Change the Key name to key_dns_record.
• Change the Value name to value_dns_record.

Check if the domain is associated with other IPs

To act on the object you looped over, reference $.value_dns_record in subsequent steps.

You can now create a condition whereby if the type under $.value_dns_record is A, the IP address of that record is checked. If the IP address for the Type A record is the same as the initial IP address that was already checked by VirusTotal, do nothing, as there is no point in checking the same IP address twice.

However, if the IP address is different from the initial address, the IP address in the type A DNS record should be checked by VirusTotal.

After handling the type A DNS record, you can break the loop since, for the purposes of this tutorial, the assumption is there is only one type A record.

Loop for polling

The following example shows how to use Torq to perform polling, i.e. waiting for a process to complete before moving on with your workflow. It is based on extracting and analyzing URLs from a suspected phishing email.

Since the analysis and the respective report can take some time to complete, you will need to implement a loop waiting for the results.

Extract URLs from a suspicious email

In this example workflow, the trigger is an email that comes from a user asking for the email to be analyzed. This occurs when a user suspects the email might be part of a phishing scam.

Use one of the extraction utilities to extract all of the URLs from the suspected email.

Submit extracted URLs for analysis

When you have all of the URLs, you can start to loop over each one using a loop over a dataset. In the input for the loop, you should enter the results of the extraction utility step.

The action you want to take on each of these URLs is scanning with VirusTotal. Enter the value from the key:value pair of the loop step as the URL to scan. In this example, that is $.loop_2_value.

Wait for analysis results retrieval to complete

Create another loop within this loop using the Range type. In this example, the loop runs a maximum of 20 times (or less if the results are retrieved sooner).

You will then pass the ID of the URL scan to a Get Analysis Results step. This attempts to retrieve the report for the URL that was submitted earlier.

If the analysis has been completed, you can check the results to determine if the URL is malicious. Otherwise, configure the workflow to wait a few seconds (in this case, 10) before checking again. Once that is completed, you can move back to the outer loop and check the next URL.

Loop for pagination

When retrieving large volumes of data where pagination is supported, consider using the optional page size parameter to maintain more manageable step output sizes. In such cases, if there are more entries than what can be returned in a single iteration, the next page token output field will hold a pointer to the next page of results. You can rerun the step and specify the next page token output value of the previous step execution in the next page token optional parameter to get the next page of results. When the next page token is empty or null, it means there are no more results to retrieve.

In the example below, the activity log is being retrieved in pages of 10 entries.

1. Retrieve the first page of results and check whether pagination is required.
2. If pagination is required, add a loop and set the loop type to Until Break.
3. The action performed in each loop iteration is to retrieve the next page of results by specifying the next page token provided by the previous execution of the same step.
4. Use the Collect operator to accumulate the output of each execution of the step responsible for retrieving the relevant data.
5. Following the retrieval of each page of results, check if there are more pages to retrieve.
6. If the next page token is empty or null, it means there are no additional pages to retrieve, and the loop should be broken (by using the Break operator). Otherwise, continue with the loop to retrieve the next page of results.