Use loops

A loop is a series of continually repeated actions until a certain condition is reached. Alternatively, a loop can also repeat a series of actions on a certain data set. In this case, the action is repeated per the number of items in a dataset.

In Torq, loops are used in workflows for several use cases.

Loop in

• Loop over a dataset and perform an action on each dataset item.
• Loop over a dataset until an object matches certain criteria.

Loop range

• Loop n times (or less) until a certain action takes place.
• Loop n times to retrieve multiple pages of results (pagination).

Key concepts

Understanding the terms and concepts associated with using loops in Torq is important.

Loop operator

The step you add to the designer to implement loop logic in a workflow. You'll define the loop parameters after adding it to the designer.

Mode

Specifies how to execute the items in the loop.

Type

In or range

Action

These are the steps you will execute for each iteration in the loop. When using an In loop, this might be a step to enrich the findings, send a message to an administrator with certain information, etc. When using a Range loop, this might be a Wait operator telling the system to pause while waiting for information or a certain outcome.

Condition

Sometimes, you only want to execute the previously discussed action if a certain condition applies. For example, when looping over an array of email addresses, you only want to execute the action if the email address belongs to a user in the VIP group.

Collect loop results

Torq has a dedicated step that collects the results of a loop's iterations. The step includes two parameters.

• RESULT: The result of a single loop iteration. The result can be a string, number, JSON object, or array representing an individual entry of a collection of results.
• FLATTEN_RESULTS: When true and RESULT is an array, merge that array into the array of collected loop values. Otherwise, it will add the RESULT as a single item in the collected loop results.

Terminate a loop

The Break operator is used within the loop to terminate the loop based on any specific condition before the loop comes to its predefined end. This means that while looping over an array or looping n times if a certain condition is met, terminate the loop.

Loop iteration limits

Loops can run up to 50,000 (fifty thousand) iterations before reaching the iteration limit. Once the iteration limit is reached, the Loop will fail and an appropriate error message will be displayed. 

How to use a loop

In many situations, you will use Torq to ingest data from a 3rd party vendor to perform further action. For example, you are ingesting data from your IAM vendor and using some of that data in your device-management vendor. Or, you are ingesting IP addresses from your threat intel system and enriching those addresses with additional information.

Additionally, you can loop over a data set and include a condition in the loop. If the condition is met, create a list of all of the objects for which the condition is true, and create a different or no list for the objects for which the condition is false.

In all of these examples, you will loop over the data set, and you will perform the same action for each item in that data set.

Loop on a dataset (Basic)

In this example, the workflow uses the In parameter to repeat an action on each member of an array. Follow the video tutorial or the step-by-step tutorial below.

This workflow is triggered by a scheduled task that pulls threat intel data from Recorded Future daily. You can substitute Recorded Future with your threat intel service.

Pull IP addresses from threat intel vendor

When triggered, the workflow pulls IP addresses from the threat intel vendor, Recorded Future. Below is the output of the Search IP Addresses step.

Extract IP addresses

Since the result of the Search IP Addresses step contains a bunch of additional information, you should use an extraction utility to extract all IPv4 addresses from the result. To extract the IP addresses, input the JSON path that points to $.search_ip_addresses.api_object.data.

The output of the extraction step provides you with a clean list of all IPv4 addresses from the event.

De-duplicate IP addresses

However, there are multiple entries for the same IP address. Passing the results of the extraction step, {{ $.extract_all_ipv4_addresses.results }} to the Unique IPs step de-duplicates the IP addresses.

The output provides you with a list of unique IP addresses from the event, meaning each IP address appears in the list only once.

Loop over unique IP addresses

The list of unique IP addresses contains 10 items. To loop (perform a series of actions) on each of those IP addresses, the input for the Loop operator is the result of the Unique IPs step {{ $.unique_ips.result }}.

Enrich each IP address with VirusTotal

The action performed on each IP address is IP enrichment using VirusTotal. You can substitute VirusTotal with your IP enrichment service.

Each IP address in the list is sent to VirusTotal. Every time an IP address is enriched is called an iteration. Each iteration results in a separate output entry in the execution log. Therefore, there are 10 entries in the execution log. This is easy to spot because they'll generally have the same or very close timestamps.

Use loop results (Intermediate)

In the previous example, a basic loop was established to iterate over an array of unique IP addresses and enrich each IP address with information from VirusTotal.

This example will show you how to use the results of the loop step in subsequent parts of the workflow.

Check if the IP is malicious

After enriching the IP addresses, you can check if the verdict of each IP address came back as malicious or benign. The following If step checks what the VirusTotal analysis showed.

Add malicious IPs to an array

 No action is required if all of the engines say that the IP address is clean. However, if at least one of the engines checked comes back with a malicious verdict, a Collect Loop Results utility step (name changed to Malicious IPs) can be used to add that IP address to an array. The array gets the name of the Collect Loop Results step that creates it. In this example, Malicious IPs.

Any IP addresses whose VirusTotal malicious score was at least 1 can be added to the Malicious IPs array by using $.value as the input for the Malicious IPs step (originally Collect Loop Results step).

Send malicious IPs to the admin

When the loop finishes and all malicious IP addresses were added to the Malicious IPs array, you can send a message with the findings to the admin whose email address is defined in the Set Admin Email Address step (originally Set Variable step).

Loop on a dataset (Advanced)

This final example will show how you can create a loop within a loop to further develop your flow. This will require you to manipulate the execution options of the Loop operator so you can access your data and act on it. In addition, this example demonstrates the usage of the Break operator.

The intermediate example showed that you can take the data on which you looped and further act on it, for example, adding the IP addresses to the malicious IP array.

This example will show how you can further examine the IP information and ‘double-check’ your findings to make sure the information you have is correct. You can cross-check the IP addresses that came back benign with another platform, and check if the IP address is associated with any problematic domains.

Get information on the domain associated with the benign IP

The IP address that came back as not malicious can be run through an additional engine, in this case, AbuseIPDB. In the IP field, you can use $.value as that contains the IP address from the initial loop, which came back as not malicious at a first glance.

AbuseIPDB provides the domain information for the IP address, which can be passed on to VirusTotal for a domain lookup.

Loop over the DNS records of a domain

VirusTotal returns all of the information about the domain, including the DNS records for the domain. You can loop over those records to check for a Type A.

It’s important to note that you can change the loop name and Execution Options to help identify each loop and make it easy to use each loop's Key and Value in context.

• Change the Key name to key_dns_record.
• Change the Value name to value_dns_record.

Check if the domain is associated with other IPs

To act on the object you looped over, reference $.value_dns_record in subsequent steps.

You can now create a condition whereby if the type under $.value_dns_record is A, the IP address of that record is checked. If the IP address for the Type A record is the same as the initial IP address that was already checked by VirusTotal, do nothing, as there is no point in checking the same IP address twice.

However, if the IP address is different from the initial address, the IP address in the type A DNS record should be checked by VirusTotal.

After handling the type A DNS record, you can break the loop since for the purposes of this tutorial, the assumption is there is only one type A record.

Add malicious IPs to an array

If the IP address comes back malicious, add the address to an array created by an additional Collect Loop Results step (name changed to Malicious IPs from Reverse DNS lookup). The array gets the name of the step that creates it. In our example, Malicious IPs from Reverse DNS lookup.

You need to add the original IP address that led us to the domain investigation to the Malicious IPs from Reverse DNS lookup array as well, as that IP address is linked to a compromised domain.

When you're done checking all IP addresses and domains you can add the Concat Arrays step (name changed to Malicious IPs and New Findings) to combine the initial malicious IPs with the most recent findings for a complete list of malicious IPs.

Finally, send an updated list of all malicious IP addresses to the admin for further attention.

Loop for pagination

The following example demonstrates how to use Torq to retrieve multiple pages of results. It is based on retrieving alerts for a defined period of time and returning a set number of results in each iteration.

Since there are more results than the amount returned per iteration, we need to loop through the different pages until all of the results are retrieved.

Query an EDR for a list of threats

The first thing you will do is set a time period 6 months ago. This is the earliest date from which you will pull alerts. Use the Get Date step (name changed to Get Date 6 months ago) with the optional parameter TIME_AGO to get the date 6 months prior.

Next, retrieve the threats from an EDR, in this case, SentinelOne.

Query SentinelOne for all threats from the date you previously calculated, meaning 6 months ago. Set the LIMIT parameter of the SentinelOne Get Threats step to 7 to retrieve 7 threats each time the step is run (7 is only for purposes of this example).

After retrieving the threats, you can add the results to an array using the Collect Loop Results step (name changed to Threats). Since we are expecting additional pages of results, this step is useful for storing current, and future results in the same array.

Loop for pagination

Next, you can create a condition that checks within the results of the Get Threats step, if a value exists for the nextCursor parameter. If this parameter has a value, that indicates that there are additional results. If this parameter is empty or null, you know that all results have been retrieved.

If you look back at the Execution Log of the Get Threats step, you can see that the current results show a value for the nextCursor parameter.

Since the condition evaluates as true, you can create a loop that runs either n times (in this example 500), or until the condition evaluates as false.

The number 500 is, on the one hand, arbitrary - it could be 100, it could be 1000. However, since you may not know in advance how many results there are, you would rather have the loop set to a higher number since it will break on its own once all of the results have been retrieved.

Now run the Get Threats step again within the loop to retrieve the next batch of results. You do this by passing the nextCursor parameter in the step.

The results are appended to the array you created earlier with the Collect Loop Results step (name changed to Threats) as long as both steps have the same name.

Lastly, you will add a condition to again check if there is a value in the nextCursor parameter. If there is, you will continue to loop to retrieve the next batch of results. Once the nextCursor comes back with a null value, the loop breaks before completing the 500 iterations as there are no more results to retrieve.

Loop for polling

The following example shows how to use Torq to perform polling, i.e. waiting for a process to complete before moving on with your workflow. It is based on extracting and analyzing URLs from a suspected phishing email.

Since the analysis, and the respective report, can take some time to complete, you will need to implement a loop waiting for the results.

Extract URLs from a suspicious email

In this example workflow, the trigger is an email that comes from a user asking for the email to be analyzed. This occurs when a user suspects the email might be part of a phishing scam.

As with previous workflows, you can set a variable containing the contact information of an admin so they can be contacted within the process.

Use one of the extraction utilities to extract all of the URLs from the suspected email. To do so, you must set the input as the email body.

Submit extracted URLs for analysis

When you have all of the URLs, you can start to loop over each one, using a loop over a dataset (more information on those is available in earlier tutorials). In the input for the loop, you should enter the results of the extraction utility step.

The action you want to take on each of these URLs is scanning with VirusTotal. Enter the value from the key:value pair of the loop step as the URL to scan. In this example, that is $.loop_2_value.

Wait for analysis results retrieval to complete

Create another loop within this loop using the Range parameter. In this example, the loop runs a maximum of 20 times (or less if the results are retrieved sooner).

You will then pass the ID of the URL scan to a Get Analysis Results step. This attempts to retrieve the report for the URL that was submitted earlier.

If the analysis has completed, you can check the results to determine if the URL is malicious. Otherwise, configure the workflow to wait a few seconds (in this case, 10) before checking again. Once that is completed, you can move back to the outer loop and check the next URL.

Notify the admin of investigation results

After all of the URLs are checked, you can create a condition determining which message to send to the admin you defined earlier.

If the Collect Malicious URLs array has any values in it, that means the email was problematic and requires further investigation. However, if all of the URLs were ok, you can send a message that everything is ok.