​
What you’ll learn

By the end of this tutorial, you’ll learn how to:

  • Set up the fetcher-processor pattern to process lists of data.
  • Create lookup tables with natural keys for upserts.

Benefits of the fetcher-processor pattern:

  1. Processor workflows can be reused by other fetcher workflows.
  2. Processor workflows can be tested in isolation with a single item.
  3. Fetcher workflows can be scheduled to run at specified intervals.
  4. Lower memory usage per workflow when processing large lists of data.

​
Fetcher-processor

This pattern consists of two workflows:

  • A fetcher workflow that fetches a list of data (e.g. list of alerts).
  • A processor workflow that receives and processes one item at a time (e.g. enrich alert and open a case)

It’s easier to start with the processor workflow, which only processes one item at a time. Once that’s working, we can add the fetcher workflow that fetches a list of data and passes it to the processor workflow.

​
Tutorial

​
Prerequisites

  • Knowledge of Tracecat for-loops and child workflows
  • URLScan and URLHaus abuse.ch API keys added as Tracecat secrets.
  • A table with url (text), download (jsonb), asn (jsonb), and geoip (jsonb) columns. url as the index.

​
Steps

1

Processor workflow

In the following example, we assume the workflow receives a url field from the TRIGGER, then looks up the URL with URLScan and extracts important fields.

2

Fetcher workflow

In the following example, we fetch a list of URLs from URLHaus and pass them to the processor workflow.

3

Lookup table

Go to tables to view the results of the processor workflow.

​
Best practices

  • Add new workflow logic into the processor workflow.
  • If the processor workflow gets too large, use child workflows in the processor workflow to split it into smaller ones.
  • The fetcher workflow should only be used to fetch data and (optionally) filter them into different processor workflows.

To implement branching logic (similar to if-conditions) with lists of data, use core.transform.filter or JSONPath filters.

  • Learn about the filter action in the data transforms tutorial.
  • Learn about JSONPath filters in the JSONPath cheatsheet.

​
Use cases

Common processor workflows include:

  • Normalizing a single alert (e.g. renaming fields)
  • Enriching indicators of compromise (IoCs) in a single alert
  • Opening a single case in a ticketing system (Tracecat cases, Jira, etc.)

These processor workflows can be reused by different fetcher workflows. Examples of fetcher workflows include:

  • Ingesting alerts from a SIEM
  • Ingesting indicators from a threat intelligence platform