Security

Infrastructure credentials

The default configuration ships with weak, well-known passwords for PostgreSQL, MinIO, and Redis. Replace them with strong, unique values before exposing Tracecat to production traffic.

# PostgreSQL
TRACECAT__POSTGRES_USER=tracecat
TRACECAT__POSTGRES_PASSWORD=<strong random password>

# Temporal PostgreSQL
TEMPORAL__POSTGRES_USER=temporal
TEMPORAL__POSTGRES_PASSWORD=<strong random password>

# MinIO / S3
MINIO_ROOT_USER=<strong random user>
MINIO_ROOT_PASSWORD=<strong random password>

# Redis — add a password and update the URL
REDIS_URL=redis://:<strong random password>@redis:6379

For Redis, also pass the password to the container. Add a command override in your docker-compose.yml:

services:
  redis:
    command: ["redis-server", "--requirepass", "<same password as above>"]

In production, prefer managed services (e.g., Amazon RDS, ElastiCache) over self-hosted PostgreSQL and Redis: they handle encryption at rest, automated backups, and credential rotation.

Platform secrets

Tracecat requires four cryptographic secrets, plus an optional keyring if you enable Temporal payload encryption.

Secret	Protects
`TRACECAT__DB_ENCRYPTION_KEY`	Credentials and sensitive settings stored in PostgreSQL. A Fernet-format key.
`TRACECAT__SERVICE_KEY`	Service-to-service JWT signing.
`TRACECAT__SIGNING_SECRET`	Webhook URL signing and HMAC operations.
`USER_AUTH_SECRET`	Password reset tokens, email verification, OAuth state, the MCP OIDC issuer keypair, and the internal OIDC client secret.
`TEMPORAL__PAYLOAD_ENCRYPTION_KEYRING`	Versioned keyring for Temporal payload encryption. Optional. Tracecat reads it only when `TEMPORAL__PAYLOAD_ENCRYPTION_ENABLED` is true.

Tracecat requires four secrets. Generate them with openssl:

# Service key (hex, 32 bytes) — used for internal service-to-service JWT signing
openssl rand -hex 32

# Signing secret (hex, 32 bytes) — used for webhook URL signing and HMAC operations
openssl rand -hex 32

# User auth secret (hex, 32 bytes) — used for password reset, email verification, OAuth state, and OIDC key derivation
openssl rand -hex 32

# DB encryption key (Fernet-compatible base64, 32 bytes) — used for encrypting secrets at rest
openssl rand 32 | base64 | tr -d '\n' | tr '+/' '-_'

Store these securely. Losing TRACECAT__DB_ENCRYPTION_KEY makes encrypted credentials unrecoverable. Losing TRACECAT__SIGNING_SECRET invalidates existing webhook URLs and agent channel endpoint URLs.

See Where to store them for the recommended storage on each deployment target, and which of these secrets can be rotated.

Temporal payload keyring

TEMPORAL__PAYLOAD_ENCRYPTION_KEYRING holds a JSON keyring rather than a single key. Each entry maps a key ID to a root secret, and current_key_id selects the key that encrypts new payloads:

{"current_key_id": "v1", "keys": {"v1": "<root secret>"}}

Generate each root secret the same way as the other secrets:

openssl rand -hex 32

Tracecat derives a separate AES-256-GCM key per workspace from the root secret, and stamps the key ID into every payload it encrypts. That stamp is what lets you rotate the keyring while older histories stay readable. On AWS, supply TEMPORAL__PAYLOAD_ENCRYPTION_KEYRING_ARN instead, and Tracecat fetches the keyring from Secrets Manager at runtime.

Database encryption

TRACECAT__DB_ENCRYPTION_ALGORITHM picks the cipher Tracecat uses when it writes an encrypted value to PostgreSQL:

Value	Cipher
`fernet` (default)	Fernet: AES-128-CBC with an HMAC-SHA256 tag.
`aes256gcm`	AES-256-GCM, keyed by HKDF-SHA256 from the 32 bytes of `TRACECAT__DB_ENCRYPTION_KEY`.

The setting controls writes only. Tracecat reads both formats whatever the setting is, so existing rows stay readable after a switch, and rows written before the switch stay Fernet until you re-encrypt them. Tracecat refuses to start when the value is anything else. TRACECAT__DB_ENCRYPTION_PREVIOUS_KEYS takes a comma-separated list of retired keys. Tracecat decrypts with TRACECAT__DB_ENCRYPTION_KEY first, then each previous key, and always encrypts with TRACECAT__DB_ENCRYPTION_KEY. Each AES-256-GCM value carries a key ID derived from its key, so Tracecat picks the right key without trying each one. Set both variables on every service that receives TRACECAT__DB_ENCRYPTION_KEY. The bundled Compose file, Helm chart, and Fargate module do not pass them through, so add them to each service’s environment, for example in a docker-compose.override.yml.

Once any service writes with aes256gcm, Tracecat versions without this setting cannot read those rows. To downgrade, set TRACECAT__DB_ENCRYPTION_ALGORITHM=fernet, run the re-encrypt command below, and only then deploy the older version.

Re-encrypt stored values

The re-encrypt command rewrites every encrypted value that is not yet in the current algorithm and key. Run it in the api container, with the same environment as the running services:

docker compose exec api python -m tracecat.secrets.reencrypt --dry-run
docker compose exec api python -m tracecat.secrets.reencrypt

On Kubernetes or Fargate, run the same module in an api pod or task.

--dry-run prints per-column counts of Fernet values, AES-256-GCM values, values already current, and values no configured key can decrypt. It writes nothing.
Without --dry-run, the command scans first and refuses to write if any value fails to decrypt. Add the missing key to TRACECAT__DB_ENCRYPTION_PREVIOUS_KEYS, or pass --skip-undecryptable to leave those values unchanged.
Writes go in committed batches of --batch-size rows (default 200). The command skips values that are already current and leaves a value alone if the application rewrites it mid-run, so you can stop it and run it again at any time.

The output reports counts only, never values.

Rotate the database key

Generate a new key with openssl rand 32 | base64 | tr -d '\n' | tr '+/' '-_'.
Add the new key to TRACECAT__DB_ENCRYPTION_PREVIOUS_KEYS on every service and deploy. Every service can now read values written under it.
Set TRACECAT__DB_ENCRYPTION_KEY to the new key, move the old key to TRACECAT__DB_ENCRYPTION_PREVIOUS_KEYS, and deploy. New writes use the new key.
Confirm that every service that receives TRACECAT__DB_ENCRYPTION_KEY has restarted with the new values. In the bundled Compose file these are api, worker, executor, agent-worker, agent-executor, litellm, and mcp. Check each service’s environment, with docker compose config <service>, the ECS task definition, or the pod spec, and check that it started after step 3. A service still running with the old key keeps writing rows under it.
Run the re-encrypt command with --dry-run to check the counts, then without it to rewrite the values. Run --dry-run again and confirm every value counts as current.
Remove the old key from TRACECAT__DB_ENCRYPTION_PREVIOUS_KEYS and deploy. Keep a backup of it until you no longer need database backups taken before the final dry run in step 5.

Removing the old key makes every row still encrypted under it unreadable, and Tracecat cannot recover those rows without the key. Remove it only after step 4 confirms that no service still uses it, and after the final dry run in step 5 shows every value as current.

Rotating the key also resets MCP OAuth state that Tracecat keeps in Redis under TRACECAT__DB_ENCRYPTION_KEY, so MCP clients sign in again. To switch algorithms without rotating, set TRACECAT__DB_ENCRYPTION_ALGORITHM on every service and deploy. Confirm as in step 4 that every service restarted with the new value, then run step 5. A service still on the old algorithm keeps writing that format, which matters when you switch back to fernet before downgrading.

Where to store them

Keep these out of your application configuration and out of version control. Use the mechanism your platform already provides.

Deployment	Recommended storage
AWS ECS Fargate	AWS Secrets Manager. Pass secret ARNs to Terraform; ECS resolves the values at task launch, so they never appear in your `.tfvars`.
Kubernetes	External Secrets Operator syncing from your secret manager, or an existing Kubernetes Secret you create out of band.
Docker Compose	A `.env` file on the host. Restrict it to the service account that runs Tracecat and exclude it from version control.

On Kubernetes, prefer External Secrets Operator over a hand-created Secret. Your secret manager stays the source of truth, the operator re-syncs values rather than copying them, and no plaintext passes through a shell history or a manifest. Reserve chart-managed secret templates for pipelines that encrypt values at rest with Sealed Secrets or SOPS.

Rotation

Rotation support differs by secret. Check the differences before you plan a rotation window.

Losing TRACECAT__DB_ENCRYPTION_KEY makes every stored credential unrecoverable. Back it up, and control access to it at least as tightly as the credentials it encrypts.

Secret	Overlap window	Rotation procedure	Blast radius
`TEMPORAL__PAYLOAD_ENCRYPTION_KEYRING`	Yes	Add the new key to `keys` and let every service pick it up, then point `current_key_id` at it. Keep retired keys as long as you retain their histories.	None for existing histories, which stay readable under the key they were written with. Between the two steps, a service still on the old keyring cannot decrypt payloads written under the new key. In ARN mode that gap lasts up to `TEMPORAL__PAYLOAD_ENCRYPTION_CACHE_TTL_SECONDS`.
`TRACECAT__DB_ENCRYPTION_KEY`	Yes	Follow Rotate the database key.	MCP clients sign in again. Removing the old key before every service has restarted with the new key, and before re-encryption finishes, makes rows still encrypted under the old key unreadable.
`TRACECAT__SIGNING_SECRET`	No	Rotate, then reissue every affected URL.	Every webhook URL and agent channel endpoint URL. Breaks in-flight Slack OAuth installs, and pending Slack approval buttons for up to 24 hours.
`TRACECAT__SERVICE_KEY`	No	Restart every service that holds it in one window, including `ui`, which uses it for the SAML ACS proxy.	While services disagree, in-flight agent turns fail their next tool or LLM call, executor calls fail, and service-to-service requests return `401`. Miss `ui` and SSO login breaks.
`USER_AUTH_SECRET`	No	Restart `api` and `mcp` together.	Active MCP access tokens (1-hour lifetime), and in-flight password reset and verification links. Sessions are database-backed, so users stay logged in. MCP refresh tokens are hashed and survive.

Isolation

Tracecat executes user-defined Python scripts, custom actions, and agents inside the executor service. You choose between nsjail isolation and no isolation. Defaults differ by deployment target.

Deployment	Default
Kubernetes	nsjail sandbox. The chart applies the required security context to the executor pods.
Docker Compose	No isolation. `TRACECAT__EXECUTOR_BACKEND=direct`.
AWS Fargate	No isolation. Fargate cannot grant the capabilities nsjail requires.

nsjail sandbox (recommended for production)

For production, enable nsjail — a process isolation tool from Google that enforces:

Filesystem isolation — scripts reach only their job directory and explicitly mounted paths. The host filesystem stays invisible.
Resource limits — nsjail caps CPU time, memory, file size, and process count per execution, so a runaway script cannot starve the host.
User namespace separation — scripts run as unprivileged users even when the container runs as root.
Network access — scripts keep network access, since they need to reach databases, APIs, and S3, but nsjail confines it to the container’s network namespace.

To enable nsjail, set the following in your .env:

TRACECAT__EXECUTOR_BACKEND=nsjail

The legacy ephemeral value remains an alias for nsjail. For Docker Compose, use the sandbox overlay in the startup instructions to supply container permissions and cgroup delegation.

The direct backend does not sandbox ordinary workflow actions. Set TRACECAT__EXECUTOR_BACKEND=nsjail for supported action isolation.

nsjail requires:

Linux with kernel 4.6+
Docker privileged mode or CAP_SYS_ADMIN capability on the executor container
The nsjail binary and sandbox rootfs (included in Tracecat images)

nsjail is not supported on macOS or Windows. Use the direct backend on those platforms.

Resource limits

When you enable nsjail, each agent, Python script, custom action, and dependency-install sandbox gets a cgroup v2 memory budget shared by its child processes.

Sandbox	Memory cap	Configure with	CPU time	File size	Processes
`core.script.run_python` and custom actions	2048 MB	`TRACECAT__SANDBOX_DEFAULT_MEMORY_MB`	300 s	256 MB	64
Dependency installation for `core.script.run_python`	2048 MB	`TRACECAT__SANDBOX_DEFAULT_MEMORY_MB`	300 s	256 MB	64
Agents	4096 MB	`TRACECAT__AGENT_SANDBOX_MEMORY_MB`	Agent timeout ceiling	256 MB	128

Pass TRACECAT__SANDBOX_DEFAULT_MEMORY_MB to your action executor and TRACECAT__AGENT_SANDBOX_MEMORY_MB to agent-executor. Docker Compose forwards both settings. Helm chart 0.7.0 or newer exposes them as executor.sandboxMemoryMib and agentExecutor.sandboxMemoryMib and provides the required executor cgroup bootstrap. Sandboxes have no per-process address-space cap (rlimit_as) by default, because the cgroup memory budget already bounds RAM. Set TRACECAT__SANDBOX_ADDRESS_SPACE_MB=8192 or TRACECAT__AGENT_SANDBOX_ADDRESS_SPACE_MB=16384 to add one. Size this cap well above your workload’s peak virtual size, since runtimes such as DuckDB reserve far more address space than the RAM they use. Stdio MCP connection probes use the same policy with a memory budget capped at 1024 MiB. NsJail requires a writable cgroup v2 subtree with the memory controller enabled. The Compose sandbox overlay starts python -m tracecat.sandbox.bootstrap for executors and python -m tracecat.agent.sandbox.cgroup for agent executors as root to prepare the container’s subtree, then drops privileges to apiuser before starting the worker. Other deployments must use that bootstrap with an isolated, writable container cgroup namespace, or delegate a subtree and set TRACECAT__SANDBOX_CGROUP_PATH=/sys/fs/cgroup/action-sandboxes for executors or TRACECAT__AGENT_SANDBOX_CGROUP_PATH=/sys/fs/cgroup/agent-sandboxes for agent executors. Both workers reject missing delegation before accepting activities. Set concurrency so all sandbox budgets plus worker overhead fit within the outer container limit. Tracecat does not resize concurrency automatically. A per-sandbox limit cannot prevent a container OOM when the sum of concurrent workloads exceeds that outer limit. Tracecat reports an action or agent run that exceeds the memory, CPU time, or file size limit as sandbox.resource_limit_exceeded. The failure is attributed to the workload, is not retried, and the error names the env var that controls the cap. Two limits fall outside that guarantee. Exceeding the process count fails process creation inside the sandbox with an ordinary error instead of killing the workload, so it surfaces as whatever the script or agent does with that error. Dependency installation is reported as a package installation failure whichever limit it hits. These limits apply only when nsjail is enabled. Without it Tracecat installs no rlimits, so a failure of the same shape is reported as an ordinary workload or platform failure.

No isolation

Without nsjail, scripts, custom actions, and agents run as regular subprocesses in the executor. This is a supported production configuration when you trust everything that runs. That means your own workflow and custom registry code, the third-party dependencies those actions install, and the agents, tools, and MCP servers you enable. Review that code and pin those dependencies as you would any code with direct access to your systems. Choose nsjail instead when you run code you have not reviewed, such as untrusted third-party packages or agents that generate and execute their own code.

Choosing a backend

Backend	Isolation	Latency	Use case
`direct`	Does not support sandboxing	~50ms	Trusted code, dependencies, and agents
`nsjail` (`ephemeral` alias)	nsjail sandbox (cold per action)	~4000ms	Untrusted code, dependencies, and agents

Authentication

Docker Compose deployments default to basic email/password authentication. For production, configure OIDC or SAML SSO. To require a hardware security key or passkey at sign-in, set TRACECAT__PUBLIC_APP_URL to an HTTPS URL and turn on multi-factor authentication. Browsers only offer security keys on HTTPS or localhost, and each key stays bound to that host. See Roles and permissions for the roles you assign to users and groups once they can sign in.

TLS

Never run production traffic over plain HTTP. See TLS and certificates for Caddy-based automatic TLS setup, custom certificates, and trusting internal CAs.

See Architecture for the platform and AI agent trust model.

Self-hosting

Manage platform

Authentication

Security

Infrastructure credentials

Platform secrets

Temporal payload keyring

Database encryption

Re-encrypt stored values

Rotate the database key

Where to store them

Rotation

Isolation

nsjail sandbox (recommended for production)

Resource limits

No isolation

Choosing a backend

Authentication

TLS

​Infrastructure credentials

​Platform secrets

​Temporal payload keyring

​Database encryption

​Re-encrypt stored values

​Rotate the database key

​Where to store them

​Rotation

​Isolation

​nsjail sandbox (recommended for production)

​Resource limits

​No isolation

​Choosing a backend

​Authentication

​TLS

​Related pages

Infrastructure credentials

Platform secrets

Temporal payload keyring

Database encryption

Re-encrypt stored values

Rotate the database key

Where to store them

Rotation

Isolation

nsjail sandbox (recommended for production)

Resource limits

No isolation

Choosing a backend

Authentication

TLS

Related pages