Skip to main content

Retrohunt Configuration and Operations Guide

This guide describes how to configure, deploy, scale, validate, and troubleshoot the Azul Retrohunt plugin.

Retrohunt provides on-demand historical YARA scanning over previously ingested content. It uses a retroingestor, one or more hunt workers, an indexer, Redis, shared persistent storage, dispatcher services, and Prometheus metrics.


Architecture overview

ComponentCommandPurpose
Retroingestorazul-plugin-retroingestorCoordinates ingestion, index maintenance, cleanup, and Retrohunt operations.
Workerazul-plugin-retroworkerExecutes broad-phase and narrow-phase YARA searches.
Content indexerazul-plugin-retroindexer --indexer-name contentBuilds long-term searchable indexes from content streams.
RedisExternal serviceStores hunt state, work queues
Persistent volumeMounted at /indicesStores bgi indexes.
Dispatcher/streamsAzul platform servicesRetrieve candidate files during narrow-phase scanning.
PrometheusMetrics consumerScrapes metrics from each Retrohunt container.

All Retrohunt containers must mount the same index volume at the path configured by PLUGIN_ROOT_PATH.

/indices

Prerequisites

Before enabling Retrohunt, ensure that:

  • The plugin-retrohunt image is available.
  • Redis is deployed and reachable.
  • The Redis secret contains redis-username and redis-password.
  • A persistent volume can be provisioned with sufficient capacity and I/O performance.
  • All Retrohunt containers can mount the same index volume.
  • Dispatcher services can retrieve files from configured sources.
  • The cluster has sufficient memory and CPU for all extra containers in the Retrohunt pod.
  • The content stream label exists and receives files that should be indexed.
  • Prometheus scraping is configured when metrics are required.
  • The Helm chart supports extraContainers using plugin.container.main.

Base configuration

retrohunt:
type: standard
enabled: true

# CronJob schedule used to clean up hunts.
schedule: "0 2 * * *"

pvc:
size: "100Gi"

# Shared configuration applied to Retrohunt containers.
config:
max_thread_count: "10" # number active narrow search threads for each container
PLUGIN_ENABLE_MEM_LIMITS: "false"
PLUGIN_ROOT_PATH: "/indices"

PLUGIN_INDEXERS:
content:
name: content
stream_labels:
- content
max_bytes_before_indexing: "10GiB"
periodic_index_frequency_min: "60"
timeout_minutes: "60"
allow_splitting_and_deletion: "true"

Enabling Retrohunt

Enable the plugin with:

retrohunt:
enabled: true

Disable it without deleting the configuration with:

retrohunt:
enabled: false

Disabling Retrohunt does not automatically delete the PVC or existing index data.


Hunt cleanup schedule

The cleanup CronJob uses:

schedule: "0 2 * * *"

This runs daily at 02:00 according to the timezone used by the Kubernetes CronJob.

Cron format is:

minute hour day-of-month month day-of-week

Examples:

# Daily at 02:00
schedule: "0 2 * * *"

# Every six hours
schedule: "0 */6 * * *"

# Every Sunday at 03:00
schedule: "0 3 * * 0"

Persistent storage

Retrohunt uses a persistent volume for indexes:

pvc:
size: "100Gi"

The size of the pvc should be configured according to your environment. e.g. 67 bgi index files comes to roughly 223Gi etc.

The plugin mounts the claim:

volumes:
- name: indices
persistentVolumeClaim:
claimName: plugin-retrohunt

Every Retrohunt container mounts it at:

volumeMounts:
- name: indices
mountPath: "/indices"

This must match:

PLUGIN_ROOT_PATH: "/indices"

Storage sizing

Required capacity depends on:

  • Total content volume.
  • Number and size of indexes.
  • Index creation frequency.
  • Temporary indexing data.
  • Failed or split index directories.
  • index retention.

Storage performance

Retrohunt can be I/O intensive. Prefer storage with good sequential throughput, low metadata latency, and sufficient IOPS for multiple workers and the indexer.

Recreate strategy

The main Retrohunt workload uses:

strategy:
type: Recreate

This stops the current pod before starting the replacement. It is appropriate when the PVC is ReadWriteOnce or when two main pods must not operate on the same index directory simultaneously.

Do not change to RollingUpdate without verifying storage access and application safety.


Shared configuration

Settings under:

retrohunt:
config:

are made available through the retrohunt ConfigMap:

envFrom:
- configMapRef:
name: retrohunt
apiVersion: v1
kind: ConfigMap
metadata:
name: retrohunt
data:
REDIS_HOST: '{{ (split ":" .Values.external.redis.endpoint)._0 }}'
REDIS_PORT: '{{ (split ":" .Values.external.redis.endpoint)._1 }}'
REDIS_DB: "{{ .Values.external.redis.db }}"
REDIS_CLEANUP_DELAY: "{{ .Values.external.redis.cleanup_delay }}"
REDIS_CLEANUP_RUNNING_DELAY: "{{ .Values.external.redis.cleanup_running_delay }}"
REDIS_TTL: "{{ .Values.external.redis.ttl }}"
REDIS_EXCEPTION_WAIT: "{{ .Values.external.redis.exception_wait }}"
MAX_THREAD_COUNT: "{{ .Values.plugins.retrohunt.config.max_thread_count }}"
REDIS_CLEANUP_DELAY: "{{ .Values.external.redis.cleanup_delay }}"

Set this value to the number of days before a stored hunt is cleaned up by the cronjob.

REDIS_CLEANUP_RUNNING_DELAY: "{{ .Values.external.redis.cleanup_running_delay }}"

Set this value to the number of days before a running hunt is cleaned up by the cronjob.

REDIS_TTL: "{{ .Values.external.redis.ttl }}"

This is the time in seconds for a worker to hold a redis jobstream (hunt job). The worker will periodically refresh this time as it works. If a worker fails at some point when processing a hunt, and stops refreshing the ttl, another worker will pick up the job when this time expires.

A pod restart is normally required after changing these values.


Narrow-phase thread count

The number of narrow-phase threads used by each worker is:

max_thread_count: "10"

This value applies independently to every worker container.

WorkersThreads per workerMaximum worker threads
11010
21020
31030
41040

Adding workers multiplies total concurrency; it does not divide the thread count between workers. We have found that this can have a heavy impact on Dispatcher CPU.

Memory guidance

Use at least 4 GiB per worker as a minimum baseline.

For 10 threads, 4 to 8 GiB per worker is a safe starting point, especially for large files or large narrow-phase candidate sets.

resources:
requests:
memory: "2Gi"
cpu: "100m"
limits:
memory: "4Gi"
cpu: "1000m"

Memory use increases with:

  • Thread count.
  • Concurrent downloads.
  • Candidate file size.
  • YARA working memory.
  • Multiple simultaneous hunts.
  • Dispatcher buffering.
  • Python and allocator behaviour.

If a worker is OOM-killed:

  1. Reduce max_thread_count.
  2. Increase worker memory.
  3. Reduce the number of simultaneous workers.
  4. Review candidate counts and file sizes.
  5. Check whether several hunts run concurrently.

CPU guidance

A limit of:

cpu: "1000m"

allows one CPU core.

More CPU helps when YARA scanning is CPU-bound. It may not improve performance when workers are waiting for dispatcher retrieval, network I/O, or persistent storage.


Plugin memory limits

The internal plugin memory limiter is disabled:

PLUGIN_ENABLE_MEM_LIMITS: "false"

This avoids incorrect memory-limit behaviour caused by cgroup detection.


Content indexer configuration

PLUGIN_INDEXERS:
content:
name: content
stream_labels:
- content
max_bytes_before_indexing: "10GiB"
periodic_index_frequency_min: "60"
timeout_minutes: "60"
allow_splitting_and_deletion: "true"

Indexer name

The indexer command must match the configured name:

command:
- azul-plugin-retroindexer
- "--indexer-name"
- "content"

Stream labels

The indexer processes data associated with:

stream_labels:
- content

Additional labels may be added, but this increases indexing and storage load.

Index byte threshold

max_bytes_before_indexing: "10GiB"

When accumulated input reaches approximately 10 GiB, Retrohunt creates a new long-term index.

Smaller values create indexes more often. Larger values create fewer, larger indexes.

Periodic indexing

periodic_index_frequency_min: "60"

This allows indexing every 60 minutes even when the byte threshold has not been reached.

Lower values make low-volume data searchable sooner but increase indexing overhead.

Index timeout

timeout_minutes: "60"

An indexing operation exceeding this duration is terminated and treated as failed.

Increase it only after checking storage, CPU, memory, and index size.

Failed-index splitting

allow_splitting_and_deletion: "true"

If an index directory fails twice, Retrohunt may split it into smaller directories and remove the failed directory.

Use this only where source data can be regenerated or re-ingested.


Main Retrohunt container

The main container runs:

command:
- azul-plugin-retroingestor

Example:

plugins:
retrohunt-worker:
additionalLabels:
allow-egress-redis: "true"

strategy:
type: Recreate

promMetricsEnabled: true
promPort: "8900"

image: plugin-retrohunt
runTimeout: "6000"
maxFileSize: "100000000"
useSmartScaler: false

command:
- azul-plugin-retroingestor

volumeMounts:
- name: indices
mountPath: "/indices"

volumes:
- name: indices
persistentVolumeClaim:
claimName: plugin-retrohunt

envFrom:
- configMapRef:
name: retrohunt

baseEnv:
- name: REDIS_USERNAME
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-username

- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-password

Run timeout

runTimeout: "6000"

This is 6,000 seconds, or 100 minutes. It allows long-running maintenance, indexing, or compaction tasks to complete.

Maximum file size

maxFileSize: "100000000"

This is approximately 100 MB in decimal bytes.

Larger files take longer to download and scan and may increase memory use. Review dispatcher timeouts and worker memory before increasing it.

Smart scaler

useSmartScaler: false

The smart scaler remains disabled because it cannot reliably detect or manage extra worker containers. Scale Retrohunt by adding or removing explicit extraContainers.


Redis configuration

Each Retrohunt container requires Redis credentials:

- name: REDIS_USERNAME
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-username

- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-password

Verify the secret exists:

kubectl get secret <redis-secret-name> -n <namespace>

Verify the expected keys without decoding them:

kubectl get secret <redis-secret-name> -n <namespace> -o jsonpath='{.data.redis-username}'
kubectl get secret <redis-secret-name> -n <namespace> -o jsonpath='{.data.redis-password}'

Do not expose decoded production credentials in documentation or tickets.


Default worker

extraContainers:
worker:
template: "plugin.container.main"
promMetricsEnabled: true
promPort: "8901"

envFrom:
- configMapRef:
name: retrohunt

env:
- name: PLUGIN_PROMETHEUS_PORT_WORKER
value: "8901"

- name: REDIS_USERNAME
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-username

- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-password

image: plugin-retrohunt

resources:
requests:
memory: "2Gi"
cpu: "100m"
limits:
memory: "4Gi"
cpu: "1000m"

command:
- azul-plugin-retroworker

volumeMounts:
- name: indices
mountPath: "/indices"

The worker command must be:

command:
- azul-plugin-retroworker

The Helm metrics port must match the environment variable:

promPort: "8901"
- name: PLUGIN_PROMETHEUS_PORT_WORKER
value: "8901"

Content indexer container

content-indexer:
template: "plugin.container.main"
promMetricsEnabled: true
promPort: "8902"

envFrom:
- configMapRef:
name: retrohunt

env:
- name: PLUGIN_PROMETHEUS_PORT_INDEXER
value: "8902"

- name: REDIS_USERNAME
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-username

- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-password

image: plugin-retrohunt

resources:
requests:
memory: "8Gi"
cpu: "100m"
limits:
memory: "16Gi"
cpu: "4000m"

command:
- azul-plugin-retroindexer
- "--indexer-name"
- "content"

volumeMounts:
- name: indices
mountPath: "/indices"

The indexer receives more CPU and memory than a standard worker because index creation can be resource intensive.

Monitor index duration, memory, CPU, temporary storage, failure count, and timeout events.


Adding extra workers

Additional workers are added under:

retrohunt:
plugins:
retrohunt-worker:
extraContainers:

Each extra worker requires:

  • A unique container name.
  • template: "plugin.container.main".
  • The plugin-retrohunt image.
  • azul-plugin-retroworker.
  • The retrohunt ConfigMap.
  • Redis credentials.
  • A unique Prometheus port.
  • Matching PLUGIN_PROMETHEUS_PORT_WORKER.
  • Resource requests and limits.
  • The /indices volume mount.

Example:

retrohunt:
debug: true

plugins:
retrohunt-worker:
debug: false

extraContainers:
worker-2:
template: "plugin.container.main"
allowAllTraffic: true
promMetricsEnabled: true
promPort: "8903"

envFrom:
- configMapRef:
name: retrohunt

env:
- name: REDIS_USERNAME
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-username

- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-password

- name: CONTAINER_MEMORY_LIMIT_MI
valueFrom:
resourceFieldRef:
containerName: worker-2
resource: limits.memory
divisor: 1Mi

- name: PLUGIN_PROMETHEUS_PORT_WORKER
value: "8903"

image: plugin-retrohunt

resources:
requests:
memory: "2Gi"
cpu: "100m"
limits:
memory: "4Gi"
cpu: "1000m"

command:
- azul-plugin-retroworker

volumeMounts:
- name: indices
mountPath: "/indices"

worker-3:
template: "plugin.container.main"
allowAllTraffic: true
promMetricsEnabled: true
promPort: "8904"

envFrom:
- configMapRef:
name: retrohunt

env:
- name: REDIS_USERNAME
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-username

- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-password

- name: CONTAINER_MEMORY_LIMIT_MI
valueFrom:
resourceFieldRef:
containerName: worker-3
resource: limits.memory
divisor: 1Mi

- name: PLUGIN_PROMETHEUS_PORT_WORKER
value: "8904"

image: plugin-retrohunt

resources:
requests:
memory: "2Gi"
cpu: "100m"
limits:
memory: "4Gi"
cpu: "1000m"

command:
- azul-plugin-retroworker

volumeMounts:
- name: indices
mountPath: "/indices"

Network policy

The base plugin includes:

additionalLabels:
allow-egress-redis: "true"

Extra workers may use:

allowAllTraffic: true

Prefer the least permissive network policy supported by the environment.

Workers normally require access to:

  • Redis.
  • Dispatcher services.
  • Supporting Azul APIs.
  • Prometheus scraping endpoints.

Avoid allowAllTraffic: true in production when specific policies are available.


Worker scaling behaviour

Additional workers primarily improve total throughput and queue processing when several hunts run at once.

They may not make a single hunt faster.

Performance depends on:

  • Hunt assignment.
  • Candidate counts.
  • Dispatcher throughput.
  • Storage I/O.
  • Redis performance.
  • Network latency.
  • Worker CPU.
  • Number of active hunts.

Adding workers can reduce performance when they compete for dispatcher, storage, Redis, node CPU, or network bandwidth.

For example, three workers using 10 threads each can generate up to 30 concurrent narrow-phase operations.


Dispatcher and streams tuning

High Retrohunt thread counts can place significant load on dispatcher file retrieval.

When worker CPU is low but hunts are slow, the bottleneck may be dispatcher or streams retrieval rather than YARA scanning.

For high thread counts, allocate approximately 3 to 4 CPUs to the dispatcher path as a starting point.

Example:

dispatcher:
...
plugin:
events:
env:
BED.LOG_LEVEL: "INFO"
BED.LOG_PRETTY: "TRUE"
DP.EVENTS.DEDUPE_CACHE_BYTES: "1Gi"
DP.EVENTS.REPLAY_PLUGIN_CACHE.SIZE_BYTES: "1Gi"

replicas: 1

resources:
requests:
memory: "12Gi"
cpu: "500m"
limits:
memory: "12Gi"
cpu: "2"

streams:
env:
BED.LOG_LEVEL: "INFO"
BED.LOG_PRETTY: "TRUE"
DP.STREAMS.CACHE.SIZE_BYTES: "2Gi"
DP.STREAMS.CACHE.SHARDS: 32

replicas: 1

resources:
requests:
memory: "8Gi"
cpu: "2"
limits:
memory: "16Gi"
cpu: "4"

Streams service

requests:
memory: "8Gi"
cpu: "2"

limits:
memory: "16Gi"
cpu: "4"

When Retrohunt is slow and workers show low CPU:

  1. Check the streams CPU is not too low.
  2. Check dispatcher request latency.
  3. Check storage latency.
  4. Check network throughput.
  5. Increase dispatcher-related CPU to 3 or 4 cores when CPU constrained.
  6. Increase streams CPU where retrieval concurrency is high.
  7. Avoid increasing worker threads until retrieval is healthy.

Increasing max_thread_count without increasing dispatcher capacity can increase contention without improving hunt duration.


Capacity planning

Small deployment

max_thread_count: "4"

Worker:

resources:
requests:
memory: "1Gi"
cpu: "100m"
limits:
memory: "2Gi"
cpu: "1000m"

Use one worker for evaluation, small indexes, and low hunt volume.

Medium deployment

max_thread_count: "8"

Worker:

resources:
requests:
memory: "2Gi"
cpu: "500m"
limits:
memory: "4Gi"
cpu: "2000m"

Use two workers for moderate concurrent hunt volume.

High-concurrency deployment

max_thread_count: "10"

Worker:

resources:
requests:
memory: "4Gi"
cpu: "500m"
limits:
memory: "8Gi"
cpu: "2000m"

Use five or more workers only after reviewing dispatcher CPU, streams CPU, Redis, storage throughput, and total pod size.


Total pod sizing

All extra containers run in one Retrohunt pod. The scheduler must place the entire pod on one node.

Example:

ContainerMemory requestMemory limit
Worker4 GiB4 GiB
Worker 28 GiB8 GiB
Worker 38 GiB8 GiB
Content indexer8 GiB16 GiB

These extra containers request 28 GiB and have a combined 36 GiB limit, excluding the retroingestor.

The node must also have room for Kubernetes services, DaemonSets, runtime overhead, and other workloads.

If the pod remains pending:

kubectl describe pod <retrohunt-pod> -n <namespace>

Look for:

Insufficient memory
Insufficient cpu
volume node affinity conflict
unbound immediate PersistentVolumeClaims

Prometheus ports

Every container must use a unique port.

ContainerPort
Retroingestor8900
Worker8901
Content indexer8902
Worker 28903
Worker 38904

For a worker:

promPort: "8903"

must match:

- name: PLUGIN_PROMETHEUS_PORT_WORKER
value: "8903"

For an indexer:

promPort: "8902"

must match:

- name: PLUGIN_PROMETHEUS_PORT_INDEXER
value: "8902"

Use 8905 for a fourth worker unless another container already uses it.


Debugging

Group-level debugging:

retrohunt:
debug: true

Plugin override:

plugins:
retrohunt-worker:
debug: false

Use debug logging temporarily. It can increase log volume, storage use, and operational noise.


Troubleshooting

Pod remains pending

Check:

kubectl describe pod <retrohunt-pod> -n <namespace>

Common causes:

  • Insufficient node memory.
  • Insufficient CPU.
  • Unbound PVC.
  • Volume node affinity.
  • The combined pod is too large.
  • Taints or selectors prevent scheduling.

Worker is OOM-killed

Symptoms:

Reason: OOMKilled
Exit Code: 137

Actions:

  1. Reduce max_thread_count.
  2. Increase worker memory.
  3. Prefer 8 GiB for 10 threads.
  4. Reduce simultaneous workers.
  5. Review file size and candidate counts.
  6. Check for concurrent hunts.

Memory remains elevated after a hunt

Container memory may remain high because Python, YARA, allocators, and filesystem caches retain reusable memory.

A stable plateau is less concerning than continuous growth over repeated hunts.

Run several comparable hunts and check whether memory stabilises or continues increasing toward the limit.

Worker CPU is low but hunts are slow

Likely causes:

  • Dispatcher retrieval bottleneck.
  • Streams CPU saturation.
  • Storage I/O saturation.
  • Network latency.
  • Redis delay.
  • Large candidate files.
  • Too much worker concurrency.

Check shared services before increasing threads.

Extra worker does not start

Check:

  • Unique container name.
  • Correct YAML indentation.
  • Correct template.
  • Correct image.
  • azul-plugin-retroworker.
  • Redis secret references.
  • /indices mount.
  • Unique metrics port.
  • Correct promMetricsEnabled capitalisation.
  • Correct resourceFieldRef.containerName.
  • Node capacity.

Extra worker has no metrics

Check:

promMetricsEnabled: true

and ensure:

promPort: "8903"

matches:

PLUGIN_PROMETHEUS_PORT_WORKER: "8903"

Also confirm Prometheus configuration and network policy.

Worker cannot connect to Redis

Check:

  • Secret name.
  • Required keys.
  • Network policy.
  • Redis DNS.
  • Authentication.
  • TLS requirements.

Worker cannot find indexes

Confirm every container uses:

PLUGIN_ROOT_PATH: "/indices"

and:

mountPath: "/indices"

Inspect the mounted contents from each container.

Indexer repeatedly times out

Check:

  • Index size.
  • Storage performance.
  • Indexer CPU.
  • Indexer memory.
  • Input volume.
  • Timeout value.
  • Failed-index splitting.

Possible changes:

timeout_minutes: "120"

or:

max_bytes_before_indexing: "5GiB"

Increase the timeout only when indexing is still making healthy progress.

Hunt restarts after narrow phase

Check:

  • Worker restart count.
  • OOM events.
  • Worker exceptions.
  • Redis hunt-state updates.
  • Dispatcher timeouts.
  • Final hunt-state persistence.