Retrohunt Configuration and Operations Guide
This guide describes how to configure, deploy, scale, validate, and troubleshoot the Azul Retrohunt plugin.
Retrohunt provides on-demand historical YARA scanning over previously ingested content. It uses a retroingestor, one or more hunt workers, an indexer, Redis, shared persistent storage, dispatcher services, and Prometheus metrics.
Architecture overview
| Component | Command | Purpose |
|---|---|---|
| Retroingestor | azul-plugin-retroingestor | Coordinates ingestion, index maintenance, cleanup, and Retrohunt operations. |
| Worker | azul-plugin-retroworker | Executes broad-phase and narrow-phase YARA searches. |
| Content indexer | azul-plugin-retroindexer --indexer-name content | Builds long-term searchable indexes from content streams. |
| Redis | External service | Stores hunt state, work queues |
| Persistent volume | Mounted at /indices | Stores bgi indexes. |
| Dispatcher/streams | Azul platform services | Retrieve candidate files during narrow-phase scanning. |
| Prometheus | Metrics consumer | Scrapes metrics from each Retrohunt container. |
All Retrohunt containers must mount the same index volume at the path configured by PLUGIN_ROOT_PATH.
/indices
Prerequisites
Before enabling Retrohunt, ensure that:
- The
plugin-retrohuntimage is available. - Redis is deployed and reachable.
- The Redis secret contains
redis-usernameandredis-password. - A persistent volume can be provisioned with sufficient capacity and I/O performance.
- All Retrohunt containers can mount the same index volume.
- Dispatcher services can retrieve files from configured sources.
- The cluster has sufficient memory and CPU for all extra containers in the Retrohunt pod.
- The
contentstream label exists and receives files that should be indexed. - Prometheus scraping is configured when metrics are required.
- The Helm chart supports
extraContainersusingplugin.container.main.
Base configuration
retrohunt:
type: standard
enabled: true
# CronJob schedule used to clean up hunts.
schedule: "0 2 * * *"
pvc:
size: "100Gi"
# Shared configuration applied to Retrohunt containers.
config:
max_thread_count: "10" # number active narrow search threads for each container
PLUGIN_ENABLE_MEM_LIMITS: "false"
PLUGIN_ROOT_PATH: "/indices"
PLUGIN_INDEXERS:
content:
name: content
stream_labels:
- content
max_bytes_before_indexing: "10GiB"
periodic_index_frequency_min: "60"
timeout_minutes: "60"
allow_splitting_and_deletion: "true"
Enabling Retrohunt
Enable the plugin with:
retrohunt:
enabled: true
Disable it without deleting the configuration with:
retrohunt:
enabled: false
Disabling Retrohunt does not automatically delete the PVC or existing index data.
Hunt cleanup schedule
The cleanup CronJob uses:
schedule: "0 2 * * *"
This runs daily at 02:00 according to the timezone used by the Kubernetes CronJob.
Cron format is:
minute hour day-of-month month day-of-week
Examples:
# Daily at 02:00
schedule: "0 2 * * *"
# Every six hours
schedule: "0 */6 * * *"
# Every Sunday at 03:00
schedule: "0 3 * * 0"
Persistent storage
Retrohunt uses a persistent volume for indexes:
pvc:
size: "100Gi"
The size of the pvc should be configured according to your environment. e.g. 67 bgi index files comes to roughly 223Gi etc.
The plugin mounts the claim:
volumes:
- name: indices
persistentVolumeClaim:
claimName: plugin-retrohunt
Every Retrohunt container mounts it at:
volumeMounts:
- name: indices
mountPath: "/indices"
This must match:
PLUGIN_ROOT_PATH: "/indices"
Storage sizing
Required capacity depends on:
- Total content volume.
- Number and size of indexes.
- Index creation frequency.
- Temporary indexing data.
- Failed or split index directories.
- index retention.
Storage performance
Retrohunt can be I/O intensive. Prefer storage with good sequential throughput, low metadata latency, and sufficient IOPS for multiple workers and the indexer.
Recreate strategy
The main Retrohunt workload uses:
strategy:
type: Recreate
This stops the current pod before starting the replacement. It is appropriate when the PVC is ReadWriteOnce or when two main pods must not operate on the same index directory simultaneously.
Do not change to RollingUpdate without verifying storage access and application safety.
Shared configuration
Settings under:
retrohunt:
config:
are made available through the retrohunt ConfigMap:
envFrom:
- configMapRef:
name: retrohunt
apiVersion: v1
kind: ConfigMap
metadata:
name: retrohunt
data:
REDIS_HOST: '{{ (split ":" .Values.external.redis.endpoint)._0 }}'
REDIS_PORT: '{{ (split ":" .Values.external.redis.endpoint)._1 }}'
REDIS_DB: "{{ .Values.external.redis.db }}"
REDIS_CLEANUP_DELAY: "{{ .Values.external.redis.cleanup_delay }}"
REDIS_CLEANUP_RUNNING_DELAY: "{{ .Values.external.redis.cleanup_running_delay }}"
REDIS_TTL: "{{ .Values.external.redis.ttl }}"
REDIS_EXCEPTION_WAIT: "{{ .Values.external.redis.exception_wait }}"
MAX_THREAD_COUNT: "{{ .Values.plugins.retrohunt.config.max_thread_count }}"
REDIS_CLEANUP_DELAY: "{{ .Values.external.redis.cleanup_delay }}"
Set this value to the number of days before a stored hunt is cleaned up by the cronjob.
REDIS_CLEANUP_RUNNING_DELAY: "{{ .Values.external.redis.cleanup_running_delay }}"
Set this value to the number of days before a running hunt is cleaned up by the cronjob.
REDIS_TTL: "{{ .Values.external.redis.ttl }}"
This is the time in seconds for a worker to hold a redis jobstream (hunt job). The worker will periodically refresh this time as it works. If a worker fails at some point when processing a hunt, and stops refreshing the ttl, another worker will pick up the job when this time expires.
A pod restart is normally required after changing these values.
Narrow-phase thread count
The number of narrow-phase threads used by each worker is:
max_thread_count: "10"
This value applies independently to every worker container.
| Workers | Threads per worker | Maximum worker threads |
|---|---|---|
| 1 | 10 | 10 |
| 2 | 10 | 20 |
| 3 | 10 | 30 |
| 4 | 10 | 40 |
Adding workers multiplies total concurrency; it does not divide the thread count between workers. We have found that this can have a heavy impact on Dispatcher CPU.
Memory guidance
Use at least 4 GiB per worker as a minimum baseline.
For 10 threads, 4 to 8 GiB per worker is a safe starting point, especially for large files or large narrow-phase candidate sets.
resources:
requests:
memory: "2Gi"
cpu: "100m"
limits:
memory: "4Gi"
cpu: "1000m"
Memory use increases with:
- Thread count.
- Concurrent downloads.
- Candidate file size.
- YARA working memory.
- Multiple simultaneous hunts.
- Dispatcher buffering.
- Python and allocator behaviour.
If a worker is OOM-killed:
- Reduce
max_thread_count. - Increase worker memory.
- Reduce the number of simultaneous workers.
- Review candidate counts and file sizes.
- Check whether several hunts run concurrently.
CPU guidance
A limit of:
cpu: "1000m"
allows one CPU core.
More CPU helps when YARA scanning is CPU-bound. It may not improve performance when workers are waiting for dispatcher retrieval, network I/O, or persistent storage.
Plugin memory limits
The internal plugin memory limiter is disabled:
PLUGIN_ENABLE_MEM_LIMITS: "false"
This avoids incorrect memory-limit behaviour caused by cgroup detection.
Content indexer configuration
PLUGIN_INDEXERS:
content:
name: content
stream_labels:
- content
max_bytes_before_indexing: "10GiB"
periodic_index_frequency_min: "60"
timeout_minutes: "60"
allow_splitting_and_deletion: "true"
Indexer name
The indexer command must match the configured name:
command:
- azul-plugin-retroindexer
- "--indexer-name"
- "content"
Stream labels
The indexer processes data associated with:
stream_labels:
- content
Additional labels may be added, but this increases indexing and storage load.
Index byte threshold
max_bytes_before_indexing: "10GiB"
When accumulated input reaches approximately 10 GiB, Retrohunt creates a new long-term index.
Smaller values create indexes more often. Larger values create fewer, larger indexes.
Periodic indexing
periodic_index_frequency_min: "60"
This allows indexing every 60 minutes even when the byte threshold has not been reached.
Lower values make low-volume data searchable sooner but increase indexing overhead.
Index timeout
timeout_minutes: "60"
An indexing operation exceeding this duration is terminated and treated as failed.
Increase it only after checking storage, CPU, memory, and index size.
Failed-index splitting
allow_splitting_and_deletion: "true"
If an index directory fails twice, Retrohunt may split it into smaller directories and remove the failed directory.
Use this only where source data can be regenerated or re-ingested.
Main Retrohunt container
The main container runs:
command:
- azul-plugin-retroingestor
Example:
plugins:
retrohunt-worker:
additionalLabels:
allow-egress-redis: "true"
strategy:
type: Recreate
promMetricsEnabled: true
promPort: "8900"
image: plugin-retrohunt
runTimeout: "6000"
maxFileSize: "100000000"
useSmartScaler: false
command:
- azul-plugin-retroingestor
volumeMounts:
- name: indices
mountPath: "/indices"
volumes:
- name: indices
persistentVolumeClaim:
claimName: plugin-retrohunt
envFrom:
- configMapRef:
name: retrohunt
baseEnv:
- name: REDIS_USERNAME
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-username
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-password
Run timeout
runTimeout: "6000"
This is 6,000 seconds, or 100 minutes. It allows long-running maintenance, indexing, or compaction tasks to complete.
Maximum file size
maxFileSize: "100000000"
This is approximately 100 MB in decimal bytes.
Larger files take longer to download and scan and may increase memory use. Review dispatcher timeouts and worker memory before increasing it.
Smart scaler
useSmartScaler: false
The smart scaler remains disabled because it cannot reliably detect or manage extra worker containers. Scale Retrohunt by adding or removing explicit extraContainers.
Redis configuration
Each Retrohunt container requires Redis credentials:
- name: REDIS_USERNAME
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-username
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-password
Verify the secret exists:
kubectl get secret <redis-secret-name> -n <namespace>
Verify the expected keys without decoding them:
kubectl get secret <redis-secret-name> -n <namespace> -o jsonpath='{.data.redis-username}'
kubectl get secret <redis-secret-name> -n <namespace> -o jsonpath='{.data.redis-password}'
Do not expose decoded production credentials in documentation or tickets.
Default worker
extraContainers:
worker:
template: "plugin.container.main"
promMetricsEnabled: true
promPort: "8901"
envFrom:
- configMapRef:
name: retrohunt
env:
- name: PLUGIN_PROMETHEUS_PORT_WORKER
value: "8901"
- name: REDIS_USERNAME
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-username
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-password
image: plugin-retrohunt
resources:
requests:
memory: "2Gi"
cpu: "100m"
limits:
memory: "4Gi"
cpu: "1000m"
command:
- azul-plugin-retroworker
volumeMounts:
- name: indices
mountPath: "/indices"
The worker command must be:
command:
- azul-plugin-retroworker
The Helm metrics port must match the environment variable:
promPort: "8901"
- name: PLUGIN_PROMETHEUS_PORT_WORKER
value: "8901"
Content indexer container
content-indexer:
template: "plugin.container.main"
promMetricsEnabled: true
promPort: "8902"
envFrom:
- configMapRef:
name: retrohunt
env:
- name: PLUGIN_PROMETHEUS_PORT_INDEXER
value: "8902"
- name: REDIS_USERNAME
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-username
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-password
image: plugin-retrohunt
resources:
requests:
memory: "8Gi"
cpu: "100m"
limits:
memory: "16Gi"
cpu: "4000m"
command:
- azul-plugin-retroindexer
- "--indexer-name"
- "content"
volumeMounts:
- name: indices
mountPath: "/indices"
The indexer receives more CPU and memory than a standard worker because index creation can be resource intensive.
Monitor index duration, memory, CPU, temporary storage, failure count, and timeout events.
Adding extra workers
Additional workers are added under:
retrohunt:
plugins:
retrohunt-worker:
extraContainers:
Each extra worker requires:
- A unique container name.
template: "plugin.container.main".- The
plugin-retrohuntimage. azul-plugin-retroworker.- The
retrohuntConfigMap. - Redis credentials.
- A unique Prometheus port.
- Matching
PLUGIN_PROMETHEUS_PORT_WORKER. - Resource requests and limits.
- The
/indicesvolume mount.
Example:
retrohunt:
debug: true
plugins:
retrohunt-worker:
debug: false
extraContainers:
worker-2:
template: "plugin.container.main"
allowAllTraffic: true
promMetricsEnabled: true
promPort: "8903"
envFrom:
- configMapRef:
name: retrohunt
env:
- name: REDIS_USERNAME
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-username
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-password
- name: CONTAINER_MEMORY_LIMIT_MI
valueFrom:
resourceFieldRef:
containerName: worker-2
resource: limits.memory
divisor: 1Mi
- name: PLUGIN_PROMETHEUS_PORT_WORKER
value: "8903"
image: plugin-retrohunt
resources:
requests:
memory: "2Gi"
cpu: "100m"
limits:
memory: "4Gi"
cpu: "1000m"
command:
- azul-plugin-retroworker
volumeMounts:
- name: indices
mountPath: "/indices"
worker-3:
template: "plugin.container.main"
allowAllTraffic: true
promMetricsEnabled: true
promPort: "8904"
envFrom:
- configMapRef:
name: retrohunt
env:
- name: REDIS_USERNAME
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-username
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: "{{ .Values.secrets.redis }}"
key: redis-password
- name: CONTAINER_MEMORY_LIMIT_MI
valueFrom:
resourceFieldRef:
containerName: worker-3
resource: limits.memory
divisor: 1Mi
- name: PLUGIN_PROMETHEUS_PORT_WORKER
value: "8904"
image: plugin-retrohunt
resources:
requests:
memory: "2Gi"
cpu: "100m"
limits:
memory: "4Gi"
cpu: "1000m"
command:
- azul-plugin-retroworker
volumeMounts:
- name: indices
mountPath: "/indices"
Network policy
The base plugin includes:
additionalLabels:
allow-egress-redis: "true"
Extra workers may use:
allowAllTraffic: true
Prefer the least permissive network policy supported by the environment.
Workers normally require access to:
- Redis.
- Dispatcher services.
- Supporting Azul APIs.
- Prometheus scraping endpoints.
Avoid allowAllTraffic: true in production when specific policies are available.
Worker scaling behaviour
Additional workers primarily improve total throughput and queue processing when several hunts run at once.
They may not make a single hunt faster.
Performance depends on:
- Hunt assignment.
- Candidate counts.
- Dispatcher throughput.
- Storage I/O.
- Redis performance.
- Network latency.
- Worker CPU.
- Number of active hunts.
Adding workers can reduce performance when they compete for dispatcher, storage, Redis, node CPU, or network bandwidth.
For example, three workers using 10 threads each can generate up to 30 concurrent narrow-phase operations.
Dispatcher and streams tuning
High Retrohunt thread counts can place significant load on dispatcher file retrieval.
When worker CPU is low but hunts are slow, the bottleneck may be dispatcher or streams retrieval rather than YARA scanning.
For high thread counts, allocate approximately 3 to 4 CPUs to the dispatcher path as a starting point.
Example:
dispatcher:
...
plugin:
events:
env:
BED.LOG_LEVEL: "INFO"
BED.LOG_PRETTY: "TRUE"
DP.EVENTS.DEDUPE_CACHE_BYTES: "1Gi"
DP.EVENTS.REPLAY_PLUGIN_CACHE.SIZE_BYTES: "1Gi"
replicas: 1
resources:
requests:
memory: "12Gi"
cpu: "500m"
limits:
memory: "12Gi"
cpu: "2"
streams:
env:
BED.LOG_LEVEL: "INFO"
BED.LOG_PRETTY: "TRUE"
DP.STREAMS.CACHE.SIZE_BYTES: "2Gi"
DP.STREAMS.CACHE.SHARDS: 32
replicas: 1
resources:
requests:
memory: "8Gi"
cpu: "2"
limits:
memory: "16Gi"
cpu: "4"
Streams service
requests:
memory: "8Gi"
cpu: "2"
limits:
memory: "16Gi"
cpu: "4"
When Retrohunt is slow and workers show low CPU:
- Check the streams CPU is not too low.
- Check dispatcher request latency.
- Check storage latency.
- Check network throughput.
- Increase dispatcher-related CPU to 3 or 4 cores when CPU constrained.
- Increase streams CPU where retrieval concurrency is high.
- Avoid increasing worker threads until retrieval is healthy.
Increasing max_thread_count without increasing dispatcher capacity can increase contention without improving hunt duration.
Capacity planning
Small deployment
max_thread_count: "4"
Worker:
resources:
requests:
memory: "1Gi"
cpu: "100m"
limits:
memory: "2Gi"
cpu: "1000m"
Use one worker for evaluation, small indexes, and low hunt volume.
Medium deployment
max_thread_count: "8"
Worker:
resources:
requests:
memory: "2Gi"
cpu: "500m"
limits:
memory: "4Gi"
cpu: "2000m"
Use two workers for moderate concurrent hunt volume.
High-concurrency deployment
max_thread_count: "10"
Worker:
resources:
requests:
memory: "4Gi"
cpu: "500m"
limits:
memory: "8Gi"
cpu: "2000m"
Use five or more workers only after reviewing dispatcher CPU, streams CPU, Redis, storage throughput, and total pod size.
Total pod sizing
All extra containers run in one Retrohunt pod. The scheduler must place the entire pod on one node.
Example:
| Container | Memory request | Memory limit |
|---|---|---|
| Worker | 4 GiB | 4 GiB |
| Worker 2 | 8 GiB | 8 GiB |
| Worker 3 | 8 GiB | 8 GiB |
| Content indexer | 8 GiB | 16 GiB |
These extra containers request 28 GiB and have a combined 36 GiB limit, excluding the retroingestor.
The node must also have room for Kubernetes services, DaemonSets, runtime overhead, and other workloads.
If the pod remains pending:
kubectl describe pod <retrohunt-pod> -n <namespace>
Look for:
Insufficient memory
Insufficient cpu
volume node affinity conflict
unbound immediate PersistentVolumeClaims
Prometheus ports
Every container must use a unique port.
| Container | Port |
|---|---|
| Retroingestor | 8900 |
| Worker | 8901 |
| Content indexer | 8902 |
| Worker 2 | 8903 |
| Worker 3 | 8904 |
For a worker:
promPort: "8903"
must match:
- name: PLUGIN_PROMETHEUS_PORT_WORKER
value: "8903"
For an indexer:
promPort: "8902"
must match:
- name: PLUGIN_PROMETHEUS_PORT_INDEXER
value: "8902"
Use 8905 for a fourth worker unless another container already uses it.
Debugging
Group-level debugging:
retrohunt:
debug: true
Plugin override:
plugins:
retrohunt-worker:
debug: false
Use debug logging temporarily. It can increase log volume, storage use, and operational noise.
Troubleshooting
Pod remains pending
Check:
kubectl describe pod <retrohunt-pod> -n <namespace>
Common causes:
- Insufficient node memory.
- Insufficient CPU.
- Unbound PVC.
- Volume node affinity.
- The combined pod is too large.
- Taints or selectors prevent scheduling.
Worker is OOM-killed
Symptoms:
Reason: OOMKilled
Exit Code: 137
Actions:
- Reduce
max_thread_count. - Increase worker memory.
- Prefer 8 GiB for 10 threads.
- Reduce simultaneous workers.
- Review file size and candidate counts.
- Check for concurrent hunts.
Memory remains elevated after a hunt
Container memory may remain high because Python, YARA, allocators, and filesystem caches retain reusable memory.
A stable plateau is less concerning than continuous growth over repeated hunts.
Run several comparable hunts and check whether memory stabilises or continues increasing toward the limit.
Worker CPU is low but hunts are slow
Likely causes:
- Dispatcher retrieval bottleneck.
- Streams CPU saturation.
- Storage I/O saturation.
- Network latency.
- Redis delay.
- Large candidate files.
- Too much worker concurrency.
Check shared services before increasing threads.
Extra worker does not start
Check:
- Unique container name.
- Correct YAML indentation.
- Correct template.
- Correct image.
azul-plugin-retroworker.- Redis secret references.
/indicesmount.- Unique metrics port.
- Correct
promMetricsEnabledcapitalisation. - Correct
resourceFieldRef.containerName. - Node capacity.
Extra worker has no metrics
Check:
promMetricsEnabled: true
and ensure:
promPort: "8903"
matches:
PLUGIN_PROMETHEUS_PORT_WORKER: "8903"
Also confirm Prometheus configuration and network policy.
Worker cannot connect to Redis
Check:
- Secret name.
- Required keys.
- Network policy.
- Redis DNS.
- Authentication.
- TLS requirements.
Worker cannot find indexes
Confirm every container uses:
PLUGIN_ROOT_PATH: "/indices"
and:
mountPath: "/indices"
Inspect the mounted contents from each container.
Indexer repeatedly times out
Check:
- Index size.
- Storage performance.
- Indexer CPU.
- Indexer memory.
- Input volume.
- Timeout value.
- Failed-index splitting.
Possible changes:
timeout_minutes: "120"
or:
max_bytes_before_indexing: "5GiB"
Increase the timeout only when indexing is still making healthy progress.
Hunt restarts after narrow phase
Check:
- Worker restart count.
- OOM events.
- Worker exceptions.
- Redis hunt-state updates.
- Dispatcher timeouts.
- Final hunt-state persistence.