Solved: Local Install: "No WARC Files, assuming crawl failed. Quitting"

Help! Browsertrix
1

Moving on with a fresh not-so-experimental install on a Ubuntu workstation (still using Minikube), we now face a new error on every crawl: Crawling seems to start but immediatly stops with “No WARC Files, assuming crawl failed. Quitting” (no other errors indicated in the UI logfile). The (simple, static) test site in question is reachable interactively and also when creating a browser profile and has been our successful first test crawl target many times.

Unfortunately, crawler pods like “crawl-manual-20260731152959-908b15e2-586-0” and redis pods like “redis-manual-20260731153103-908b15e2-586 " are created dynamically and disappear, before we can get any status information or logs. Also, a timestamp is part of there their names so whe can’t prepare a 'kubectl logs” cmd easily.

Is there any way to make these dynamic pods more persistent, so we are able to do some debugging? Or an environment variable to have them save their logs?

TIA
Heinz

2

Using this script we were able to capture crawler pod log files:

#!/bin/bash

# Try to get debug info from dynamic crawling pods

stdbuf -i0 -o0 -e0 kubectl get pods -n crawlers -w  |
while read pod numbers state dummy
do
    echo "$pod" "$numbers" "$state" "$dummy"
    if [[ "$state" == "Terminating" ]]
    then
        kubectl logs "$pod" -n crawlers > "$pod.log" 
    fi
done

Turns out it simply was

{“timestamp”:“2026-08-04T06:54:07.702Z”,“logLevel”:“info”,“context”:“general”,“message”:“Disk utilization threshold reached 92% > 90%, stopping”,“details”:{}}

Browsertrix on Windows: How to log Container Crashes etc?
3

Hm, does the pod start with very small disk space? It shouldn’t be running out of disk space on startup. If so, can disable the disk utilization check..
How much storage space are you configuring the pods with via crawler_storage ?

4

Yes, indeed, problem is (potentially) solved, will free some space soon: