How can we debug and avoid such errors on container level? (Sorry, still Kubernetes noob here, just followed your excellent guides for installation.)
This crawl initially ran fine, only the second and subsequent calls no longer run (after finetuning some parameters). Also other crawls still run just fine.
With access to a log file, this “Crash” turned out to be just an expired login in the browser profile, so the crawler was redirected to a login page and could not find any matching URLs.
The message “Crawler Instance Crashed” seems a little over-dramatic in this case.