I have a crawl I am trying to do that keeps getting bogged down and then crashing out, sometimes saying the browser needs to restart.
However, it gets a random mix of messages and I am curious if anyone has insight into what they mean in practical sense and if there are any things I could tweak to better pull it. I have tried from 2-100 concurrent threads and get the same regardless. If I keep restarting eventually it gets the pages.
Messages I see:
Skipping behaviors for slow page
Custom page load check timed out
New Window Timed Out
Finishing Fetch Timed Out
Unable to load new page, browser needs restart
I know some of these messages are all related to timeouts, but it is weird that it happens with some threads and not others and isn’t any kind of blanket block or server timeout.
So I am just curious if anyone has ideas for what I could try to stop running into the timeouts and needing to restart the entire process. The text file is a list of ~1000 specific urls.
All of the pages do load fine if loaded individually. I don’t see any timeouts or slowdowns when directly accessing them.
docker run \
-v /storage/data:/crawls/ \
-v $PWD/url_list.txt:/app/url_list.txt \
-v /storage/collections/data/crawls/20251031175519580-753445a030f5-data.yaml:/app/crawl.yaml \
–publish 9037:9037 \
webrecorder/browsertrix-crawler:latest crawl \
–collection data \
–workers 16 \
–config /app/crawl.yaml \
–seedFile /app/url_list.txt \
–description “Pull of webpage data” \
–exclude site.com/browse \
–scopeType domain \
–saveState always \
–saveStateInterval 15 \
–screenshot thumbnail \
–text final-to-warc \
–generateWACZ \
–screencastPort 9037 \
–extraHops 0 \
–depth 1 \
–behaviorTimeout 300 \
–pageLoadTimeout 300 \
–maxPageRetries 5 \
–restartsOnError true