On various websites, during my crawls when it comes to archiving large files it’s been hitting a navigation timeout. I’ve increased the Page Load Timeout and Behavior Timeout substantially but it continues to timeout at 30000ms. This I believe is a separate timer but I don’t see any documentation about it or reference it. I’ve also looked into the browsertrix-crawler project but no clear solution to this. Any help would be greatly appreciated.
Hi, it looks like you’ve hit the Pupeteer page.goto() method’s default timeout, which is set at 30 seconds. It should be getting overriden by the Page Load Timeout but perhaps it isn’t. Can you try with an older crawler release and see if you have the same issue with e.g. 0.12.4?
If you’re using our hosting service, you should be able to select “Previous” from the crawler channel options, otherwise if you’re self-hosting, you can set multiple crawler channels in the helm chart: Customizing Browsertrix Deployment - Browsertrix Docs