How can i crawl more pages than total?

General Discussion
1

In Browsertrix, how is this possible?

image
image257×130 3.09 KB

2

There was a bug related to direct-fetched (non HTML) pages that were double counted. It was fixed in: Direct fetched pages may be double-counted · Issue #995 · webrecorder/browsertrix-crawler · GitHub
Which version are you using? It should be fixed in browsertrix crawler 1.12.3+

3

Currently using v1.21.5-8149bef
But job was started 9. mar. 2026. Large job 466GB

Also: We are limited by the 999.999 max pages cap.

4

That’s the Browsertrix app version, the crawler has a separate version and can be configured via the channels in the helm chart:

The default is:

crawler_channels:
  - id: default
    image: "docker.io/webrecorder/browsertrix-crawler:latest"
    imagePullPolicy: Always

so if you haven’t changed it, it’s already using the latest. If it was started with an older version, there’s not an easy way to fix that, but it shouldn’t be adding any more of these

The current crawler version if displayed next to the channel in the workflow UI as well.