I’m struggling with how to best accomplish this in Browsertrix, so all tips and ideas would be helpful
We would like to do a broadcrawl of something like 1mio domains with 1 hop and max 10 pages per domain.
Is that possible without creating a job per domain?
I can get 1 page per domain by doing crawllists with 1000 domains frontpageurls in each job. But I can’t figure out how to set limits per domain.
Thank you
Hi Thomas,
Browsertrix doesn’t currently support per-domain limits but I can see how this would be very useful for a broad domain crawl like this!
The other issue I can see that you hint at is that our UI also doesn’t currently support setting a scope type other than page for URL lists, which means that the kind of scoping you are describing is only possible via the API (and likely by programmatically specifying all the seeds in the POST request, rather than using a seed list file). That might require batching the domains into separate workflows.
Otherwise, at the moment creating a job per-domain is the best supported way of doing this in the UI.
You might also be interested in looking at this crawler PR, which we haven’t yet had time to review/merge but I believe introduces a number of changes that would be helpful for the type of broad domain crawl you’re describing: Finalize per-domain stats and completeness for staged domain crawls by aponb · Pull Request #1027 · webrecorder/browsertrix-crawler · GitHub
Thank you Tessa. I have been doing API crawlgeneration with seedfiles.
It sounds like it might be possible to do scope type other than page for URL lists via the API.
These broadcrawls are run entirely by API so that would be fine. But does that mean I could add 1000 seeds to different domains, and get them all crawled by setting scope to domain?
Anyway - as long as I can not set per-domain limits also, it is not going to solve my problem.