Crawler Instance Crashed

[Mareike]

Good Morning

Trying to crawl https://www.unisg.ch/de with different kind of Crawler Channels, but all brought me after 12 to 18min to “Crawler Instance Crashed” and before there were many pages with a “Link extraction timed out” - but all these looked fine to me when trying to open them. After short time it ends the Crawl without having (of course) everthing crawled

Does somebody have an idea what could help or what could cause that error?

Thank you so much

Mareike

[ilya]

Hi Mareike,

It is normal for crawler to occasionally crash and then resume. It looks like all of those crawls were cancelled, so hard to say exactly what was happening. It’s possible the site started blocking the crawler. I’ve started a crawl with 2 windows instead of 4 and with a profile to see if that works more smoothly - we’ll keep any eye on it to see if something else is happening.