PathologicalPathDecideRule on Browsertrix

I’ve been struggling in crawling a wordpress with a crappy theme producing an endless loop of links, with many identical path segments.

What could be a preferred way to implement a feature like PathologicalPathDecideRule of Heritrix in Browsertrix? A js behaviour?


Rule REJECTs any URI which contains an excessive number of identical, 
consecutive path-segments 
(eg == 3 ‘/a’ segments)

1 Like