HTTP Status 400 & 500 - LineTooLong

Tools Q&A
#1

I am running Pywb and started to receive HTTP Status “500 Internal Server Error” and “400 Bad Request” errors with below message:

Internal Error: ('Connection aborted.', LineTooLong('got more than 131072 bytes when reading header line'))

At the beginning it was working, but now I have 4.000 index (cdxj) files with 4.000 *.warc.gz and it started to generate errors. How can I solve it?

#2

It’s hard to say exactly, it seems perhaps one of the cdxj may be corrupt…
Can you reindex the WARCs using the wb-manager reindex <collection> command, or directly using the new cdxj-indexer with cdxj-indexer -s </path/to/warcs> (the -s flag is to generate a sorted output). Hopefully that will fix the issue…

#3

Thanks for your reply @ilya. I was using cdxj-indexer and re-indexed again with cdxj-indexer, however Pywb fails when I create more than 1.000 cdxj files. Anyway I decided to run my own ReplayWeb.page.