How to run custom behaviors with crawler?

Help! Browsertrix Crawler
1

Hi, I followed the tutorial to write a custom behavior for a site I am archiving. I followed the tutorial here: https://github.com/webrecorder/browsertrix-behaviors/blob/main/docs/TUTORIAL.md

And the behavior works as expected when I build, copy the behaviors.js to sources and run self.__bx_behaviors.run() in the console. But when trying to run with the actual crawler I can’t get the behavior to happen.

I ran docker run -p 9037:9037 -v $PWD/crawls:/crawls -v $PWD/tests/custom-behaviors/:/custom-behaviors/ webrecorder/browsertrix-crawler crawl --url https://roosterteeth.com/g/rtnews --customBehaviors /custom-behaviors/ --generateWACZ --text --collection test_behavior --behaviors "siteSpecific" --behaviorTimeout 0 --screencastPort 9037

And watching the screen cast I did not see my behavior run.

I tried placing the built behavior.js mybehavior.js and the ts file of my behavior in the /tests/custom-behavior folder and none of those did anything, and I don’t see any logging from my behavior(Is that normal)

Any help would be appreciated.

2

Hi,

I faced the same issue.

I’m not sure the --customBehaviors is still working, looking at the codebase it looks like it dates from the age where behaviors were still plain JS files.

Since the browsertrix-behaviors contains all regular behaviors, I’ve achieved to successfully run my custom behavior by replacing the whole behavior file:

docker run -v $PWD/custombehaviors/behaviors.js:/app/node_modules/browsertrix-behaviors/dist/behaviors.js --rm webrecorder/b
rowsertrix-crawler:0.12.3 crawl --profile /crawls/profiles/profile.tar.gz --depth 0 --url "https://octo-help.refined.com" --behaviors "autoplay,autofetch,autoscroll,siteSpecific"

Where the custombehaviors/behaviors.js file is the file built with yarn run build or yarn run build-dev.