I’m having trouble getting a URL-list crawl to work with a short list of anchor-style URLS. Here are a couple examples:
The page in full where these anchors appear is crawled and replays fine, but in replay, if a user clicks on a top-of-page link to any of the lower sections like those listed above, they get a “Sorry, this page was not found in this archive.”
Should crawling of these kinds of URLs work? Or is there something that needs to be configured on the replay side to make them work?
Clarification: I’m using the Browsertrix Cloud in-app replay right now. I haven’t yet tried loading the WACZ into my hosted page where replay.web is embedded. But either way, the crawl of these URLs is failing.
What you seem to be describing here is the URL List working as expected. URL List Workflows will only download the pages specified in the URL List. Optionally, they can visit every link on those pages if Include Any Linked Page is turned on.
It sounds like you might want to try a seeded crawl for https://countryofwords.supdigital.org/ instead? I think the default settings should capture what you seem to be after?
As bonus info: For most sites, ID links like the examples here don’t need to be specifically captured as discrete pages because they usually just jump to content that is already present in the webpage and doesn’t need to be loaded from an external source. Sometimes that also isn’t the case, with “single page applications” that do load new content and surface that as the same URL but with a # link. For these cases, we offer the Hashtag Links Only scope option
Thank you, Hank. My first pass at archiving this site was actually to do the seeded crawl of https://countryofwords.supdigital.org/. It didn’t capture the # links on the homepage (which, you’re absolutely right, are just sections of content on the page). This was not consistent with other pages in the project that also contain anchor links. But whereas the anchor links on those other pages are linked to from a list in the body of the page, the ones on this homepage are linked top from a graphic header that’s been giving me some other problems. It may very well be acting as a single page application (but only on this one page?). In any case, I tried the Hashtag Links Only option (thank you for pointing that out!), but I’m still getting the same results.
Since the issue is limited to this one page, and a user could still access the content by simply scrolling own the page, I’m inclined to just put a note on the HTML page I’m embedding the player into, informing users to just scroll that page rather than access the content via the links.
I appreciate the help, though, and the suggestions. I had completely overlooked the hashtags option in the seeded crawl!
Curious… Either way, if you’re finding different levels of success with multiple different scopes, if you haven’t already, try adding them all to a collection! It will effectively merge them and allow each crawl to access data from the others, patching the parts that don’t work!
If you’re unable to capture the content within Browsertrix, you could also try manually capturing the interactions on the page with the ArchiveWeb.page extension, uploading that, and adding it to the same collection to patch the crawl.
It appears that I should have had a deeper look at your site before making suggestions…
I have some thoughts, but will send to the team first. It is looking more like a replay issue to me. Do you know if there is a specific reason your timeline links are prefaced with /timeline/?? This may the the cause of some of the behavior.
Thanks for the further investigation! The /timeline/ situation is due to that being the page (the Timeline page) the anchors are on. For instance the other pages–Visualizations, Network, and Audio Interviews–are /visualisations/ [sic], /network/, and /audio-interviews/, respectively. I did crawl the full anchor links (e.g. countryofwords dot supdigital dot org/timeline/#literary-diasporas-post-nakba-scattering), but you’re right that on replay, everything after the /timeline/ disappears. This is not the case, however on the individual pages like countryofwords dot supdigital dot org/periods/literary-diasporas-the-mahjar, where anchor links like countryofwords dot supdigital dot org/periods/literary-diasporas-the-mahjar/#heading-332b5ff37526 are indeed working and showing correctly.
On the public facing page where I’ve embedded the replay, I’ve simply made a note that these links do not work and the user should simply scroll down the page to access each section. Since the issue is so minor for this particular archive and only limited to this one page I ended up going with this (non)solution
This is not the case, however on the individual pages […] where anchor links […] are indeed working and showing correctly.
I would hazard a guess that this is because they don’t include the page prefix. I’ve never seen in-text links include that on other sites and while browsers seem to deal with it properly ReplayWeb.page is having trouble with re-writing those links correctly.
Any chance you can remove the page prefix for them on the site? Prefixing ID links with a path should only be required if the path navigates to a different page.
If it helps, here’s the exact same SVG code the site has for the timeline but with the /timeline/ prefixes removed. Nothing should change for users, but it will hopefully archive correctly!
Well it’s all a bit complicated. I’m not the site developer, so I can only relay what I can observe in the live hosted site files (which I do maintain). It appears the timeline is being generated by JavaScript (“svelte” if that makes sense). So the svg is not actually in that index page code. Instead there’s simply a div that’s pulling in further JS files, including the svelte. I was able to locate the file where the links are being generated, and I was able to change them (removing the “/timeline” so it was just “/#”. I confirmed they were relinked, but after that change the functionality of the timeline was no longer the same and the new anchor links did not actually resolve to the section headings as expected. Since I’m not the developer, I don’t feel comfortable digging too much further into the source files to try and track down all the various pieces that are coming together to create this functionality. I think the statement on the replay page will have to do for this particular archive. It definitely seems like a pretty fringe case to try and dedicate too much time to on you y’all’s end. Nevertheless, I do appreciate the digging! It’s helped me to get a better understanding of the site itself and certain red flags for archivability in the future. So thank you!
It should just start with the #. Typically in-page ID links look something like this:
<a href="#in-page-link">visible link text<a/>
I don’t feel comfortable digging too much further into the source files to try and track down all the various pieces that are coming together to create this functionality
Absolutely fair enough! I was hoping this would be an easy asset swap but that may not be the case. Generally it’s best practice not to edit the generated files from Svelte so that’s the right move.
We’ll see what can be done on our end at some point. I’ll try to file issues on Monday.
