Deleting crawl failure

I’m running Browsertrix with OpenStack with SWIFT S3 storage. Seems to be working fine, mostly. Uploading WACZs works, storing WACZs from a crawl works. But deleting a crawl doesn’t work. The Backend shows the error:

[ERROR] Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/", line 419, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/", line 84, in __call__
    return await, receive, send)
  File "/usr/local/lib/python3.10/site-packages/fastapi/", line 292, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/", line 162, in __call__
    await, receive, _send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/", line 68, in __call__
    await, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/", line 20, in __call__
    raise e
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/", line 17, in __call__
    await, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/", line 276, in handle
    await, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/", line 66, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/site-packages/fastapi/", line 273, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/site-packages/fastapi/", line 190, in run_endpoint_function
    return await**values)
  File "/app/btrixcloud/", line 824, in delete_crawls_all_types
    return await ops.delete_crawls_all_types(delete_list, org, user)
  File "/app/btrixcloud/", line 667, in delete_crawls_all_types
    deleted, _, quota_reached = await self.delete_crawls(
  File "/app/btrixcloud/", line 337, in delete_crawls
    crawl_size = await self._delete_crawl_files(crawl, org)
  File "/app/btrixcloud/", line 376, in _delete_crawl_files
    if not await self.storage_ops.delete_crawl_file_object(org, file_):
  File "/app/btrixcloud/", line 493, in delete_crawl_file_object
    return await self._delete_file(org, crawlfile.filename,
  File "/app/btrixcloud/", line 509, in _delete_file
    response = await client.delete_object(Bucket=bucket, Key=key)
  File "/usr/local/lib/python3.10/site-packages/aiobotocore/", line 408, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the DeleteObject operation: Unknown

The S3 credential I’m using has full access to the bucket. It is a little strange to me that uploading works, but deletion not.

One thing I noticed is that StorageOps#_delete_file includes s3storage.use_access_for_presign when getting the S3 client. So in the storages.json I set use_access_for_presign explicitely to false to be sure it doesn’t use the access endpoint. No difference.

I’m not really sure how to debug this, without either having to set up a fully local Backend, or building custom browsertrix backend images and run them the in the cluster. Are there ways to increase log detail (e.g. boto3 logging)?

Submitted PR #1600 which fixes this.

1 Like