I’m running Browsertrix with OpenStack with SWIFT S3 storage. Seems to be working fine, mostly. Uploading WACZs works, storing WACZs from a crawl works. But deleting a crawl doesn’t work. The Backend shows the error:
[ERROR] Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 419, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
return await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 292, in __call__
await super().__call__(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
raise exc
File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
raise exc
File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
await self.app(scope, receive, sender)
File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
raise e
File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
response = await func(request)
File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 273, in app
raw_response = await run_endpoint_function(
File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 190, in run_endpoint_function
return await dependant.call(**values)
File "/app/btrixcloud/basecrawls.py", line 824, in delete_crawls_all_types
return await ops.delete_crawls_all_types(delete_list, org, user)
File "/app/btrixcloud/basecrawls.py", line 667, in delete_crawls_all_types
deleted, _, quota_reached = await self.delete_crawls(
File "/app/btrixcloud/basecrawls.py", line 337, in delete_crawls
crawl_size = await self._delete_crawl_files(crawl, org)
File "/app/btrixcloud/basecrawls.py", line 376, in _delete_crawl_files
if not await self.storage_ops.delete_crawl_file_object(org, file_):
File "/app/btrixcloud/storages.py", line 493, in delete_crawl_file_object
return await self._delete_file(org, crawlfile.filename, crawlfile.storage)
File "/app/btrixcloud/storages.py", line 509, in _delete_file
response = await client.delete_object(Bucket=bucket, Key=key)
File "/usr/local/lib/python3.10/site-packages/aiobotocore/client.py", line 408, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the DeleteObject operation: Unknown
The S3 credential I’m using has full access to the bucket. It is a little strange to me that uploading works, but deletion not.
One thing I noticed is that StorageOps#_delete_file includes s3storage.use_access_for_presign when getting the S3 client. So in the storages.json I set use_access_for_presign explicitely to false to be sure it doesn’t use the access endpoint. No difference.
I’m not really sure how to debug this, without either having to set up a fully local Backend, or building custom browsertrix backend images and run them the in the cluster. Are there ways to increase log detail (e.g. boto3 logging)?