Browsertrix deployment stalls when initializing container migrations

I am trying to deploy Browsertrix on Red Hat Enterprise Linux 8.10 using k3s and helm.

I have followed the instructions for remote deployment and manage to get some services running, but no backend.

Running kubectl get pods shows the two backend pods are in ‘init’ status.

NAME                                          READY   STATUS     RESTARTS   AGE
browsertrix-cloud-backend-7d5f9c6cc-tw4pn     0/2     Init:0/1   0          2m8s
browsertrix-cloud-backend-b895df648-xvwff     0/2     Init:0/1   0          2m8s
browsertrix-cloud-frontend-76f584cbbc-q69t5   1/1     Running    0          2m7s
btrix-metacontroller-helm-0                   1/1     Running    0          2m6s
local-minio-7cf5b78b4b-29v56                  1/1     Running    0          2m7s
local-mongo-0                                 1/1     Running    0          2m6s

The frontend is definitely running because I get the login screen at http://129.67.246.94:30870/log-in

Running kubectl describe pod on one of the initializing pods shows:

Name:             browsertrix-cloud-backend-b895df648-xvwff
Namespace:        default
Priority:         0
Service Account:  default
Node:             algorithmic-archive-dev/129.67.246.94
Start Time:       Tue, 10 Jun 2025 14:49:36 +0100
Labels:           app=browsertrix-cloud
                  pod-template-hash=b895df648
                  role=backend
Annotations:      helm.update: gB6A0
Status:           Pending
IP:               10.42.0.21
IPs:
  IP:           10.42.0.21
Controlled By:  ReplicaSet/browsertrix-cloud-backend-b895df648
Init Containers:
  migrations:
    Container ID:  containerd://511b8fad7e8a558bcf0417e16239620840ae4c9b9dc403ac74059f629fb07f4e
    Image:         docker.io/webrecorder/browsertrix-backend:1.16.2
    Image ID:      docker.io/webrecorder/browsertrix-backend@sha256:a2c5b1c8e915a31e5b05c7820058d6e2952a079540782a38d7d898d1a99b5071
    Port:          <none>
    Host Port:     <none>
    Command:
      python3
      -m
      btrixcloud.main_migrations
    State:          Running
      Started:      Tue, 10 Jun 2025 14:49:37 +0100
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  350Mi
    Requests:
      cpu:     100m
      memory:  350Mi
    Environment Variables from:
      backend-env-config  ConfigMap  Optional: false
      backend-auth        Secret     Optional: false
      mongo-auth          Secret     Optional: false
    Environment:
      MOTOR_MAX_WORKERS:  1
    Mounts:
      /app/btrixcloud/email-templates/ from email-templates (rw)
      /app/btrixcloud/templates/ from app-templates (rw)
      /config from config-volume (rw)
      /ops-configs/ from ops-configs (rw)
      /ops-proxy-configs/ from ops-proxy-configs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mslvv (ro)
Containers:
  api:
    Container ID:  
    Image:         docker.io/webrecorder/browsertrix-backend:1.16.2
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      gunicorn
      btrixcloud.main:app_root
      --bind
      0.0.0.0:8000
      --access-logfile
      -
      --workers
      1
      --worker-class
      uvicorn.workers.UvicornWorker
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  350Mi
    Requests:
      cpu:      100m
      memory:   350Mi
    Liveness:   http-get http://:8000/healthz delay=5s timeout=1s period=30s #success=1 #failure=15
    Readiness:  http-get http://:8000/healthz delay=5s timeout=1s period=30s #success=1 #failure=5
    Startup:    http-get http://:8000/healthzStartup delay=0s timeout=1s period=10s #success=1 #failure=8640
    Environment Variables from:
      backend-env-config  ConfigMap  Optional: false
      backend-auth        Secret     Optional: false
      mongo-auth          Secret     Optional: false
    Environment:
      MOTOR_MAX_WORKERS:       1
      BTRIX_SUBS_APP_API_KEY:  <set to the key 'BTRIX_SUBS_APP_API_KEY' in secret 'btrix-subs-app-secret'>  Optional: true
      BTRIX_SUBS_APP_URL:      <set to the key 'BTRIX_SUBS_APP_URL' in secret 'btrix-subs-app-secret'>      Optional: true
    Mounts:
      /app/btrixcloud/email-templates/ from email-templates (rw)
      /app/btrixcloud/templates/ from app-templates (rw)
      /ops-configs/ from ops-configs (rw)
      /ops-proxy-configs/ from ops-proxy-configs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mslvv (ro)
  op:
    Container ID:  
    Image:         docker.io/webrecorder/browsertrix-backend:1.16.2
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      gunicorn
      btrixcloud.main_op:app_root
      --bind
      0.0.0.0:8756
      --access-logfile
      -
      --workers
      1
      --worker-class
      uvicorn.workers.UvicornWorker
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  350Mi
    Requests:
      cpu:      100m
      memory:   350Mi
    Liveness:   http-get http://:8756/healthz delay=5s timeout=1s period=30s #success=1 #failure=15
    Readiness:  http-get http://:8756/healthz delay=5s timeout=1s period=30s #success=1 #failure=5
    Startup:    http-get http://:8756/healthz delay=5s timeout=1s period=5s #success=1 #failure=5
    Environment Variables from:
      backend-env-config  ConfigMap  Optional: false
      backend-auth        Secret     Optional: false
      mongo-auth          Secret     Optional: false
    Environment:
      MOTOR_MAX_WORKERS:  1
    Mounts:
      /app/btrixcloud/email-templates/ from email-templates (rw)
      /app/btrixcloud/templates/ from app-templates (rw)
      /config from config-volume (rw)
      /ops-configs/ from ops-configs (rw)
      /ops-proxy-configs/ from ops-proxy-configs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mslvv (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 False 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      shared-job-config
    Optional:  false
  ops-configs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ops-configs
    Optional:    false
  ops-proxy-configs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ops-proxy-configs
    Optional:    true
  app-templates:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      app-templates
    Optional:  false
  email-templates:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      email-templates
    Optional:  false
  kube-api-access-mslvv:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  3m23s  default-scheduler  Successfully assigned default/browsertrix-cloud-backend-b895df648-xvwff to algorithmic-archive-dev
  Normal  Pulled     3m23s  kubelet            Container image "docker.io/webrecorder/browsertrix-backend:1.16.2" already present on machine
  Normal  Created    3m23s  kubelet            Created container: migrations
  Normal  Started    3m22s  kubelet            Started container migrations

So I can see it started doing some migrations, but it never moves on from this state, it’s stuck initializing forever.

I tried to check the logs but can’t get anything because the pod is initializing.

Error from server (BadRequest): container "api" in pod "browsertrix-cloud-backend-b895df648-xvwff" is waiting to start: PodInitializing

Can anyone troubleshoot what might be going on here?