S3 backup to backblaze for Vintage Story fails with CERTIFICATE_VERIFY_FAILED

OS Name/Version: Pop_OS 22.04 LTS

Product Name/Version: AMP

Problem Description:
I’m attempting to configure my Vintage Story instance for S3 cloud backups using Backblaze. I’ve successfully configured the AMP server as well as containerized and non-containerized Minecraft instances, but I’m unable to configure VintageStory instances.

When I try to upload through the web portal, I get the following error:

This task could not be completed: Uploading Backup to S3 - Scheduled Backup - State: 3.

A WebException with status TrustFailure was thrown.

Digging into the logs, I see the following exception:

[01:46:58] [Core:bill Error]      : WebException
[01:46:58] [Core:bill Error]      : [1] (WebException) : Error: TrustFailure (Authentication failed, see inner exception.)
[01:46:58] [Core:bill Error]      :   at Net.WebOperation.Run ()
  at Net.WebCompletionSource`1[T].WaitForCompletion ()
  at Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult)
  at Amazon.Runtime.Internal.HttpHandler`1[TRequestContent].InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext)
  at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext)
  at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext)
  at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext)
  at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext)
[01:46:58] [Core:bill Error]      : AuthenticationException
[01:46:58] [Core:bill Error]      : [2] (AuthenticationException) : Authentication failed, see inner exception.
[01:46:58] [Core:bill Error]      : FrameworkBtlsException
[01:46:58] [Core:bill Error]      : [3] (FrameworkBtlsException) : Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /usr/src/mono/external/boringssl/ssl/handshake_client.c:1132
[01:46:58] [Core:bill Error]      :   at Framework.Btls.FrameworkBtlsContext.ProcessHandshake ()
  at Framework.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Framework.Net.Security.AsyncOperationStatus status, Boolean renegotiate)
  at (wrapper remoting-invoke-with-check) Framework.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Framework.Net.Security.AsyncOperationStatus,bool)
  at Framework.Net.Security.AsyncHandshakeRequest.Run (Framework.Net.Security.AsyncOperationStatus status)

Steps to reproduce:

  • Create new containerized VintageStory instance
  • Configure S3 backups
  • Attempt to upload backup to cloud

Actions taken to resolve so far:
I’ve tried creating a new containerized instance, but the issue persists there. I’ve tried refreshing certs with ampinstmgr --sync-certs from the amp account. I’ve verified that I can access the Backblaze endpoint from within the docker container and that SSL verification passes:

bill@server:~$ sudo docker exec -it AMP_VintageStory01 curl -v https://s3.us-west-000.backblazeb2.com
*   Trying
* Connected to s3.us-west-000.backblazeb2.com ( port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.backblazeb2.com
*  start date: Nov  9 20:18:59 2022 GMT
*  expire date: Feb  7 20:18:58 2023 GMT
*  subjectAltName: host "s3.us-west-000.backblazeb2.com" matched cert's "s3.us-west-000.backblazeb2.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: s3.us-west-000.backblazeb2.com
> User-Agent: curl/7.74.0
> Accept: */*
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 403
< x-amz-request-id: 1c0dc62d7bbcfef9
< x-amz-id-2: adRluKWuKbq5vvXf8btI=
< Cache-Control: max-age=0, no-cache, no-store
< Content-Type: application/xml
< Content-Length: 180
< Date: Tue, 24 Jan 2023 02:02:17 GMT
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
    <Message>Unauthenticated requests are not allowed for this api</Message>
* Connection #0 to host s3.us-west-000.backblazeb2.com left intact

This is unfortunately a known issue for any applications that have a dependency on Mono. The cause of the issue is unknown at this time. It prevents AMP from being able to talk to most HTTPS endpoints for some reason.