S3 backup to backblaze for Vintage Story fails with CERTIFICATE_VERIFY_FAILED

OS Name/Version: Pop_OS 22.04 LTS

Product Name/Version: AMP 2.4.1.2

Problem Description:
I’m attempting to configure my Vintage Story instance for S3 cloud backups using Backblaze. I’ve successfully configured the AMP server as well as containerized and non-containerized Minecraft instances, but I’m unable to configure VintageStory instances.

When I try to upload through the web portal, I get the following error:

This task could not be completed: Uploading Backup to S3 - Scheduled Backup - State: 3.

A WebException with status TrustFailure was thrown.

Digging into the logs, I see the following exception:

[01:46:58] [Core:bill Error]      : WebException
[01:46:58] [Core:bill Error]      : [1] (WebException) : Error: TrustFailure (Authentication failed, see inner exception.)
[01:46:58] [Core:bill Error]      :   at Net.WebOperation.Run ()
  at Net.WebCompletionSource`1[T].WaitForCompletion ()
  at Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult)
  at Amazon.Runtime.Internal.HttpHandler`1[TRequestContent].InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext)
  at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext)
  at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext)
  at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext)
  at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext)
[01:46:58] [Core:bill Error]      : AuthenticationException
[01:46:58] [Core:bill Error]      : [2] (AuthenticationException) : Authentication failed, see inner exception.
[01:46:58] [Core:bill Error]      : FrameworkBtlsException
[01:46:58] [Core:bill Error]      : [3] (FrameworkBtlsException) : Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /usr/src/mono/external/boringssl/ssl/handshake_client.c:1132
[01:46:58] [Core:bill Error]      :   at Framework.Btls.FrameworkBtlsContext.ProcessHandshake ()
  at Framework.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Framework.Net.Security.AsyncOperationStatus status, Boolean renegotiate)
  at (wrapper remoting-invoke-with-check) Framework.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Framework.Net.Security.AsyncOperationStatus,bool)
  at Framework.Net.Security.AsyncHandshakeRequest.Run (Framework.Net.Security.AsyncOperationStatus status)

Steps to reproduce:

  • Create new containerized VintageStory instance
  • Configure S3 backups
  • Attempt to upload backup to cloud

Actions taken to resolve so far:
I’ve tried creating a new containerized instance, but the issue persists there. I’ve tried refreshing certs with ampinstmgr --sync-certs from the amp account. I’ve verified that I can access the Backblaze endpoint from within the docker container and that SSL verification passes:

bill@server:~$ sudo docker exec -it AMP_VintageStory01 curl -v https://s3.us-west-000.backblazeb2.com
*   Trying 206.190.208.254:443...
* Connected to s3.us-west-000.backblazeb2.com (206.190.208.254) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.backblazeb2.com
*  start date: Nov  9 20:18:59 2022 GMT
*  expire date: Feb  7 20:18:58 2023 GMT
*  subjectAltName: host "s3.us-west-000.backblazeb2.com" matched cert's "s3.us-west-000.backblazeb2.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: s3.us-west-000.backblazeb2.com
> User-Agent: curl/7.74.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 403
< x-amz-request-id: 1c0dc62d7bbcfef9
< x-amz-id-2: adRluKWuKbq5vvXf8btI=
< Cache-Control: max-age=0, no-cache, no-store
< Content-Type: application/xml
< Content-Length: 180
< Date: Tue, 24 Jan 2023 02:02:17 GMT
<
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
    <Code>AccessDenied</Code>
    <Message>Unauthenticated requests are not allowed for this api</Message>
</Error>
* Connection #0 to host s3.us-west-000.backblazeb2.com left intact

This is unfortunately a known issue for any applications that have a dependency on Mono. The cause of the issue is unknown at this time. It prevents AMP from being able to talk to most HTTPS endpoints for some reason.

Is there any way to tell amp to ignore this trust failure? I’m trying to send AMP backups to an S3 endpoint on my NAS but I’m running across this as well - to be fair my cert is a self-generated one.

Is this still the case? is there any progress on understanding what is going on?