S3 Backup Error

Brian · 15 September 2023 02:07

OS Name/Version: Unraid

Product Name/Version: 2.4.6.4

Problem Description:
I have an S3 self hosted bucket setup for AMP using Garage. I am able to connect to this bucket using Kopia, WinSCP, and Rclone without issue. I am unable to connect to it using AMP however. I receive an SSL error in the AMP console:

AmazonServiceException
[0] (AmazonServiceException) : A WebException with status SecureChannelFailure was thrown.
at Amazon.Runtime.Internal.WebExceptionHandler.HandleException (Amazon.Runtime.IExecutionContext executionContext, Net.WebException exception) at Amazon.Runtime.Internal.WebExceptionHandler.HandleExceptionAsync (Amazon.Runtime.IExecutionContext executionContext, Net.WebException exception) at Amazon.Runtime.Internal.ExceptionHandler`1[T].HandleAsync (Amazon.Runtime.IExecutionContext executionContext, Exception exception) at Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync (Amazon.Runtime.IExecutionContext executionContext, Exception exception) at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.Signer.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at LocalFileBackupPlugin.BackupProvider+<>c__DisplayClass41_0.<UploadToS3>b__0 (ModuleShared.RunningTask t)
WebException
[1] (WebException) : Error: SecureChannelFailure (Authentication failed, see inner exception.)
at Net.WebOperation.Run () at Net.WebCompletionSource`1[T].WaitForCompletion () at Amazon.Runtime.Internal.HttpHandler`1[TRequestContent].InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.HttpHandler`1[TRequestContent].InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext)
AuthenticationException
[2] (AuthenticationException) : Authentication failed, see inner exception.
FrameworkBtlsException
[3] (FrameworkBtlsException) : Ssl error:10000458:SSL routines:OPENSSL_internal:TLSV1_UNRECOGNIZED_NAME at /usr/src/mono/external/boringssl/ssl/tls_record.c:462

Steps to reproduce:

Hit “Upload to S3” button
Receive error

Actions taken to resolve so far:
I researched a lot of posts and I’m seeing some previous issues with insecure certificates. I’m using NPM as a reverse proxy with a signed certificate. This is only run locally and it is just game files so I wouldn’t mind if there was some option to run insecure.

Mike · 15 September 2023 06:08

If it’s a self signed certificate then it won’t work, it either needs to be no certificate at all over plain HTTP or it needs to be a well trusted certificate.

Brian · 15 September 2023 12:46

Thank you for the quick reply. The certificate is not self signed. I did see in a different post that AMP was not able to use a self signed certificate, however their error was not the same as mine. Kopia is also seeing my certificate as signed.

Mike · 15 September 2023 12:48

And the domain name you’re giving to AMP matches the name on the certificate? I.e. not using an internal hostname with a certificate issued for a publicly routable domain?

Brian · 15 September 2023 12:53

Correct. The certificate is issued through my domain registrar, and served by NPM as a wildcard subdomain for local addresses. I went ahead and turned off HTTPS entirely for the domain and AMP does attempt to upload the file, but I receive this error now:

AmazonS3Exception
[0] (AmazonS3Exception) : Error making request with Error Code MethodNotAllowed and Http Status Code MethodNotAllowed. No further error information was returned by the service.
at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionStream (Amazon.Runtime.IRequestContext requestContext, Amazon.Runtime.Internal.Transform.IWebResponseData httpErrorResponse, Amazon.Runtime.Internal.HttpErrorResponseException exception, IO.Stream responseStream) at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionAsync (Amazon.Runtime.IExecutionContext executionContext, Amazon.Runtime.Internal.HttpErrorResponseException exception) at Amazon.Runtime.Internal.ExceptionHandler`1[T].HandleAsync (Amazon.Runtime.IExecutionContext executionContext, Exception exception) at Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync (Amazon.Runtime.IExecutionContext executionContext, Exception exception) at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.Signer.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at LocalFileBackupPlugin.BackupProvider+<>c__DisplayClass41_0.<UploadToS3>b__0 (ModuleShared.RunningTask t)
HttpErrorResponseException
[1] (HttpErrorResponseException) : The remote server returned an error: (405) Not Allowed.
at Amazon.Runtime.Internal.HttpHandler`1[TRequestContent].InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T] (Amazon.Runtime.IExecutionContext executionContext)
WebException
[2] (WebException) : The remote server returned an error: (405) Not Allowed.
at Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult)

Mike · 15 September 2023 12:55

Does the S3 server itself provide any logs that might give a clue as to why it’s unhappy? AMP uses the official Amazon S3 library but there are some settings that may need tweaking depending on the particular provider.

Brian · 15 September 2023 17:59

Thank you for the suggestion. I set my log level to debug for Garage and this is the output (I removed my domain and key ID):

2023-09-15T17:48:00.649379Z  INFO garage_api::generic_server: 172.17.0.1 (via [::ffff:172.17.0.1]:45522) GET /
2023-09-15T17:48:00.649396Z DEBUG garage_api::generic_server: Request { method: GET, uri: /, version: HTTP/1.1, headers: {"host": "subdomain.mydomain.com", "x-forwarded-scheme": "http", "x-forwarded-proto": "http", "x-forwarded-for": "172.17.0.1", "x-real-ip": "172.17.0.1", "user-agent": "aws-sdk-dotnet-45/3.7.203.6 aws-sdk-dotnet-core/3.7.202.0 .NET_Runtime/4.0 .NET_Framework/Unknown OS/Unix_6.1.49.0 ClientAsync", "amz-sdk-invocation-id": "4f124581-a295-4241-8ab9-2293b7536795", "amz-sdk-request": "attempt=1; max=5", "x-amz-date": "20230915T174800Z", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "authorization": "AWS4-HMAC-SHA256 Credential=mykeyid/20230915/garage/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=3c590d370e8efc791d9647ef17f47935a42727403ec889b07590464f17cbff5e"}, body: Body(Empty) }
2023-09-15T17:48:00.649428Z DEBUG garage_api::generic_server: Endpoint: ListBuckets
2023-09-15T17:48:00.649884Z DEBUG garage_api::generic_server: 200 OK {"content-type": "application/xml"}

Also ran it on trace level logging:

2023-09-15T17:56:55.782694Z  INFO garage_api::generic_server: 172.17.0.1 (via [::ffff:172.17.0.1]:43246) GET /
2023-09-15T17:56:55.782710Z DEBUG garage_api::generic_server: Request { method: GET, uri: /, version: HTTP/1.1, headers: {"host": "subdomain.mydomain.com", "x-forwarded-scheme": "http", "x-forwarded-proto": "http", "x-forwarded-for": "172.17.0.1", "x-real-ip": "172.17.0.1", "user-agent": "aws-sdk-dotnet-45/3.7.203.6 aws-sdk-dotnet-core/3.7.202.0 .NET_Runtime/4.0 .NET_Framework/Unknown OS/Unix_6.1.49.0 ClientAsync", "amz-sdk-invocation-id": "bbb994cc-cff8-4930-a9ba-c0dea06a0ca4", "amz-sdk-request": "attempt=1; max=5", "x-amz-date": "20230915T175655Z", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "authorization": "AWS4-HMAC-SHA256 Credential=mykeyid/20230915/garage/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=f0f8135145d4177bb6bdb3dddb172119e0b71545fbd8e8c448ba1c3035e78c68"}, body: Body(Empty) }
2023-09-15T17:56:55.782751Z DEBUG garage_api::generic_server: Endpoint: ListBuckets
2023-09-15T17:56:55.782800Z TRACE garage_api::signature::payload: canonical request:
GET
/

host:subdomain.mydomain.com
user-agent:aws-sdk-dotnet-45/3.7.203.6 aws-sdk-dotnet-core/3.7.202.0 .NET_Runtime/4.0 .NET_Framework/Unknown OS/Unix_6.1.49.0 ClientAsync
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20230915T175655Z

host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2023-09-15T17:56:55.782804Z TRACE garage_api::signature::payload: string to sign:
AWS4-HMAC-SHA256
20230915T175655Z
20230915/garage/s3/aws4_request
8f9925641912aab96ac51df0cb9d2cd5542510388d2c0bdecb5cec6a6375f296
2023-09-15T17:56:55.783211Z TRACE garage_api::s3::bucket: xml: <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult><Buckets><Bucket><CreationDate>2023-09-14T20:24:11.914Z</CreationDate><Name>amp</Name></Bucket></Buckets><Owner><DisplayName>amp-app-key</DisplayName><ID>mykeyid</ID></Owner></ListAllMyBucketsResult>
2023-09-15T17:56:55.783237Z DEBUG garage_api::generic_server: 200 OK {"content-type": "application/xml"}

I’m no expert by any means, but nothing particularly stands out to me.

Mike · 15 September 2023 18:00

Are you able to access it with AWSCLI?

Brian · 15 September 2023 18:18

Yup, just gave it a try and it shows my amp bucket.

>aws --endpoint-url http://subdomain.mydomain.com s3 ls
2023-09-14 15:24:11 amp

system · 15 October 2023 18:19

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.