AMP gets stuck in Recovery mode since update

Curtis_Retherford · 19 July 2026 21:41

OS Name/Version: Debian GNU/Linux 13 x86_64

Product Name/Version: 2.8.0.2 - 20260712.1

Problem Description: Since latest update, AMP will go down a few times a day. When I check the AMP Application Manager, it says “AMP Recovery Mode: if you’re stuck in this for more than 5 minutes, try…” and stays there. Trying the fix listed doesn’t help. Restarting gets it working, but only for a while.

The logs don’t seem to show anything odd when it goes down, but here are some other errors:

[20:14:04] [System Info/13]       : Updating existing remote source https://github.com/CubeCoders/AMPTemplates.git...
No stash entries found.
[20:14:04] [System Notice/10]     : Using keypair with fingerprint TPqCjmz6Cd4ywtC3zgjMEdMyPVJh+KtbXH7xhjtfFjg=
[20:14:04] [System Info/10]       : SFTP Server started on 0.0.0.0:2223
[20:14:07] [System Warning/9]     : Failed to locate AWSSDK.S3 via late assembly resolver (tried /home/amp/.ampdata/instances/ADS01/Plugins/System.Reflection/AWSSDK.S3.dll)
[20:14:07] [System Error/9]       : Missing plugin/assembly: AWSSDK.S3 
[20:14:07] [System Error/9]       : Tried: /home/amp/.ampdata/instances/ADS01/Plugins/AWSSDK.S3/AWSSDK.S3.dll, /home/amp/.ampdata/instances/ADS01/Plugins/AWSSDK.S3/AWSSDK.S3.dll, /home/amp/.ampdata/instances/ADS01/AWSSDK.S3.dll
[20:14:07] [System Warning/9]     : Failed to locate AWSSDK.S3 via late assembly resolver (tried /home/amp/.ampdata/instances/ADS01/Plugins/System.Reflection/AWSSDK.S3.dll)
[20:14:07] [System Error/9]       : Missing plugin/assembly: AWSSDK.S3 
[20:14:07] [System Error/9]       : Tried: /home/amp/.ampdata/instances/ADS01/Plugins/AWSSDK.S3/AWSSDK.S3.dll, /home/amp/.ampdata/instances/ADS01/Plugins/AWSSDK.S3/AWSSDK.S3.dll, /home/amp/.ampdata/instances/ADS01/AWSSDK.S3.dll
[20:14:07] [System Warning/9]     : Failed to locate AWSSDK.S3 via late assembly resolver (tried /home/amp/.ampdata/instances/ADS01/Plugins/System.Reflection/AWSSDK.S3.dll)
[20:14:07] [System Error/9]       : Missing plugin/assembly: AWSSDK.S3 
[20:14:07] [System Error/9]       : Tried: /home/amp/.ampdata/instances/ADS01/Plugins/AWSSDK.S3/AWSSDK.S3.dll, /home/amp/.ampdata/instances/ADS01/Plugins/AWSSDK.S3/AWSSDK.S3.dll, /home/amp/.ampdata/instances/ADS01/AWSSDK.S3.dll
[20:14:07] [System Warning/9]     : Failed to locate AWSSDK.S3 via late assembly resolver (tried /home/amp/.ampdata/instances/ADS01/Plugins/System.Reflection/AWSSDK.S3.dll)
[20:14:07] [System Error/9]       : Missing plugin/assembly: AWSSDK.S3 
[20:14:07] [System Error/9]       : Tried: /home/amp/.ampdata/instances/ADS01/Plugins/AWSSDK.S3/AWSSDK.S3.dll, /home/amp/.ampdata/instances/ADS01/Plugins/AWSSDK.S3/AWSSDK.S3.dll, /home/amp/.ampdata/instances/ADS01/AWSSDK.S3.dll
[20:14:07] [System Warning/9]     : Failed to locate Open.Nat via late assembly resolver (tried /home/amp/.ampdata/instances/ADS01/Plugins/System.Reflection/Open.Nat.dll)
[20:14:07] [System Error/9]       : Missing plugin/assembly: Open.Nat 
[20:14:07] [System Error/9]       : Tried: /home/amp/.ampdata/instances/ADS01/Plugins/Open.Nat/Open.Nat.dll, /home/amp/.ampdata/instances/ADS01/Plugins/Open.Nat/Open.Nat.dll, /home/amp/.ampdata/instances/ADS01/Open.Nat.dll
[20:14:07] [Core Info/9]          : Webserver started on http://0.0.0.0:8080
[20:14:07] [System Info/9]        : Checking for AMP updates...
[20:14:07] [System Info/6]        : AMP is up to date.
[20:15:32] [System Error/32]      : WebSocket error: The operation was canceled.
[20:15:32] [Core Error/32]        : OperationCanceledException
[20:15:32] [Core Error/32]        : [0] (OperationCanceledException) : The operation was canceled.
[20:15:32] [Core Error/32]        :    at IO.Pipelines.Pipe.GetReadAsyncResult()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1UpgradeMessageBody.ReadAsyncInternalAwaited(ValueTask`1 readTask, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 destination, CancellationToken cancellationToken)
   at IO.Stream.ReadAtLeastAsyncCore(Memory`1 buffer, Int32 minimumBytes, Boolean throwOnEndOfStream, CancellationToken cancellationToken)
   at Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken)
   at Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
   at GSMyAdmin.WebServer.ApiService.HandleWebsocket(HttpContext context, String sessionId)
[20:15:32] [Webserver Error/32]   : Connection id "0HNN5Q94S82BR", Request id "0HNN5Q94S82BR:00000001": An unhandled exception was thrown by the application.
[20:30:00] [System Info/44]       : Updating remote source CubeCoders/AMPTemplates:main
[20:30:00] [System Info/44]       : Updating existing remote source https://github.com/CubeCoders/AMPTemplates.git...
No stash entries found.

WebSocket error: The WebSocket is in an invalid state ('Aborted') for this operation. Valid states are: 'Open, CloseReceived, CloseSent'
Connection id "0HNN5RCQQDMJP", Request id "0HNN5RCQQDMJP:00000001": An unhandled exception was thrown by the application.

Steps to reproduce:

Step 1 Not sure; it goes down at random times during the day

Actions taken to resolve so far: Tried updating AMP again. Tried fix listed in “Recovery Mode” page. Restarting.

Mike · 19 July 2026 22:25

Do you have crash reporting turned on?

Curtis_Retherford · 20 July 2026 05:32

I do. The last crash report was uploaded at 15/07/2026 21:36:51. For the other errors, AMP seems stuck in recovery mode even if I do ` ampinstmgr restart ads01, and I can’t see anything in the logs before that happened.

Here is the log file from right before the stall today:

\[20:09:26\] \[System:Perdith Info/37\] : \[signal SIGSEGV: segmentation violation code=0x1 addr=0x3 pc=0x586578d5bd80\]
\[20:09:26\] \[System:Perdith Info/37\] : goroutine 1 \[running\]:
\[20:09:26\] \[System:Perdith Info/37\] : strconv.Atoi({0x0, 0x0})
\[20:09:26\] \[System:Perdith Info/37\] : 	/usr/local/go/src/strconv/number.go:148 +0x20
\[20:09:26\] \[System:Perdith Info/37\] : github.com/docker/cli/vendor/google.golang.org/grpc/grpclog.newLoggerV2()
\[20:09:26\] \[System:Perdith Info/37\] : 	/go/src/github.com/docker/cli/vendor/google.golang.org/grpc/grpclog/loggerv2.go:77 +0x175
\[20:09:26\] \[System:Perdith Info/37\] : github.com/docker/cli/vendor/google.golang.org/grpc/grpclog.init.0()
\[20:09:26\] \[System:Perdith Info/37\] : 	/go/src/github.com/docker/cli/vendor/google.golang.org/grpc/grpclog/grpclog.go:33 +0x17
\[20:09:26\] \[System:Perdith Info/6\] : panic: runtime error: invalid memory address or nil pointer dereference
\[20:09:26\] \[System:Perdith Info/6\] : \[signal SIGSEGV: segmentation violation code=0x1 addr=0x3 pc=0x63724173cd80\]
\[20:09:26\] \[System:Perdith Info/6\] : goroutine 1 \[running\]:
\[20:09:26\] \[System:Perdith Info/6\] : strconv.Atoi({0x0, 0x0})
\[20:09:26\] \[System:Perdith Info/6\] : /usr/local/go/src/strconv/number.go:148 +0x20
\[20:09:26\] \[System:Perdith Info/6\] : github.com/docker/cli/vendor/google.golang.org/grpc/grpclog.newLoggerV2()
\[20:09:26\] \[System:Perdith Info/6\] : /go/src/github.com/docker/cli/vendor/google.golang.org/grpc/grpclog/loggerv2.go:77 +0x175
\[20:09:26\] \[System:Perdith Info/6\] : github.com/docker/cli/vendor/google.golang.org/grpc/grpclog.init.0()
\[20:09:26\] \[System:Perdith Info/6\] : /go/src/github.com/docker/cli/vendor/google.golang.org/grpc/grpclog/grpclog.go:33 +0x17

Mike · 20 July 2026 08:40

That’s not part of AMPs own logs (AMP doesn’t use Go for anything) - could you humour me and run Memtest86+ on the host system?

Curtis_Retherford · 20 July 2026 20:31

Yup, that looks like that was the problem. Thank you!