AMP Controller Random Crashing

OS Name/Version: Ubuntu 20.04.5 LTS & Windows 10

Product Name/Version: (AMP Release “Halimede”
v2.4.0.8, built 10/10/2022 21:18)

FYI had to remove all the http and https as it was only letting me post 2 links from the log.

[15:32:58] [Core Info]            : Starting AMP version 2.4.0.8 (Halimede), built 10/10/2022 21:18
[15:32:58] [Core Info]            : Stream: Mainline (Release) - built by CUBECODERS/buildbot on CCL-DEV
[15:32:59] [Core Info]            : Running in a VMware environment.
[15:32:59] [Core Info]            : OS: Linux / x86_64
[15:32:59] [Core Info]            : CPU: Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz (24C/24T)
[15:32:59] [Core Info]            : AMP Instance ID: dd8134b1-c9f8-465d-9267-28b5d6414663
[15:32:59] [ModuleLoader Info]    : Loaded ADSModule version 1.0.0.0 by CubeCoders Limited
[15:32:59] [ModuleLoader Info]    : Loaded FileManagerPlugin version 1.0.0.0 by CubeCoders Limited
[15:32:59] [ModuleLoader Info]    : Loaded EmailSenderPlugin version 1.0.0.0 by CubeCoders Limited
[15:32:59] [ModuleLoader Info]    : Loaded WebRequestPlugin version 1.0.0.0 by CubeCoders Limited
[15:32:59] [ModuleLoader Info]    : Loaded LocalFileBackupPlugin version 1.0.0.0 by CubeCoders Limited
[15:32:59] [ModuleLoader Info]    : Loaded CommonCorePlugin version 1.0.0.0 by CubeCoders Limited
[15:32:59] [ModuleLoader Info]    : ADSModule requests dependency InstanceManagerPlugin...
[15:33:00] [ModuleLoader Info]    : Loaded InstanceManagerPlugin version 1.0.0.0 by CubeCoders Limited
[15:33:00] [ModuleLoader Info]    : ADSModule requests dependency SystemUserManagerPlugin...
[15:33:00] [ModuleLoader Info]    : Loaded SystemUserManagerPlugin version 1.0.0.0 by CubeCoders Limited
[15:33:00] [ModuleLoader Info]    : Loaded steamcmdplugin version 1.0.0.0 by CubeCoders Limited
[15:33:01] [Core Info]            : Metrics server started OK on port 12820
[15:33:01] [ADS Info]             : Updating remote source CubeCoders/AMPTemplates
[15:33:01] [ADS Info]             : Updating existing remote source ://github.com/CubeCoders/AMPTemplates.git...
[15:33:01] [Auth Activity]        : Authentication token for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
[15:33:01] [FileManager Notice]   : Using keypair with fingerprint SKHUHs+lc7yG400UEr15qwOVbKAadw0sA3JDdWscSdo=
[15:33:01] [FileManager Info]     : SFTP Server started on 0.0.0.0:2223
[15:33:01] [WebServer Info]       : Websockets are enabled.
[15:33:01] [WebServer Info]       : Webserver started on ://0.0.0.0:8080
[15:33:01] [Logger Warning]       : RouterTimer@10Hz with 2 jobs started
[15:33:01] [Core Info]            : Checking for AMP updates...
Already up to date.
[15:33:02] [Core Info]            : AMP is up-to-date.
[15:35:01] [ADS:admin Activity]   : Managed remote instance SGAConanExiles01 at ://amp-linux-01.sga-gamers.com:8085/
[15:35:01] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[15:35:12] [ADS:admin Activity]   : Managed remote instance SatisfactoryManagedByAZ01 at ://amp-linux-01.sga-gamers.com:8086/
[15:35:12] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[15:35:27] [ADS:admin Activity]   : Managed remote instance FTBREV01 at ://amp-linux-01.sga-gamers.com:8081/
[15:35:27] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[15:38:30] [ADS:admin Activity]   : Managed remote instance SGAConanExiles01 at ://amp-linux-01.sga-gamers.com:8085/
[15:38:31] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[15:38:54] [ADS:admin Activity]   : Managed remote instance SatisfactoryManagedByAZ01 at ://amp-linux-01.sga-gamers.com:8086/
[15:38:54] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[15:39:07] [ADS:admin Activity]   : Managed remote instance SatisfactoryManagedByAZ01 at ://amp-linux-01.sga-gamers.com:8086/
[15:39:07] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[15:50:52] [Auth:admin Activity]  : Authentication token for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of admin
[15:52:16] [ADS:admin Activity]   : Managed remote instance SpaceEngineersManagedByAZ01 at ://amp-win-01.sga-gamers.com:8081/
[15:52:16] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[15:55:49] [ADS:admin Activity]   : Managed remote instance SpaceEngineersManagedByAZ01 at ://amp-win-01.sga-gamers.com:8081/
[15:55:49] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[15:59:09] [ADS:admin Activity]   : Managed remote instance FTBREV01 at ://amp-linux-01.sga-gamers.com:8081/
[15:59:09] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[15:59:32] [ADS:admin Activity]   : Managed remote instance FTBREV01 at ://amp-linux-01.sga-gamers.com:8081/
[15:59:32] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[15:59:59] [ADS:admin Activity]   : Managed remote instance SpaceEngineersManagedByAZ01 at ://amp-win-01.sga-gamers.com:8081/
[15:59:59] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[16:00:21] [ADS:admin Activity]   : Managed remote instance SpaceEngineersManagedByAZ01 at ://amp-win-01.sga-gamers.com:8081/
[16:00:21] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[16:00:36] [ADS:admin Activity]   : Managed remote instance SpaceEngineersManagedByAZ01 at ://amp-win-01.sga-gamers.com:8081/
[16:00:36] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[16:01:06] [ADS:admin Activity]   : Managed remote instance SpaceEngineersManagedByAZ01 at ://amp-win-01.sga-gamers.com:8081/
[16:01:06] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[16:04:38] [ADS:admin Activity]   : Managed remote instance SGAConanExiles01 at ://amp-linux-01.sga-gamers.com:8085/
[16:04:38] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[16:07:09] [ADS:admin Activity]   : Managed remote instance ADS01 at ://amp-linux-01.sga-gamers.com:8080/
[16:07:09] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[16:07:27] [ADS:admin Activity]   : Managed remote instance ADS01 at ://amp-linux-01.sga-gamers.com:8080/
[16:07:27] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[16:07:39] [ADS:admin Activity]   : Managed remote instance ADS01 at ://amp-win-01.sga-gamers.com:8080/
[16:07:39] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[16:07:47] [ADS:admin Activity]   : Managed remote instance SpaceEngineersManagedByAZ01 at ://amp-win-01.sga-gamers.com:8081/
[16:07:47] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[16:14:20] [ADS:admin Activity]   : Upgrading instance SpaceEngineersManagedByAZ01
[16:42:48] [ADS:admin Activity]   : Managed remote instance SpaceEngineersManagedByAZ01 at ://amp-win-01.sga-gamers.com:8081/
[16:42:48] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[16:43:02] [ADS:admin Activity]   : Managed remote instance SpaceEngineersManagedByAZ01 at ://amp-win-01.sga-gamers.com:8081/
[16:43:02] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[16:43:22] [ADS:admin Activity]   : Deleting instance 9c7a70aa-95ed-4db6-b364-749906c948a7
[16:43:39] [ADS:admin Activity]   : Managed remote instance ADS01 at ://amp-win-01.sga-gamers.com:8080/
[16:43:39] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[16:43:39] [ADS:Anonymous Error]  : Failure to make API call to SpaceEngineersManagedByAZ01 (://amp-win-01.sga-gamers.com:8080/API/ADSModule/Servers/9c7a70aa-95ed-4db6-b364-749906c948a7/API/Core/GetAPISpec) - retried 10 times : The operation was canceled.
[16:43:39] [Core:Anonymous Error] : TaskCanceledException
[16:43:39] [Core:Anonymous Error] : [0] (TaskCanceledException) : The operation was canceled.
[16:43:39] [Core:Anonymous Error] :   at ADSModule.WebMethods.MakeInstanceRequest (m.Http.IHttpRequest request, String REQ_RAWJSON, String requesturi) 
  at ADSModule.WebMethods.Servers (m.Http.IHttpRequest request, String id, String REQ_RAWJSON) 
[16:43:39] [Core:Anonymous Error] : IOException
[16:43:39] [Core:Anonymous Error] : [1] (IOException) : Unable to read data from the transport connection: Operation aborted.
[16:43:39] [Core:Anonymous Error] :   at Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.ThrowException (Net.Sockets.SocketError error) 
  at Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.GetResult (Int16 token) 
  at Net.Http.HttpConnection.FillAsync () 
  at Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync (Boolean foldedHeadersAllowed) 
[16:43:39] [Core:Anonymous Error] : SocketException
[16:43:39] [Core:Anonymous Error] : [2] (SocketException) : Operation aborted
[16:43:52] [ADS:Anonymous Error]  : Failure to make API call to SpaceEngineersManagedByAZ01 (://amp-win-01.sga-gamers.com:8080/API/ADSModule/Servers/9c7a70aa-95ed-4db6-b364-749906c948a7/API/Core/GetAPISpec) - retried 10 times : The operation was canceled.
[16:43:52] [Core:Anonymous Error] : TaskCanceledException
[16:43:52] [Core:Anonymous Error] : [0] (TaskCanceledException) : The operation was canceled.
[16:43:52] [Core:Anonymous Error] :   at ADSModule.WebMethods.MakeInstanceRequest (m.Http.IHttpRequest request, String REQ_RAWJSON, String requesturi) 
  at ADSModule.WebMethods.Servers (m.Http.IHttpRequest request, String id, String REQ_RAWJSON) 
[16:43:52] [Core:Anonymous Error] : IOException
[16:43:52] [Core:Anonymous Error] : [1] (IOException) : Unable to read data from the transport connection: Operation aborted.
[16:43:52] [Core:Anonymous Error] :   at Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.ThrowException (Net.Sockets.SocketError error) 
  at Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.GetResult (Int16 token) 
  at Net.Http.HttpConnection.FillAsync () 
  at Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync (Boolean foldedHeadersAllowed) 
[16:43:52] [Core:Anonymous Error] : SocketException
[16:43:52] [Core:Anonymous Error] : [2] (SocketException) : Operation aborted
[16:45:15] [ADS:admin Activity]   : Managed remote instance SpaceEngineersManagedByAZ01 at ://amp-win-01.sga-gamers.com:8081/
[16:45:15] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[16:51:12] [Core:admin Warning]   : Returned exception from API call FileManagerPlugin/GetDirectoryListing
[16:51:12] [Core:admin Error]     : NullReferenceException
[16:51:12] [Core:admin Error]     : [0] (NullReferenceException) : Object reference not set to an instance of an object
[16:51:12] [Core:admin Error]     :   at FileManagerPlugin.WebMethods.GetDirectoryListing (String Dir) 
  at GSMyAdmin.WebServer.WebAttributes.InvokeMethod (String MethodName, Newtonsoft.Json.Linq.JObject Data, m.Http.IHttpRequest request, ModuleShared.IWebSession Session, ModuleShared.WebMethodsBase MethodsClass, String RawRequest) 
  at GSMyAdmin.WebServer.APIService.InvokeAPI (m.Http.IHttpRequest request, ModuleShared.IWebSession Session, Newtonsoft.Json.Linq.JObject Data, String RequestModule, String RequestMethod, String RawRequest) 
[16:51:20] [ADS:admin Activity]   : Managed remote instance SpaceEngineersManagedByAZ01 at ://amp-win-01.sga-gamers.com:8081/
[16:51:20] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[16:51:46] [ADS:admin Activity]   : Managed remote instance SatisfactoryManagedByAZ01 at ://amp-linux-01.sga-gamers.com:8086/
[16:51:46] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[16:52:55] [ADS:admin Activity]   : Managed remote instance SatisfactoryManagedByAZ01 at ://amp-linux-01.sga-gamers.com:8086/
[16:52:55] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[16:54:07] [ADS:admin Activity]   : Managed remote instance SatisfactoryManagedByAZ01 at ://amp-linux-01.sga-gamers.com:8086/
[16:54:07] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[16:55:08] [ADS:admin Activity]   : Managed remote instance SpaceEngineersManagedByAZ01 at ://amp-win-01.sga-gamers.com:8081/
[16:55:08] [Auth:admin Activity]  : Authentication token for admin requested by ManageInstance on behalf of admin
[17:17:00] [Auth Activity]        : Authentication token for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of admin
[17:17:02] [ErrorReporter Error]  : Unhandled Exception: Please report this to the AMP developers.
[17:17:02] [ErrorReporter Error]  : [Error] --- Begin Exception Data ---
[Error] OS: Linux (x86_64/VMware)
[Error] Module: ADSModule, Version: AMP 2.4.0.8
[Error] Release spec: Release - built by CUBECODERS/buildbot on CCL-DEV
[Error] Instance ID: dd8134b1-c9f8-465d-9267-28b5d6414663
[Error] InvalidOperationException
[Error] [0] (InvalidOperationException) : The WebSocket has already been started.
[Error]   at Net.WebSockets.ClientWebSocketOptions.ThrowIfReadOnly () 
  at Net.WebSockets.ClientWebSocketOptions.SetRequestHeader (String headerName, String headerValue) 
  at ADSModule.RemoteTasksMonitor.Connect () 
[Error] --- End Exception Data ---

[17:17:02] [ErrorReporter Info]   : ####################################
[17:17:02] [ErrorReporter Info]   : #         Self Diagnostics         #
[17:17:02] [ErrorReporter Info]   : ####################################
[17:17:02] [ErrorReporter Info]   : AMP Diagnostics could not establish a solution for this exception.

This is a known issue, the nightly build has a fix for it already and we’ll be pushing it up to mainline this week.

thanks for the update Mike do we have an ETA on the mainsteam update timeline?

Actually I think that particular issue might have been fixed as a silent update. Try doing an ampinstmgr --nocache upgradeall

So i perfomed this lastnight and today the server still crahsed to I revered back to nightly. just and FYI

Hi Mike, any news for the update? my server’s crashing randomly all day’s.

Cheers

Yes - update to the latest version that just came out :slight_smile:

your the best thanks for all your hard work