AMP Controller Blank/ Not showing targets

OS Name/Version: Windows 11 / Ubuntu Desktop /

Product Name/Version: AMP 2.6.2

Problem Description:
AMP Controller instance manager is blank, not showing any connection errors just does the pulse animation and fails to load anything. Prior to upgrade to 2.6.2 if there was a connection error it would show red at the top of the target name. Now just blank. Target and Controller can ping each other. I can access each AMP target locally, but cannot manage as it’s not from the target. The targets and controller can ping each other for certain, it is not a network issue. This issue only arised after I updated to 2.6.2. I have 20+ servers I have no access to manage at this moment. A little stressed…

Deployment Log
Search...
Services Running
play_circle
:Anonymous09:32:12
   at GSMyAdmin.WebServer.WebAttributes.InvokeMethod(String MethodName, JObject Data, HttpContext context, IWebSession Session, WebMethodsBase MethodsClass, IPAddress RealIP)
   at GSMyAdmin.WebServer.ApiService.InvokeAPI(HttpContext context, IWebSession session, JObject data, String requestModule, String requestMethod)
Exception in API call Core/GetUpdates
UnauthorizedAccessException
[0] (UnauthorizedAccessException) : You do not have permission to use this method (GSMyAdmin.WebServer.GetUpdates) at this time. This method requires the Session.Exists permission.
   at GSMyAdmin.WebServer.WebAttributes.InvokeMethod(String MethodName, JObject Data, HttpContext context, IWebSession Session, WebMethodsBase MethodsClass, IPAddress RealIP)
   at GSMyAdmin.WebServer.ApiService.InvokeAPI(HttpContext context, IWebSession session, JObject data, String requestModule, String requestMethod)
09:32:14
Slow method invocation: Login took 8943ms to complete.
Slow response: Core.Login took 8944ms to complete.
Slow method invocation: Login took 8511ms to complete.
Slow response: Core.Login took 8512ms to complete.
09:32:19
Slow method invocation: Login took 4682ms to complete.
Slow response: Core.Login took 4684ms to complete.
09:32:21
Slow method invocation: Login took 4188ms to complete.
Slow response: Core.Login took 4189ms to complete.
09:32:30
Slow method invocation: Login took 3105ms to complete.
Slow response: Core.Login took 3106ms to complete.
09:32:38
Websocket closed unexpectedly. State: CloseReceived
09:33:00
Slow method invocation: Login took 3014ms to complete.
Slow response: Core.Login took 3016ms to complete.
09:33:34
Slow method invocation: Login took 2706ms to complete.
Slow response: Core.Login took 2712ms to complete.
09:33:46
Timed out waiting for response from remote instance at http://10.8.0.9:8080/
TaskCanceledException
[0] (TaskCanceledException) : The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
   at Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at AMPAPI.RemoteInstance.GetResponseDynamic(String Module, String Method, Object Parameters)
   at AMPAPI.RemoteInstance.API_Core.Login(String username, String password, String token, Boolean rememberMe)
   at InstanceManagerPlugin.RemoteInstanceManager.Login(String username, String password, String token)
TimeoutException
[1] (TimeoutException) : A task was canceled.
TaskCanceledException
[2] (TaskCanceledException) : A task was canceled.
   at Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
Failed to update AMPlinux2 R720 - Could not connect to the remote instance at http://10.8.0.9:8080/. The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
Authentication token 3fe0320cbb7b for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:33:49
Authentication token 6f50c6d02433 for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:33:51
Authentication token 89339ef0809f for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
:kenny09:33:53
Timed out waiting for response from remote instance at http://10.8.0.9:8080/
Failed to get target info from http://10.8.0.9:8080/: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
09:33:53
Connection id "0HND87HADO7RK", Request id "0HND87HADO7RK:00000001": An unhandled exception was thrown by the application.
Authentication token 95d3432861b1 for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:33:56
Authentication token d7d6f2b90be9 for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:34:02
Authentication token 7d3e0e66f0b7 for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:34:04
Authentication token 411ab8ba0e3d for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:35:44
Timed out waiting for response from remote instance at http://10.8.0.9:8080/
TaskCanceledException
[0] (TaskCanceledException) : The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
   at Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at AMPAPI.RemoteInstance.GetResponseDynamic(String Module, String Method, Object Parameters)
   at AMPAPI.RemoteInstance.API_Core.Login(String username, String password, String token, Boolean rememberMe)
   at InstanceManagerPlugin.RemoteInstanceManager.Login(String username, String password, String token)
TimeoutException
[1] (TimeoutException) : A task was canceled.
TaskCanceledException
[2] (TaskCanceledException) : A task was canceled.
   at Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
Failed to update AMPlinux2 R720 - Could not connect to the remote instance at http://10.8.0.9:8080/. The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
09:35:45
Authentication token bf4919c6cf18 for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:35:47
Authentication token c95113782fbc for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:35:52
Authentication token fa8446c29358 for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:35:54
Authentication token f7a6ad9190ed for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:35:56
Authentication token 89d2b6c43ad5 for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:35:58
Authentication token 3620a94f82aa for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:36:01
Authentication token c8f0ce52460e for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:37:41
Timed out waiting for response from remote instance at http://10.8.0.9:8080/
TaskCanceledException
[0] (TaskCanceledException) : The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
   at Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at AMPAPI.RemoteInstance.GetResponseDynamic(String Module, String Method, Object Parameters)
   at AMPAPI.RemoteInstance.API_Core.Login(String username, String password, String token, Boolean rememberMe)
   at InstanceManagerPlugin.RemoteInstanceManager.Login(String username, String password, String token)
TimeoutException
[1] (TimeoutException) : A task was canceled.
TaskCanceledException
[2] (TaskCanceledException) : A task was canceled.
   at Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
Failed to update AMPlinux2 R720 - Could not connect to the remote instance at http://10.8.0.9:8080/. The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
09:37:42
Authentication token 2620c78fbe4b for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:37:44
Authentication token 451006fba816 for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:37:46
Authentication token d4d991208846 for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:37:47
Authentication token 55c27f30da41 for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:37:49
Authentication token 40b8f1190042 for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:37:51
Authentication token 3544253583ae for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:39:31
Timed out waiting for response from remote instance at http://10.8.0.9:8080/
TaskCanceledException
[0] (TaskCanceledException) : The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
   at Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at AMPAPI.RemoteInstance.GetResponseDynamic(String Module, String Method, Object Parameters)
   at AMPAPI.RemoteInstance.API_Core.Login(String username, String password, String token, Boolean rememberMe)
   at InstanceManagerPlugin.RemoteInstanceManager.Login(String username, String password, String token)
TimeoutException
[1] (TimeoutException) : A task was canceled.
TaskCanceledException
[2] (TaskCanceledException) : A task was canceled.
   at Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
Failed to update AMPlinux2 R720 - Could not connect to the remote instance at http://10.8.0.9:8080/. The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
Authentication token b50b0ef1b6cf for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:39:33
Authentication token 570a944e73b6 for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:39:35
Authentication token aedb40871f89 for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:39:38
Authentication token 21a2a325a9b9 for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:39:40
Authentication token a8e196711912 for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:39:42
Authentication token b27484e9e236 for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL
09:39:48
Authentication token 94427f78c05c for AMP_SYSTEM requested by UpdateRemoteInstancesList on behalf of INTERNAL

Steps to reproduce:

  • Step 1
    Access AMP Controller via Proxy & Direct (to eliminate possible causes)
  • Step 2
    Browse AMP Controller
  • Step 3
    See that it is blank

Actions taken to resolve so far:

Cleared cache, used different browser.
Used different computer to access
Updated AMP controller, Updated AMP targets, Updated Instances
Restarted all (controller, instances, machines) 10+ times
logged in and updated individual instance locally via direct connection (cannot manage but can upgrade)