ADS01 Instance stuck in recovery

OS Name/Version:
OS: Ubuntu 22.04.4 LTS x86_64
Kernel: 5.15.0-101-generic
Shell: bash 5.1.16
Resolution: 1280x800
Terminal: tmux
CPU: Common KVM (4) @ 2.495GHz
Memory: 5302MiB / 24029MiB
/dev/sda1 35% used

Product Name/Version: 2.5.0.4

Problem Description:

VM was moved to secondary Proxmox host, all vm hardware information such as mac addresses, processor info, memory, network IPs, etc. stayed the same.
On boot all instances load with no errors given but the web interface will not load.
The game instances (minecraft and palworld) are loading and are functional to join as a player but they can’t be managed.
Web requests get a 502 bad gateway message. Even when connecting to the local IP:Port rather than by DNS

At a bit of a loss of what else to do.

So far I have:
Rebooted of course
moved from mainline to dev branch - Didn’t fix
Reinstalled with apt - didnt fix
Reconfigured ADS01
check ngnix config, seems setup right.

Reconfig output:

ampinstmgr reconfigure ADS01 +Core.Monitoring.LogLevel 0
[Info] AMP Instance Manager v2.5.0.4 built 04/04/2024 18:29
[Info] Stream: Mainline / Release - built by CUBECODERS/buildbot on CCL-DEV
[Info] Reconfiguring instance ADS01...
[21:53:31] [Core Info]            : Starting AMP version 2.5.0.4 (Callisto), built 05/04/2024 21:00
[21:53:31] [Core Info]            : Stream: Development / Development - built by CUBECODERS/buildbot on CCL-DEV
[21:53:31] [Core Info]            : Loading configuration from provisionargs.kvp...
[21:53:31] [Core Debug]           : Loading locale data for xx from xx.json...
[21:53:31] [Logger Info]          : Purging old audit logs (this may take a moment)...
[21:53:31] [Core Info]            : Running in a QEMU KVM environment.
[21:53:31] [Core Debug]           : Running command line: "/usr/bin/sudo -n bash -c "touch /dev/null"" from ""
[21:53:31] [Core Warning]         : Current time zone is set to EST - It is recommended you switch to UTC+0 to avoid time-zone related issues
[21:53:31] [Core Info]            : OS: Linux / x86_64
[21:53:31] [Core Info]            : CPU: Common KVM processor (4C/4T)
[21:53:31] [Core Info]            : RAM: 24029MB
[21:53:31] [Core Info]            : AMP Instance ID: b9a60cc4-33c0-4940-9ef5-a20642fc1a43
[21:53:31] [Core Debug]           : Configuring licencing, crypto and webserver...
[21:53:31] [Core Debug]           : Primary network interface node is ens18 with physical address 525400DFCD06
[21:53:31] [Config Debug]         : Setting up secure value store with machine hash: ZW5zMTg1MjU0MDBERkNEMDY=
[21:53:31] [Core Debug]           : Configuring shared components...
[21:53:31] [Core Debug]           : Using internal login provider
[21:53:31] [Core Debug]           : Configuring application module...
[21:53:31] [SignatureVerification Debug] : Performing signature verification for ADSModule...
[21:53:31] [SignatureVerification Debug] : Performing signature verification for InstanceManagerPlugin...
[21:53:31] [SignatureVerification Debug] : Performing signature verification for FileManagerPlugin...
[21:53:32] [ADS Debug]            : Loaded 0 remote targets from Datastore
[21:53:32] [SignatureVerification Debug] : Performing signature verification for steamcmdplugin...
[21:53:32] [Loader Info]          : Loaded ADSModule version 1.0.0.0 by CubeCoders Limited
[21:53:32] [Core Debug]           : Configuring scheduler...
[21:53:32] [Loader Debug]         : Deprecated plugin FileManager swapped for FileManagerPlugin
[21:53:32] [Loader Info]          : Loaded FileManagerPlugin by CubeCoders Limited
[21:53:32] [Loader Debug]         : Deprecated plugin EmailSender swapped for EmailSenderPlugin
[21:53:32] [SignatureVerification Debug] : Performing signature verification for EmailSenderPlugin...
[21:53:32] [Loader Info]          : Loaded EmailSenderPlugin by CubeCoders Limited
[21:53:32] [SignatureVerification Debug] : Performing signature verification for WebRequestPlugin...
[21:53:32] [Loader Info]          : Loaded WebRequestPlugin by CubeCoders Limited
[21:53:32] [SignatureVerification Debug] : Performing signature verification for LocalFileBackupPlugin...
[21:53:32] [FileBackup Debug]     : Read manifests for 0 backups.
[21:53:32] [Loader Info]          : Loaded LocalFileBackupPlugin by CubeCoders Limited
[21:53:32] [Loader Info]          : ADSModule requests dependency InstanceManagerPlugin...
[21:53:32] [InstanceManager Debug] : Loading instances from /home/amp/.ampdata/instances.json...
[21:53:32] [InstanceManager Debug] : Loaded 4 instances.
[21:53:32] [Loader Info]          : Loaded InstanceManagerPlugin by CubeCoders Limited
[21:53:32] [Loader Info]          : ADSModule requests dependency SystemUserManagerPlugin...
[21:53:32] [SignatureVerification Debug] : Performing signature verification for SystemUserManagerPlugin...
[21:53:32] [Loader Info]          : Loaded SystemUserManagerPlugin by CubeCoders Limited
[21:53:32] [Core Debug]           : Checking settings...
[21:53:32] [Core Info]            : -configonly specified or activation performed. Saving settings and shutting down. This is perfectly normal.

$ ampinstmgr start ADS01
[Info] AMP Instance Manager v2.5.0.4 built 04/04/2024 18:29
[Info] Stream: Mainline / Release - built by CUBECODERS/buildbot on CCL-DEV
[Info] Starting Instance: 'ADS01'
[Info] Waiting for AMP instance to start...
[Notice] AMP instance ADS01 is now running.
[Info] You can browse to this instance at http://127.0.0.1:8080/
[Info] Instance started in new session. Run 'ampinstmgr View ADS01' to view this instances live output

View and last log output of ADS01

[22:19:21] [Core Info]            : Starting AMP version 2.5.0.4 (Callisto), built 05/04/2024 21:00                                                                                                                                                                                   [0/1880]
[22:19:21] [Core Info]            : Stream: Development / Development - built by CUBECODERS/buildbot on CCL-DEV
[22:19:21] [Core Debug]           : Loading locale data for xx from xx.json...
[22:19:22] [Logger Info]          : Purging old audit logs (this may take a moment)...
[22:19:22] [Core Info]            : Running in a QEMU KVM environment.
[22:19:22] [Core Debug]           : Running command line: "/usr/bin/sudo -n bash -c "touch /dev/null"" from ""
[22:19:22] [Core Warning]         : Current time zone is set to EST - It is recommended you switch to UTC+0 to avoid time-zone related issues
[22:19:22] [Core Info]            : OS: Linux / x86_64
[22:19:22] [Core Info]            : CPU: Common KVM processor (4C/4T)
[22:19:22] [Core Info]            : RAM: 24029MB
[22:19:22] [Core Info]            : AMP Instance ID: b9a60cc4-33c0-4940-9ef5-a20642fc1a43
[22:19:22] [Core Debug]           : Configuring licencing, crypto and webserver...
[22:19:22] [Core Debug]           : Primary network interface node is ens18 with physical address 525400DFCD06
[22:19:22] [Config Debug]         : Setting up secure value store with machine hash: ZW5zMTg1MjU0MDBERkNEMDY=
[22:19:22] [Core Debug]           : Configuring shared components...
[22:19:22] [Core Debug]           : Using internal login provider
[22:19:22] [Core Debug]           : Configuring application module...
[22:19:22] [SignatureVerification Debug] : Performing signature verification for ADSModule...
[22:19:22] [SignatureVerification Debug] : Performing signature verification for InstanceManagerPlugin...
[22:19:22] [SignatureVerification Debug] : Performing signature verification for FileManagerPlugin...
[22:19:22] [ADS Debug]            : Loaded 0 remote targets from Datastore
[22:19:22] [SignatureVerification Debug] : Performing signature verification for steamcmdplugin...
[22:19:22] [Loader Info]          : Loaded ADSModule version 1.0.0.0 by CubeCoders Limited
[22:19:22] [Core Debug]           : Configuring scheduler...
[22:19:22] [Loader Debug]         : Deprecated plugin FileManager swapped for FileManagerPlugin
[22:19:22] [Loader Info]          : Loaded FileManagerPlugin by CubeCoders Limited
[22:19:22] [Loader Debug]         : Deprecated plugin EmailSender swapped for EmailSenderPlugin
[22:19:22] [SignatureVerification Debug] : Performing signature verification for EmailSenderPlugin...
[22:19:22] [Loader Info]          : Loaded EmailSenderPlugin by CubeCoders Limited
[22:19:22] [SignatureVerification Debug] : Performing signature verification for WebRequestPlugin...
[22:19:22] [Loader Info]          : Loaded WebRequestPlugin by CubeCoders Limited
[22:19:22] [SignatureVerification Debug] : Performing signature verification for LocalFileBackupPlugin...
[22:19:22] [FileBackup Debug]     : Read manifests for 0 backups.
[22:19:22] [Loader Info]          : Loaded LocalFileBackupPlugin by CubeCoders Limited
[22:19:22] [Loader Info]          : ADSModule requests dependency InstanceManagerPlugin...
[22:19:22] [InstanceManager Debug] : Loading instances from /home/amp/.ampdata/instances.json...
[22:19:22] [InstanceManager Debug] : Loaded 4 instances.
[22:19:22] [Loader Info]          : Loaded InstanceManagerPlugin by CubeCoders Limited
[22:19:22] [Loader Info]          : ADSModule requests dependency SystemUserManagerPlugin...
[22:19:22] [SignatureVerification Debug] : Performing signature verification for SystemUserManagerPlugin...
[22:19:23] [Loader Info]          : Loaded SystemUserManagerPlugin by CubeCoders Limited
[22:19:23] [Core Debug]           : Checking settings...
[22:19:23] [Loader Info]          : Loaded steamcmdplugin by CubeCoders Limited
[22:19:23] [InstanceManager Debug] : User: amp
[22:19:23] [InstanceManager Debug] : Home Dir: /home/amp
[22:19:23] [Core Info]            : Metrics server started OK on port 12820
[22:19:23] [ADS Info]             : ADS startup complete in 76ms
[22:19:23] [ADS Info]             : Updating remote source CubeCoders/AMPTemplates
[22:19:23] [ADS Info]             : Updating existing remote source https://github.com/CubeCoders/AMPTemplates.git...
[22:19:23] [Core Debug]           : Running command line: "/usr/bin/git pull" from "Plugins/ADSModule/DeploymentTemplates/CubeCoders-AMPTemplates"
[22:19:23] [InstanceManager Debug] : Fetching metadata for ADS from /home/amp/.ampdata/instances/ADS01/Plugins/ADSModule/ADSModule.dll...
[22:19:23] [InstanceManager Debug] : Fetching metadata for FileManagerPlugin from /home/amp/.ampdata/instances/ADS01/Plugins/FileManagerPlugin/FileManagerPlugin.dll...
[22:19:23] [InstanceManager Debug] : Fetching metadata for EmailSenderPlugin from /home/amp/.ampdata/instances/ADS01/Plugins/EmailSenderPlugin/EmailSenderPlugin.dll...
[22:19:23] [InstanceManager Debug] : Fetching metadata for WebRequestPlugin from /home/amp/.ampdata/instances/ADS01/Plugins/WebRequestPlugin/WebRequestPlugin.dll...
[22:19:23] [InstanceManager Debug] : Fetching metadata for LocalFileBackupPlugin from /home/amp/.ampdata/instances/ADS01/Plugins/LocalFileBackupPlugin/LocalFileBackupPlugin.dll...
[22:19:23] [InstanceManager Debug] : Fetching metadata for CommonCorePlugin from /home/amp/.ampdata/instances/ADS01/Plugins/CommonCorePlugin/CommonCorePlugin.dll...
[22:19:23] [InstanceManager Debug] : Fetching metadata for McMyAdmin from /home/amp/.ampdata/instances/ADS01/Plugins/MinecraftModule/MinecraftModule.dll...
[22:19:23] [InstanceManager Debug] : Fetching metadata for GenericModule from /home/amp/.ampdata/instances/ADS01/Plugins/GenericModule/GenericModule.dll...
Already up to date.
[22:19:23] [FileManager Notice]   : Using keypair with fingerprint N3pbdDekRFLwN7faFJqMmr6GMpL5hGKVwBK17iQHPIA=
[22:19:23] [FileManager Info]     : SFTP Server started on 0.0.0.0:2223
[22:19:23] [Webserver Debug]      : Using webroot: /home/amp/.ampdata/instances/ADS01/WebRoot
[22:19:23] [Webserver Info]       : Websockets are enabled.
[22:19:23] [API Debug]            : Building setting specification cache...
[22:19:23] [API Debug]            : Setting spec cache built, 114 items added.
[22:19:23] [Logger Info]          : RouterTimer@10Hz with 2 jobs started
[22:19:23] [Webserver Info]       : Webserver started on http://127.0.0.1:8080
[22:19:23] [Scheduler Debug]      : Scheduler timer will start in 36144ms.
[22:19:23] [Core Info]            : Checking for AMP updates...
[22:19:23] [Core Info]            : Startup mode is UpdateAndStart.
[22:19:23] [Core Debug]           : Startup Complete.
[22:19:24] [Core Info]            : A new patch is available!
[22:19:28] [RunningTasksManager Debug] : Task Updating remote sources (Fetching latest configurations...) ended: Finished

nginx logs

nginx access

$ tail /var/log/nginx/access.log
172.27.183.3 - - [05/Apr/2024:18:22:55 -0400] "GET /favicon.ico HTTP/2.0" 502 11072 "https://amp.spartan196.me/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
172.27.183.3 - - [05/Apr/2024:18:22:57 -0400] "GET /ServiceWorker.js HTTP/2.0" 502 11072 "https://amp.spartan196.me/ServiceWorker.js" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
172.27.183.3 - - [05/Apr/2024:18:23:10 -0400] "GET / HTTP/2.0" 502 11072 "https://amp.spartan196.me/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
172.27.183.3 - - [05/Apr/2024:18:23:10 -0400] "GET /shared/installState.json?t=1712355789544 HTTP/2.0" 200 15 "https://amp.spartan196.me/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
172.27.183.3 - - [05/Apr/2024:18:23:10 -0400] "GET /favicon.ico HTTP/2.0" 502 11072 "https://amp.spartan196.me/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
172.27.183.3 - - [05/Apr/2024:18:23:12 -0400] "GET /ServiceWorker.js HTTP/2.0" 502 11072 "https://amp.spartan196.me/ServiceWorker.js" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
172.27.183.3 - - [05/Apr/2024:18:23:25 -0400] "GET / HTTP/2.0" 502 11072 "https://amp.spartan196.me/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
172.27.183.3 - - [05/Apr/2024:18:23:25 -0400] "GET /shared/installState.json?t=1712355804631 HTTP/2.0" 200 15 "https://amp.spartan196.me/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
172.27.183.3 - - [05/Apr/2024:18:23:25 -0400] "GET /favicon.ico HTTP/2.0" 502 11072 "https://amp.spartan196.me/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
172.27.183.3 - - [05/Apr/2024:18:23:27 -0400] "GET /ServiceWorker.js HTTP/2.0" 502 11072 "https://amp.spartan196.me/ServiceWorker.js" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"

nginx error

$ tail /var/log/nginx/error.log
2024/04/05 18:22:57 [crit] 699#699: *2154 connect() to [::1]:8080 failed (99: Unknown error) while connecting to upstream, client: 172.27.183.3, server: amp.spartan196.me, request: "GET /ServiceWorker.js HTTP/2.0", upstream: "http://[::1]:8080/ServiceWorker.js", host: "amp.spartan196.me", referrer: "https://amp.spartan196.me/ServiceWorker.js"
2024/04/05 18:23:10 [crit] 699#699: *2154 connect() to [::1]:8080 failed (99: Unknown error) while connecting to upstream, client: 172.27.183.3, server: amp.spartan196.me, request: "GET / HTTP/2.0", upstream: "http://[::1]:8080/", host: "amp.spartan196.me", referrer: "https://amp.spartan196.me/"
2024/04/05 18:23:10 [crit] 699#699: *2154 connect() to [::1]:8080 failed (99: Unknown error) while connecting to upstream, client: 172.27.183.3, server: amp.spartan196.me, request: "GET /favicon.ico HTTP/2.0", upstream: "http://[::1]:8080/favicon.ico", host: "amp.spartan196.me", referrer: "https://amp.spartan196.me/"
2024/04/05 18:23:12 [crit] 699#699: *2154 connect() to [::1]:8080 failed (99: Unknown error) while connecting to upstream, client: 172.27.183.3, server: amp.spartan196.me, request: "GET /ServiceWorker.js HTTP/2.0", upstream: "http://[::1]:8080/ServiceWorker.js", host: "amp.spartan196.me", referrer: "https://amp.spartan196.me/ServiceWorker.js"
2024/04/05 18:23:25 [crit] 699#699: *2154 connect() to [::1]:8080 failed (99: Unknown error) while connecting to upstream, client: 172.27.183.3, server: amp.spartan196.me, request: "GET / HTTP/2.0", upstream: "http://[::1]:8080/", host: "amp.spartan196.me", referrer: "https://amp.spartan196.me/"
2024/04/05 18:23:25 [crit] 699#699: *2154 connect() to [::1]:8080 failed (99: Unknown error) while connecting to upstream, client: 172.27.183.3, server: amp.spartan196.me, request: "GET /favicon.ico HTTP/2.0", upstream: "http://[::1]:8080/favicon.ico", host: "amp.spartan196.me", referrer: "https://amp.spartan196.me/"
2024/04/05 18:23:27 [crit] 699#699: *2154 connect() to [::1]:8080 failed (99: Unknown error) while connecting to upstream, client: 172.27.183.3, server: amp.spartan196.me, request: "GET /ServiceWorker.js HTTP/2.0", upstream: "http://[::1]:8080/ServiceWorker.js", host: "amp.spartan196.me", referrer: "https://amp.spartan196.me/ServiceWorker.js"
2024/04/05 18:23:40 [crit] 699#699: *2154 connect() to [::1]:8080 failed (99: Unknown error) while connecting to upstream, client: 172.27.183.3, server: amp.spartan196.me, request: "GET / HTTP/2.0", upstream: "http://[::1]:8080/", host: "amp.spartan196.me", referrer: "https://amp.spartan196.me/"
2024/04/05 18:23:40 [crit] 699#699: *2154 connect() to [::1]:8080 failed (99: Unknown error) while connecting to upstream, client: 172.27.183.3, server: amp.spartan196.me, request: "GET /favicon.ico HTTP/2.0", upstream: "http://[::1]:8080/favicon.ico", host: "amp.spartan196.me", referrer: "https://amp.spartan196.me/"
2024/04/05 18:23:42 [crit] 699#699: *2154 connect() to [::1]:8080 failed (99: Unknown error) while connecting to upstream, client: 172.27.183.3, server: amp.spartan196.me, request: "GET /ServiceWorker.js HTTP/2.0", upstream: "http://[::1]:8080/ServiceWorker.js", host: "amp.spartan196.me", referrer: "https://amp.spartan196.me/ServiceWorker.js"

just do ampinstmgr start ads01 and it should kick it back up as the amp user sudo su -l amp

Yeah that’s been tried.
I also interactively switched uses to the amp user and ran ampinstmgr start ADS01
Its output is posted in first code block above.

Listing of amp instances,
The game instances are able to be connected to by clients but even direct links into what should be their instance https://$host/?instance={firstpart-of-instanceID} also give a 502 and error page that says “amp is being installed”

UPDATE
Now resolved after running getamp postSetupHTTPS and letting it rebind all instances.
followed by ampinstmgr --UpgradeAll

Side note why prevent us from replying to our own posts if there has been no other comments yet?

Wait for others to participate

so the instance direct url should be that yes before the - but it can be your browser cache just not being cleared and the part of not being able to replay to your own post is just part of spam protection

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.