ADS01 and Instances no longer start after Failed Upgrade and Attempted fix

Technical Support
, ,
1

OS Name/Version:
Ubuntu 20.04.6 LTS - 5.6.0-1054-oem

Product Name/Version: (Always use the full version number - not ‘Latest’)
AMP version 2.4.3.0 (Decadeus), built 14/03/2023 23:15

Problem Description:
Running a selfhosted AMP with one Docker container for a controller and currently one target for hosting images. I’ll refer to them as docker and beast.

Docker was updated to 2.4.3.0 from best as I can guess was 2.4.2.0. I believe though the correct process is to upgrade targets first. From what I saw in the web interface, it was due for an upgrade to 2.4.3.0, so I told beast to.

Before and after the upgrade, docker was not able to talk to beast and I saw failed auth key attempts on beast. I tried to fix it, could not, and made it worse. Now the web interface (ADS01) no longer starts.

Attempting to start ADS01 now gives a lastlog like this:

[04:23:29] [Core Info]            : Starting AMP version 2.4.3.0 (Decadeus), built 14/03/2023 23:15
[04:23:29] [Core Info]            : Stream: Mainline (Release) - built by CUBECODERS/buildbot on CCL-DEV
[04:23:29] [Core Debug]           : Running command line: "/usr/bin/sudo -n bash -c "touch /dev/null"" from ""
[04:23:29] [Core Warning]         : Current time zone is set to EST - It is recommended you switch to UTC+0 to avoid time-zone related issues
[04:23:29] [Core Info]            : OS: Linux / x86_64
[04:23:29] [Core Info]            : CPU: Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz (8C/16T)
[04:23:29] [Core Info]            : RAM: 63949MB
[04:23:29] [Core Info]            : AMP Instance ID: 92298491-3311-49cc-baf6-0b19e97c1f81

The actual output has a little more:

  CubeCoders AMP  💻amp@beast  📁~/.ampdata/instances  ampinstmgr start ADS01
[Info] AMP Instance Manager v2.4.3 built 14/03/2023 21:50
[Info] Release spec: Release - built by CUBECODERS/buildbot on CCL-DEV
[Info] Starting Instance: 'ADS01'
[Info] Waiting for AMP instance to start...
[Error] Failed to start AMP. Exit code was 0
[Info] This instance failed to start. Run 'ampinstmgr LastLog ADS01' to view it's most recent log.
  CubeCoders AMP  💻amp@beast  📁~/.ampdata/instances  ampinstmgr LastLog ADS01
[Info] AMP Instance Manager v2.4.3 built 14/03/2023 21:50
[Info] Release spec: Release - built by CUBECODERS/buildbot on CCL-DEV

If I run AMP_Linux_x86_64 directly, I get:

  CubeCoders AMP  💻amp@beast  📁~/.ampdata/instances/ADS01  ./AMP_Linux_x86_64
[04:28:44] [Core Info]            : Starting AMP version 2.4.3.0 (Decadeus), built 14/03/2023 23:15
[04:28:44] [Core Info]            : Stream: Mainline (Release) - built by CUBECODERS/buildbot on CCL-DEV
[04:28:44] [Core Debug]           : Running command line: "/usr/bin/sudo -n bash -c "touch /dev/null"" from ""
[04:28:44] [Core Warning]         : Current time zone is set to EST - It is recommended you switch to UTC+0 to avoid time-zone related issues
[04:28:44] [Core Info]            : OS: Linux / x86_64
[04:28:44] [Core Info]            : CPU: Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz (8C/16T)
[04:28:44] [Core Info]            : RAM: 63949MB
[04:28:44] [Core Info]            : AMP Instance ID: 92298491-3311-49cc-baf6-0b19e97c1f81

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
  at GSMyAdmin.Core.SetupAuthProvider (GSMyAdmin.Config Config) [0x00015] in <b5681ed5587e43f0853017e6ab190ac4>:0
  at GSMyAdmin.Core.Run () [0x00475] in <b5681ed5587e43f0853017e6ab190ac4>:0
  at GSMyAdmin.Program.Main () [0x00005] in <b5681ed5587e43f0853017e6ab190ac4>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
  at GSMyAdmin.Core.SetupAuthProvider (GSMyAdmin.Config Config) [0x00015] in <b5681ed5587e43f0853017e6ab190ac4>:0
  at GSMyAdmin.Core.Run () [0x00475] in <b5681ed5587e43f0853017e6ab190ac4>:0
  at GSMyAdmin.Program.Main () [0x00005] in <b5681ed5587e43f0853017e6ab190ac4>:0

None of the instances (all Minecraft related) fully start, looks like they all get errors trying to talk back to Core:

[04:11:00] [Core Error]           : SocketException
[04:11:00] [Core Error]           : [0] (SocketException) : Connection refused
[04:11:00] [Core Error]           :   at Net.Sockets.Socket.Send (Byte[] buffer, Int32 offset, Int32 size, Net.Sockets.SocketFlags socketFlags)
  at Net.Sockets.UdpClient.Send (Byte[] dgram, Int32 bytes)
  at ModuleShared.MetricsInfoPublisher.ScheduleHandler (Object sender, EventArgs e)

I see that many times in the log.

I’m also having a problem logging into any of the instances. May be related.

Any suggestions outside resetting everything? I’m about ready to wipe and rebuild at this rate.

Steps to reproduce:

  • Controller was updated, Target was not.
  • Target got updated.
  • Still unable to connect to Target

Some of the commands I ran to try to fix were:

  • ampinstmgr --repairauth
  • ampinstmgr --reactivate ADS01 {key}
  • ampinstmgr --ReconfigureInstance ADS01 (this would not even run)

Actions taken to resolve so far:

Aftert I broke it more (nothing starts), I tried:

  • resetting permissions
  • taking a full backup (no real prior backups though)
  • restoring ADS01 from a 2 month old zip
  • Resetting things back
  • running AMP_Linux_x86_64 directly (above)
  • restoring AMP_Linux_x86_64 from 2.4.2.0’s Mainline zip along with Plugins and WebRoot
  • Resetting things back…
  • Finally coming here.
2

You said you had a controller and a target, but it’s not clear from the logs which ADS log is from which?

3

All the logs is from the Target. Controller is up and running fine as far as I can tell.

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.