Instances failed to start after 2.4.3.6 update

Hello everyone,

I recently did an update across all instances to 2.4.3.6 and have been unable to start up instances including ADS after.

So far, the issues only persist on Fedora Server 37. I was able to successfully update AlmaLinux 8.7 and Ubuntu 20.04 without a problem.

Currently 3 systems are running Fedora Server 37 with only one of those having docker enabled. The result is the same as all 3.

I have been able to update flawlessly during the years, but somehow, I am stuck on this one. Any assistance would be appreciated.

Troubleshooting:

  • Attempted --nocache updating on all instances with problems.
  • Rebooted several times
  • Manually tried to start instances (Log below of core dump)
  • Performed a full system update – dnf update ; upgrade
  • Rebooted some more

‘ampinstmgr -a’ reports instances are on and running when indeed are not running as displayed in ‘ampinstmgr -l’

Running the command again displays: Error failed to start amp with exit code 0


Logs

Process 1931 (AMP_Linux_x86_6) of user 1001 dumped core.

Module /home/amp/.ampdata/instances/ADS01/AMP_Linux_x86_64 with build-id 3d907dfd51def3970966f9333d958f141cb0b2d9
Module /tmp/mono-bundle-pgFMhI/libmonoboehm-2 with build-id ffa713e2049ff10d753792397938018a5f704c87
Module /tmp/mono-bundle-pgFMhI/libmono-native with build-id 424cccd5f4a6ca2103a099ea101b5df63d36b6f7
Module /tmp/mono-bundle-pgFMhI/MonoPosixHelper with build-id 79d773e6b09bd366320ec19858e7654c17740885
Module /tmp/mono-bundle-pgFMhI/libmono-btls-shared with build-id d476a235acd8357c6be855f9d22b4d3ca476bdf1
Module /tmp/mono-bundle-pgFMhI/e_sqlite3 with build-id 6bcc1fa2345785aad27567aa9a689067f40401d0
Module /tmp/mono-bundle-pgFMhI/libsqlite3 with build-id 5e4d89ec762208ca668b4dfac5f761fbdb69e3e6
Module linux-vdso.so.1 with build-id 02d41dea1be42d17709e5c411f68ba964cbe65c3
Module ld-linux-x86-64.so.2 with build-id bb6fec54c7521fddc569a2f4e141dfb97bf3acbe
Module libc.so.6 with build-id 81daba31ee66dbd63efdc4252a872949d874d136
Module libgcc_s.so.1 with build-id 6baf7a1f1451ec21c7a598b1ea165a2d563edb07
Module libpthread.so.0 with build-id 622fcaae1f5e94890852b15a1705dff85830347b
Module libdl.so.2 with build-id ff36f66b23c53809d27933a667b1f5d6b55e033b
Module librt.so.1 with build-id 6080d9df60161f18df8e640eea23089bbc05e13b
Module libm.so.6 with build-id 9ca49f056adbf7edc2bda92b95a74d491abe03f2
Stack trace of thread 1931:
#0  0x00000000004ad954 n/a (/home/amp/.ampdata/instances/ADS01/AMP_Linux_x86_64 + 0xad954)
ELF object binary architecture: AMD x86-64
CODE_FILE
../src/coredump/coredump.c
CODE_FUNC
submit_coredump
CODE_LINE
1034
COREDUMP_CGROUP
/system.slice/ampinstmgr.service
COREDUMP_CMDLINE
/home/amp/.ampdata/instances/ADS01/AMP_Linux_x86_64 +Core.Webserver.Port 8080 +Core.Webserver.IPBinding 0.0.0.0 +Core.AMP.InstanceID 83896f41-c229-43d0-b5d3-bafb7bc4d456 +FileManagerPlugin.SFTP.SFTPPortNumber 2223 +ADSModule.Network.MetricsServerPort 12820
COREDUMP_COMM
AMP_Linux_x86_6
COREDUMP_CWD
/home/amp/.ampdata/instances/ADS01
COREDUMP_ENVIRON
SHELL=/bin/bash TERM_PROGRAM_VERSION=3.3a TMUX=/tmp/tmux-1001/default,1929,0 PWD=/home/amp/.ampdata/instances/ADS01 LOGNAME=amp SYSTEMD_EXEC_PID=1012 HOME=/home/amp LANG=en_US.UTF-8 INVOCATION_ID=83a5c1fcb25046c3b96a730d71937341 TERM=tmux-256color USER=amp TMUX_PANE=%0 SHLVL=0 LD_LIBRARY_PATH=./:/opt/cubecoders/amp/ JOURNAL_STREAM=8:27015 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin TMUX_SESSION_NAME=AMP_ADS01 OLDPWD=/home/amp/.ampdata/instances/ADS01 TERM_PROGRAM=tmux _=/home/amp/.ampdata/instances/ADS01/AMP_Linux_x86_64
COREDUMP_EXE
/home/amp/.ampdata/instances/ADS01/AMP_Linux_x86_64
COREDUMP_FILENAME
/var/lib/systemd/coredump/core.AMP_Linux_x86_6.1001.a0d002609fcd447abcb19f772b3cc29a.1931.1684213427000000.zst
COREDUMP_GID
1001
COREDUMP_OPEN_FDS
0:/dev/pts/0 pos: 0 flags: 02 mnt_id: 27 ino: 3 1:/dev/pts/0 pos: 0 flags: 02 mnt_id: 27 ino: 3 2:/dev/pts/0 pos: 0 flags: 02 mnt_id: 27 ino: 3 3:/home/amp/.ampdata/instances/ADS01/AMP_Linux_x86_64 pos: 101645432 flags: 0100000 mnt_id: 94 ino: 473060
COREDUMP_PACKAGE_JSON
{"elfType":"coredump","elfArchitecture":"AMD x86-64"}
COREDUMP_PID
1931
COREDUMP_PROC_AUXV
[336 bytes of binary data]
COREDUMP_PROC_CGROUP
0::/system.slice/ampinstmgr.service
COREDUMP_PROC_LIMITS
Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 63751 63751 processes Max open files 1024 524288 files Max locked memory 8388608 8388608 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 63751 63751 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
COREDUMP_PROC_MAPS
00400000-0084e000 r-xp 00000000 fd:00 473060 /home/amp/.ampdata/instances/ADS01/AMP_Linux_x86_64 00a4e000-00a5a000 rw-p 0044e000 fd:00 473060 /home/amp/.ampdata/instances/ADS01/AMP_Linux_x86_64 00a5a000-00a76000 rw-p 00000000 00:00 0 02360000-02740000 rw-p 00000000 00:00 0 [heap] 7f71ece00000-7f71ed23f000 r-xp 00000000 00:22 73 /tmp/mono-bundle-pgFMhI/libmonoboehm-2 7f71ed23f000-7f71ed43e000 ---p 0043f000 00:22 73 /tmp/mono-bundle-pgFMhI/libmonoboehm-2 7f71ed43e000-7f71ed44a000 rw-p 0043e000 00:22 73 /tmp/mono-bundle-pgFMhI/libmonoboehm-2 7f71ed44a000-7f71ed6ab000 rw-p 00000000 00:00 0 7f71eea00000-7f71eec78000 r--p 00000000 fd:00 16927259 /usr/lib/locale/en_US.utf8/LC_COLLATE 7f71eee00000-7f71eeebd000 r-xp 00000000 00:22 62 /tmp/mono-bundle-pgFMhI/libmono-native 7f71eeebd000-7f71ef0bd000 ---p 000bd000 00:22 62 /tmp/mono-bundle-pgFMhI/libmono-native 7f71ef0bd000-7f71ef0be000 rw-p 000bd000 00:22 62 /tmp/mono-bundle-pgFMhI/libmono-native 7f71ef200000-7f71ef257000 r-xp 00000000 00:22 61 /tmp/mono-bundle-pgFMhI/MonoPosixHelper 7f71ef257000-7f71ef456000 ---p 00057000 00:22 61 /tmp/mono-bundle-pgFMhI/MonoPosixHelper 7f71ef456000-7f71ef458000 rw-p 00056000 00:22 61 /tmp/mono-bundle-pgFMhI/MonoPosixHelper 7f71ef458000-7f71ef459000 rw-p 00000000 00:00 0 7f71ef600000-7f71ef7a1000 r-xp 00000000 00:22 60 /tmp/mono-bundle-pgFMhI/libmono-btls-shared 7f71ef7a1000-7f71ef9a0000 ---p 001a1000 00:22 60 /tmp/mono-bundle-pgFMhI/libmono-btls-shared 7f71ef9a0000-7f71ef9b6000 rw-p 001a0000 00:22 60 /tmp/mono-bundle-pgFMhI/libmono-btls-shared 7f71efe00000-7f71efeb8000 r-xp 00000000 00:22 59 /tmp/mono-bundle-pgFMhI/e_sqlite3 7f71efeb8000-7f71f00b7000 ---p 000b8000 00:22 59 /tmp/mono-bundle-pgFMhI/e_sqlite3 7f71f00b7000-7f71f00ba000 r--p 000b7000 00:22 59 /tmp/mono-bundle-pgFMhI/e_sqlite3 7f71f00ba000-7f71f00bd000 rw-p 000ba000 00:22 59 /tmp/mono-bundle-pgFMhI/e_sqlite3 7f71f0200000-7f71f02c3000 r-xp 00000000 00:22 58 /tmp/mono-bundle-pgFMhI/libsqlite3 7f71f02c3000-7f71f04c2000 ---p 000c3000 00:22 58 /tmp/mono-bundle-pgFMhI/libsqlite3 7f71f04c2000-7f71f04c5000 r--p 000c2000 00:22 58 /tmp/mono-bundle-pgFMhI/libsqlite3 7f71f04c5000-7f71f04c7000 rw-p 000c5000 00:22 58 /tmp/mono-bundle-pgFMhI/libsqlite3 7f71f04c7000-7f71f04c8000 rw-p 00000000 00:00 0 7f71f0577000-7f71f4ef0000 r--p 01777000 fd:00 473060 /home/amp/.ampdata/instances/ADS01/AMP_Linux_x86_64 7f71f5059000-7f71f50b0000 r--p 00000000 fd:00 8390429 /usr/lib/locale/C.utf8/LC_CTYPE 7f71f50b0000-7f71f50b1000 r--p 00000000 fd:00 16927263 /usr/lib/locale/en_US.utf8/LC_NUMERIC 7f71f50b1000-7f71f50b2000 r--p 00000000 fd:00 25167086 /usr/lib/locale/en_US.utf8/LC_TIME 7f71f50b2000-7f71f50b3000 r--p 00000000 fd:00 25167082 /usr/lib/locale/en_US.utf8/LC_MONETARY 7f71f50b3000-7f71f50b4000 r--p 00000000 fd:00 25167095 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 7f71f50b4000-7f71f50b5000 r--p 00000000 fd:00 8390043 /usr/lib/locale/en_US.utf8/LC_PAPER 7f71f50b5000-7f71f50b6000 r--p 00000000 fd:00 16927262 /usr/lib/locale/en_US.utf8/LC_NAME 7f71f50b6000-7f71f50b7000 r--p 00000000 fd:00 25167077 /usr/lib/locale/en_US.utf8/LC_ADDRESS 7f71f50b7000-7f71f50ba000 rw-p 00000000 00:00 0 7f71f50ba000-7f71f50e0000 r--p 00000000 fd:00 8389706 /usr/lib64/libc.so.6
COREDUMP_PROC_MOUNTINFO
22 94 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:13 - proc proc rw 23 94 0:22 / /sys rw,nosuid,nodev,noexec,relatime shared:2 - sysfs sysfs rw,seclabel 24 94 0:5 / /dev rw,nosuid shared:9 - devtmpfs devtmpfs rw,seclabel,size=4096k,nr_inodes=1048576,mode=755,inode64 25 23 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:3 - securityfs securityfs rw 26 24 0:23 / /dev/shm rw,nosuid,nodev shared:10 - tmpfs tmpfs rw,seclabel,inode64 27 24 0:24 / /dev/pts rw,nosuid,noexec,relatime shared:11 - devpts devpts rw,seclabel,gid=5,mode=620,ptmxmode=000 28 94 0:25 / /run rw,nosuid,nodev shared:12 - tmpfs tmpfs rw,seclabel,size=3272264k,nr_inodes=819200,mode=755,inode64 29 23 0:26 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:4 - cgroup2 cgroup2 rw,seclabel,nsdelegate,memory_recursiveprot 30 23 0:27 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:5 - pstore pstore rw,seclabel 31 23 0:28 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:6 - efivarfs efivarfs rw 32 23 0:29 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:7 - bpf bpf rw,mode=700 94 1 253:0 / / rw,relatime shared:1 - xfs /dev/mapper/fedora-root rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota 34 23 0:19 / /sys/fs/selinux rw,nosuid,noexec,relatime shared:8 - selinuxfs selinuxfs rw 33 22 0:30 / /proc/sys/fs/binfmt_misc rw,relatime shared:14 - autofs systemd-1 rw,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=21788 35 24 0:18 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:15 - mqueue mqueue rw,seclabel 36 24 0:31 / /dev/hugepages rw,relatime shared:16 - hugetlbfs hugetlbfs rw,seclabel,pagesize=2M 37 23 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:17 - debugfs debugfs rw,seclabel 38 23 0:12 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:18 - tracefs tracefs rw,seclabel 39 23 0:32 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:19 - fusectl fusectl rw 40 23 0:33 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:20 - configfs configfs rw 86 94 0:34 / /tmp rw,nosuid,nodev shared:40 - tmpfs tmpfs rw,seclabel,nr_inodes=1048576,inode64 89 94 8:2 / /boot rw,relatime shared:46 - xfs /dev/sda2 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota 92 89 8:1 / /boot/efi rw,relatime shared:48 - vfat /dev/sda1 rw,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro 96 33 0:35 / /proc/sys/fs/binfmt_misc rw,nosuid,nodev,noexec,relatime shared:50 - binfmt_misc binfmt_misc rw 152 94 0:36 / /var/lib/nfs/rpc_pipefs rw,relatime shared:71 - rpc_pipefs sunrpc rw
COREDUMP_PROC_STATUS
Name: AMP_Linux_x86_6 Umask: 0022 State: S (sleeping) Tgid: 1931 Ngid: 0 Pid: 1931 PPid: 1929 TracerPid: 0 Uid: 1001 1001 1001 1001 Gid: 1001 1001 1001 1001 FDSize: 64 Groups: 5 1001 NStgid: 1931 NSpid: 1931 NSpgid: 1931 NSsid: 1931 VmPeak: 134776 kB VmSize: 113648 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 29544 kB VmRSS: 8900 kB RssAnon: 4164 kB RssFile: 3968 kB RssShmem: 768 kB VmData: 6836 kB VmStk: 132 kB VmExe: 4408 kB VmLib: 10736 kB VmPTE: 120 kB VmSwap: 0 kB HugetlbPages: 0 kB CoreDumping: 1 THP_enabled: 1 Threads: 1 SigQ: 0/63751 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000000300000 SigCgt: 0000000000000000 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: 000001ffffffffff CapAmb: 0000000000000000 NoNewPrivs: 0 Seccomp: 0 Seccomp_filters: 0 Speculation_Store_Bypass: thread vulnerable SpeculationIndirectBranch: conditional enabled Cpus_allowed: ff Cpus_allowed_list: 0-7 Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001 Mems_allowed_list: 0 voluntary_ctxt_switches: 63 nonvoluntary_ctxt_switches: 10
COREDUMP_RLIMIT
18446744073709551615
COREDUMP_ROOT
/
COREDUMP_SIGNAL
11
COREDUMP_SIGNAL_NAME
SIGSEGV
COREDUMP_SLICE
system.slice
COREDUMP_TIMESTAMP
1684213427000000
COREDUMP_UID
1001
COREDUMP_UNIT
ampinstmgr.service
MESSAGE_ID
fc2e22bc6ee647b6b90729ab34a250b1
PRIORITY
2
SYSLOG_IDENTIFIER
systemd-coredump
_BOOT_ID
a0d002609fcd447abcb19f772b3cc29a
_CAP_EFFECTIVE
0
_CMDLINE
/usr/lib/systemd/systemd-coredump
_COMM
systemd-coredum
_EXE
/usr/lib/systemd/systemd-coredump
_GID
1001
_MACHINE_ID
d22d412af0ba41dd922fbfcb3bfc68fa
_PID
1948
_SELINUX_CONTEXT
system_u:system_r:systemd_coredump_t:s0
_SOURCE_REALTIME_TIMESTAMP
1684213427519332
_SYSTEMD_CGROUP
/system.slice/system-systemd\x2dcoredump.slice/systemd-coredump@0-1947-0.service
_SYSTEMD_INVOCATION_ID
16e55225c9dd4c169a4eee7f2661a3a7
_SYSTEMD_SLICE
system-systemd\x2dcoredump.slice
_SYSTEMD_UNIT
systemd-coredump@0-1947-0.service
_TRANSPORT
journal
_UID
1001
__CURSOR
s=dc6298c3f36f4e9f9a032f43ecc0e300;i=26f01;b=a0d002609fcd447abcb19f772b3cc29a;m=5883eb;t=5fbc87f12bf73;x=3f3b7f6028346b6c
__MONOTONIC_TIMESTAMP
5800939
__REALTIME_TIMESTAMP
1684213427519347

Show what you see when you type ampinstmgr status

Instance Name      │ Friendly Name    │ Module     │ IP              │ Port  │ Up
───────────────────┼──────────────────┼────────────┼─────────────────┼───────┼───
ADS01              │ ADS              │ ADS        │ 0.0.0.0         │ 8080  │
Instance1 │ Instance1  │ Minecraft  │ 0.0.0.0         │ 8081  │
 Instance2 │ Instance2    │ Minecraft  │ 0.0.0.0         │ 8082  │

Can you include the bits above that as well? Trying to get a full picture of what you’re dealing with.

$ ampinstmgr status
[Info] AMP Instance Manager v2.4.3.6 built 07/05/2023 12:05
[Info] Release spec: Release - built by CUBECODERS/buildbot on CCL-DEV
Instance Name      │ Friendly Name    │ Module     │ IP              │ Port  │ Up
───────────────────┼──────────────────┼────────────┼─────────────────┼───────┼───
ADS01              │ ADS              │ ADS        │ 0.0.0.0         │ 8080  │
Instance1 │ Instance1  │ Minecraft  │ 0.0.0.0         │ 8081  │
 Instance2 │ Instance2    │ Minecraft  │ 0.0.0.0         │ 8082  │

Just to add – I updated all instances to the new AMP release 2.4.4.0. Issue persists with the same type of log dumped from the original post.

  • Rebooted
  • Attempted to manually boot instances

Failed to start AMP. Exit code was 0

This appears to be related to Fedora 37, still unsure if it’s an AMP issue or something else within that OS causing the issue. Temporarily the user is switched to the LTS version that was still functioning, but that lacks the needed S3 support. Since this isn’t an enterprise supported OS, it was discussed in Discord to move to Debian or Ubuntu. Closing since it’s “resolved” for this user, but there still may be an issue running AMP on Fedora currently.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.