When servers crashes unexpectedly, it doesn't start back up

OS Name/Version: Windows Server 2019 Standard

Product Name/Version: AMP Release “Decadeus” (v2.4.6.6, built 05/10/2023 11:56)

┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Key                  ┃ Value                                     ┃
┣━━━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Operating System     ┃ Windows - Windows Server 2019 on x86_64   ┃
┃ Product              ┃ AMPService 'Decadeus' v2.4.6.6 (Mainline) ┃
┃ Virtualization       ┃ None                                      ┃
┃ Application          ┃ Application Deployment                    ┃
┃ Module               ┃ ADSModule                                 ┃
┃ Running in Container ┃ No                                        ┃
┃ Current State        ┃ Indeterminate                             ┃
┗━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

Problem Description: We host a Ark: Survival Ascended server and when it crashes, the panel will does not start the server back up correctly. I can share more logs if need be.

I removed a large portion of the logs because it spams the same “Slow response” and “Slow method invocation” message and it’s too big to post in here.

[00:00:00] [Logger:jack404 Info]  : Rollover from previous log file: AMP_Logs\AMPLOG 2023-11-10 00-00-00.log
[00:00:00] [FileBackup Activity]  : Creating Backup: Scheduled Backup
[01:43:00] [Core Info]            : Checking for AMP updates...
[01:43:00] [Core Info]            : AMP is up to date.
[07:43:00] [Core Info]            : Checking for AMP updates...
[07:43:00] [Core Info]            : AMP is up to date.
[08:25:27] [Generic:jack404 Warning] : The application stopped unexpectedly. Exit code 3
[08:25:27] [Generic:jack404 Warning] : Application failed to start 0 times, not restarting.
[10:01:11] [Core Activity]        : Authentication attempt for user jack404 from 127.0.0.1
[10:01:13] [Core Activity]        : Authentication success
[10:01:26] [API:jack404 Activity] : Updating the application.
CPID: 8064
Redirecting stderr to 'C:\AMPDatastore\Instances\Gr8Server01\ark-sa\logs\stderr.txt'
Logging directory: 'C:\AMPDatastore\Instances\Gr8Server01\ark-sa/logs'
[  0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation - version 1698778122
-- type 'quit' to exit --
Loading Steam API...OK
force_install_dir 2430930
Connecting anonymously to Steam Public...OK
Waiting for client config...OK
Waiting for user info...OK
app_update 2430930 validate
 Update state (0x5) verifying install, progress: 0.00 (0 / 9498536863)
 Update state (0x5) verifying install, progress: 0.49 (46584384 / 9498536863)
 Update state (0x5) verifying install, progress: 2.47 (234422189 / 9498536863)
 Update state (0x5) verifying install, progress: 10.07 (956701341 / 9498536863)
 Update state (0x5) verifying install, progress: 19.81 (1881545373 / 9498536863)
 Update state (0x5) verifying install, progress: 25.01 (2375245025 / 9498536863)
 Update state (0x5) verifying install, progress: 34.32 (3260087665 / 9498536863)
 Update state (0x5) verifying install, progress: 44.02 (4181101285 / 9498536863)
 Update state (0x5) verifying install, progress: 53.71 (5101751013 / 9498536863)
 Update state (0x5) verifying install, progress: 63.37 (6019596432 / 9498536863)
 Update state (0x5) verifying install, progress: 73.09 (6942343312 / 9498536863)
 Update state (0x5) verifying install, progress: 82.59 (7844725357 / 9498536863)
 Update state (0x5) verifying install, progress: 92.34 (8771295869 / 9498536863)
 Update state (0x0) unknown, progress: 0.00 (0 / 0)
Success! App '2430930' fully installed.
quit
[10:01:57] [steamcmdplugin:jack404 Info] : SteamCMD Update successful
[10:01:57] [Generic:jack404 Info] : Skipping download - target already exists (C:\AMPDatastore\Instances\Gr8Server01\ark-sa\2430930\ShooterGame/Saved/Config/WindowsServer/Game.ini)
[10:01:57] [Generic:jack404 Info] : Skipping download - target already exists (C:\AMPDatastore\Instances\Gr8Server01\ark-sa\2430930\ShooterGame/Saved/Config/WindowsServer/GameUserSettings.ini)
[10:01:57] [Generic:jack404 Info] : Skipping download - target already exists (C:\AMPDatastore\Instances\Gr8Server01\ark-sa\2430930\.backupExclude)
[10:01:57] [Generic:jack404 Info] : Skipping download - target already exists (C:\AMPDatastore\Instances\Gr8Server01\ark-sa\2430930\ShooterGame\.backupExclude)
[10:01:57] [Generic:jack404 Info] : Merging config file ./ark-sa/2430930/ShooterGame/Saved/Config/WindowsServer/GameUserSettings.ini (INI format) using Unicode (UTF-8) encoding.
[10:01:57] [Generic:jack404 Info] : Merging config file ./ark-sa/2430930/ShooterGame/Saved/Config/WindowsServer/Game.ini (INI format) using Unicode (UTF-8) encoding.
[10:02:02] [API:jack404 Activity] : Starting the application.
[10:02:02] [Generic:jack404 Info] : Merging config file ./ark-sa/2430930/ShooterGame/Saved/Config/WindowsServer/GameUserSettings.ini (INI format) using Unicode (UTF-8) encoding.
[10:02:02] [Generic:jack404 Info] : Merging config file ./ark-sa/2430930/ShooterGame/Saved/Config/WindowsServer/Game.ini (INI format) using Unicode (UTF-8) encoding.
[13:43:00] [Core Info]            : Checking for AMP updates...
[13:43:00] [Core Info]            : AMP is up to date.
[14:56:09] [Core:jack404 Warning] : Slow method invocation: GetUpdates took 2002ms to complete.
[14:56:09] [Core:jack404 Warning] : Slow response: Core.GetUpdates took 2002ms to complete.
[14:56:09] [Core:jack404 Warning] : Slow method invocation: GetUpdates took 2018ms to complete.
[14:56:09] [Core:jack404 Warning] : Slow response: Core.GetUpdates took 2018ms to complete.
[14:56:09] [Core:jack404 Warning] : Slow method invocation: GetUpdates took 2034ms to complete.
[14:56:09] [Core:jack404 Warning] : Slow response: Core.GetUpdates took 2035ms to complete.
[14:56:09] [Core:jack404 Warning] : Slow method invocation: GetUpdates took 2042ms to complete.
[14:58:45] [Core:jack404 Warning] : Slow response: Core.GetUpdates took 7935ms to complete.
[14:58:45] [Core:jack404 Warning] : Slow method invocation: GetUpdates took 7910ms to complete.
[14:58:45] [Core:jack404 Warning] : Slow response: Core.GetUpdates took 7911ms to complete.
[14:58:45] [Core:jack404 Warning] : Slow method invocation: GetUpdates took 7910ms to complete.
[14:58:45] [Core:jack404 Warning] : Slow response: Core.GetUpdates took 7910ms to complete.
[15:52:52] [Core Activity]        : Authentication attempt for user admin from 127.0.0.1
[15:52:53] [Core Activity]        : Authentication success
[15:53:10] [Core Activity]        : Authentication attempt for user admin from 127.0.0.1
[15:53:10] [Core Activity]        : Authentication success

Steps to reproduce: I have not attempted to reproduce it, just happens when the server crashes.

Actions taken to resolve so far: I setup an Event Trigger for when the application state changes, then set it to start the server back up. This was working for a small time, then it just caused issues with it being in a state of limbo.

Logs are indeed needed from the point the server stops and what AMP tries to do afterwards.

That was the full log for that day other than the 100+ lines of the same message that is already included in there. It just doesn’t fit into the full code block.

Here is the rest of the previous day, continuation of logs in first post:

CPID: 7584
Redirecting stderr to 'C:\AMPDatastore\Instances\Gr8Server01\ark-sa\logs\stderr.txt'
Logging directory: 'C:\AMPDatastore\Instances\Gr8Server01\ark-sa/logs'
[  0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation - version 1698778122
-- type 'quit' to exit --
Loading Steam API...OK
force_install_dir 2430930
Connecting anonymously to Steam Public...OK
Waiting for client config...OK
Waiting for user info...OK
app_update 2430930 validate
 Update state (0x5) verifying install, progress: 0.10 (9437184 / 9498536863)
 Update state (0x5) verifying install, progress: 5.41 (513958709 / 9498536863)
 Update state (0x5) verifying install, progress: 14.68 (1393957533 / 9498536863)
 Update state (0x5) verifying install, progress: 24.43 (2320374317 / 9498536863)
 Update state (0x5) verifying install, progress: 33.90 (3220241777 / 9498536863)
 Update state (0x5) verifying install, progress: 43.52 (4133915365 / 9498536863)
 Update state (0x5) verifying install, progress: 53.28 (5060856549 / 9498536863)
 Update state (0x5) verifying install, progress: 63.01 (5984993424 / 9498536863)
 Update state (0x5) verifying install, progress: 72.49 (6885720208 / 9498536863)
 Update state (0x5) verifying install, progress: 81.99 (7787905645 / 9498536863)
 Update state (0x5) verifying install, progress: 91.50 (8691370605 / 9498536863)
Success! App '2430930' fully installed.
quit
[22:05:20] [steamcmdplugin:admin Info] : SteamCMD Update successful
[22:05:20] [Generic:admin Info]   : Skipping download - target already exists (C:\AMPDatastore\Instances\Gr8Server01\ark-sa\2430930\ShooterGame/Saved/Config/WindowsServer/Game.ini)
[22:05:20] [Generic:admin Info]   : Skipping download - target already exists (C:\AMPDatastore\Instances\Gr8Server01\ark-sa\2430930\ShooterGame/Saved/Config/WindowsServer/GameUserSettings.ini)
[22:05:20] [Generic:admin Info]   : Skipping download - target already exists (C:\AMPDatastore\Instances\Gr8Server01\ark-sa\2430930\.backupExclude)
[22:05:20] [Generic:admin Info]   : Skipping download - target already exists (C:\AMPDatastore\Instances\Gr8Server01\ark-sa\2430930\ShooterGame\.backupExclude)
[22:05:20] [Generic:admin Info]   : Merging config file ./ark-sa/2430930/ShooterGame/Saved/Config/WindowsServer/GameUserSettings.ini (INI format) using Unicode (UTF-8) encoding.
[22:05:20] [Generic:admin Info]   : Merging config file ./ark-sa/2430930/ShooterGame/Saved/Config/WindowsServer/Game.ini (INI format) using Unicode (UTF-8) encoding.
[22:05:59] [API:admin Activity]   : Starting the application.
[22:05:59] [Generic:admin Info]   : Merging config file ./ark-sa/2430930/ShooterGame/Saved/Config/WindowsServer/GameUserSettings.ini (INI format) using Unicode (UTF-8) encoding.
[22:05:59] [Generic:admin Info]   : Merging config file ./ark-sa/2430930/ShooterGame/Saved/Config/WindowsServer/Game.ini (INI format) using Unicode (UTF-8) encoding.
[22:07:55] [Core Activity]        : Authentication attempt for user admin from 127.0.0.1
[22:07:57] [Core Activity]        : Authentication success
[22:14:32] [API:admin Activity]   : Starting the application.
[22:14:32] [Generic:admin Info]   : Merging config file ./ark-sa/2430930/ShooterGame/Saved/Config/WindowsServer/GameUserSettings.ini (INI format) using Unicode (UTF-8) encoding.
[22:14:32] [Generic:admin Info]   : Merging config file ./ark-sa/2430930/ShooterGame/Saved/Config/WindowsServer/Game.ini (INI format) using Unicode (UTF-8) encoding.

Recent crash from this morning.

[00:00:00] [Logger:jack404 Info]  : Rollover from previous log file: AMP_Logs\AMPLOG 2023-11-11 00-00-00.log
[01:43:00] [Core Info]            : Checking for AMP updates...
[01:43:00] [Core Info]            : AMP is up to date.
[07:43:00] [Core Info]            : Checking for AMP updates...
[07:43:00] [Core Info]            : AMP is up to date.
[13:35:45] [Generic:admin Warning] : The application stopped unexpectedly. Exit code 3
[13:35:45] [Generic:admin Warning] : Application failed to start 0 times, not restarting.
[13:43:00] [Core Info]            : Checking for AMP updates...
[13:43:00] [Core Info]            : AMP is up to date.
[14:12:00] [Core Activity]        : Authentication attempt for user admin from 127.0.0.1
[14:12:02] [Core Activity]        : Authentication success

How much RAM does the host have?

OS Windows
Platform Windows Server 2019
System Type x86_64
CPU Model AMD Ryzen 5 3600X 6-Core Processor
CPU Layout 1S/6C/12T
Installed RAM 32660

So what’s happening is the server is crashing, AMP tries to restart it, it crashes again immediately, AMP detects this and doesn’t try again.

Given that you’re on Windows this is likely a memory issue, as in not having a large enough free block available after the first crash. This would explain why starting it manually later on is fine because everything has had a chance to tidy up.

Do you have any other game servers or memory heavy applications running?

We host The Isle and ASA on the machine, including AMP of course. That’s all we run though. The usage of both servers usually hover around 50-60% while idle, and 70-75% while they see some action. I’m just not understanding the not being able to boot back up part. Is there no way to set a timer to restart it after so many seconds? The other server, The Isle, remains up just fine even when the ASA server crashes. ASA servers crashing seems to be a common trend, even among the official servers.

You can set a schedule task every 10 minutes say to start the server - which won’t do anything if the server is already running.