Problem Description:
Around 60% into the backup, TPS will start to fluctuate from between normal ~20TPS and 1TPS - within 10 minutes this will lead toa full crash, and the server will not recover or restart.
This issue has only appeared in the last ~2 weeks, after I had an issue where my backup exclusions were being ignored (I’ve had to delete my .backupExclude files and re-add twice now in this time, backup keeps randomly ignoring them after no changes).
I am unable to get any Spark report at this time - as I’ve not been able to get a Spark report before shutdown. There are no performance issues during normal server operation that would suggest my hardware is the issue (i9-9900k & 64 GB DDR4, running on an NVME) my problem appears only when taking a backup.
Backup size is 150GB - 190GB
Steps to reproduce:
Step 1
Take a backup
Step 2
Wait 15-20 minutes
Step 3
1TPS / Console Spam about thread starvation & missed ticks…eventual death
Actions taken to resolve so far:
seeked help in CubeCoders discord
remade backup exclusions
excluded large log files
Firstly, to add some more detail:
I have now noticed that CPU usage in AMP spikes to 100+% as the TPS drops to 1 during a backup.
Normal usage is rarely above 15-35% and it doesn’t increase by much during a backup at any stage except when the entire server decides to implode
Secondly, to your point:
That would be 15-35+ minutes of downtime - which isn’t really doable for an active SMP, the map is 20kx20k so not huge by any means…as I said backups DID work, and while I do appreciate potential workarounds, what I am really looking for is progress towards the same functionality I had before…
When performing a large backup, a large amount of disk read and write load will be generated, which may compete with the read and write operations of the server during normal operation, especially when the backup is performed on the same disk. Minecraft’s main game loop is single-threaded, which means that any I/O delays (such as waiting for backup files to be loaded) will directly affect TPS, leading to performance degradation or even crashes.
Writing hot spares to different SSDs should reduce I/O contention, thereby mitigating TPS degradation and server crashes.
It is recommended to also consider using tools that support asynchronous backup and adjust the backup time.
Based on my experience with AMP, a compressed archive of the backup is created during the backup process. However, the compression process itself requires CPU resources. Even if I/O contention is reduced, a full CPU load may still indirectly impact the server.
