Minecraft Instance fails restart. Failed to start AMP. Exit code was 0

Technical Support
,
1

OS Name/Version: Ubuntu

Product Name/Version: AMP v2.4.4

Problem Description: Minecraft server crashes - log shows that it cannot write to disk. Restarting instance causes failure with error “Failed to start AMP. Exit code was 0”.

Steps to reproduce:

Honestly, this has happened several times, and I have no idea how to reproduce it. The last time, I ended up rebuilding the entire AMP cluster because I could not pinpoint the issue.

Actions taken to resolve so far:

I have checked the Hypervisor Disk (300GB) and it is not full. I have checked the Datastore in AMP (90GB) and it is not full.

Looking at the AMPLOG, I see the a plugin timing out with a REST request and then the following:

2023-07-04 13:22:59,146 Log4j2-AsyncAppenderEventDispatcher-1-Async ERROR Unable to write to stream logs/latest.log for appender File org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to RandomAccessFile logs/latest.log

The Minecraft log has the following:

[13:21:04] [Server thread/INFO]: rlberry05 lost connection: Timed out
[13:21:04] [Server thread/INFO]: rlberry05 left the game
[13:22:00] [Server thread/INFO]: TPS from last 1m, 5m, 15m: 20.0, 20.0, 20.0
[13:22:59] [RegionFile I/O Thread #0/ERROR]: Failed to save chunk [36, -24]
java.io.IOException: No space left on device
	at sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method) ~[?:?]
	at sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:68) ~[?:?]
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:130) ~[?:?]
	at sun.nio.ch.IOUtil.write(IOUtil.java:97) ~[?:?]
	at sun.nio.ch.IOUtil.write(IOUtil.java:67) ~[?:?]
	at sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:878) ~[?:?]
	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:864) ~[?:?]
	at net.minecraft.world.level.chunk.storage.RegionFile.write(RegionFile.java:830) ~[paper-1.19.2.jar:git-Paper-307]
	at net.minecraft.world.level.chunk.storage.RegionFile$ChunkBuffer.close(RegionFile.java:1047) ~[paper-1.19.2.jar:git-Paper-307]
	at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:249) ~[?:?]
	at java.io.FilterOutputStream.close(FilterOutputStream.java:188) ~[?:?]
	at java.io.FilterOutputStream.close(FilterOutputStream.java:188) ~[?:?]
	at net.minecraft.world.level.chunk.storage.RegionFileStorage.write(RegionFileStorage.java:305) ~[?:?]
	at net.minecraft.world.level.chunk.storage.ChunkStorage.write(ChunkStorage.java:189) ~[?:?]
	at net.minecraft.server.level.ChunkMap.a(ChunkMap.java:899) ~[paper-1.19.2.jar:git-Paper-307]
	at net.minecraft.server.level.ServerLevel$1.writeData(ServerLevel.java:341) ~[?:?]
	at io.papermc.paper.chunk.system.io.RegionFileIOThread$ChunkDataTask.run(RegionFileIOThread.java:1286) ~[paper-1.19.2.jar:git-Paper-307]
	at ca.spottedleaf.concurrentutil.executor.standard.PrioritisedThreadedTaskQueue$PrioritisedTask.executeInternal(PrioritisedThreadedTaskQueue.java:351) ~[paper-1.19.2.jar:git-Paper-307]
	at ca.spottedleaf.concurrentutil.executor.standard.PrioritisedThreadedTaskQueue.executeTask(PrioritisedThreadedTaskQueue.java:11```
2

It’s telling you what’s wrong :slight_smile:

3

However, there is space left on the device. The hypervisor that runs AMP has 300GB of memory. 150 are allocated to AMP, of which 60GB has been used.
HV

The datastore that AMP places all instances on is 90GB. Looking at AMP, 27% of that 90GB has been used.

DD

There is enough space, I cannot figure out why AMP can’t see or use it. Because it is there.

4

First, check with df -h that there really is enough space as per the partitioning scheme. What ESXi shows you doesn’t take everything into account.

Second, the only other thing it could be is an excessive number of open files, in which case you need to find where that’s coming from or increase the open file limit.

5

df-h showed 3 dev/sda entries. 2 were small, the third large, but 100% consumed.

I was able to mount a Ubuntu desktop ISO and use GParted to look at the disk. The dev/sda3 that was reported 100% full showed 50% full using GParted.

GP
GP778×200 21 KB

None of this makes sense, as the entire AMP datastore is only ~20GB. I am expanding the dev/sda3 to fill the entire 148GB space available to it. Is there any configuration in AMP that needs to reflect the new space?

6

Nope. Have you got /home/ mounted on a different partition perchance? That’ll cause some problems.

7

Not that I can tell. I used a single partition to install Ubuntu and AMP.

8

Show me the actual output of df -h

Edit: Looks like you’re using LVM - Extra steps are required to properly expand LVM partitions. Check your distros documentation.

9

DFH below. I will see what Ubuntu has on LVM. df -h is still not showing properly.
DFH

10

Yeah you’ve not extended your LVM volume. Expanding it in GParted is not sufficient. You’ll need to consult your distros documentation for how you go about doing this.

11

Solution follows for anyone who needs it:

Ubuntu by default does not allocate all space and uses LVM as Mike suspected. GParted is not enough to expand the disk.
Steps to expand the disk: (NOTE: I am on VMWare and had already allocated more raw disk space to the Ubuntu server running AMP.)

  1. Check for free space with vgdisplay
  2. Check the logical volume size with lvdisplay
  3. Expand the logical volume to 100% of free space using lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv and run lvdisplay to verify the change.
  4. Run df -h to verify the root file system then resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv and finally df -h to verify that the space has indeed changed.
    Source: Ubuntu: Extend your default LVM space - Packet Pushers
12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.