See this guide here:
https://discourse.cubecoders.com/t/setting-up-a-modded-java-minecraft-server/26535#p-50642-h-4-additional-java-information-43
Though most likely those are poorly written client-only mods that crash when loading them on the server, you should be able to remove them fine.