Startup Issue with Application Deployment - New SCP:SL instances crashing 100% of the time

System Information

Field Value
Operating System Linux - Debian GNU/Linux 12 on x86_64
Product AMP ‘Decadeus’ v2.4.6.8 (Mainline)
Virtualization QEMU_KVM
Application Application Deployment
Module ADSModule
Running in Container No
Current State Indeterminate

Problem Description

Issue

[08:39:00] [steamcmdplugin Info]  : SteamCMD Update successful
[08:39:00] [Logger Info]          : Merging config file ./scp-secret-laboratory/996560/AppData/config/7777/config_gameplay.txt (KVP format) using Unicode (UTF-8) encoding.
[08:39:00] [Core Error]           : ArgumentNullException
[08:39:00] [Logger Error]         : [0] (ArgumentNullException) : Value cannot be null.
Parameter name: input
[08:39:00] [Core Error]           :   at Text.RegularExpressions.Regex.Replace (String input, Text.RegularExpressions.MatchEvaluator evaluator) 
  at GenericModule.GenericModuleConfig+ApplicationConfig.SubstituteTemplates (String input, Collections.Generic.Dictionary`2[TKey,TValue] variables) 
  at GenericModule.GenericModuleConfig+ApplicationConfig.GetValue (String input) 
  at GenericModule.ModuleMain.MergeKVP (String finalPath, GenericModule.MetaConfigFile fileInfo) 
  at GenericModule.ModuleMain.MergeConfigFile (String finalPath, GenericModule.MetaConfigFile fileInfo) 
  at GenericModule.ModuleMain.WriteMetaconfigs () 
  at GenericModule.ModuleMain.SaveChanges (String key, String value) 
  at GenericModule.GenericApp.DoAppStartup (Boolean IsUpdating) 
  at GenericModule.GenericApp.Start () 
[08:39:00] [Logger Info]          : Merging config file ./scp-secret-laboratory/996560/AppData/config/7777/config_gameplay.txt (KVP format) using Unicode (UTF-8) encoding.
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Key                  ┃ Value                                 ┃
┣━━━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Operating System     ┃ Linux - Debian GNU/Linux 12 on x86_64 ┃
┃ Product              ┃ AMP 'Decadeus' v2.4.6.8 (Mainline)    ┃
┃ Virtualization       ┃ QEMU_KVM                              ┃
┃ Application          ┃ Application Deployment                ┃
┃ Module               ┃ ADSModule                             ┃
┃ Running in Container ┃ No                                    ┃
┃ Current State        ┃ Indeterminate                         ┃
┗━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

Reproduction Steps

  • Create new SCP: Secret Laboratory Instance

you need to run it in docker on amp for it to work. go to edit settings on your scp instance and then select Run in Docker Container

Not actually true.

The error in this case is something different. OP, can you share your config_gameplay.txt?

During all my issue recreation runs, the instance was already set to Run in Docker Container using the default image cubecoders/ampbase:mono

I suspect the issue is with the template grabbing your public IP to put into the config file

What’s ampinstmgr version show?

➜  ~ ampinstmgr version
[Info] AMP Instance Manager v2.4.6.6 built 05/10/2023 11:57
[Info] Stream: Mainline / Release - built by CUBECODERS/buildbot on CCL-DEV

Additionally, here are the contents of the default generated SCP:SL configuration file below.

File: ./scp-secret-laboratory/996560/AppData/config/7777/config_gameplay.txt

Contents:

#Server main settings
server_name: My Server Name
#default - uses server_name
player_list_title: default
player_list_title_rate: default
serverinfo_pastebin_id: 7wV681fT
server_ip: auto
max_players: 20
server_tickrate: 60
use_reserved_slots: true
lobby_waiting_time: default
use_native_sockets: true
ipv4_bind_ip: 0.0.0.0
ipv6_bind_ip: ::
contact_email: default

#Spawn settings
minimum_MTF_time_to_spawn: 280
maximum_MTF_time_to_spawn: 350
maximum_MTF_respawn_amount: 15
maximum_CI_respawn_amount: 15
priority_mtf_respawn: true
use_crypto_rng: false
team_respawn_queue: default
late_join_time: 0

## Player Info Range ##
# This controls the default range that the player information text appears on clients.
player_info_range: 10 # Default is 10.

## Respawn ticket system settings ##

# Enables the respawn tickets system.
# Each NTF or CI player spawned in consumes one ticket.
# We recommend keeping this on for game balance!
#
# Warning: Even if you disable the system, the tickets ratio for MTF/CI will still determine their spawn chance. 
respawn_tickets_enable: true

# Ticket counts (initial)
# The ratio of MTF-to-CI tickets directly determines which team has a higher chance of spawning.
# The initial settings (24:18) give around 42.8% chance for the CI to spawn instead of the MTF.
# The CI spawn chance can be calculated by this formula: CI_TICKETS / (MTF_TICKETS + CI_TICKETS) * 100%
respawn_tickets_mtf_initial_count: default
respawn_tickets_ci_initial_count: default

# Ticket counts (events)
respawn_tickets_mtf_classd_cuffed_escape_count: default
respawn_tickets_mtf_scientist_escape_count: default
respawn_tickets_mtf_scp_hurt_escape_count: default
respawn_tickets_ci_classd_escape_count: default
respawn_tickets_ci_scientist_cuffed_escape_count: default
respawn_tickets_ci_scientist_died_count: default
respawn_tickets_ci_scp_item_count: default

# Ticket misc settings
# The percentage of health an SCP must lose to trigger the 'scp_hurt' tickets.
respawn_tickets_mtf_scp_hurt_interval: default

## Stamina system settings ##

# The percentage of stamina used per second while sprinting.
# 0.05 = 5%, 0.33 = 33%, etc.
# NOTE: Stamina is currently only utilized on humans.
# 0 to disable, but this *will* cause weirdness with status effects
# and other mechanics, so we recommend keeping it on!
stamina_balance_use: 0.05

# The time (in seconds) that newly-spawned players can sprint
# without consuming stamina. Used to reduce visual artifacts.
# We recommend to put it at least 2 seconds
stamina_balance_immunity: 3

# The seconds that must pass before stamina starts regenerating after
the player stops sprinting.
# Lower values are more liekly to be exploited by players
# Higher higher can be annoying to players.
stamina_balance_regen_cd: 1

# How fast the stamina regenerates.
# This is a multiplier for the default setting,
# which is based on a curve - not a specific value.
stamina_balance_regen_speed: 1

#Random-stuff settings (-1: random)
server_forced_class: -1
map_seed: -1

#Misc gameplay settings
intercom_cooldown: 120
intercom_max_speech_time: 20
auto_round_restart_time: 10
friendly_fire: false
friendly_fire_multiplier: 0.4
warhead_tminus_start_duration: 90
lock_gates_on_countdown: true
isolate_zones_on_countdown: false
open_doors_on_countdown: true
keep_items_after_escaping: true
allow_playing_as_tutorial: true
disable_decontamination: false
096_destroy_locked_doors: true
no_holidays: false
allow_disarmed_interaction: false
914_mode: default
sinkhole_slow_amount: 30
sinkhole_spawn_chance: 0
disconnect_drop: true
end_round_on_one_player: false
cuffed_escapee_change_team: true
pd_exit_count: 2

#How many seconds someone is AFK sitting in their spawn location before they are kicked. Set to 0 or lower to disable. (Due to the 079 intro screen 079 has 13 extra seconds to be AFK.)
afk_time: 90
constantly_check_afk: false
afk_kick_message: AFK

#Spawn Protect settings
spawn_protect_enabled: default
spawn_protect_time: default
spawn_protect_can_shoot: default
spawn_protect_prevent_all: default
spawn_protect_team: [1, 2]

#Auto Event settings
auto_warhead_start_minutes: 0
auto_warhead_lock: true
auto_warhead_broadcast_enabled: true
auto_warhead_broadcast_message: Alpha Warhead is being automatically detonated and cannot be cancelled.
auto_warhead_broadcast_time: 10
auto_warhead_detonate_broadcast: Automatic Alpha Warhead has been detonated.
auto_warhead_detonate_broadcast_time: 10
auto_decon_broadcast_enabled: false
auto_decon_broadcast_message: Light Containment Zone is now decontaminated
auto_decon_broadcast_time: 10

#Security & Anticheat (default settings are good enough in most cases)
#Please type !private in your server console, if your server is verified, but you want to keep it hidden from the list. Type !public to make it public again.
online_mode: true
authentication_timeout: 45
ip_banning: true
enable_whitelist: false
forward_ports: true
enable_query: false
query_port_shift: 0
query_use_IPv6: true
administrator_query_password: none
connections_delay_time: 5
enable_sync_command_binding: false
ratelimit_kick: true
same_account_joining: false
anticheat_console_output: false
enable_fast_round_restart: false
fast_round_restart_delay: 3.2
display_preauth_logs: true
rejection_suppression_threshold: 60
challenge_issuance_suppression_threshold: 50

#Enables challenge during preauthentication
preauth_challenge: true

#Reply - prevents from flooding server with preauth requests from spoofed IP addresses
#MD5 or SHA1 - prevents from flooding with preauth requests from legitimate IP addresses as well. Requires additional time to join the server.
preauth_challenge_mode: reply
preauth_challenge_base_length: 10
preauth_challenge_time_window: 12
preauth_challenge_clean_period: 4

#Only for MD5 and SHA1 challenges (higher value = longer time required to join the server)
preauth_challenge_secret_length: 2

#This enforces the player to be using same IP address to connect to the game and authenticate.
#Enabling this blocks proxies. Not enforced if player joined from IPv4 and authenticated from IPv6 or viceversa.
#Situations with combinating IPv4 and IPv6 addresses are being checked on central servers after authentication (only for servers on public list).
enforce_same_ip: true
no_enforcement_for_local_ip_addresses: true

#Connections ratelimiting
#Ratelimit time windows define how often user with the same IP/UserID can initialize new connection (once per X seconds).
enable_ip_ratelimit: true
enable_userid_ratelimit: true
ip_ratelimit_window: 3
userid_ratelimit_window: 5

# - Item category limits -
# Values of 0 are NOT unlimited and will prevent item pickups of that type entirely!
# The inventory can hold a max of 8 items, so a limit of 8 is effectively unlimited.
limit_category_grenade: default
limit_category_keycard: default
limit_category_medical: default
limit_category_scpitem: default
limit_category_firearm: default

# - Ammo type limits -
# Values from 1 to 65k.
limit_ammo12ga: default
limit_ammo556x45: default
limit_ammo44cal: default
limit_ammo762x39: default
limit_ammo9x19: default

#Friendly fire punishment
#Action is performed if someone either kills enough teammates OR deals enough damage to teammates.
#Set kills or damage to 0 to disable that threshold.
#Detectors priority: Respawn (kill), Window (kill), Life (kill), Round (kill), Respawn (damage), Window (damage), Life (damage), Round (damage)
#Actions: kill, kick, ban, noop (no operation - do nothing, eg. to enable logging)
#Damage dealt AFTER round end is ignored.

ff_detector_global_broadcast_seconds: 5
ff_detector_global_adminchat_seconds: 6

#If enabled Class D personnel can damage or kill other Class D personnel without being punished for it
ff_detector_classD_can_damage_classD: false

#If set to "none" then the webhook for cheaters reporting will be used
ff_detector_webhook_url: none


#Per round
ff_detector_round_enabled: false
ff_detector_round_kills: 6
ff_detector_round_damage: 500

ff_detector_round_action: ban
ff_detector_round_ban_time: 24h
ff_detector_round_bankick_reason: You have been automatically banned for teamkilling.
ff_detector_round_kill_reason: You have been automatically killed for teamkilling.

ff_detector_round_adminchat_enable: false
ff_detector_round_adminchat_message: %nick has been banned for teamkilling (round detector).

ff_detector_round_broadcast_enable: true
ff_detector_round_broadcast_message: %nick has been automatically banned for teamkilling.

ff_detector_round_webhook_report: true


#Per life (resets on respawn)
ff_detector_life_enabled: false
ff_detector_life_kills: 4
ff_detector_life_damage: 300

ff_detector_life_action: ban
ff_detector_life_ban_time: 24h
ff_detector_life_bankick_reason: You have been automatically banned for teamkilling.
ff_detector_life_kill_reason: You have been automatically killed for teamkilling.

ff_detector_life_adminchat_enable: false
ff_detector_life_adminchat_message: %nick has been banned for teamkilling (life detector).

ff_detector_life_broadcast_enable: true
ff_detector_life_broadcast_message: %nick has been automatically banned for teamkilling.

ff_detector_life_webhook_report: true


#In a specified time window
ff_detector_window_enabled: false
ff_detector_window_seconds: 180
ff_detector_window_kills: 3
ff_detector_window_damage: 250

ff_detector_window_action: ban
ff_detector_window_ban_time: 16h
ff_detector_window_bankick_reason: You have been automatically banned for teamkilling.
ff_detector_window_kill_reason: You have been automatically killed for teamkilling.

ff_detector_window_adminchat_enable: false
ff_detector_window_adminchat_message: %nick has been banned for teamkilling (window detector).

ff_detector_window_broadcast_enable: true
ff_detector_window_broadcast_message: %nick has been automatically banned for teamkilling.

ff_detector_window_webhook_report: true


#In a specified time window AFTER RESPAWN
ff_detector_spawn_enabled: true
ff_detector_spawn_window_seconds: 120
ff_detector_spawn_kills: 2
ff_detector_spawn_damage: 180

ff_detector_spawn_action: ban
ff_detector_spawn_ban_time: 48h
ff_detector_spawn_bankick_reason: You have been automatically banned for teamkilling.
ff_detector_spawn_kill_reason: You have been automatically killed for teamkilling.

ff_detector_spawn_adminchat_enable: false
ff_detector_spawn_adminchat_message: %nick has been banned for teamkilling (spawn detector).

ff_detector_spawn_broadcast_enable: true
ff_detector_spawn_broadcast_message: %nick has been automatically banned for teamkilling.

ff_detector_spawn_webhook_report: true


#Kills made with explosions (eg. grenades) after DISCONNECTING from a server (eg. throwing a grenade and immediately closing the game)
ff_detector_explosion_after_disconnecting_enabled: true

ff_detector_explosion_after_disconnecting_action: ban
ff_detector_explosion_after_disconnecting_ban_time: 48h
ff_detector_explosion_after_disconnecting_bankick_reason: You have been automatically banned for teamkilling.

ff_detector_explosion_after_disconnecting_adminchat_enable: false
ff_detector_explosion_after_disconnecting_adminchat_message: %nick has been banned for teamkilling (explosion after disconnecting detector).

ff_detector_explosion_after_disconnecting_broadcast_enable: true
ff_detector_explosion_after_disconnecting_broadcast_message: %nick has been automatically banned for teamkilling.

ff_detector_explosion_after_disconnecting_webhook_report: true


#Enable this if and ONLY if you use plugin or modification that provides custom whitelist.
#This is only to mark the server on the public list as using a whitelist.
#More information can be found in the Verified Server Rules.
#Ignore this config key if your server is not verified.
custom_whitelist: false

#Enable this if and ONLY if you use plugin or modification that restricts access to the server (other than whitelist, eg. password).
#This is only to mark the server on the public list as using access restriction.
#More information can be found in the Verified Server Rules.
#Ignore this config key if your server is not verified.
server_access_restriction: false

#Enable this if and ONLY if your server is a Transparently Modded Server according to the definition that can be found in the Verified Server Rules.
#Ignore this config key if your server is not verified.
transparently_modded_server: false

#port_queue is for NONDEDICATED SERVERS ONLY
port_queue:
 - 7777
 - 7778
 - 7779
 - 7780
 - 7781
 - 7782
 - 7783
 - 7784

#Enabling this will automatically ban IP of players on the server that receives a Global Ban by a Global Moderator. (Defaults to false)
gban_ban_ip: default

ban_nickname_maxlength: default
ban_nickname_trimunicode: default

#Nickname filtering, using regex
nickname_filter: default
nickname_filter_replacement: default

#Administrative actions broadcast options
broadcast_kicks: false
broadcast_kick_text: %nick% has been kicked from this server.
broadcast_kick_duration: 5
broadcast_bans: true
broadcast_ban_text: %nick% has been banned from this server.
broadcast_ban_duration: 5

#Server idle mode
idle_mode_enabled: true
idle_mode_time: 5000
idle_mode_preauth_time: 30000
idle_mode_tickrate: 1

#Player report
report_send_using_discord_webhook: false
report_discord_webhook_url: PleaseSetWebhookUrlHere
report_username: Cheater Report
report_avatar_url: default
report_color: 14423100
report_server_name: My SCP:SL Server
report_header: Player Report
report_content: Player has just been reported.

#Restart options
#Requires LocalAdmin or a fully compatible tool.

#You can set autorestart of the server after a specified amount of rounds (0 - disabled).
restart_after_rounds: 0

#Amount of time after client should rejoin after a full server restart.
#Adjust depending on your server restart time.
full_restart_rejoin_time: 25


#Geoblocking
#If your server is on the public list, please refer to Verified Server Rules for more details.
#Modes: none, whitelist, blacklist
geoblocking_mode: none

#If enabled, players on the whitelist are able to ignore geoblocking.
geoblocking_ignore_whitelisted: true

#ISO country codes, eg. PL, US, DE
geoblocking_whitelist:
 - AA
 - AB
 - AC

geoblocking_blacklist:
 - AA
 - AB
 - AC

Try updating ampinstmgr via apt

I have updated ampinstmgr from v2.4.6.6 to v2.4.6.8

➜  ~ ampinstmgr version
[Info] AMP Instance Manager v2.4.6.8 built 23/11/2023 12:29
[Info] Stream: Mainline / Release - built by CUBECODERS/buildbot on CCL-DEV

I created a new SCP:SL instance and content-wise the templated configuration file seems to be the same:

➜  Downloads md5sum old_7777.conf new_7777.conf
52dad90bdc36c64834c8bdf891cc0d2d  old_7777.conf
52dad90bdc36c64834c8bdf891cc0d2d  new_7777.conf

That’s simply because it is the default file. AMP is not writing to it because it is throwing an error.

There used to be a bug that meant the AMP mono docker image interfered with AMP getting the public IP over https. That was addressed many weeks ago tho.

And you seem to be having the same issue outside docker.

Can you run docker pull cubecoders/ampbase:mono as the amp user to ensure you have the latest image, and then try again running the instance in docker?

Thanks for the explanation, that makes sense.

The SCP:SL instance is already running the latest cubecoders/ampbase:mono (e373e646333631e2e6178af95ebc99a28a6b70171ab2a727d6afc9693ab7048f) image available.

amp@amp:~$ docker image ls
REPOSITORY                             TAG       IMAGE ID       CREATED        SIZE
cubecoders/ampbase                     mono      9049f324b509   2 months ago   1.21GB

amp@amp:~$ docker pull cubecoders/ampbase:mono
mono: Pulling from cubecoders/ampbase
Digest: sha256:e373e646333631e2e6178af95ebc99a28a6b70171ab2a727d6afc9693ab7048f
Status: Image is up to date for cubecoders/ampbase:mono
docker.io/cubecoders/ampbase:mono

amp@amp:~$ docker ps
CONTAINER ID   IMAGE                     COMMAND                  CREATED         STATUS         PORTS     NAMES
e6b8eeb77351   cubecoders/ampbase:mono   "/ampstart.sh +Core.…"   3 minutes ago   Up 3 minutes             AMP_SCP0101

I’m out of ideas then. I can spin up an instance both in and outside Docker, and it is fine. I don’t have visibility on exactly how the AMP code grabs the external IP so can’t troubleshoot further. But perhaps you have some sort of network issue

In regards to networking, the server is in a DMZ without the ability to accurately obtain its own public IP as the outbound NAT is different from the inbound NAT.

Now all is revealed lol

You can edit configmanifest.json to take out the first setting block but it will be put back on any further refresh of the template config

AMP should though have better error handling

Thanks! I didn’t realize the WAN networking outside of the server was relevant to the local game server crashing otherwise I would have mentioned it earlier :sweat_smile:

Where can I find configmanifest.json for SCP: Secret Laboratory and how can I make it survive software updates? Is the issue I’m facing right now a bug or is it more of a limitation of the AMP-generated SCP: Secret Laboratory template?

The configmanifest.json is in the instance datastore. Access through the main file manager. Stop the instance first.

The SCP:SL server has a server_ip field in the config, used to report the server’s public IP to the master server. It is by default set to auto, meaning the server itself will try to determine the IP. But often that doesn’t work - eg it selects an IPv6 address that can’t be used with the master server. So I set up the template so that AMP automatically supplies the IPv4 public IP, bypassing all these issues.

Until you come along…

I could have it as a manually configurable field but that is more prone to user error.

Thank you! I was able to get past the configuration generation issue by setting the IP manually to 0.0.0.0. I do wonder if this field has to match the public IP or if 0.0.0.0 will suffice instead of auto.

If this is a required field, my suggestion/feature request is to optionally be able to set a DNS record per instance which could reference a record updated by DDNS for edge cases like mine where the server is not directly exposed on the Internet and is behind network devices performing different Inbound/Outbound NAT.

Pretty sure 0.0.0.0 won’t cut it

You are correct - I just saw this in the console Could not update data on server list (legacy)- Blacklisted IP address (bind all address)

DDNS support in the future would be awesome as the modified template may not survive an update :slight_smile:

Understand the request, but don’t hold your breath. xD

I had to at least ask :smile:

Thanks for the help!