Gmod Source RCON - 'bad password'

Bryantdl7 · 10 November 2023 13:24

Note - If you do not fill in every section below, your post won’t be answered - you must provide the steps you have followed so far and the actions you’ve already taken. Make sure to remove this notice from your post too.

OS Name/Version:
Windows 10

Product Name/Version: (Always use the full version number - not ‘Latest’)
AMP Release “Decadeus”

v2.4.6.6, built 05/10/2023 11:56
Problem Description:
Per my discussion with Mike on discord last night, here’s my logs from the most recent interaction where RCON breaks. I have left the detailed logging on from last time we troubleshooted this, and I am not seeing a password mentioned in the logs, it seems like it isn’t storing the password at all by the line break…?

[13:04:00] [Core Debug]           : Login request from 127.0.0.1 for bryantdl7
[13:04:00] [Core Activity]        : Authentication attempt for user bryantdl7 from 127.0.0.1
[13:04:00] [Core Debug]           : Logging in via remote auth server http://localhost:8080/...
[13:04:02] [Core Activity]        : Authentication success
[13:04:02] [API Debug]            : Ending session 698682c5-2847-4cfd-b892-798557aaaf48 (bryantdl7 at 127.0.0.1, API Login) - No activity for 5 minutes.
[13:04:02] [API Debug]            : Ending session 37298317-2a22-4c32-aacb-727cfacf3d74 (bryantdl7 at 127.0.0.1, API Login) - No activity for 5 minutes.
[13:04:02] [API Debug]            : Creating new session for bryantdl7 at 127.0.0.1 (AMP_ADS_REDIR/10) API Login
[13:04:02] [API Debug]            : Login success from 127.0.0.1 assigned session ID efc95043-688f-4cf9-8508-fbeaf91a0d6e
[13:04:02] [Core Warning]         : Slow method invocation: Login took 2458ms to complete.
[13:04:02] [Core Warning]         : Slow response: Core.Login took 2458ms to complete.
[13:04:12] [API:bryantdl7 Activity] : Sent to console: `status`
rcon from "10.136.143.5:60439": Bad Password

[13:04:17] [API:bryantdl7 Activity] : Sent to console: `status`pa
rcon from "10.136.143.5:60439": Bad Password

[13:10:00] [Scheduler Debug]      : Scheduler timer will start in 59496ms.

Here’s a log from yesterday when the problem started happening:

[21:49:02] [Core Debug]           : Login request from 127.0.0.1 for bryantdl7
[21:49:02] [Core Activity]        : Authentication attempt for user bryantdl7 from 127.0.0.1
[21:49:02] [Core Debug]           : Logging in via remote auth server http://localhost:8080/...
[21:49:02] [Core Activity]        : Authentication success
[21:49:02] [API Debug]            : Creating new session for bryantdl7 at 127.0.0.1 (AMP_ADS_REDIR/10) API Login
[21:49:02] [API Debug]            : Login success from 127.0.0.1 assigned session ID 698682c5-2847-4cfd-b892-798557aaaf48
11/09/2023 - 16:49:03: Player [1][pixel demon humanoid bunny thing] is arming a dupe, size: 11097
11/09/2023 - 16:49:03: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/xqm/cylinderx1huge.mdl
11/09/2023 - 16:49:03: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube1x4x1.mdl
11/09/2023 - 16:49:03: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube1x1x05.mdl
11/09/2023 - 16:49:03: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube05x1x025.mdl
11/09/2023 - 16:49:03: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube1x1x025.mdl
11/09/2023 - 16:49:03: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube4x6x2.mdl
11/09/2023 - 16:49:03: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube05x1x025.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube1x4x1.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube1x1x025.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/xqm/cylinderx1huge.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube1x4x1.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/xqm/cylinderx1big.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube1x4x1.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube1x1x025.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube1x4x1.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube4x4x2.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube1x4x1.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned vehicle models/nova/airboat_seat.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube1x4x1.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube1x1x025.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube2x2x2.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/xqm/cylinderx1big.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube1x1x05.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube1x2x05.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/xqm/cylinderx1huge.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube1x4x1.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/hunter/blocks/cube1x2x05.mdl
11/09/2023 - 16:49:04: [ULX] pixel demon humanoid bunny thing<STEAM_0:0:562159524> spawned model models/xqm/cylinderx1huge.mdl
[21:49:19] [API:bryantdl7 Activity] : Sent to console: `ulx uptime`
rcon from "10.136.143.5:60439": Bad Password

[21:58:00] [Scheduler Debug]      : Scheduler timer will start in 59491ms.

Steps to reproduce:

Step 1 - run the server for a significant amount of time without rebooting ADS / srcds instance
Step 2 - have many map changes and server crashes
Step 3 - boom problem begins after 1-7 days

Actions taken to resolve so far:
ampinstmgr.exe --stopall; ampinstmgr.exe --start ADS01 as a scheduled job (now once a week at least…!)

Mike · 10 November 2023 13:49

Need to know which exact game this is for. The password would only show up at the point the game server is first started (assuming it’s a command line flag) - otherwise it’ll be in the config file if applicable.

Bryantdl7 · 10 November 2023 14:13

Apologies - Garry’s mod

Looks like the rcon password is starred out in the logs, and no longer visible in my ampconfig.conf.

[10:04:27] [Core Debug]           : Arguments: -game garrysmod -console -usercon -ip 10.136.143.5 -port 27015 +rcon_password ********* +map gm_flatgrass +maxplayers 40 +hostname test +sv_pure 1 -disableluarefresh " " +exec autoexec.cfg -jitdisabled " " -disableluajit " " -pingboost 2 +log on -tickrate 66 -authkey ********** +host_workshop_collection 692212058 +gamemode sandbox +sv_setsteamaccount [sanitized by bryant] +logaddress_add 10.136.143.5:51704

Mike · 10 November 2023 14:13

That’s because you didn’t turn on the option to log sensitive arguments

Bryantdl7 · 10 November 2023 14:20

I have the following value in my ampconfig.conf (of the related gmod instance)

Security.LogSensitiveProcArgs=True

Is that the wrong value to change?

Mike · 10 November 2023 14:25

That is indeed the correct setting. You also need to ask AMP to produce startup scripts which will have all the information in them.

Bryantdl7 · 10 November 2023 14:37

I just set

Monitoring.ProduceStartupScripts=True

While the instance was fully turned off, booted it back up, and the rcon_password is still censored.

I ended up powering the server off inside the instance and clicking generate RCON password, which put the custom RCON password back into the srcds module kvp. It was not there prior to this, even though I know for a fact I clicked that button last month so there would be a password in there.

Really feels like something inside AMP is losing the rcon password and getting confused.

P.S. You can reproduce a similar problem by running the console command mp_loglevel without any arguments, that will immediately break the RCON tab, and also stop a server from properly rebooting or shutting down via AMP, it will instead permanently hang until you click kill. Maybe that will help you troubleshoot since that can be reproduced every time?

Mike · 10 November 2023 14:40

You need to find the scripts it generated and check those - They’ll be in the Instance directory.

Based on what you’re saying - I can be pretty certain that something is resetting the RCON password in a config file somewhere. Have you tried grepping the entire directory tree for anything that might reset it?

Bryantdl7 · 10 November 2023 14:55

If you let me know which subdirectory of the instance, or what file extension those end in, or even a wiki where I can see how to do this properly I’d be happy to poke around in those.

My server runs two executed configurations on startup. autoexec.cfg and server.cfg - those are the only two files I modify things in. Here’s the contents of both of those logs, with our prop and entity limits sanitized, as those have no pertinence to RCON.

These can easily be executed from one file e.g. autoexec or server.cfg, I just keep them separate because autoexec only runs at startup and server.cfg is on each map reload.

autoexec


gamemode "sandbox"
sv_setsteamaccount [redacted]
log 1

fps_max 0
changelevel gm_flatgrass


//lower console log outputs = less cpu usage
developer 0


// fix angular velocity? 
lua_run local tbl = physenv.GetPerformanceSettings() tbl.MaxAngularVelocity = 30000 physenv.SetPerformanceSettings(tbl)

lua_run local tbl = physenv.GetPerformanceSettings() tbl.MaxVelocity = 20000 physenv.SetPerformanceSettings(tbl)
map gm_flatgrass
host_workshop_collection "[redacted]"

server.cfg


log 1
echo =====================================
echo executing Darkstorm Build Server.cfg
echo =====================================



// Make sure the right people are banned
exec banned.cfg
exec banned_user.cfg
exec banned_ip.cfg
///////////////////////////////////////


// mount games!
exec "mount.cfg"
/////////////////////

//prevent server from hibernating
sv_hibernate_think 1
/////////////////////////////////


//////////////////////////////////
// Our name to the world
hostname "Darkstorm Build [#1] - Loads fast + new players welcome"
/////////////////////////////////


sv_location US


// fix for maps?
net_maxfilesize 300

//antihack measures, as i find them...
sv_allowcslua 0
sv_allowupload 0
sv_allowdownload 0
OrbitalStrikeAdminOnly 1



// Some stuff to tweak lag ... Possibly
sv_ya3dmg_smoke_enabled 0
acf_gunfire 1


//=================limits============================================================
sv_maxvelocity 9000000
sbox_noclip 1

sv_minrate 100000
sv_maxrate 0
sv_maxupdaterate 120
sv_minupdaterate 10
net_splitpacket_maxrate 100000
fps_max 0




// server stuff bryant added
wac_air_nodamage 0
sv_noclipspeed 10
spraymon_nooverspraying 0
sv_kickerrornum 0
sv_pausable 0
sv_lan 0
//sv_loadingurl "https://loading.darkstorm.cc/build/index.php?sid=%s"
sv_loadingurl "https://loading.darkstorm.cc/build-new-2/?steamid=%s&mapname=%m"
//sv_loadingurl "https://darkstorm.cc/?page_id=229?steamid=%s&mapname=%m22"
sv_alltalk 3
g_ragdoll_maxcount 0
wire_holograms_modelany 2
acf_ammomod 1
acf_setdefaultpermissionmode "build"
acf_setpermissionmode "build"
sv_manualweaponpickup_autodraw 1
wd_admin_mode 1




//==============================================================================================================







// General HL Cvars
mp_fraglimit 0
mp_footsteps 1
mp_flashlight 1
sv_contact "contact@darkstorm.cc"
decalfrequency 60
mp_decals 100
mp_falldamage 0
sv_cheats 0
sv_timeout 0

writeid


// Rcon Stuff
//rcon_password "nope not in use - this is a security vulnerability so that is why we keep it commented out"

//files
mapcyclefile mapcycle.txt

// disable lag compensation
//sv_unlag 1

heartbeat

// important settings 
sv_stats 0
sv_region 1


sv_password ""

tv_enable 0


//=======================2023 NETWORKING SETTINGS=============================



//finishing touches - makes sure we yell at GSLT to wake me up (cant wake up)

heartbeat

say =====================================
say Server Configuration Update Completed
say =====================================

echo =====================================
echo "ServerConfiguration Update Completed"
echo =====================================

Mike · 10 November 2023 15:05

They’re shell scripts, generated in the root of the instance directory.

Apollo - I’ve been helping this user with an RCON issue. After a while it stops working in AMP. I suspect the issue is that the rcon password is being reset somewhere between level loads in some unknown config file. Could you take a look please?

CubeCoders-Bot · 10 November 2023 15:05

Hey there, Bryant!

Sorry to hear about the RCON troubles. Taking a look at your server setup and config, it does seem like AMP is able to start the server with the RCON password you set originally, but it’s getting lost somewhere along the way.

Firstly, given the debug logs from AMP and the information you’ve provided, it’s clear the initial startup is correct and carries the appropriate RCON password. Your autoexec and server config files look fine, although I see you’ve previously commented out the RCON password line.

Here’s what I suggest we check step by step:

Step 1: Please make sure that there are no other config files or plugins that might be overwriting the RCON password during runtime or on map changes. The line you commented out in server.cfg is correct, as we prefer settings like RCON passwords to be handled outside such files for better security.

//rcon_password "nope not in use - this is a security vulnerability so that is why we keep it commented out"

Step 2: Check the shell scripts in the root of your instance directory that AMP generates on startup. Make sure the RCON password is set correctly there.

Step 3: Ensure that the Monitoring.ProduceStartupScripts setting remains True in your AMPConfig.conf to continuously check if it’s being overridden elsewhere.

Step 4: Let’s take a look at any other custom scripts that could possibly be impacting your server’s cfg files or any addon configurations that might interact with RCON settings.

If everything checks out and you’re sure there aren’t any external changes to the RCON password, I’d like to suggest:

Step 5: Restart your server and test the RCON connection immediately upon a fresh start, and then again after a map change, to see if the issue correlates with specific server events.

Would you mind starting with these steps and reporting back what you find? Let’s narrow down where the loss of the RCON password could be occurring.

Bryantdl7 · 10 November 2023 15:32

I was able to confirm the startup script is giving the desired RCON password.

I just was able to reproduce the issue exactly this time, and finally have concise steps! Furthermore, my external RCON client source admin tool is still working, while AMP is unable to connect. The RCON password has not changed.

Start a server after generating your rcon password
Confirm rcon connectivity within amp + source admin tool (or other RCON program)
Cause a server crashing event to occur (easiest repro is to kill the gmod server with task manager)
3A. When source throws an engine error it is the same type of program halt as clicking end process via task manager, so that’s why it is a good route to use for repro
Server will acknowledge it has stopped, reboot it, and show as back online
AMP now no longer is able to control the instance, but external RCON clients continue to work fine

Logs to back this up:

11/10/2023 - 10:22:33: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:34: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:34: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:34: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:34: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:34: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:34: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:34: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:34: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:34: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:34: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:34: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:34: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:35: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:35: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:35: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:35: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:35: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:35: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:35: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
11/10/2023 - 10:22:35: [ULX] [DSTM] Bryant<STEAM_0:0:26745653> spawned model models/props_phx/construct/metal_plate_curve360x2.mdl
[15:22:41] [API:bryantdl7 Activity] : Sent to console: `status`
[15:23:54] [Core Debug]           : Application state changed from Ready to Stopped
[15:23:54] [Core Debug]           : Application state changed from Stopped to PreStart
[15:23:54] [srcds Debug]          : UDP logger listener started on 10.136.143.5:54952
[15:23:54] [Core Debug]           : Working Directory: ./srcds/4020
[15:23:54] [Core Debug]           : Arguments: -game garrysmod -console -usercon -ip 10.136.143.5 -port 27015 +rcon_password ********** +map gm_flatgrass +maxplayers 40 +hostname test +sv_pure 1 -disableluarefresh " " +exec autoexec.cfg -jitdisabled " " -disableluajit " " -pingboost 2 +log on -tickrate 66 -authkey ********** +host_workshop_collection 692212058 +gamemode sandbox +sv_setsteamaccount [redact] +logaddress_add 10.136.143.5:54952
[15:23:54] [Core Debug]           : Application state changed from PreStart to Starting
11/10/2023 - 10:23:58: server_cvar: "mp_flashlight" "1"
11/10/2023 - 10:23:58: Log file closed.
11/10/2023 - 10:23:58: Log file started (file "logs\L1110023.log") (game "D:\AMPDatastore\Instances\dstm-build01\srcds\4020\garrysmod") (version "9033")
11/10/2023 - 10:23:58: Loading map "gm_flatgrass"
11/10/2023 - 10:23:58: server cvars start
11/10/2023 - 10:23:58: "ttt_detective_min_players" = "8"
11/10/2023 - 10:23:58: "sv_maxspeed" = "10000"
11/10/2023 - 10:23:58: "sbox_maxballoons" = "100"
11/10/2023 - 10:23:58: "ttt_preptime_seconds" = "30"
11/10/2023 - 10:23:58: "sv_maxusrcmdprocessticks" = "24"
11/10/2023 - 10:23:58: "sv_rollangle" = "0"
11/10/2023 - 10:23:58: "r_AirboatViewZHeight" = "0.0"
11/10/2023 - 10:23:58: "sv_cheats" = "0"
11/10/2023 - 10:23:58: "r_VehicleViewDampen" = "0"
11/10/2023 - 10:23:58: "sbox_maxwheels" = "50"
11/10/2023 - 10:23:58: "sv_hl2mp_item_respawn_time" = "30"
11/10/2023 - 10:23:58: "physgun_maxrange" = "4096"
11/10/2023 - 10:23:58: "sbox_maxlights" = "5"
11/10/2023 - 10:23:58: "mp_allowNPCs" = "1"
11/10/2023 - 10:23:58: "sbox_maxvehicles" = "4"
11/10/2023 - 10:23:58: "sbox_maxdynamite" = "10"
11/10/2023 - 10:23:58: "sv_allowcslua" = "0"
11/10/2023 - 10:23:58: "ttt_credits_starting" = "2"
11/10/2023 - 10:23:58: "mp_flashlight" = "1"
11/10/2023 - 10:23:58: "sv_stepsize" = "18"
11/10/2023 - 10:23:58: "mp_teamlist" = "hgrunt;scientist"
11/10/2023 - 10:23:58: "sv_alltalk" = "0"
11/10/2023 - 10:23:58: "ttt_round_limit" = "6"
11/10/2023 - 10:23:58: "sbox_maxhoverballs" = "50"
11/10/2023 - 10:23:58: "mp_footsteps" = "1"
11/10/2023 - 10:23:58: "ttt_detective_max" = "32"
11/10/2023 - 10:23:58: "ttt_detective_hats" = "1"
11/10/2023 - 10:23:58: "sbox_maxlamps" = "3"
11/10/2023 - 10:23:58: "sv_noclipaccelerate" = "5"
11/10/2023 - 10:23:58: "sv_wateraccelerate" = "10"
11/10/2023 - 10:23:58: "mp_autocrosshair" = "1"
11/10/2023 - 10:23:58: "sbox_maxsents" = "100"
11/10/2023 - 10:23:58: "sv_rollspeed" = "200"
11/10/2023 - 10:23:58: "ttt_detective_karma_min" = "600"
11/10/2023 - 10:23:58: "ttt_haste_starting_minutes" = "5"
11/10/2023 - 10:23:58: "sv_airaccelerate" = "10"
11/10/2023 - 10:23:58: "mp_timelimit" = "0"
11/10/2023 - 10:23:58: "sbox_noclip" = "1"
11/10/2023 - 10:23:58: "mp_fadetoblack" = "0"
11/10/2023 - 10:23:58: "ttt_allow_discomb_jump" = "0"
11/10/2023 - 10:23:58: "ttt_roundtime_minutes" = "10"
11/10/2023 - 10:23:58: "ttt_namechange_bantime" = "10"
11/10/2023 - 10:23:58: "sbox_maxnpcs" = "10"
11/10/2023 - 10:23:58: "sv_waterfriction" = "1"
11/10/2023 - 10:23:58: "sv_steamgroup" = ""
11/10/2023 - 10:23:58: "ttt_debug_preventwin" = "0"
11/10/2023 - 10:23:58: "coop" = "0"
11/10/2023 - 10:23:58: "nextlevel" = ""
11/10/2023 - 10:23:58: "sv_stopspeed" = "10"
11/10/2023 - 10:23:58: "sv_specaccelerate" = "5"
11/10/2023 - 10:23:58: "ttt_haste" = "1"
11/10/2023 - 10:23:58: "sv_hl2mp_weapon_respawn_time" = "20"
11/10/2023 - 10:23:58: "sv_report_client_settings" = "0"
11/10/2023 - 10:23:58: "deathmatch" = "1"
11/10/2023 - 10:23:58: "tv_enable" = "0"
11/10/2023 - 10:23:58: "sbox_maxbuttons" = "50"
11/10/2023 - 10:23:58: "ttt_detective_pct" = "0.125000"
11/10/2023 - 10:23:58: "sbox_playershurtplayers" = "1"
11/10/2023 - 10:23:58: "mp_falldamage" = "0"
11/10/2023 - 10:23:58: "sv_voiceenable" = "1"
11/10/2023 - 10:23:58: "sbox_maxcameras" = "10"
11/10/2023 - 10:23:58: "ttt_teleport_telefrags" = "1"
11/10/2023 - 10:23:58: "sv_bounce" = "0"
11/10/2023 - 10:23:58: "r_AirboatViewDampenFreq" = "7.0"
11/10/2023 - 10:23:58: "sbox_godmode" = "0"
11/10/2023 - 10:23:58: "mp_weaponstay" = "0"
11/10/2023 - 10:23:58: "mp_friendlyfire" = "0"
11/10/2023 - 10:23:58: "sv_noclipspeed" = "5"
11/10/2023 - 10:23:58: "ttt_namechange_kick" = "1"
11/10/2023 - 10:23:58: "r_JeepViewDampenDamp" = "1.0"
11/10/2023 - 10:23:58: "r_AirboatViewDampenDamp" = "1.0"
11/10/2023 - 10:23:58: "ttt_postround_dm" = "0"
11/10/2023 - 10:23:58: "sbox_persist" = ""
11/10/2023 - 10:23:58: "sbox_bonemanip_player" = "0"
11/10/2023 - 10:23:58: "physgun_limited" = "0"
11/10/2023 - 10:23:58: "sbox_bonemanip_npc" = "1"
11/10/2023 - 10:23:58: "sv_footsteps" = "1"
11/10/2023 - 10:23:58: "tv_relaypassword" = ""
11/10/2023 - 10:23:58: "ttt_no_nade_throw_during_prep" = "1"
11/10/2023 - 10:23:58: "mp_teamplay" = "0"
11/10/2023 - 10:23:58: "r_JeepViewDampenFreq" = "7.0"
11/10/2023 - 10:23:58: "sbox_bonemanip_misc" = "0"
11/10/2023 - 10:23:58: "sv_specnoclip" = "1"
11/10/2023 - 10:23:58: "sv_friction" = "8"
11/10/2023 - 10:23:58: "ttt_ragdoll_pinning_innocents" = "0"
11/10/2023 - 10:23:58: "ttt_haste_minutes_per_death" = "0.500000"
11/10/2023 - 10:23:58: "sbox_maxragdolls" = "10"
11/10/2023 - 10:23:58: "sv_contact" = ""
11/10/2023 - 10:23:58: "ttt_traitor_pct" = "0.250000"
11/10/2023 - 10:23:58: "ttt_det_credits_starting" = "1"
11/10/2023 - 10:23:58: "ttt_traitor_max" = "32"
11/10/2023 - 10:23:58: "sbox_maxeffects" = "200"
11/10/2023 - 10:23:58: "ai_disabled" = "0"
11/10/2023 - 10:23:58: "sbox_maxthrusters" = "50"
11/10/2023 - 10:23:58: "ai_ignoreplayers" = "0"
11/10/2023 - 10:23:58: "ttt_time_limit_minutes" = "75"
11/10/2023 - 10:23:58: "sbox_maxprops" = "200"
11/10/2023 - 10:23:58: "ttt_firstpreptime" = "60"
11/10/2023 - 10:23:58: "mp_fraglimit" = "0"
11/10/2023 - 10:23:58: "mp_forcerespawn" = "1"
11/10/2023 - 10:23:58: "decalfrequency" = "10"
11/10/2023 - 10:23:58: "sbox_weapons" = "1"
11/10/2023 - 10:23:58: "sv_pausable" = "0"
11/10/2023 - 10:23:58: "sv_accelerate" = "10"
11/10/2023 - 10:23:58: "sv_gravity" = "600"
11/10/2023 - 10:23:58: "sv_password" = ""
11/10/2023 - 10:23:58: "ttt_ragdoll_pinning" = "1"
11/10/2023 - 10:23:58: "r_JeepViewZHeight" = "10.0"
11/10/2023 - 10:23:58: "ttt_posttime_seconds" = "30"
11/10/2023 - 10:23:58: "tv_password" = ""
11/10/2023 - 10:23:58: "sbox_maxemitters" = "20"
11/10/2023 - 10:23:58: "sv_specspeed" = "3"
11/10/2023 - 10:23:58: server cvars end
[15:23:58] [srcds Info]           : Setting up RCON...
11/10/2023 - 10:23:59: Started map "gm_flatgrass" (CRC "f8d50e3ab59e69bc4c89907606864be0")
11/10/2023 - 10:24:00: server_cvar: "awarn_decay_rate" "760"
11/10/2023 - 10:24:00: server_cvar: "awarn_reset_warnings_after_ban" "1"
11/10/2023 - 10:24:00: server_cvar: "awarn_logging" "1"
11/10/2023 - 10:24:02: server_cvar: "M9KDynamicRecoil" "0"
11/10/2023 - 10:24:02: server_cvar: "M9KAmmoDetonation" "0"
11/10/2023 - 10:24:02: server_cvar: "M9KDefaultClip" "2"
11/10/2023 - 10:24:02: server_cvar: "M9KClientGasDisable" "1"
11/10/2023 - 10:24:02: server_cvar: "DavyCrockettAllowed" "0"
11/10/2023 - 10:24:02: server_cvar: "OrbitalStrikeAdminOnly" "1"
[15:24:03] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
[15:24:08] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
11/10/2023 - 10:24:12: server_cvar: "sv_streamradio_rebuildplaylists_community_auto" "1"
[15:24:13] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
11/10/2023 - 10:24:14: server_cvar: "trackassembly_maxtrmarg" "0.020000"
11/10/2023 - 10:24:14: server_cvar: "trackassembly_curvefact" "0.500000"
11/10/2023 - 10:24:17: Lua Error: [SF] Attempt to load bad module: starfall/libs_sv/acffunctions.lua
starfall/libs_sv/acffunctions.lua:63: attempt to call field 'AddHook' (a nil value)
stack traceback:
	starfall/libs_sv/acffunctions.lua:63: in function 'r'
	addons/starfall-ex/lua/starfall/sflib.lua:1828: in function <addons/starfall-ex/lua/starfall/sflib.lua:1828>
	[C]: in function 'xpcall'
	addons/starfall-ex/lua/starfall/sflib.lua:1828: in function 'compileModule'
	addons/starfall-ex/lua/starfall/sflib.lua:1842: in function 'addModule'
	addons/starfall-ex/lua/starfall/sflib.lua:1858: in function 'loadModules'
	addons/starfall-ex/lua/starfall/sflib.lua:1863: in main chunk
	[C]: in function 'include'
	addons/starfall-ex/lua/autorun/server/sf_init.lua:34: in main chunk
[15:24:18] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
11/10/2023 - 10:24:20: server_cvar: "stacker_delay" "0.500000"
11/10/2023 - 10:24:20: server_cvar: "stacker_improved_force_stayinworld" "0"
[15:24:23] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
11/10/2023 - 10:24:26: server_cvar: "wire_expression2_find_rate" "0.050000"
11/10/2023 - 10:24:28: server_cvar: "wire_holoemitter_interval" "0.300000"
[15:24:28] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
11/10/2023 - 10:24:29: server_cvar: "wire_turret_delay_minimum" "0.010000"
11/10/2023 - 10:24:29: Lua Error: Refusing to load m9k_ammo_nuke because it is missing Type and Base keys!
[15:24:33] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
11/10/2023 - 10:24:37: Log file closed.
11/10/2023 - 10:24:37: Log file started (file "logs\L1110024.log") (game "D:\AMPDatastore\Instances\dstm-build01\srcds\4020\garrysmod") (version "9033")
11/10/2023 - 10:24:37: server_cvar: "M9KDisablePenetration" "1"
11/10/2023 - 10:24:37: server_cvar: "sbox_maxtextscreens" "3"
11/10/2023 - 10:24:37: server_cvar: "sbox_maxsents" "10"
11/10/2023 - 10:24:37: server_cvar: "sbox_maxprops" "3000"
11/10/2023 - 10:24:37: server_cvar: "sbox_maxragdolls" "1"
11/10/2023 - 10:24:37: server_cvar: "sbox_maxnpcs" "20"
11/10/2023 - 10:24:37: server_cvar: "sbox_maxballoons" "10"
11/10/2023 - 10:24:37: server_cvar: "sbox_maxeffects" "50"
11/10/2023 - 10:24:37: server_cvar: "sbox_maxdynamite" "3"
11/10/2023 - 10:24:37: server_cvar: "sbox_maxlamps" "10"
11/10/2023 - 10:24:37: server_cvar: "sbox_maxwheels" "15"
11/10/2023 - 10:24:37: server_cvar: "sbox_maxhoverballs" "20"
11/10/2023 - 10:24:37: server_cvar: "sbox_maxvehicles" "20"
11/10/2023 - 10:24:37: server_cvar: "sbox_maxbuttons" "20"
11/10/2023 - 10:24:37: server_cvar: "sbox_maxemitters" "1"
11/10/2023 - 10:24:37: server_cvar: "pk_pill_admin_restrict" "0"
11/10/2023 - 10:24:37: server_cvar: "M9KDisablePenetration" "0"
11/10/2023 - 10:24:37: server_cvar: "sv_noclipspeed" "10"
11/10/2023 - 10:24:37: server_cvar: "sv_alltalk" "3"
11/10/2023 - 10:24:37: server_cvar: "sv_contact" "contact@darkstorm.cc"
11/10/2023 - 10:24:37: server_cvar: "decalfrequency" "60"
11/10/2023 - 10:24:37: "Console<0><Console><Console>" say "====================================="
11/10/2023 - 10:24:37: "Console<0><Console><Console>" say "Server Configuration Update Completed"
11/10/2023 - 10:24:37: "Console<0><Console><Console>" say "====================================="
11/10/2023 - 10:24:37: AWarn: Message Table Exists
11/10/2023 - 10:24:37: AWarn: Player Data Table Exists
11/10/2023 - 10:24:37: AWarn: Offline Player Data Table Exists
[15:24:38] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
11/10/2023 - 10:24:41: "[DSTM] Bryant<2><STEAM_0:0:26745653><>" connected, address "100.4.162.114:27005"
11/10/2023 - 10:24:42: AWarn: Your server info has been updated to the online statistics tracking
11/10/2023 - 10:24:42: "[DSTM] Bryant<2><STEAM_0:0:26745653>" STEAM USERID validated
[15:24:43] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
[15:24:48] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
[15:24:53] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
[15:24:58] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
[15:25:03] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
[15:25:08] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
[15:25:11] [API:bryantdl7 Activity] : Sent to console: `status`
[15:25:13] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
[15:25:15] [API:bryantdl7 Activity] : Sent to console: ``
[15:25:15] [API:bryantdl7 Activity] : Sent to console: ``
[15:25:17] [API:bryantdl7 Activity] : Sent to console: `status`
[15:25:18] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
[15:25:23] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
[15:25:28] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
[15:25:33] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
[15:25:38] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
[15:25:43] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...

CubeCoders-Bot · 10 November 2023 15:33

Thank you for providing such detailed reproduction steps, Bryant. It definitely helps narrow down the cause of the issue. Given that external RCON clients still work while AMP’s RCON functionality does not after a server crash, it suggests that the RCON subsystem within AMP may not be recovering correctly after an unexpected server termination.

Here’s what I suggest as the next troubleshooting steps:

Step 1: Confirm that the RCON password AMP is using to connect matches the RCON password in use by the server when it comes back up. Given the internal state of AMP could be desynchronized from the server’s actual state after a crash, this is a good starting point.

Step 2: Check the RCON configuration in the AMP instance directly after such a crash and restart. Within the instance’s configuration, the RCON password should be visible and correct if it was saved properly.

Step 3: Observe if the connection attempts from AMP’s RCON module get any response at all from the server or if there is some sort of silent failure happening.

Step 4: After the server is back up from a manual restart if AMP still can’t connect, you can try restarting the AMP instance (not the server itself) to see if that resolves the connectivity issue for AMP’s RCON system.

Step 5: Examine any potential logs from AMP after the server crash that might indicate where the RCON connection is breaking down. Running ampinstmgr lastlog [INSTANCENAME] might provide some useful insights.

If these steps do not resolve the issue, it might require a deeper investigation into AMP’s RCON subsystem and how it’s handling disconnections or crashes of the game server process. Please try out these steps and let us know the results. If the problem persists, we may need to consider reporting this behavior to the AMP development team for further analysis.

Bryantdl7 · 10 November 2023 16:51

Dear almighty bot of wisdom,

Step 1: Confirmed it has not changed
Step 2: Confirmed it has not changed
Step 3:

[15:25:18] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
[15:25:23] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
[15:25:28] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
[15:25:33] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
[15:25:38] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...
[15:25:43] [RCON Debug]           : Source RCON connecting to 10.136.143.5:27015...

Step 4: This is known to fix the issue, and is the band-aid I have been using for years
Step 5: see my logs in the prior message for lastlog output

This appears to require a deeper investigation into AMP’s RCON subsystem.

Mike · 10 November 2023 18:43

Can you try the Development build out?

Bryantdl7 · 14 November 2023 01:19

Just made a fresh gmod instance using the following dev version via ampinstmgr.exe --changeinstancestream scuff development
https://downloads.cubecoders.com/AMP/Development/20231108.1/AMP_x86_64.zip

When I end task on srcds.exe I am still greeted with the server eternally on “starting” and rcon being broken.

Mike · 14 November 2023 10:19

Just to sanity check - where are you pulling the RCON password from to give to your external clients? I’m assuming you used AMPs ‘Generate’ option?

Bryantdl7 · 14 November 2023 11:24

Correct, the generate option.

Mike · 14 November 2023 11:28

Run this from the directory with the instance data in:

grep -rl "rcon_password" ./

It’ll search more thoroughly for anything that could be changing the RCON password.

Bryantdl7 · 14 November 2023 13:28

This is a windows machine, I cannot grep.

Furthermore, this is a fresh gmod install with zero addons or configuration files. I reproduced this yesterday with a fresh windows 10 install and a fresh amp install with no other programs installed on the machine.