Problems with a GG5 server running in deathmatch mode

[DreamState]

Ok,

GG version: 5.0.581
"Mattie's EventScripts, http://mattie.info/cs, Version:2.0.0.250"
"Mani Admin Plugin 2007 V1.2BetaS VSP, www.mani-admin-plugin.com"
"GunGame Utils 0.0.2, Predator, www.TwistedPlay.co.uk"

My community owns a dedicated server and we have set up a 24 slot GunGame server on it, amongst a few others. GG5 is excellent, truly a major upgrade to the previous versions.

Basically, the server seems to lag after a certain period of time. I'm not totally sure as to why. But when playing on the server, and watching net_graph 3, the 'in' (ie. what comes from the server) eventually ends up around the 10 mark instead of the 100 (its a 100 tick server). Now, I can expect a drop in this when busy but usually we are around the 50/60 mark when full, depending on the map.

When the server is lagging badly (ie. when the 'in' of net_graph is showing around the 10 mark), our other servers are fine and do not suffer lag. We have looked at server rates etc and tried various combinations of them and also of loaded mods (es version, etc). The server load is not excessive nor is the bandwidth.

Tonight it started lagging badly so we es_unloaded GG and then load it back up and it sorted the lag. The 'in' from the server instantly improved - back up to 80/90 again. Is it something to do with the DM part? We have tried an es_reload in the server.cfg but this crashed the server. Could potentially put an es_unload in server.cfg and use map configs to reload the mod? It definately seems to be something which worsens over time.

Has anyone any ideas on this as I can't find a cause at the moment?

Many thanks in advance,

DS.

93.89.83.35:27050 [UK] TWKgaming.com | TNT Gungame_Deathmatch [T100-FastDL-HLstatsX]

//----------------------
// SERVER RATE SETTINGS
//----------------------
sv_minrate 20000
sv_maxrate 30000
sv_minupdaterate 80
sv_maxupdaterate 101
sv_mincmdrate 80
sv_maxcmdrate 101
sv_client_predict 1
sv_client_interpolate 1
sv_client_interp 0.01
sv_client_cmdrate_difference 30
decalfrequency 5
r_decal_cullsize 5
think_limit 10
fps_max 600

[Don]

Which version of Eventscripts are you using? Are you using ES Tools or GG Utils for the deathmatch respawn?

Edit: Seems I didn't read your post very well. Sorry about that.

[Warren]

"Mattie's EventScripts, http://mattie.info/cs, Version:2.0.0.250"
"Mani Admin Plugin 2007 V1.2BetaS VSP, http://www.mani-admin-plugin.com"
"GunGame Utils 0.0.2, Predator, http://www.TwistedPlay.co.uk"

[DreamState]

Just some further information.

We monitored this again today - the server has been very busy. The stats of the server load are here should this help in anyway.

http://tnt.hlstatsx.com/?mode=servers&s ... 2&game=css

About 9ish the 'in' being recieved from the server had dropped to 10/20 again. This didnt affect the other servers (I personally was in a clan match at the time and the clan server was absolutely fine). An es_unload and es_load of gungame sorted it back out and it went back up to 80ish.

Is there a log file getting too big or some other file that is taking longer to reference the more the mod is loaded? Something 'seems' to be building up over time and a reload of the mod clears it.

Any other information I can provide just let me know

Many thanks,

DS.

[DreamState]

Have just unloaded the logging mod to check if this is causing any issues over time.

This was in the file for today if it helps in anyway I should probably say this gungameWarmupTimer expection seems to have started after the upgrade to the latest version of GG (the errors start appearing on Friday 23 - there is nothing before that in the file). We were having issues with the 'in' dropping before the upgrade so its probably unrelated, but as I was looking at the logs I decided to pop it here anyways.

Exception caught: 25/01/2009 @ [21:20:20] [Occurences: 106]
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Traceback (most recent call last):

File "csspub/cstrike/addons/eventscripts/es.py", line 207, in tick
x()

File "csspub/cstrike/addons/eventscripts/_libs/python/gamethread.py", line 179, in tick
_executenode(task)

File "csspub/cstrike/addons/eventscripts/_libs/python/gamethread.py", line 156, in _executenode
function(*a, **kw)

File "csspub/cstrike/addons/eventscripts/_libs/python/testrepeat.py", line 276, in fire
raise RepeatError('Cannot fire repeat: \"%s\" does not exist.' %repeatName) RepeatError: Cannot fire repeat: "gungameWarmupTimer" does not exist

[DreamState]

We still seem to be having this problem and we have noticed its seems to go hand in hand with the fps on the server. The fps seems to dip quite low most of the time and returns to the server setting (we have fps at 600 atm). At other times the fps goes extremely low but then remains low for map after map (less than 50 fps) - this is where the lag kicks in and the server is giving 'out' 10 (as shown by the 'in' of netgraph 3) instead of 100.

Its important to note that at these times our other servers on the same dedicated box are not lagging. Just the GG.

I've attached a small image from our stats comparing our GG and Warcraft server. Now, both are slightly different ofc, but it literally just shows the dramatic dips associated with GG_DM. Ofc, DM will likely be more bodies, etc I guess (might have a look at these settings) but warcraft is also heavy on the effects and the maps in GG 'tend' to be slightly simpler. Both also have the exact same mods installed (aside from WCS and GG). Eventscripts, ES_tools and mani (we've now altered mani to be loaded as part of metamod to see if that helps).

Any thoughts would be welcomed on why the FPS dips occur and why sometimes the FPS dips occur and remain low. As stated in the first post, an es_unload and es_load of GG rectifies this issue.

gg_dm_wcs.jpg (48.88 KiB) Viewed 29731 times

Many thanks in advance,

DS.

[Cisco]

Hmmmm you're on to something here, I have been trying to figure out some server crashing with my classic GG5 server, you just made me realize that my FPS dips at the beginning of the map that my server is going to crash on.

I have ES2.250 and GG5 v577 with no ES Tools or GG Utils because I don't use deathmatch/elimination.

Do you use Mani Admin? I have been considering migrating to Sourcemod for the functions Mani provides, with the hope that doing so will resolve some unknown conflict.

[DreamState]

We do use mani atm.

First it was loaded via vdf, now via metamod.

I've done a lot of searching but I've never really found any explanation as to why the FPS drops so low when it gets a higher number of players. Hoping someone can explain how this works and why its seems to be such a noticable drop using GG5. I've no way to compare it to other versions of GG, just our other servers.

Also curious as if ggwin.php has something to do with it... We use it, doesn't make much sense for it to cause anything unless it keeps logging instead of just logging the win (and even so...).

We could certainly try without mani (in addition to sourcemod, eXtensible admin is looking promising), although I'm not entirely convinced its mani and we don't want to take that off at the moment.

We have adopted the load and unload daily, but that is no solution, we are just trying to help out on figuring out what makes the server drop its in. Its deffo something to do with GG5, as again unloading and reloading gg5 solves the issue and as on the same dedicated server we run 5 more css mods without the same issues, so we don't feel its really something to do with the server itself...

EDIT:

Just as I was writing this I noticed it has happened again - which is both useful and frustrating! Its important to note I was on one of our other servers at the time which is not suffering any issues at all. Just our GG.

Here is the image - notice the sharp drop in FPS on the server and an in game shot of when I joined show the problem with the server output.

Some feedback on this would be appreciated, even if its just to tell us to post this somewhere else...We are making an effort to use GG5 and help out with issues we find, and do not want to bash at a great mod and do understand the amount of work involved, but some feedback from the creators would be great...

[VoGon]

Adding to Dreamstate's posts (I too admin TWKGaming GG Deathmatch) we have unloaded and reloaded gungame5 and here is the stats pic after it (immediately after the last image on his post).

Notice the immediate fps increase to normal levels and In went back to a more confortable 60-40 (still low though...) as per screen below. Pay no attention to my fps's as I'm on my laptop atm and they dont relate to actual server fps's...

[Cisco]

Just curious, what is your server's sv_maxrate setting?

Also, if any developers are reading this, would the .mdmp files help at all from the crashes?

[DreamState]

We've tried a couple of settings. Currently its set at "sv_maxrate" = "30000".

[DreamState]

We have disabled mani and left only GG, eventscripts and GG-utils. We use es-tools for an anticamp but this has only just been added and the problem was occuring before that.

What can be causing this fps drop? It must be something that is not reset on a map to map basis - something that persists and accumlates to such a point that it causes the fps to drop a little each map? A reload of GG or a server restart clears it.

Please, any ideas, as we are at a loss (excuse the pun).

[erazor]

Hi,

I have the exact same problem, how can i fix this?


Thanks

[Cisco]

EDIT: THIS CAUSED A SERVER CRASH, BEWARE OF ATTEMPTING IT.

Well, I was toying with the idea of a ES2.0 script that runs "stats" in console and parses out the FPS, then reloads gungame on the next map change after it goes below a certain threshold (say 250 FPS).

But, I realized, it's much less effort to just reload gungame every map so that's what I am going to try. I have added the following to the bottom of my server.cfg:

Expand/Contract | Select All

// reload gungame
es_xunload gungame
es_xdelayed 1.0 es_xload gungame

We will see if this causes any instability.

[erazor]

Well, I was toying with the idea of a ES2.0 script that runs "stats" in console and parses out the FPS, then reloads gungame on the next map change after it goes below a certain threshold (say 250 FPS).

But, I realized, it's much less effort to just reload gungame every map so that's what I am going to try. I have added the following to the bottom of my server.cfg:

Expand/Contract | Select All

// reload gungame
es_xunload gungame
es_xdelayed 1.0 es_xload gungame

We will see if this causes any instability.

Awesome, let me know if its fully stable doing this. I have added this code on my server too, ill get back with results.

Thanks

[Cisco]

Wow, that went really badly.

It crashed the very first map, got stuck in a loading loop where it was spamming all the cvar changes over and over and over again until it crashed. Server FPS dropped like a rock to <50.

For some reason this makes me suspect the warmup round...going to try having no warmup round for a while, see what happens then.

Edit:

Here is the graph so you can see for yourself, I put in the changes on gg_little_bridge.

[erazor]

Well, I was toying with the idea of a ES2.0 script that runs "stats" in console and parses out the FPS, then reloads gungame on the next map change after it goes below a certain threshold (say 250 FPS).

But, I realized, it's much less effort to just reload gungame every map so that's what I am going to try. I have added the following to the bottom of my server.cfg:

Expand/Contract | Select All

// reload gungame
es_xunload gungame
es_xdelayed 1.0 es_xload gungame

We will see if this causes any instability.

Awesome, let me know if its fully stable doing this. I have added this code on my server too, ill get back with results.

Thanks

Damn that went really badly, the game doesnt start, all i see it a message saying spawn in 2 secs. I am running with no warmup and i still get the dips.

[Cisco]

Just wanted to report back that removing the warmup round does not solve the crashing.

Also, we've had no luck with reloading gungame itself, our only "comfortable" fix so far has been a controlled server restart.

I guess I'll be thorough and run without mani admin for a day and see if that stabilizes things.

[Warbucks]

What revision are you running? Many people have had to revert back to an older revision (557 works fine for myself and a few others) because some of the later revisions cause crashing. You may want to do the same if you haven't already tried that.

[Cisco]

Well I'm currently running rev 577, and I have tried v586 and v548 with no luck.

Going to try v557 now, first just reverting gungame.py followed by reverting gungamelib.py as well if that doesn't work.