So what's with the lag recently?

Server/script information: rules, feature documentation and connection information.

Moderators: In-Game Leads, Lead Admins

User avatar
Mmartin
Head of Development
Head of Development
Posts: 8184
Joined: Thu Mar 01, 2012 10:05 pm
Ingame name: Lucas Foreman
Location: Basement

So what's with the lag recently?

Post by Mmartin » Mon Jun 05, 2017 10:29 pm

For about a month now, we're facing issues with increased lag at higher player counts. It's mostly noticeable once we reach three hundred or more players. Since we've been hearing complaints left and right and it's difficult to provide everyone a full answer, I'm creating this thread to log progress on trying to resolve the issue in effort to provide more transparency into the efforts we've conducted.

Terminology I'll be using
Tickrate - Server FPS, i.e. how many times a second does the server process new information.
Packet loss - When I say this, I mean an arithmetic mean of packet loss per each player on the server. This is when information is sent to or from the server, but doesn't reach its destination as it should.

Issue at hand
Every couple seconds, there's 0.5-1 second lag with chat and vehicles go out of sync, this is most noticeable when switching lanes, making turns or accelerating/stopping.

Initial situation
After the issues first began, we began monitoring server's performance to see anything unusual. It was obvious this isn't a regular old attack, because the flow of packets to and from the server seemed normal and we have pretty sophisticated methods of capturing packets that don't belong (I won't go into detail with this for obvious reasons). There were two things we originally noticed:
 
  • Increased packet loss - what we're normally used to is 0.1% - 0.2% of average packet loss per player on the server, unless there's a heavy attack going on. We noticed an increase in the packet loss to at least 0.3% at all times, sometimes reaching as high as 0.8%.
  • Drops in Tickrate - another thing that was very obvious a dip in server FPS ever so often; about every couple seconds. This was fine at lower player count, but critical during peak. Once we reached 350-400 players, the server FPS would drop down to even 1-10 once every few seconds.
We saw the second anomaly as the primary culprit for our issues. Something's clearly hogging the server resources and causes it to send data to clients with delays — makes sense, right?

Revision reversion
The lag began, conveniently, around the time we released the 5.6.005 update. Initially I thought we must've messed something up in the release that'd cause the lag, so the first thing we did was to go over the changes made. This was a challenge, as it was many things spanning across three months of work, but as much as I tried, I couldn't find anything that'd make sense, given what I was looking for.

The next logical step here was to reverse to previous revision anyway, to see if it wasn't the reason anyway. We reversed and there were signs of the lag as early as with 300 players online, plus the packet loss was still as high as ever, so we moved back to the new revision within a few hours.

Hardware Fault & Upgrade
Another possible cause of lag coming out of blue could be faulty hardware. Back then we were running on a, what is by today's industry standards considered an old machine, fourth generation i7 CPU and a high-speed SSD. In order to rule out any possibility of the issue being in the hardware or the datacenter's routing, we opted for an upgrade to something better. We're now running on a seventh generation i7-7700k and NVMe drives, which are even faster than already high performing SSD's. We even changed our datacenter. The issue however persisted and despite getting a much needed upgrade for our flag ship, hardware wasn't the issue.

Performance Profiling
Still convinced the issue must be the throttling of server's FPS, we ran zeex's performance profiler a few times. Here's the result after our first run:
Image
What does that tell us? Well the first thing that popped into my mind was "who the hell calls that much!". It turns out that our Phone timer, which contains calculations for the player's signal (and then further decides what to do, e.g. hang up their call if they've no signal) was very inefficient. Each iteration of the player loop (one iteration per player online) contained roughly seventy iterations, each of which container proximity checks to radio towers. For those of you who didn't understand a word I just said: with 500 players on, that's 35000 distance checks every time the loop is performed. That's a lot. And proximity checks are very demanding computations.

Further investigation showed that the loop is performed exactly every three seconds. Ho! The spike happens every few, say, three seconds! That must be it, right? Holy shit I saved the day yet again.

We put a patch in the same day and well, the spike that happens every couple seconds was milder, but still there. We are still looking at drops down to 10-30 server FPS every now and then, which could definitely cause the issues we're having. Second run of the performance profiler confirmed the phone issue was gone, but there were some new obvious suspects.
Image

OnPlayerUpdate is a callback that is ran every time there's anything at all new about your client (when you walk, move your camera, etc). I found an old feature still residing there, which contained a loop of each player online every thirty OnPlayerUpdate hits. This would explain why the issue gets more severe with more players on, as with more entities and activity, the calls would rise exponentially. Anyway, like I said — old feature, I just removed that.

There wasn't anything suspicious really, except OnPlayerPropertyTickCount. That's a property system related callback executed every second. It primarily handled displaying checkpoints to players near houses. Albeit some optimizations for it were in place, at peak (let's say 500 players), it could yield about a million iterations a second (500 players x 2000 properties), and you guessed it, a lot of them contain proximity checks (player to house). Okay, that's really gotta be it. I mean it's just so, so obvious this is it, and I've saved th--

Yeah well, this is where it gets interesting. I did manage to stabilize the tick rate at 200-300 at peak, with mild drops to 80-120 every now and then. While that could still use some work, that's far better than what it used to be. However, the lag did not care. We ran another round of profiling, proving that there isn't a single function that'd be unnecessarily hogging server resources.

Image

Back to square... 1?
I guess. At this point I question whether my first instinct — chasing the server performance, was correct. With the server FPS drops being gone, both packet loss and lag are still present. Bet that's it, then. Originally, I'd assume that if network was the issue, it'd be the same regardless of the amount of players online. Perhaps as the amount of players increases, the amount of packets increases — our protection gets more sensitive and begins filtering out packets it shouldn't.

Since then, we tried many things, including but not limited to: server config tweaking - different stream rates, update rates and other settings. No avail. We tried disabling or tweaking our DDOS protection and firewalls, both hardware and software. No significant difference was shown, either. Among other things, we also upgraded all of our software and optimized our MySQL tables.

Next in store
There's still a couple more options we can try. We'll be looking into changing our database engine and doing further optimizations in areas we deem could be responsible, I'll keep this post updated.

Some of you expressed interest in helping us diagnose the issue, but given the circumstances and how unique the issue is, that'd probably cause more harm and headache than good. We do appreciate anyone that's patient while we work on trying to resolve this; we've been hard at work trying to fix it before the anniversary even started. I know very well how frustrating it is and none of us will rest until it's fixed. I also wanted to show you a different side of development that you're used to see, so I hope the read was at least a bit interesting.

User avatar
Konichiwa
Senior Forum Moderator
Posts: 4160
Joined: Sun May 03, 2015 9:33 pm
Ingame name: Derrick_Layne
Contact:

Re: So what's with the lag recently?

Post by Konichiwa » Mon Jun 05, 2017 10:35 pm

Hopefully this can be fixed up soon.

User avatar
Ruskie
Retired Administrator
Retired Administrator
Posts: 10101
Joined: Tue Jan 05, 2010 11:44 am
Ingame name: The heroin Triumvirate
Location: Heinzen x Sines x Fleming

Re: So what's with the lag recently?

Post by Ruskie » Mon Jun 05, 2017 10:37 pm

Thanks for keeping us in the loop.
And as a fading red iron-cross shaped tail light disappeared down a bombed out highway, the only sound louder than a tricked out Shovelhead skull bound for glory, was the rusty laughter of the devil himself. His work here was finally done.

User avatar
Sneezy!
Gangster
Gangster
Posts: 45
Joined: Fri May 20, 2016 9:07 pm
Ingame name: Uthkarsh Singh

Re: So what's with the lag recently?

Post by Sneezy! » Mon Jun 05, 2017 10:37 pm

holy heck

Minst
Wannabe Mafia
Wannabe Mafia
Posts: 226
Joined: Thu Nov 12, 2015 11:16 pm
Ingame name: Clarence Reich

Re: So what's with the lag recently?

Post by Minst » Mon Jun 05, 2017 10:42 pm

I am fairly understanding about this, and I would hope others would be as well. At least during these times, people should be considerate while driving, and shouldn't take it IC if it happens due to lag.

User avatar
xdrime
Gangster
Gangster
Posts: 138
Joined: Tue Jun 14, 2016 4:18 pm
Location: Money land

Re: So what's with the lag recently?

Post by xdrime » Mon Jun 05, 2017 11:07 pm

That's what you get from a 12 y/o game, thank you for the effor dho.
Last edited by xdrime on Mon Jun 05, 2017 11:23 pm, edited 1 time in total.
Image

User avatar
Sheffield
Game Administrator
Posts: 2035
Joined: Fri May 10, 2013 1:47 pm
Ingame name: Marcus_Rothberg

Re: So what's with the lag recently?

Post by Sheffield » Mon Jun 05, 2017 11:11 pm

A lot of time and effort goes into this. I appreciate what you do for the server so it hopefully will work soon!
Weekly lyric:

''And mom, wish I'd've had the chance to have one last heart-to heart honest and open talk with you''. - Eminem

cataplasia
meme lord
meme lord
Posts: 827
Joined: Thu Oct 16, 2014 12:38 am

Re: So what's with the lag recently?

Post by cataplasia » Mon Jun 05, 2017 11:17 pm

cheers for the update, good luck sorting it 8)

User avatar
Ethan_Brumley
Российская Федерация
Российская Федерация
Posts: 3859
Joined: Tue Aug 03, 2010 11:27 pm

Re: So what's with the lag recently?

Post by Ethan_Brumley » Mon Jun 05, 2017 11:44 pm

Ruskie wrote:
Mon Jun 05, 2017 10:37 pm
Thanks for keeping us in the loop.

User avatar
Maid_Doreen
Wannabe Mafia
Wannabe Mafia
Posts: 161
Joined: Sun Nov 20, 2016 5:03 pm
Ingame name: Carlo_Doreen
Location: Bosnia and Herzegovina
Contact:

Re: So what's with the lag recently?

Post by Maid_Doreen » Mon Jun 05, 2017 11:57 pm

Thank you staff for your effort
Image

User avatar
Mark_Warner
Mafia
Mafia
Posts: 474
Joined: Sun May 08, 2016 10:00 pm
Ingame name: Joe Warner
Location: Mulholland Hills, Vinewood.

Re: So what's with the lag recently?

Post by Mark_Warner » Mon Jun 05, 2017 11:59 pm

Great work, thank you for keeping us informed!

User avatar
Script
Mafia
Mafia
Posts: 347
Joined: Sat Sep 14, 2013 2:33 pm
Ingame name: Valentin
Location: India
Contact:

Re: So what's with the lag recently?

Post by Script » Tue Jun 06, 2017 8:00 am

Hope you guys find the issue soon. :) Thank you for keeping us informed.
Script
Other Games Moderator
Reserved Foreign Moderator (Hindi)
Current Character(s)
Valentin Payne - Former DoT Supervisor
Valentin Quinn - Former Correctional Officer II
Valentin Roussea - Former LSU Professor
Valentin Lumiere - Civilian

Previous Character(s)
Dennis Cavazos- LSGOV CITY

User avatar
Stefanovic
Wannabe Don
Wannabe Don
Posts: 921
Joined: Mon Oct 14, 2013 4:48 am
Ingame name: PERALTA
Location: MELBOURNE

Re: So what's with the lag recently?

Post by Stefanovic » Tue Jun 06, 2017 9:20 am

I could never become a developer.
Image

THE NOTORIOUS

User avatar
DRUNKY
Retired Administrator
Retired Administrator
Posts: 2080
Joined: Sun Jan 27, 2013 3:51 pm
Ingame name: Mauricio Echevarria
Location: Tunisia, North africa.
Contact:

Re: So what's with the lag recently?

Post by DRUNKY » Tue Jun 06, 2017 9:32 am

About time and surely everything will be back on track.
(وَإِنَّ رَبَّكَ لَذُو فَضْلٍ عَلَى النَّاسِ وَلَكِنَّ أَكْثَرَهُمْ لا يَشْكُرُونَ (النمل73

creepy lamb
- Gold Member -
- Gold Member -
Posts: 1091
Joined: Thu Aug 20, 2015 5:11 pm
Location: Amsterdam

Re: So what's with the lag recently?

Post by creepy lamb » Tue Jun 06, 2017 10:50 am

i understand every word :P :P

Locked

Return to “Server Information”

Who is online

Users browsing this forum: Kenyatta and 1 guest