[BUG] Server lag Spikes

[pimpmyname2]

Describe the bug
We are encountering significant lag spikes on our Surftimer server, which we believe are attributed to the size of our database. This issue has only recently emerged and is particularly noticeable when top-ranked players achieve time records(doesnt matter if its new or not). Interestingly, new players do not trigger these lag spikes, very very strange. When the spikes occur, the server comes to a complete stop for approximately 0.5 seconds.
These lag spikes have been persisting for the past 3 months. Surftimer server has been operational for approximately 2 years with many players on. So it must be a database related issue. Maybe:

SQL_db_CalcAvgRunTimeCallback
sql_CountRankedPlayersCallback
sql_CountRankedPlayers2Callback

Additional context
To address the issue, we have already taken several measures to troubleshoot. We attempted unloading most plugins, deleted all replay bots, and even migrated to a localhost database, but unfortunately, none of these actions have resolved the problem. As a result, we can conclude that the issue is not related to hardware, network, or server settings.

Systeminfo:
threadripper 2950x
samsung evo 970 nvme m2
32gb ddr4

Mariadb settings:
skip-name-resolve = ON
thread_stack = 192K
thread_cache_size = 8
query_cache_type = 1
query_cache_limit = 10M
query_cache_size = 80M
tmp_table_size = 64M
max_heap_table_size = 64M
innodb_buffer_pool_size = 1024M
innodb_log_file_size = 256M

EDIT:

I just noticed when using "incomplete maps" in !nominate menu from SurfTimer-Mapchooser a lag spike also occurs.

[pimpmyname2]

After testing many things out, i finally figured out why.

SurfTimer/addons/sourcemod/scripting/surftimer/sql.sp

Line 646 in 38e1e7b

public void sql_CalcuatePlayerRankCallback(Handle owner, Handle hndl, const char[] error, any pack)

The lag spike is being triggered by the player rank calculation. By temporarily commenting out this calculation(look img below), I was able to completely eliminate the lag spike.

Interestingly, it appears that the lag spike is not triggered by new players. I have a strong suspicion about the underlying cause: