Issue Details#559 Serious lag with lookups after 30+ mins of server uptime.

  • Priority: Normal
  • Number: 559
  • Type: Bug
  • Status: Closed
  • Age: 2271 Days

Description

There seems to be a bug since day 1 with prism, it starts lagging really bad. Only when you have more params then p:Name and r:global.

No errors, when you click with the wand, server completely freezes for up to 30 seconds.

Not sure why, but it is getting to be really bad, only way i've found to fix it is restart the server, which I do not like to do every 30 minutes.

History

superpyromaniac Gravatar

superpyromaniac March 31, 2013 6:39 PM

Issue reported.

botsko Gravatar

botsko March 31, 2013 6:48 PM

i've never had this issue and no one's reported it either. Any ideas what the issue is? Any errors?

superpyromaniac Gravatar

superpyromaniac March 31, 2013 7:33 PM

Not very often an error shows up, but the few times it does, it's this

2013-03-31 13:09:24 [INFO] [Prism]: Database connection error: Connection has already been closed. 2013-03-31 13:09:24 [SEVERE] java.sql.SQLException: Connection has already been closed. 2013-03-31 13:09:24 [SEVERE] at org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:118) 2013-03-31 13:09:24 [SEVERE] at sun.proxy.$Proxy16.commit(Unknown Source) 2013-03-31 13:09:24 [SEVERE] at me.botsko.prism.actionlibs.ActionRecorder.insertActionsIntoDatabase(ActionRecorder.java:201) 2013-03-31 13:09:24 [SEVERE] at me.botsko.prism.actionlibs.ActionRecorder.save(ActionRecorder.java:81) 2013-03-31 13:09:24 [SEVERE] at me.botsko.prism.actionlibs.ActionRecorder.run(ActionRecorder.java:230) 2013-03-31 13:09:24 [SEVERE] at org.bukkit.craftbukkit.v1_5_R2.scheduler.CraftTask.run(CraftTask.java:53) 2013-03-31 13:09:24 [SEVERE] at org.bukkit.craftbukkit.v1_5_R2.scheduler.CraftAsyncTask.run(CraftAsyncTask.java:53) 2013-03-31 13:09:24 [SEVERE] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 2013-03-31 13:09:24 [SEVERE] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 2013-03-31 13:09:24 [SEVERE] at java.lang.Thread.run(Unknown Source)

superpyromaniac Gravatar

superpyromaniac April 1, 2013 12:58 AM

The issue is getting so bad, I may have to stop using prism.. It's at the point where the server crashes for 20 minutes when using the want. Other forms off lookups work perfectly fine, as well as rollbacks and restores.

botsko Gravatar

botsko April 1, 2013 3:31 AM

I hate to hear that but I don't know what the issue is. We're using Prism without issue, and I have no idea what might be causing lag. The only time there's an issue with lag is when the db connection errors out, and you'd know because of console errors.

Please list the version of CB, Prism you have, any plugins you use, and an estimate as to when this started, and a description of what you're doing when it occurs.

dariusc93 Gravatar

dariusc93 April 1, 2013 6:57 PM

This is the same issue I was referring too on our server. If the database is new and clean it wont lag out (not alot of data) but over time it would become a issue. I still use prism but i try to limit the use of it now since the new updates. Could be related to how the data is pulled on the main thread (local or remote database). I know with coreprotect it supports multithreading to prevent that issue from happening. Maybe Prism can do the same?

botsko Gravatar

botsko April 1, 2013 7:04 PM

How much data are we talking about though? I don't recall but I'm pretty sure you both have no more data than most of our users, my live server included (about 30-40 million rows, 5GB, etc). Our system performs just fine.

Granted the only thing left on the main thread is the lookup query but all that could do is cause a slight delay as it queries data, it shouldn't ever be crashing the server unless it's an extremely costly query. You almost have to go out of your way to do those.

First of all, maybe give the async lookup branch a try:

http://botsko.s3.amazonaws.com/Prism/Prism-v.1.5.7-38-g81fcb44-async-lookup.jar

The near and lookup commands have been moved async there, though not the wands.

Let me know if that has any impact. I want to fix any performance issues but it's extremely hard when I can't reproduce it.

Can you both give me the following info:

  • What version of MySQL are you using. I'm assuming you both use MySQL locally.
  • How much data do you have
  • Your average query, or what actions most seem to cause these problems

Unrelated to this, I've pulled a fresh copy of our live data so I can run through a new round of profiling and possible improvements on the database side itself, both for generic stuff, but also for some web ui improvements.

superpyromaniac Gravatar

superpyromaniac April 8, 2013 2:38 PM

Size of Prism_actions is 1.4 GB Remote Host MySQL Not sure on MySQL version, but the Phpmyadmin is 3.5.5 What causes the error: Well now any type of command after a few minutes of uptime.

I am going to try the build you posted there, sorry I took a while, I was about 3 states away from my PC to do anything.

botsko Gravatar

botsko April 8, 2013 2:41 PM

So far I see nothing that would indicate a problem. Having remote mysql is always going to be sub-par but it shouldn't be this problematic unless you just have a below average connection speed to it, etc.

superpyromaniac Gravatar

superpyromaniac April 8, 2013 3:34 PM

Yea I have a solid connection and when prism is working, the speed is almost identical to localhost. It can take about 2,000 blocks a second as well as instant lookup speeds, until recently with this issue.

Also with this dev build, I did a world edit clearing out some of the creative world, about 300,000 edits.

Prism froze for about 2 minutes spamming errors before processing the queue.

http://pastebin.com/2fxRUv8Z

superpyromaniac Gravatar

superpyromaniac April 8, 2013 3:35 PM

Other then that, prism has been working good so far, but i'll see how i plays out for the rest of the day.

superpyromaniac Gravatar

superpyromaniac April 8, 2013 5:12 PM

Oh I think I found the issue, prism was working 100% perfect, I go offline for a while, come back, prism is not working again, got thousands of these errors.

http://pastebin.com/K1ShvU0m

botsko Gravatar

botsko April 8, 2013 5:42 PM

What version are you using now?

superpyromaniac Gravatar

superpyromaniac April 8, 2013 5:50 PM

I was using Prism-v.1.5.7-38-g81fcb44-async-lookup.jar - the one you posted in this thread. I am now trying the latest RB of it, 1.5.8 as the other failed after about 10 minutes.

botsko Gravatar

botsko April 8, 2013 5:51 PM

You should use 1.5.8.1.

superpyromaniac Gravatar

superpyromaniac April 8, 2013 6:00 PM

I am now, still same issues.

botsko Gravatar

botsko April 8, 2013 6:03 PM

Don't use a dev build, make sure you're on 1.5.8.1 release. That and the 1.5.8 has a ton of work that resolved all of these db errors for other users. No one else is having these probs with 1.5.8.1 so I'll need more information about your current config and what causes the issue.

superpyromaniac Gravatar

superpyromaniac April 8, 2013 6:15 PM

I am using the recommended build, 1.5.8.1, I reset the config Config: http://pastebin.com/c0u4bKeb

Server: Localhost via UNIX socket Software: MySQL Software version: 5.1.68-cll - MySQL Community Server (GPL)

After restart, prism works flawlessly, but after about 5 - 10 minutes of up time the wand starts becoming very slow, to the point where you click the block, it takes about 30 - 90 seconds to show any results, in the meantime freezing the entire server. After a few hours of up time, if you use the wand, it will crash the server, where I have to force stop it. Non specific lookups or rollbacks, etc work fine. Such as /pr l p:name r:global or /pr rb p:blah r:global, but adding more params seems to also show the lag. I did some tests with coreprotect and logblock, I put so much pressure on them with millions of logs at once and they handled fine, no lag, but prism does for some reason. I do not want to use LB or CO because prism is by far superior and much nicer, so hopefully this can get fixed.

Also, i'm on the IRC as SuperPyroManiac if you need anything else

botsko Gravatar

botsko April 12, 2013 12:01 AM

Closing. This seems to be an issue with your setup or some conflict I can't reproduce. Hit me up in IRC if you have any new information.

  • Status → Closed

Update Issue

Add your comments to this issue!

Creating a free account will let you immediately report issues on this project, as well as comment on existing issues, save your filters, and more!!