See other 4play Articles
Title: Recent Problems on 4um
Source:
[None]
URL Source: [None]
Published: Mar 4, 2009
Author: Neil
Post Date: 2009-03-04 13:05:15 by Pinguinite
Ping List: *4um Admin News* Subscribe to *4um Admin News*
Keywords: None
Views: 671
Comments: 26
Lately there have been slowdowns and outages on 4um. I do believe that one of them recently was a DOS (denial of service) attack, which worked by simply overwhelming the server with bogus requests with lots of web pages that were more difficult to respond to. That was on the 14th or 15th of Feb. That's the first and only instance where I am confident 4um was attacked. There may have been more but this is the only one I'd certify as such. BUT... there has apparently been another innocent factor which is likely the cause of most of the problems of late (and made the attack more effective). In a nutshell, the ever growing size of 4um's DB seems to have broken a limit which exceeded the capacity of the DB to handle quickly. This type of weakness makes DOS type attacks much more effective, but I believe this one has been remedied in a very reasonably acceptable fashion. Here's a hopefully layman's explanation of what has been happening. First, mysql is a program that runs on the server. It's responsible for storing all the data on 4um. Articles, comments, PMs, everything about the site that ordinary use by 4um members and admins can change. Mysql is a widely used program in the linux world and is very useful for storing and retrieving data like 4um's articles, comments, etc. When a web page on 4um is visited, my own forum software takes requests for various articles and threads on 4um and in the case of someone visiting the LC or LA page, asks mysql for information about what was previously posted. Mysql takes those requests, processes them to dig up from (in the case of 4um) its vast DB and sends the raw articles, comments etc it finds to my software, which then formats it into a nice looking web page and sends that to your browser. One of the features I built into Pinguinite long ago was the ability to filter out categories that people may not be interested in. The category filter. Works similar to the bozo filter but works against the category instead of the person who posted it. But because of a nuance in mysql, I found asking it for articles that were NOT in a category took it much longer to process than asking it for articles that WERE in every other category. So that's how it's been working. So even if someone had NO category filtering at all, the request put to mysql was "send me articles in category" and then proceeded to list every category in the DB. Up until recently, mysql could respond to this request by doing everything in it's memory. And apparently, at around 96,000 articles which is what 4um has now, the request started to go over the limit for what mysql could process in it's memory, and it then needed to use the hard disk to figure out what to send which takes a LOT longer. As the number of articles approached a limit, it more and more often had to struggle to service those requests, hense the increasing problems of late. It seems that at about 5:30 AM this morning, it crossed that threshold for good and not even "rebooting" either mysql alone or the entire server returned things to normal as has been the case in the recent past. But the good news there is it makes it very easy for me to see what the problem is and address it. So, I've disabled the Category filter. This allows mysql to respond to requests for information without using the hard disk which is much, much faster, and, it seems, it can now easily keep up with the demands from 4um. Normally, mysql should be able to serve requests from much bigger DBs without using the HD and it can -- if the conditions are right. The fix today was rather drastic in one sense, because it involved NOT asking mysql to do as much work -- my software is now being nicer to it. Maybe mysql should have been able to do the category filtering and the current mysql version just couldn't handle it due to some inefficiency in it. I've read of one discussion on the net where it was debated whether a similar issue was the fault of mysql OR the fault of a program that asked it for data, so the question of "blame" in this case can be a bit fuzzy. Since it appears no one was using the category filter, I think disabling it entirely can be considered an appropriate fix. But the next step is to just wait and over the next month or two and see if there are any more problems.
Post Comment Private Reply Ignore Thread
Top • Page Up • Full Thread • Page Down • Bottom/Latest
#1. To: Pinguinite (#0)
Thanks for the update. Off topic (sorta), but I've been playing with getting your software to run on a Macbook. I'll let you know if I ever get it running.
96,000 articles? I know that you know more about this than I ever will, but sometimes a private can offer a good suggestion to a general... Can that number be pared down? Perhaps put them into a separate and searchable archive? Would that help?
Click for Privacy and Preparedness files
Should be using Oracle 11G.
You got $40k? ; )
Sure, any reduction can help, but it's not the best solution. Separate archives could help reduce load except it complicates things significantly for me if I want to allow people to search both of them/more than one archive. Sure, 96,000 is a lot but not really for a computer (except perhaps the fact that the articles are fully searchable on both title and body, which about doubles its size). Some DB's can search 10's of millions of records with no problem, but it depends on more than just that number. The rate of growth of 4um is probably exceeded by the growth of computer hardware and software tech. I'm tentatively planning on replacing the server this year with something much faster and much more memory. If that were already done, it's likely none of these problems would have happened. Or been noticed is more accurate, as the extra ram alone would likely have permitted the server to handle the software inefficiencies without noticeable slowdown. The idea of purging old articles was entertained in the distant past, but I don't think that it's worth pursuing for performance purposes.
Got it, and thanks
Click for Privacy and Preparedness files
I'm curious, how many articles have the word "turtle" in the title?
Depends on what you are running it on. Remember, you don't have to license the product for support for more than the first year. And Oracle will usually negotiate up to 40% discount on their database software.
What hardware is 4um on? Private owned, ASP ???? CPUs, memory etc.
You know, THAT might be the real problem!!
Which knocks it down to some 20 grand? Where do I sign up!!??? :^)
It's my hardware. But it is a few years old so a replacement is in order. And RAM is a lot cheaper than when it was when I got this system so that will be a no-brainer to max it out.
The new Nehalem-based Xeon processors will be hitting the streets in a few weeks (currently shipping in the new Mac Pros), these have four cores on one die and have a new approach to memory management (each core has a dedicated memory controller) and a faster memory bus. Certainly worth checking out.
#2. To: Pinguinite (#0)
#3. To: Pinguinite (#0)
#4. To: bush_is_a_moonie (#3)
#5. To: PSUSA (#2)
Can that number be pared down? Perhaps put them into a separate and searchable archive?
#6. To: Pinguinite (#5)
#7. To: Pinguinite (#0)
...at around 96,000 articles which is what 4um has now...
Mike Flaherty: Yeah, it's been there a long time. I wish I knew which corner.
My Man Godfrey (1936)
#8. To: bluegrass (#4)
#9. To: Pinguinite (#5)
#10. To: Esso (#7)
I'm curious, how many articles have the word "turtle" in the title?
#11. To: bush_is_a_moonie (#8)
And Oracle will usually negotiate up to 40% discount on their database software.
#12. To: bush_is_a_moonie (#9)
What hardware is 4um on? Private owned, ASP ???? CPUs, memory etc.
#13. To: Pinguinite (#12)
 | Science flies you to the moon. Religion flies you into buildings. |
#14. To: PSUSA (#2)
And half of them have the word "Turtle" in them!
96,000 articles?
No place is better than Turtle Island.
#15. To: Elliott Jackalope (#13)
At any point of time, when choosing hardware/software where reliability is essential, I prefer to stay away from the bleeding edge, as often the latest stuff can have small bugs that can cause big headaches, whether it's compatibility with matching software or something else. Here at home I've got a 64 bit quad core system with a 64 bit linux OS and I really wish it could handle multimedia as well as my 5 year old 32 bit laptop. But it doesn't yet do it. I pretty can't watch movies on it at all, except for youtube. No other format works at the moment.
#16. To: Turtle (#14)
daggone, time for me to start charging you half for this joint! ;)
96,000 articles? And half of them have the word "Turtle" in them!
#17. To: christine, turtle (#16)
daggone, time for me to start charging you half for this joint! ;)
"Controlling carbon is a bureaucrat's dream. If you control carbon, you control life." — Dr. Richard Lindzen, MIT Professor of Meteorology
#18. To: christine, Turtle (#16)
Well, he DOES run an island resort: he's got deep pockets from all the money he saves by employing Buckwheat bellhops ;-)
_________________________________________________________________________
"This man is Jesus,” shouted one man, spilling his Guinness as Barack Obama began his inaugural address. “When will he come to Kenya to save us?”
-Schweizerische Schuetzenzeitung (Swiss Shooting Federation) April, 1941
#19. To: Pinguinite (#12)
Just curious. Been thinking about getting rid of some of the hardware I have around here. But most of what I am considering dumping is Sun boxes and Solaris 10 and maybe, a couple of 4 CPU 3GHz Xeon compaq servers with 8GB. Like yours they are a few years old but I hate kinda hate to get rid of them because they are running RH AS5 and I use them to test Oracle stuff that I then may implement at work. Right now I am running 10G Oracle RAC on them along with 3 quad core boxes. I got a steal on the quad cores - 3GHz 8G mem and 1 TB sata each for $499.00. Compulsive computer shopper I guess. But I don't have to run the heater during the winter. LOL.
#20. To: christine (#16)
For flight and hotel expenses I'll swear in any court of law that you have legitimate copyright to the word "turtle".
daggone, time for me to start charging you half for this joint! ;)
The ultimate effect of shielding men from the effects of folly, is to fill the world with fools. - Herbert Spencer
#21. To: X-15, christine, farmfriend, Dakmar (#18)
Turtle supports Freedom4um with his Brain.
No place is better than Turtle Island.
#22. To: Turtle (#21)
You and Bob Dole both, eh? If you experience enlightenment lasting past about 1947 please call in your agents, as prolonged infiltration may result in incurable infestation.
Turtle supports Freedom4um with his Brain.
The ultimate effect of shielding men from the effects of folly, is to fill the world with fools. - Herbert Spencer
#23. To: X-15, christine, Turtle, dakmar (#18)
He plays a mean game of frisbe.
"Controlling carbon is a bureaucrat's dream. If you control carbon, you control life." — Dr. Richard Lindzen, MIT Professor of Meteorology
#24. To: Turtle (#21)
Turtle's brain is an athletic supporter.
Turtle supports Freedom4um with his Brain.
Make awkward sexual advances, not war.
Used Tires Albany, NY
#25. To: farmfriend (#23)
ROFL!!!
_________________________________________________________________________
"This man is Jesus,” shouted one man, spilling his Guinness as Barack Obama began his inaugural address. “When will he come to Kenya to save us?”
-Schweizerische Schuetzenzeitung (Swiss Shooting Federation) April, 1941
#26. To: Critter (#24)
It's Brain, not brain, and in French it's Jacque Supportre.
Turtle's brain is an athletic supporter.
No place is better than Turtle Island.
Top • Page Up • Full Thread • Page Down • Bottom/Latest