Wikichan Second Downtime
The Second Downtime of Wikichan was a three day period where the new host of Wikichan, Hostgator, decided that Wikichan was too "heavy" for the hosting that The Wikisysop paid for, and just shut it off claiming it took too much resources. This led into a long series of e-mails between The Wikisysop and the company, ending in nothing but Hostgator sucking it's own dick and demanding money. This was when Wikichan was saved by a good friend of Carcer who hosts it at this moment, as he works at a hosting company which does some awesome shit (Note from host: The machine Wikichan runs on is a Dual Xeon 2.8Ghz with a gig of RAM and a 100mbps pipe. And very little else running on it (Read: Nearly nothing)).
Every website in the world, pure HTTP or not, has load on a given server. For example, when I would query google.com, a certain big of processor from some server is required to process my request. This is usually (if not always) minimal and easy to deal with, except in the case of scripts (such as PHP, which is what the Wiki runs on).
On March 23d, HostGator's Abuse department e-mailed the Wikisysop with an e-mail basically saying "We locked Wikichan down because it was causing high load". Immediately, the Wikisysop (remembering The First Downtime) immediately went to back up the MySQL, but it was "missing". Of course, there were backups on the server, but it still made him livid. Quickly e-mailing HostGator back, it became apparent (through about 4 e-mails and some live chat) that HostGator had no intention of putting the website back up, yet still expected Wikisysop to pay for the hosting and use it for other purposes.
Wikisysop eventually, through a bunch of arguments, made four demands (all four of which having yet to be met), primarily that of the return of our image database (which is still lost). As well, he also requested for information about the company's hierarchy numerous times, especially that of any supervisors (in which no-one responded, especially when they blatantly avoided reporting who their RA was).
Currently, Wikichan is hosted on a kinda-sorta dedicated server setup graciously hosted by a good friend of Carcer, who works for a hosting company and actually spent his time setting the server up to work with Wikichan. Best of all, so long as we don't kill him in use of bandwidth or disk size, we can hopefully stay free to all. Huzzah!
Hello, Your account "[WIKICHAN]" was causing very high load on the eldorado server due to your wiki script which was running constant MySQL and PHP processes. Instead of suspending your account, we have disabled the wiki directory. Please reply back to this e-mail as soon as possible.
[LOGS]
Thank you.
If there is anything else we can assist you with please do not hesitate to ask.
Regards, David Collins HostGator Technical Support
Hostgator,
I understand if there was a problem regarding the directory, I would be glad to discuss that. However, I must ask- what the hell happened to my MySQL database? We both know completely that the database was not part of the problem, and simply closing the Wiki directory would have been sufficient. This is not something I take lightly- I'm already in the process of filing a lawsuit against one hosting company for shutting down a database of mine without prior notification or approval, I would have no problem starting another.
Send me every single bit of information you have on this abuse of usage, from the database to your logs. I would appreciate it if the person who shut it down explained what was going on as well. While I understand if one server is taxed too much, completely destroying a MySQL DB is too far.
[WIKISYSOP]
Hello,
The logs are below what we saw on the server and thus the cause for the suspension. This is covered in the Terms of Service under resource usage. Below I have pasted the information. You also have authorized to take actions to disable your site or scripts at any time for an abuse reason. I would advise that you review the Terms of Service as we do sometimes disable MySQL databases. Please note you agreed to the Terms of Service when you signed up for service. The Terms of Service can be found at http://www.hostgator.com/tos.shtml.
[ATTACHED]
Regards, Rich G. HostGator Technical Support / Abuse Team Supervisor.
That's all great and wonderful, but I would be led to believe that those logs are not exactly trustworthy or indicative of issues on my end. Indeed, one connection was up to 11 minutes- but speaking as someone who has run a server on my own, we both know that the server should have killed it would it have been on that long or downloaded that much. You can look yourself at the logs on the MySQL for the Wiki, as well as the other associated scripts- there is no evidence of any connection that would last 11 minutes. The script being used is the exact same as Wikipedia- it is not an error of programming, nor an error of a repeating script of some kind.
While I understand the terms of service, I simply argue with your methodology. I would appreciate it if you would show me where there was any kind of connection beyond some sort of log- even a history of what was accessed or any other kind of evidence. Just sending me something that says "someone was connected to PHP for 11 minutes through the /wiki/ directory" isn't evidence enough that it was an issue on my end. It could have very well been a server error on your company's end, or some other kind of pinging-out scenario. As I'm sure you know many web scripts yourself, there is absolutely no way in which a Wiki could naturally take 11+ minutes to process something- and the log of the database shows no editing or any kind (nor any uploads or downloads) at the time of this supposed abuse. In fact, the time in which your scripts cite (today at 4:00 PM), there is absolutely nothing in the database nor anywhere near any proof of access- basically, no-one was on, we were all getting home from work. Hell, the pageviews barely increased. Where did these massive world-changing PHP processes come from?
I chose your company based on your value and services, but it is becoming increasingly obvious that your methods of handling things are unprofessional. I would be fully willing to take fault if there was a legitimate error on my part, but it simply seems to be incredulous that the script I use could in any way tax your systems. If you can show me further logs (Even access logs or some form of proof that it was directly at fault of the PHP), then I will simply take the blame and move on. However, if this is something that is not of my fault and you're simply punishing me for something out of my power, then I will have no qualms about contacting your Registered Agent on this subject to handle this in a court rather than in cyberspace.
As well, your policy involving the MySQL is far from legitimate. Your ToS in no way has any stipulations for targeted shutdown or deletion of material. You have simply supposed that if you can completely cancel an account, you have full right to edit or delete anything you want- which is not permitted to you in your contracts. What you have essentially done is given yourself the permission to edit anything on my website based on your permission to shut down my service- your only stated right in there is to turn the switch on or off, not decide how bright the light will be.
Interestingly enough, I found my MySQL database (which was untouched) yet you gave me a canned answer about having the right to shut it down- do you even have an idea of my accounts or what is going on? If not, I would obviously prefer to deal with someone who actually had an idea of the situation. I understand you probably are simply tasked with being "the guy to answer e-mails", but obviously when you're dealing with accounts tied to real money, I want to know what actually happened, not a copy and pasted blob of text you found by making two clicks on your company's front page.
The long story short here is that your accusatory methods and poor reporting of the error will no doubt make me cancel my account, but my other accounts as well. Even if I do not file any actions via legal methods, please be advised that I will be forwarding this e-mail, along with a complete explanation, to the Better Business Bureau. I in no means want to be your typical "angry webmaster", but the way you are handling this obviously demands further attention. Because of this, I would appreciate that any further e-mails include the technicians and any/all supervisors and assistants involved in this decision, as I obviously want to know who was making these decisions.
Thanks, [WIKISYSOP]
[WIKISYSOP],
Those logs are directly from the server at the time of the incident. The fact something was running up to 11 minutes you may believe impossible but does happen. The server doesn't just kill connections that have had a long time of running only defunct processes. Since the process was working as it should the operating system will not kill it. We were able to provide you the proof of the suspension and what brought us to the conclusion that your site was violating the Terms of Service.
Regardless if someone was connected to PHP for longer then 11 minutes your account had an active process that didn't terminate the connection after a period of time. Such as a time out scripts run and if the connection hasn't been used it times out and removes the connection. We by no means are saying that was the only reason your account or the files suspended. We in fact showed you what was the cause of the load based on the other scripts. The timestamp in this is 16:36 for those processes. A sample of your access logs for your domain are:
[LOGS]
If no one was one why all that traffic being present in your access logs? It appears that your logs are not showing the correct incorrect information or not accurate. Below is the line count (and the command I ran to figure it) this is total lines in the log file for the 16:36:00 - 16:36:59.
root@eldorado [~]# cat /usr/local/apache/domlogs/vetari.com |grep 22/Mar/2007:16:36: |grep wiki -c 275
I disagree that you had no traffic and that it is impossible you couldn't have cause this load. The logs we showed you were not just one process. They were of more then one. Again that was just the sample we took at that time.
Our company does provide top value and great services. It is that when we see customers causing a highload and causing outages on the boxes that we take the action to disable it to allow the other customers access to their sites that they pay for just as you do. Unprofessional handling would have been us disabling your account and not even notifying you. Which we didn't do. We notified you that we saw a high load of resources and these processes were the top of the list. After figuring that it was only one directory causing the load we felt it best to just suspend the directory instead of the entire account. Which under the terms of service we can do. Even if we take it to court we can (and have) shown that your account was causing a high load and in violation of the Resource Limits. Which is what I provided you with.
Our policy does state we reserve the right to require or disable as necessary any website, account, database, or other component that doesn't comply with our established policies, or to make a modification in an emergency at our sole discretion. We can disable the database in the event of an emergency. Which causing a server load which prevents other customers from use of the services is considered an emergency situation and we handle it as such. Below I have copied the Terms of Service wording for your review.
HostGator reserves the right to require changes or disable as necessary any web site, account, database, or other component that does not comply with its established policies, or to make any such modifications in an emergency at its sole discretion
The canned answer as you call it was actually typed out. I do not use canned answers and ensure that my responses are thought out before I give them to a customer. The response was directed to notify you that if we had or in the future have to disable your database we are permitted to by our Terms of Service. Which you agreed to when you accepted service with HostGator.Com. I am by no means the "email response guy". I am the Abuse Team Lead and am tasked with various duties in the company. One of which is to handle the Abuse Emails and ensure that our customers and our servers are taken care of. This includes enforcement of our Terms of Service.
As to your canceling your account. This of course is your choice and is not ours to make. I am sorry you feel that our "reporting" is not to your standards. The reporting that we have in place is that of what we see inside of the server and what is going on. I do not know how much better the reporting can get. We even included to you where the processes were running from (that is the ls -al /proc/pidnumber). If you have any suggestions on how to make our reporting better I would gladly be welcome to address them with the CTO to see if we can get them implemented.
Regards, Rich G. HostGator Technical Support / Abuse Team Supervisor.
Rich,
Let us quickly get this over with. It's obvious to many people (my staff, primarily) that you are showing a great deal of animosity, and I'm not sure how or why that's being displayed. It's unprofessional and outright rude. I run a company with a customer support division, and if any of my staff were to act in the way you have (god forbid), they would be fired immediately without warning. You have shown that you have little knowledge of the situation, a genuinely brash attitude, and worst of all, absolutely no will to correct it or help in this situation. We can only hope you keep your job there, because I assure you the many other business owners I know would rather sell their company than hire someone of your unprofessional attitude.
However, let us redeem this problem. Because it's very obvious that you intend to stand your childish position of "you are wrong and my company is saintly", let's do some things that can alleviate this problem, or at least try to expedite this process so neither of us have to deal with each other any more.
First of all, you absolutely must unlock the directory of the Wiki (now /2/ instead of /wiki/, as I had to explain to people what happened). I don't care if you delete all of the .php scripts in there, I need access to the image database inside. Leaving that there would not pose a threat to your system, as it would be impossible for anyone to find the images on there, I would simply back them up and then delete them. There is absolutely no reason why you would not do this, as it would not pose a threat to 25% of your resources, therefore it is only logical and right that you do so.
Second of all, I want real information on the systems that I was running (specifically, the eldorado server). To name, I want information on the server itself (the system specs, available memory and CPU, available bandwidth, number of other clients on the server, and the percentage of the server my hosting section took up), information on the logs you have sent me (A general guideline of what-means-what, or where these logs came from (what application and/or logging system)), and a text file with approximately two hour's worth of the logs for the Wikichan system (from about 3:30 to 5:30 or so, including both the access logs and the PHP error logs). Most importantly, I want any and all output from PHP through CGI or however you have it- most importantly the error logs. All of these things can be found with minimal effort on your part and on ANY Apache+MySQL+PHP setup, so this is not difficult, and I will expect it to be sent.
Third of all, as I have requested previously, I want a listing of who was responsible for every part of this operation. I do not want "us" or "the abuse department"- I'm looking for the names of the person(s) who decided to shut it down, whomever was involved in it at any time, and when this was initially discovered. As well, I want information regarding any past problems with this domain, including all times in which you have checked it for errors. It has run on your service for about a month now, no doubt you have checked it at least once before. If not, and since nothing has changed on the server since a month ago, it simply strengthens my argument that it was not a fault of mine.
Fourth and finally, I want the name of your supervisor- the person or persons directly above you in your company's hierarchy. Name and whatever method I can contact him/her- E-mail, phone, whatever.
These four things can be done by any person with even read access to a server (which you obviously have since you've been disabling things), so it is absolutely imperative and non-negotiable that you send me these things. If you are the head of the abuse department, then the responsibility of running these things is yours, and there is no reason why you cannot do these minor tasks. From then on, would these things be completed, I can assure you I will not take up your time any longer. If you wish to call me on this matter, my number is below, and I will be available throughout the weekend.
Thanks, [WIKISYSOP]
Hi,
There was no animosity in Rich's reply. It was giving you all of the logs you wanted as well as the reasoning behind it all.
It appears that you are attacking our employees based on your first two paragraphs of your response. You continually questioned the kind of people we hire based on him providing you with your logs and reasoning. Attacks on our employees are not tolerated and can end in a suspension or termination.
As for the disabled directory: The database is still active due to it never being disabled. Only the wiki directory was disabled. You will have access to this from the cpanel using phpmyadmin. I have allowed cpanel and ftp access to the directory so you can remove the directory yourself or implement changes to ensure that it does not cause loads again by keeping connections open for hundreds of seconds.
As for your request on the server, we cannot provide you with all of the information you requested due to the privacy agreements. We can give you the specs on the server but nothing regarding clients on the box.
The servers are dual 2.4 gig Xeon processors with 4 gigs of ram. Your site was using 132% of the available processors at the time of the disabling of the site. This created a queue for the processes to complete. The server uses PHP as a cgi for security issues.
As for the logs of any previous problems with this domain, I do not see any so there is nothing we can provide you for that. We provided you with the logs showing that you were to blame for the load and you refuse to accept that. There is nothing more we can do to show you as we cannot make someone believe something they don't want to believe.
As for the supervisor, it is Rich G, who you stated had a great deal of animosity when he explained things to you. Thank you.
If there is anything else we can assist you with please do not hesitate to ask.
Regards, Matt Johnson HostGator Technical Support / Abuse Administrator
Mr. Johnson,
I fully understand that your job is to manage the Abuse systems, but it seems as if you are no better than your subordinate. The reason in which I am aggravated with your support team is very simple: your team has done absolutely nothing to attempt to fix the problem, and has only come to me with an accusation. Instead of handling this in a proactive and professional manner (offering solutions), you essentially immediately blamed me and at no point have even entertained the thought that it could have indeed been a technical issue. I have run many servers of my own: spikes like this happen for more reasons than "someone's script exploded". But, of course, I have gotten backup on this- from your own company. Since you were obviously unwilling to release information, I researched it out myself. Let's face it- over 200 people on one server is bound to cause issues like this not related to any client. You've oversold your servers, and this should have been expected on your end. I'm not going to act as a martyr just because you have poor business practices.
Allow me to explain the issue in a very easy way. Your department is, for all intents and purposes, an arm of Customer Service that deals with accounts management- much like a Risk Management department at a bank. Risk Management departments are the people who watch accounts and determine when someone is overdrawn or has issues with their account- let's say I ran off and unknowingly withdrew $1,000 more than I had, essentially overdrafting my account. Would I do that (which I'm not foolish enough to do), Accounts Management would not send me a curt yet pointless e-mail saying "we closed your account because you (unknowingly) overdrafted" and basically sent me a bill for $1000. That's incredibly stupid- banks know better because they don't know if I might turn around and deposit $30,000 INTO the account, thus fixing the issue in their favor. What they would do is ask me about the problem- they would attempt to repair the problem, not make a knee jerk reaction and go on a vendetta to kill my credit score.
The same applies to your department. The issue here, in a nutshell, is that your company has assumed a defensive instead of proactive stance. The impression your company is giving my company is one of an angry pissed-off teenager who refuses to admit that he/she may be partially at fault for a problem. Your blunt and childish refusal to look deeper into the problem is completely unprofessional, and is quite insulting to those of us who actually operate on a business standpoint. I am a businessman, not your team's peon- therefore, I would appreciate being treated with some form of respect or deference, rather than arrogance. What is even a worse slap in the face is your blatant lie about your privacy policy simply to hide your overselling of the server I was on. Trying to hide something like that seems quite indicative of knowledge of guilt.
I, naturally, had a friend contact Customer Support. who got all of the information you refused to give to me (because of your imaginary privacy policy). It is incredulous to me that you have over 200 users on a 2.4 gig Xeon with that kind of RAM. It is no wonder to me that my logs spiked so high- it wouldn't be hard to take up 300% when you have a small country running on a computer about the power of the one I'm typing on right now. This is the very reason I think your logs are heinous to throw at me. If I was running 200 people on my computer, there's no doubt that any page running any kind of script would fluctuate like crazy: if the computer is being taxed anyway, going over any kind of limit gets easier. I have a chat log sitting right on my computer where one of your customer support representatives directly tells my associate that the fact that there are 200 people on the server would possibly cause my script to spike easier- and this is from your end, not mine.
You have completely ignored my requests. For the sake of actually trying to get them done, let me say them again: 1. Unlock the /2/ directory from the Vetari account. Feel free to prevent it from running, but I want my material back from that directory. The MySQL is irrelevant. I cannot edit anything in the directory if I have no FTP access to it. I need full FTP access- the domain is no longer pointing to it, so it is not even an issue if you unlock it. Do so. 2. This one (information on your server load) is done, but of course, not by your own hands. I appreciate your uselessness and blatant lie. 3. I did not ask for the person who e-mailed me, I asked for a list of those responsible. A list- I can read the bottom of my e-mails. Obviously, from the e-mails I have received, I can count three- David Collins, Rich G., and yourself. I want to know if there are more. 4. I want Rich G.'s supervisor, not his name. If this fails, I will simply start from your Registered Agent and work outward, though that would obviously be more trouble to your company than it would for you to simply tell me who is above him. Even if the person directly above him is the CEO/COO/Manager, I want to know- this is not a difficult question (and there is no privacy policy protecting this).
I would appreciate it if you would be so kind as to actually put forward an effort to fix these issues. It should be an embarrassment to you that you were not only caught lying at a piece of vital information, but that I was so easily able to prove you incorrect. The least you can do in this situation is make an effort to actually resolve this issue, and stop dragging it along and attempting to continue to shift blame on my end without making any effort on your part. Again, I will be glad to contact you in any manner of ways, not limited to e-mail. I will gladly call your offices upon request.
[WIKISYSOP]
Quote from the Hostgator Abuse Department:
As for your request on the server, we cannot provide you with all of the information you requested (amount of people on the server, server specs, et al) due to the privacy agreements. We can give you the specs on the server but nothing regarding clients on the box.
Chatlog with Hostgator Support:
(No, my name is not Balthsazar Demitiri)
Chat Information Welcome to HostGator Live Chat! You are now chatting with 'Thino'
Thino: Welcome to HostGator Live Chat, how may I assist you?
Balthsazar Demitiri: I have a question regarding a server that I've heard I may be put on if I buy one of the shared hosting accounts.
Balthsazar Demitiri: The server name I was given by previous tech support was "eldorado". They said I would most likely be put on that.
Balthsazar Demitiri: My tech department wants to know a somewhat simple question- how many clients are on the eldorado server and what are the specs on the server?
Thino: One moment please.
Balthsazar Demitiri: Are you there?
Thino: What is the primary domain name?
Balthsazar Demitiri: I haven't purchased it yet, I apologize. I was just told by my tech guys to check out that server.
Balthsazar Demitiri: Do you mind?
Thino: Theres more than 200.
Balthsazar Demitiri: On a Dual Xeon?
Thino: Only 1 account.
Balthsazar Demitiri: What do you mean?
Thino: On a dedicated you are on there yourself.
Balthsazar Demitiri: Oh, I know that. But I was told the eldorado was a Dual Xeon. So basically, on that server, you have over 200 clients? I'm new to this, is that much?
Thino: Correct.
Balthsazar Demitiri: But is that much?
Thino: Yes.
Balthsazar Demitiri: I'm looking into making a corporate site with some scripts. How easy would it be for those scripts to "go over your processor limit"? I'm typing these off a list I got from the tech section, so you may have to help me with this.
Thino: It's better if you would send a ticket to support regarding what types of script that you would be trying to run.
Balthsazar Demitiri: I have basic specs written down, would you be able to tell me if I typed them out?
Balthsazar Demitiri: It's just whatever they have made up.
Thino: Probably not.
Thino: The admins would be able to answer that for you
Balthsazar Demitiri: Well, what about in general?
Thino: Sure.
Balthsazar Demitiri: Let's just say that it's going to be some blog or something. Not static.
Balthsazar Demitiri: How easy would it be for that to tax that server?
Thino: That would be very hard.
Thino: It would run fine.
Balthsazar Demitiri: But, wouldn't a bunch of people on the server cause problems? I may get dedicated, but I'm just trying to figure out how much of a problem it would be. Would there be the possibility of my scripts going over your 25% limit or something?
Thino: not exactly sure.
Thino: It would just depend on the traffic.
Thino: I would just go ahead and start off small first.
Thino: You can upgrade anytime.
Balthsazar Demitiri: What about when we get big? I'm aiming up- let's say like maybe 100+ users a day.
Thino: Probably need to go dedicated.
Balthsazar Demitiri: So basically, my chances of going over the limit are much higher because I'd be sharing a server with a lot of other people?
Thino: Correct.
Thino: Is there anything else I can help you with?
Balthsazar Demitiri: Uhhh, one last question. I'm writing all of this down, so sorry for my delays
Balthsazar Demitiri: You have over 200 people on eldorado, right? Would it be much different if it was like 50 people? I'm just curious, trying to learn.
Thino: Yes.
Thino: You can also go with a semi-dedicated. which only has 4.
Balthsazar Demitiri: I see. So, that would significantly decrease the chances of me overtaxing the server system?
Thino: Correct.
Balthsazar Demitiri: That's all I really need, I'll be sure to contact my tech department and see if we can't use your service.
Hostgator oversold the server we were on. The reason the abuse department will not release that information is simply because they know that OVER 200 people on a server (the server being somewhat mid-grade) will severely accentuate any use on it, because of the severe load. Like I frequently mentioned to the abuse department, it is obvious that the server Wikichan was running on had flaws that accentuated these problems- NOT our scripts.
I've removed my name as well as the information about the guy who's hosting it. If he wants to be known, then I'll put him on here. But yeah, we're all trying to stay anonymous.