server kicks

71 posts / 0 new
Last post
JDMF's picture
JDMF
Offline
Joined: 21-05-04
Oct 27 2006 21:46
server kicks

should we record these? i did one today 22:06, db was not responding.

Mike Harman
Offline
Joined: 7-02-06
Oct 28 2006 05:38

just did one now, load was 44.

Jacques Roux's picture
Jacques Roux
Offline
Joined: 17-07-06
Nov 9 2006 19:30

- John said server crashed this morning, i dont know who restarted it.

- I just did one right now.

Jacques Roux's picture
Jacques Roux
Offline
Joined: 17-07-06
Nov 9 2006 20:31

And again

JDMF's picture
JDMF
Offline
Joined: 21-05-04
Nov 9 2006 20:46

are you getting some high loads or crawlers going through the site to cause this after so many days of good behaviour?

Jacques Roux's picture
Jacques Roux
Offline
Joined: 17-07-06
Nov 9 2006 22:48

Loads were pretty high when i restarted it.

Not sure about crawlers

Jacques Roux's picture
Jacques Roux
Offline
Joined: 17-07-06
Nov 13 2006 15:19

Restarted.

Load was hovering around 20, and site froze for 5 mins

Jacques Roux's picture
Jacques Roux
Offline
Joined: 17-07-06
Nov 13 2006 15:32

And again.

High load. Was around 70-80 users online

Steven.'s picture
Steven.
Offline
Joined: 27-06-06
Nov 13 2006 15:49

should we ban some people?

seems to be pretty slow still...

Mike Harman
Offline
Joined: 7-02-06
Nov 13 2006 15:56

31 guests now. Could be loads of crawlers today maybe?

Mike Harman
Offline
Joined: 7-02-06
Nov 13 2006 15:56
John. wrote:
should we ban some people?

grin

Mike Harman
Offline
Joined: 7-02-06
Nov 13 2006 16:01

just kicked it again.

Mike Harman
Offline
Joined: 7-02-06
Nov 13 2006 16:30

Load was around 5 for a bit but it's calmed down again.

JDMF's picture
JDMF
Offline
Joined: 21-05-04
Nov 14 2006 13:16

twice today (load average: 27.02, 13.00, 5.57, roughly the same both times, spiked up like that).

need to start looking through the logs of what kinds of requests are being made to spike the load so high (could be search engine crawlers, could be something else...).

Steven.'s picture
Steven.
Offline
Joined: 27-06-06
Nov 14 2006 13:19

hmmm is this because of the upgrade? it can't all be the prol position stuff i put in the library!

JDMF's picture
JDMF
Offline
Joined: 21-05-04
Nov 14 2006 13:21

dont know, someone (rkn, catch, me?) should go through access logs and see what kind of queries are being made just before the load peaks, that will be fucking hard though!

Mike Harman
Offline
Joined: 7-02-06
Nov 14 2006 22:21
John. wrote:
hmmm is this because of the upgrade? it can't all be the prol position stuff i put in the library!

It's very unlikely to be the upgrade, this is a bit of code that doesn't work properly somewhere that hasn't been upgraded, or it might be we're getting hit by bursts of traffic from something. If we do what JDMF says, we might be able to get an idea where that code is, or where the burst comes from.

Jason Cortez
Offline
Joined: 14-11-04
Nov 14 2006 23:12

yeah, so get on with you lazy feks.

No, idea what your talking about. wink

Jacques Roux's picture
Jacques Roux
Offline
Joined: 17-07-06
Nov 15 2006 13:40

Just restarted it.

Jacques Roux's picture
Jacques Roux
Offline
Joined: 17-07-06
Nov 15 2006 14:03

And again.

Catch - where are the logs hiding?

Steven.'s picture
Steven.
Offline
Joined: 27-06-06
Nov 15 2006 14:05

could you disable some things for a bit to try to reduce the server load maybe? who's online box or something?

have all the cache refresh rates been reset maybe?

JDMF's picture
JDMF
Offline
Joined: 21-05-04
Nov 15 2006 14:56
rkn wrote:
And again.

me too.

Steven.'s picture
Steven.
Offline
Joined: 27-06-06
Nov 15 2006 14:59

thanks jdmf, sorry to keep texting you, is that ok though?

Jacques Roux's picture
Jacques Roux
Offline
Joined: 17-07-06
Nov 15 2006 15:05

im here all today so dont bother texting him john. I will try and remember to keep checking.

JDMF's picture
JDMF
Offline
Joined: 21-05-04
Nov 15 2006 15:10
John. wrote:
thanks jdmf, sorry to keep texting you, is that ok though?

hey man, i didn't get your text actually!

no i dont mind doing this, once things calm down at my work i would like to have a closer look at the logs and stuff. I found something which i PMd to catch, but there is nothing obvious...

Jacques Roux's picture
Jacques Roux
Offline
Joined: 17-07-06
Nov 15 2006 15:40

One more time. Im pretty sure this is bots from all the 404s we are getting in drupal log. Maybe google updating all those old links it has to the old forums eek Hmm but googling one of the IP's

http://en.wikipedia.org/wiki/User:210.87.251.107

Mike Harman
Offline
Joined: 7-02-06
Nov 15 2006 16:20

spambots then - they'll ignore any nice robots.txt rules we set (like 10 second delay, although google ignores that as well). Might be able to block really bad ips one by one at the server level.

Few things other things we can do

1. kill all old sections on libcom.org.uk - if bots are crawling .org.uk every broken link and image on there to here is generating a 404 which won't help at all.

2. kill who's online block completely (currently only available to logged in users)

3. tidy up the database - we could empty 1. sessions (logs everyone out once) 2. watchdog table 3. accesslog

4. consider a pm cull longer term to reduce overall db size (it's about 10% of it iirc) - have to give at least 2-4 weeks warning for that.

5. Generally disable a few blocks or have them appear on fewer pages, or set the throttle thresholds lower.

Steven.'s picture
Steven.
Offline
Joined: 27-06-06
Nov 15 2006 16:26
Mike Harman wrote:
1. kill all old sections on libcom.org.uk - if bots are crawling .org.uk every broken link and image on there to here is generating a 404 which won't help at all.

yeah that's cool now.

Quote:
2. kill who's online block completely (currently only available to logged in users)

could we swap it for a thing saying number of users online - no. registered, no. anonymous, then a link to just a page which displays online users for those interested?

Mike Harman
Offline
Joined: 7-02-06
Nov 15 2006 16:29

We can take just the names out yeah. The link to online users I'm not sure about - the standard user list is sorted by most recent users though so that's almost the same thing (considering it's not perfectly real time anyway).

BB
Offline
Joined: 12-08-04
Nov 15 2006 16:43

Glad it's being sorted, nice one.

Steven.'s picture
Steven.
Offline
Joined: 27-06-06
Nov 15 2006 16:51
Mike Harman wrote:
We can take just the names out yeah. The link to online users I'm not sure about - the standard user list is sorted by most recent users though so that's almost the same thing (considering it's not perfectly real time anyway).

hmmm, i spose it's nice if you wanna PM someone... especially users online who want to talk to an admin, but hey