server kicks

71 posts / 0 new
Last post
Jacques Roux's picture
Jacques Roux
Offline
Joined: 17-07-06
Nov 15 2006 16:53

Yeah i was thinking we should cull PM's, cos they will be getting large.

Mike Harman
Offline
Joined: 7-02-06
Nov 15 2006 17:19

alright I killed names on who's online.

Mike Harman
Offline
Joined: 7-02-06
Nov 15 2006 17:21
John. wrote:
hmmm, i spose it's nice if you wanna PM someone... especially users online who want to talk to an admin, but hey

Yeah it's nice when the site doesn't crash every ten minutes as well wink

Who's online page should be possible, just a bit later on.

Jacques Roux's picture
Jacques Roux
Offline
Joined: 17-07-06
Nov 15 2006 17:23

kicked it

Mike Harman
Offline
Joined: 7-02-06
Nov 15 2006 17:27

OK I'm flushing cache and sessions tables.

cache is 120mb with 30mb overhead (our problem perhaps?)

sessions is 10mb with 80,000 entries.

Mike Harman
Offline
Joined: 7-02-06
Nov 15 2006 18:16
John. wrote:
Mike Harman wrote:
1. kill all old sections on libcom.org.uk - if bots are crawling .org.uk every broken link and image on there to here is generating a 404 which won't help at all.

yeah that's cool now.

Rkn - you up for doing this - we could do with a 301 of the whole thing to .org as well if possible

Jacques Roux's picture
Jacques Roux
Offline
Joined: 17-07-06
Nov 15 2006 18:24

well i know how to delete, but dunno how to do a 301 on there. If u dont know either i will find out.

Mike Harman
Offline
Joined: 7-02-06
Nov 15 2006 18:29

OK I thought you knew.

Might just be redirect / http://libcom.org

I'm in now and will start deleting.

Jacques Roux's picture
Jacques Roux
Offline
Joined: 17-07-06
Nov 16 2006 11:45

just restarted, load 45. had been down for a while

Steven.'s picture
Steven.
Offline
Joined: 27-06-06
Nov 16 2006 11:59

yeah at least 2 hours sad

Jacques Roux's picture
Jacques Roux
Offline
Joined: 17-07-06
Nov 16 2006 12:08

it was earliest i could get to a computer sad

Unless can figure out a way to do it from my phone, which would be cool!

edit - found it:
http://s2putty.sourceforge.net/

JDMF's picture
JDMF
Offline
Joined: 21-05-04
Nov 16 2006 21:48

kicked, load 45+

next time i will be patient and not kick it for couple minutes while getting more info on whats going on smile

fort-da game
Offline
Joined: 16-02-06
Nov 16 2006 22:02

also happened on the a/p site for about same amount of time. I was the last poster on both

P.

Lone Wolf's picture
Lone Wolf
Offline
Joined: 1-03-06
Nov 16 2006 22:21
Jason Cortez wrote:
yeah, so get on with you lazy feks.

No, idea what your talking about. ;)

Really Jason?? I am down with all this - I was about to educate Rkn actually... cool wink

Seriously you don't get tech stuff either?? Soulmate! smile

Mike Harman
Offline
Joined: 7-02-06
Nov 17 2006 19:14

Just had to kick about four times.

Two things set it off:

* deleting a post
* saving a "view".

JDMF's picture
JDMF
Offline
Joined: 21-05-04
Nov 17 2006 20:56

mate that has to be just drupal being fucking inefficient, or some setting is wrong somewhere...

how about if server grinds to halt lets do things like:

top >> file
top -u mysql >> file
top -u www-data >> file
netstat >> file

And it would be cool to go in mysql and do show connections and stuff like that, but if the fucker is ground to a halt what can you do. I think it is the apache though which is grinding, but that could be easily tested by going to mysql from the command line while the site is in halt.

Mike Harman
Offline
Joined: 7-02-06
Nov 17 2006 20:57

yeah no worries, will do some of that.

Jacques Roux's picture
Jacques Roux
Offline
Joined: 17-07-06
Nov 19 2006 02:49

seems to have got better last 2 days. Less crashing.

Hopefully drupals performance is going to be tightened up in 5.0

Mike Harman
Offline
Joined: 7-02-06
Nov 19 2006 08:51
rkn wrote:
Hopefully drupals performance is going to be tightened up in 5.0

Yeah I think it'll be a mixture, but hopefully whatever it is that's messing us up will disappear - I think it's just one or two requests that are a mess but almost impossible to work out what.

There's going to be new different kinds of caching though, and I think at least part of our problem is we have loads of pages viewed a few times a day each rather than a few pages viewed thousands of times a day - if it'll work better with that it'd be great.

Mike Harman
Offline
Joined: 7-02-06
Nov 19 2006 10:10

just did one.

JDMF's picture
JDMF
Offline
Joined: 21-05-04
Nov 19 2006 11:36
Mike Harman wrote:
just did one.

me too just now. And got all the data as well. analysing it now...

Jacques Roux's picture
Jacques Roux
Offline
Joined: 17-07-06
Nov 19 2006 20:05

Just did one.

JDMF's picture
JDMF
Offline
Joined: 21-05-04
Nov 19 2006 21:06

i did one now as well.

I think its clear why it keeps jamming: the memory runs out as the mysql proces sizes grow and apache keeps spawning new processes as well.

Could you try lowering the number of mysql and apache processes a bit to see what happens. It could affect the performance though if connections need to be queued, but there must be a balance where the memory doesn't leak out and responses are quick enough not to jam up.

Mike Harman
Offline
Joined: 7-02-06
Nov 20 2006 10:40

OK I set apache max children to 30 (from 45) and mysql max_connections to 45 (from no setting). There was no max_connections in mysql, so that might be the issue. May not be able to check in much until much later today, so anyone who's around please keep an eye on it!

JDMF's picture
JDMF
Offline
Joined: 21-05-04
Nov 20 2006 10:53

so at the moment one apache child process takes 4.5% of memory.

one mysql process takes 6.5%.

30 x 4.5 + 45 x 6.5 = fucked (427.5%)

you have only 200% to spend before swap runs out - thats when the system starts slowing down considerably and the queues build up grinding everything to halt.

i wonder if you would allow only say 10-15 mysql connections, what would happen then? Basically apache would queue the connections, might be worth trying out some day but preferably on a day when you can stay online and watch it.

Mike Harman
Offline
Joined: 7-02-06
Nov 20 2006 15:54

that makes sense, will do when I get home.

Mike Harman
Offline
Joined: 7-02-06
Nov 20 2006 16:25

set apache max clients to 10, mysql max connections to 11. Should run really tight now, see what happens!

Mike Harman
Offline
Joined: 7-02-06
Nov 20 2006 16:34

http://www.mkdoc.org/docs/howto/server-tuning/

for my ref.

JDMF's picture
JDMF
Offline
Joined: 21-05-04
Nov 24 2006 22:53

kicked the apache at 10.50pm. It was non responsive. It was running and it looked ok on the box, but it was not using any CPU so it was hanging for some reason. Lots of connections in SYN_RECV state which could have been a reason but who knows. Mysterious...

Looks like the box has been performing loads better lately though!

Jacques Roux's picture
Jacques Roux
Offline
Joined: 17-07-06
Jan 11 2007 19:04

Have ppl stopped logging kicks or have there not been any?