http___www.stochasticgeometry.ie_ga_2010_03_10_performance-tuning-a-server-in-three-minutes-while-being-slashdotte__1.xml
1.1 1.2 2.1 2.2 2.3 2.4 2.5 3.1 3.2 3.3 3.4 3.5 3.6 4.1 4.2 4.3 5.1 5.2 5.3 5.4 6.1 6.2 6.3 6.4 7.1 7.2 8.1 9.1 9.2 9.3 9.4 9.5 9.6 9.7 9.8 10.1 10.2 10.3 11.1 11.2 11.3 11.4 11.5 11.6 12.1 12.2 12.3 13.1 13.2 13.3 13.4 14.1 14.2 14.3 14.4 14.5 14.6 14.7 15.1 15.2 15.3 15.4 15.5 15.6 16.1 16.2 17.1 18.1 19.1 19.2 19.3 20.1 20.2 21.1 22.1 23.1 24.1
So you wrote a blog post about something that seemed fairly innocuous, but for whatever reason, it caught the attention of one of the major sites and now your server load is at 110 and climbing, the ssh command line session is taking thirty seconds to respond to anything at all, and given that your post is on the front page of slashdot at primetime, this doesn’t look like it’s a temporary blip. What do you do? Okay, first things first. You don’t have time to do a proper fine-tuning session. You need a quick & dirty tuneup here. Proper fine tuning you leave till after the traffic spike, and you can then come back at it with a plan and decent tools like siege and so on – but this is the “fix it in three minutes” version. So if you see stuff here that looks crude to you, despair not, everything looks crude when you’re trying to do it in that kind of timeframe. First, you have to be able to actually use the server command line. The load’s at 110 – that’s not coming down anytime soon and until it does you can’t do the rest of the work you need to since it takes so much time for even a shell to respond. The load is being caused by Apache and MySQL taking on more work than the server can handle, causing the server to swap excessively; and you’ve got to dump that work or shut off the incoming new work to recover. You can try sudo killall -9 apache2 if you can get an open ssh terminal in to do the job (and it’s the first thing you should try), but the odds are that that server has to be reset. Whether that means a phone call to the data centre, or a walk down the hall to the server room, or just clicking a button in a web interface, that’s the first thing to do. Don’t hold off, because unless everyone stops reading your page right now, that load’s not coming down. Once the box has rebooted (and I mean immediately – sit there watching ping until it comes back), ssh in and shut down apache. MySQL is okay for now, but the work is coming from Apache, so that has to get shut down until everything’s ready to go again. You’re going to lose a few minutes of service, yes, but that’s recoverable from for a blog (and if this is for something more serious than a blog, you’re going to be in trouble for not properly spec’ing the hardware to begin with anyway, so limit the damage now and take your licks later). At this point — whether you’ve just logged in or whether you managed to run killall successfully – if it’s a debian server you should run sudo /etc/init.d/apache2 stop (did I mention I love debian for having scripts like that? No man-paging apachectl, just an easy to remember standard interface for every service on the box. Wonderful). It’ll tidy up from killall, or it’ll shut down apache cleanly, depending on how you got here. I’m going to use my server as the example here, by the way, since it’s what got burned last night and it’s what prompted me to write this refresher — it’s been two or three years since I last had to maintain a server that was near its capacity, the experience was a bit of a flashback So, some background on my server – I moved my blog from wordpress.com to here, on a Hetzner server. It’s their entry-level dedicated server offering 2Gb of RAM, a 64-bit Athlon, 160Gb of hard drive in a hardware RAID-1 array and a 1Gbit NIC) — all running Debian Lenny (and no, I’ve no relationship with Hetzner, they were just the cheapest of the places various friends recommended). WordPress is up to date on my server (2.9.2 at the time of writing), as is the Lenny install — if you don’t have the latest security fixes and such in place, or your WordPress is outdated, then that’s probably adding to your problem, but for a quick fix like this, that’s too big a job. Get through the traffic spike and deal with it later. And yes, that server spec is overkill for my needs really – but I had a bunch of side projects like RangeClerk (don’t bother, not much is up yet) and the blog for Herself Indoors and her book and some other things I wanted to run as well that would be using wierd php and python modules and libraries and the like; and I just hate cpanel and not being able to install anything I wanted. Plus, it was cheap Right, back to it. The first thing we need to do is to sort out MySQL’s configuration. Open up the my.cnf file, whereever you’ve put it (it’ll be /etc/mysql/my.cnf for a stock Debian install). We need to tweak just a few settings. First off, key_buffer. This is probably the most critical variable here because by default all the tables will be MyISAM tables (if they’re not, then this isn’t so critical). It’s set to about 16Mb by default; we’re going to turn that up quite a bit. On a dedicated database box, this would be set very high – anything up to 50% of the total available memory. In this case, with a full stack on the one box, we set it a bit lower since Apache’s going to want a lot of RAM too – 256Mb will do for a starting value. Next we’re going to disable the InnoDB engine completely to cut down on MySQL’s footprint. Again, WordPress by default isn’t using it. Just ensure skip-innodb is either uncommented or inserted into my.cnf. Lastly, we’re going to enable the query cache. The thing is, MySQL’s query cache is a fairly blunt instrument. If a query is precisely the same the second time it comes in, it’ll hit the cache – but any change at all, no matter how small, and it misses the cache. So it’s not as enormously useful as you’d first imagine. However, it does help, so we’ll increase its size modestly (48Mb of RAM is sufficient here). So our changes to my.cnf look like so: Once those changes are made, sudo /etc/init.d/mysql restart will get the MySQL server up and running with the new setup. Once that’s done, let’s look to the next level in the stack – Apache. Under debian the config files are arranged differently than normal; the configuration changes we’ll make will be in /etc/apache2/apache2.conf but in other installations they would be in httpd.conf or elsewhere. The default Apache install uses the prefork MPM setup – one thread per process. It’s older, slower, less efficient, but less buggy than the worker MPM which isn’t threadsafe. So find the prefork MPM config settings in apache2.conf. They should look like this in a default install: We’re going to cut down a lot on how much work Apache takes on at once here. Yes, some users will have to wait a few seconds to see your page – but right now, with the load at 110 and climbing, they could wait until their browser timed out and they’d never see anything. So we reduce slightly the number of servers that Apache will farm off to handle requests at any one time from 5 to 4; we’ll increase the number of spare servers it’ll keep around to hand off requests to (we want to reduce the overhead of starting and stopping those processes) from 10 to 12. We’ll set an upper limit on how many we can have though, and we’ll keep it to just under 100. This works on my system, which is an entry-level system; you might get away with more, but for now use these settings and it’ll get you up and running and you can increase a bit and check again as you go (and this guide really isn’t aimed at big sites anyway, just small ones like mine which were caught on the hop). We’re also going to ensure no apache process takes on too much at once by creating a limit of how many requests any process can take on – we’ll keep it low for now (3), but it can be increased later. So our changed config settings now look like this: Okay. At this stage, you have two options. The first is to start Apache up again and get back to work. Odds are, this will hold up pretty well – but you want to keep a window open with htop running in the background to keep an eye on things (and mainly you’re watching the swap space usage and the load. The former’s critical, the latter indicative that a problem’s arising – if either go sideways, kill apache and edit apache2.conf setting even lower values for ServerLimit, MaxClients and MaxRequestsPerChild before restarting apache). If that’s your preferred option, skip to the end of this post. However, if you want to take that extra step, we could install memcached quickly here. It’s a very effective load reducer and under debian, it’s far easier than you’d expect: And let that haul in whatever other libraries it needs, then: And once that’s done, edit the php.ini file (in Debian, that’ll be /etc/php5/apache2/php.ini ) and insert this (anywhere in the file will do, but the extensions section is the tidiest): That should be memcached installed and running in a default configuration (we can finetune later). We now need to drop in the backend that WordPress uses to take advantage of memcached. Download object-cache.php and copy it into the wp-content directory of your website and change the permissions and ownership of the file: And that’s it done. Quick, dirty, and everything at default, but that’s a three-minute setup for you (well, maybe five if you do the memcached setup as well, and I am assuming you have a fast net connection for the aptitude step, but still). Now, restart apache and everything should fire up with memcached caching a lot of requests and keeping the server load to a managable level. And once that traffic spike is past… take the time to tune it properly! Like this: Like Loading...
1.1 1.2 2.1 2.2 2.3 2.4 2.5 3.1 3.2 3.3 3.4 3.5 3.6 4.1 4.2 4.3 5.1 5.2 5.3 5.4 5.5 5.6 6.1 6.2 6.3 6.4 7.1 7.2 8.1 9.1 9.2 9.3 9.4 9.5 9.6 9.7 9.8 10.1 10.2 10.3 11.1 11.2 11.3 11.4 11.5 11.6 12.1 12.2 12.3 12.4 13.1 13.2 13.3 13.4 14.1 14.2 14.3 14.4 14.5 14.6 14.7 15.1 15.2 15.3 15.4 15.5 15.6 16.1 16.2 17.1 18.1 19.1 19.2 19.3 20.1 20.2 21.1 22.1 23.1
Mar sin, scríobh tú blog post faoi rud éigin go bhfuil an chuma cothrom innocuous, ach ar chúis ar bith, ghabh sé aird ar cheann de na suímh móra agus anois tá do ualach freastalaí ag 110 agus dreapadóireachta, tá an seisiún ssh-orduithe ag cur tríocha soicind chun freagairt le rud ar bith ar chor ar bith, agus ós rud é go bhfuil do phost ar an leathanach tosaigh de slashdot ag Primetime, ní hionann sin is cuma mhaith tá sé ina blip sealadach. Cad a dhéanann tú a dhéanamh? Maith go leor, rudaí chéad chéad uair. Ní gá duit am a dhéanamh seisiún fíneáil-tiúnta cuí. Ní mór duit tuneup mear agus salach anseo. Mionchoigeartú cheart fhágann tú go dtí tar éis an Spike tráchta, agus is féidir leat teacht ansin ar ais ag sé le plean agus uirlisí réasúnta cosúil le léigear, agus mar sin de - ach is é seo an "a shocrú i trí nóiméad" leagan. Mar sin, má fheiceann tú rudaí anseo go Breathnaíonn amh a thabhairt duit, ní, Breathnaíonn gach rud éadóchas amh nuair a bhíonn tú ag iarraidh a dhéanamh i chineál sin de fhráma ama. Gcéad dul síos, caithfidh tú a bheith in ann úsáid a bhaint as i ndáiríre na n-orduithe freastalaí. An t-ualach ag 110 - ní ar sin ag teacht síos ag am ar bith go luath agus go dtí go ndéanann sé nach féidir leat a dhéanamh ar an chuid eile den obair is gá duit a ós rud é a thógann sé an oiread sin ama le haghaidh fiú bhlaosc chun freagra a thabhairt. Tá an t-ualach a bheith de bharr Apache agus MySQL ag cur ar obair níos mó ná mar is féidir leis an bhfreastalaí láimhseáil, is cúis leis an bhfreastalaí a mhalartú excessively; agus tá tú chun Dumpáil an obair nó a stoptha an obair nua ag teacht isteach a ghnóthú. Is féidir leat triail sudo killall -9 apache2 más féidir leat a fháil críochfort ssh oscailte i chun an jab a (agus tá sé an chéad rud ba chóir duit iarracht a dhéanamh), ach tá an odds go bhfuil an freastalaí a athshocrú. Cibé Ciallaíonn sé glao gutháin chuig an ionad sonraí, nó ag siúl síos an halla chun an seomra freastalaí, nó díreach cliceáil an cnaipe i comhéadan gréasáin, go bhfuil an chéad rud a dhéanamh. Ná shealbhú amach, mar gheall ar mura stopann gach duine a léamh do leathanach ceart anois, ní ar sin ualach ag teacht anuas. Nuair a bheidh an bosca rebooted (agus ciall agam láithreach - suí ann breathnú ping dtí go dtiocfaidh sé ar ais), ssh isteach agus stoptar síos Apache. Tá MySQL ceart go leor do anois, ach tá an obair ag teacht ó Apache, ionas go mbeidh a fháil t síos go dtí gach rud réidh le dul arís. Tá tú ag dul a chailleadh cúpla nóiméad de sheirbhís, yes, ach sin inghnóthaithe ó do bhlag (agus má tá sé seo rud éigin níos tromchúisí ná ar bhlag, agus tú ag dul a bheith i dtrioblóid le haghaidh gan spec'ing i gceart na crua-earraí chun tús a chur leis ar aon nós, agus mar sin a theorannú leis an damáiste anois agus a ghlacadh do licks níos déanaí). (did I mention I love debian for having scripts like that? No man-paging apachectl, just an easy to remember standard interface for every service on the box. Wonderful). Ag an bpointe seo - an bhfuil tú logáilte isteach díreach i nó cibé a bhainistiú tú a reáchtáil go rathúil killall - má tá sé ina fhreastalaí Debian ba chóir duit a reáchtáil sudo / stad etc/init.d/apache2 (luaigh mé breá liom Debian le haghaidh a bhfuil scripteanna mar sin ? Níl apachectl fear-glaoireachta, ach éasca le cuimhneamh comhéadan caighdeánach do gach seirbhís ar an mbosca. iontach). Beidh sé slachtmhar suas ó killall, nó beidh sé stoptar síos Apache cleanly, ag brath ar conas a fuair tú anseo. Tá mé ag dul chun úsáid as mo freastalaí mar an sampla anseo, dála an scéil, ós rud é tá sé cad a fuair dóite an oíche seo caite agus tá sé cad a spreag mé a scríobh an athnuachana - é a bhí dhá nó trí bliana ó shin bhí mé go deireanach a choimeád ar bun freastalaí go raibh in aice ar a cumas, bhí an taithí le beagán de flashback Mar sin, roinnt cúlra ar an bhfreastalaí agam - Bhog mé mo bhlag ó wordpress.com anseo, ar fhreastalaí Hetzner. Tá sé a n- iontrála-leibhéal fhreastalaí tiomnaithe a thairiscint 2Gb RAM, le Athlon 64-giotán, 160GB tiomáint crua i crua-earraí Raid-1 sraith agus NIC 1Gbit) - iad ar fad ag rith Debian Lenny (agus gan aon, tá mé aon ghaol leis Hetzner , bhí siad díreach an saoire de na háiteanna cairde éagsúla a moladh). Tá WordPress suas chun dáta ar an bhfreastalaí agam (2.9.2 ag an am seo á scríobh), mar go bhfuil an Lenny shuiteáil - más rud é nach bhfuil tú ag an ceartúcháin slándála is déanaí agus den sórt sin i bhfeidhm, nó go bhfuil do WordPress as dáta, ansin is dócha go bhfuil cur le d'fhadhb, ach le haghaidh a shocrú mear mar seo, go bhfuil ró-mhór post. Faigh tríd an Spike tráchta agus déileáil leis níos déanaí. Agus tá, is é sin spec freastalaí overkill do mo chuid riachtanas i ndáiríre - ach bhí mé a bunch de thionscadail taobh cosúil RangeClerk (nach bodhraigh, nach bhfuil i bhfad suas go fóill) agus an mblog le haghaidh í féin Laistigh agus a leabhar agus roinnt rudaí eile a bhí mé a reáchtáil chomh maith a bheadh ag baint úsáide as php agus python modúil agus leabharlanna aisteach agus a leithéidí; agus is fuath liom ach cpanel agus gan a bheith in ann aon rud a bhí mé a shuiteáil. Plus, bhí sé saor Ceart, ar ais dó. Is é an chéad rud is gá dúinn a dhéanamh a shórtáil amach chumraíocht MySQL ar. for a stock Debian install). Oscail suas an comhad my.cnf, cibé áit atá tú a chur air (go mbainfidh sé a bheith / etc / MySQL / my.cnf do stoc Debian a shuiteáil). Ní mór dúinn a tweak ach cúpla suíomhanna. Ar dtús, key_buffer. Is dócha an athróg is tábhachtaí anseo mar gheall ar de réir réamhshocraithe beidh na táblaí a bheith táblaí MyISAM (más rud é nach bhfuil siad, ansin nach bhfuil sé seo amhlaidh criticiúil). Tá sé leagtha síos go dtí thart ar 16MB de réir réamhshocraithe; táimid ag dul chun dul suas go leor le beagán. Ar bosca bunachar tiomnaithe, go mbeadh sé seo a shocrú an-ard - rud ar bith suas go dtí 50% den chuimhne iomlán atá ar fáil. Sa chás seo, le Stack iomlán ar an mbosca amháin, a leag muid sé beagán níos ísle ó Apache ag dul go dtí gur mian a lán de RAM ró - Beidh 256MB a dhéanamh le haghaidh luach tosaigh. Next táimid ag dul a chur ar ceal an t-inneall InnoDB go hiomlán a ghearradh síos ar lorg MySQL ar. Arís, nach bhfuil WordPress réir réamhshocraithe baint úsáide as é. . Just a chinntiú go bhfuil scipe-InnoDB ceachtar uncommented nó isteach i my.cnf. Ar deireadh, táimid ag dul chun cur ar chumas an taisce cheist. Is é an rud, is taisce cheist MySQL ar ionstraim cothrom mhaol. Má tá ceist go beacht mar an gcéanna leis an dara huair a thagann sé i, beidh sé bhuail an taisce - ach aon athrú ar chor ar bith, is cuma cé chomh beag, agus misses sé an taisce. Mar sin, nach bhfuil sé chomh úsáideach go mór mar gur mhaith leat a shamhlú dtús. Mar sin féin, a dhéanann sé cabhrú, mar sin beidh orainn a mhéadú a mhéid modestly (48MB RAM bhfuil go leor anseo). Mar sin, ár n-athruithe a my.cnf breathnú cosúil sin: Nuair a bheidh na hathruithe a dhéantar, beidh sudo / etc / init.d / MySQL atosú a fháil ar an fhreastalaí MySQL suas agus a reáchtáil leis an thus nua. Chomh luath agus a dhéanamh, a ligean ar breathnú ar an chéad leibhéal eile sa Stack - Apache. Faoi Debian na comhaid cumraíochta eagraithe difriúil ná mar is gnách; or elsewhere. Beidh na hathruithe chumraíocht beidh orainn a dhéanamh a bheith i / etc/apache2/apache2.conf ach i suiteálacha eile a bheadh siad a bheith i httpd.conf nó in áit eile. Úsáideann an réamhshocraithe Apache shuiteáil prefork MPM thus - snáithe amháin de réir an phróisis. Tá sé níos sine, níos moille, Buggy níos lú éifeachtach, ach níos lú ná an t-oibrí MPM nach bhfuil threadsafe. Mar sin, teacht ar an prefork suímh cumraíochta MPM i apache2.conf. Ba chóir dóibh breathnú mar seo i mainneachtain a shuiteáil: Táimid ag dul a ghearradh síos ar a lán ar cé mhéad oibre a thógann Apache ar ag an am céanna anseo. Sea, beidh roinnt úsáideoirí ag fanacht cúpla soicind a fheiceáil do leathanach - ach anois, leis an ualach ar 110 agus dreapadóireachta, d'fhéadfadh siad a fanacht go dtí a n-bhrabhsálaí thar am agus riamh gur mhaith leo aon rud a fheiceáil. Mar sin, laghdaímid beagán ar líon na freastalaithe go mbeidh Apache feirmeoireacht ar thalamh ar iarratais ag aon am amháin ó 5 láimhseáil go 4; beidh méadú ar líon na n freastalaithe breise beidh sé a choinneáil ar fud a lámh amach iarratais (ba mhaith linn a laghdú ar an forchostais a thosú agus a stopadh na bpróiseas sin) 10-12. Beidh muid a leagtar uasteorainn ar cé mhéad is féidir linn a bheith áfach, agus beidh orainn é a choinneáil go dtí díreach faoi 100. Oibríonn sé seo ar mo chóras, a bhfuil córas iontrála-leibhéal; go dtiocfadh leat a fháil amach le níos mó, ach do anois úsáid a bhaint as na socruithe seo agus beidh sé a gheobhaidh tú suas agus a reáchtáil agus is féidir leat a mhéadú beagán agus seiceáil arís mar a théann tú (agus an treoir seo nach bhfuil i ndáiríre atá dírithe ar láithreáin móra ar aon nós, ach beag cinn cosúil le mianach a bhí gafa atá ar an hop). Táimid ag dul freisin chun a chinntiú go dtarlaíonn aon phróiseas Apache ar an iomarca ag an am céanna ag a chruthú teorainn de cé mhéad is féidir iarratais a chur ar aon phróiseas - beidh orainn a choinneáil íseal do anois (3), ach is féidir é a mhéadú níos déanaí. Mar sin, ár n-suímh cumraíochta athraithe breathnú anois mar seo: Maith go leor. Ag an gcéim seo, tá dhá rogha. Is é an chéad chun tús a chur Apache suas arís agus a fháil ar ais ag obair. and MaxRequestsPerChild before restarting apache). Odds, beidh sé seo a shealbhú suas go maith go leor -. Ach ba mhaith leat a choinneáil fuinneog oscailte le htop ag rith sa chúlra a choimeád ar súil ar rudaí (agus den chuid is mó go bhfuil tú ag faire ar an úsáid spáis babhtála agus an t-ualach na iar-chriticiúil, an sin táscach go bhfuil fadhb ag eascairt - má théann ceachtar sideways, a mharú apache2.conf Apache agus eagar a leagan luachanna fiú níos ísle le haghaidh ServerLimit, MaxClients agus MaxRequestsPerChild roimh restarting Apache). Má tá go bhfuil do rogha is fearr, skip go dtí deireadh an phoist . Mar sin féin, más mian leat a ghlacadh go céim breise, d'fhéadfadh muid a shuiteáil memcached go tapa anseo. Tá sé ina ualach reducer an-éifeachtach agus faoi Debian, tá sé i bhfad níos éasca ná gur mhaith leat a bheith ag súil: Agus lig an tharraingt i cibé leabharlanna eile ní mór é, ansin: ) and insert this (anywhere in the file will do, but the extensions section is the tidiest): Agus nuair a tá go dhéanamh, in eagar an comhad php.ini (i Debian, beidh a bheith / etc/php5/apache2/php.ini) agus cuir isteach seo (in áit ar bith sa chomhad a dhéanamh, ach tá an t-alt síntí an slachtmhaire): Ba chóir é sin a memcached suiteáilte agus a reáchtáil i cumraíocht réamhshocraithe (is féidir linn a finetune níos déanaí). Ní mór dúinn anois chun titim san inneall a úsáideann WordPress leas a bhaint as memcached. Íosluchtaigh réad-cache.php agus chóipeáil sé isteach ar an eolaire WP-ábhar ar do láithreán gréasáin agus a athrú na ceadanna agus úinéireacht an chomhaid: Agus sin déanta air. Thapa, salach, agus gach rud ag réamhshocraithe, ach sin thus trí nóiméad ar do shon (go maith, b'fhéidir cúig má dhéanann tú an thus memcached chomh maith, agus tá mé ag glacadh leis go bhfuil tú nasc glan tapa le haghaidh an chéim inniúlachta, ach fós) . Anois, atosú Apache agus ba chóir gach rud a dóiteáin suas le memcached caching a lán de na hiarratais agus a choimeád ar an ualach freastalaí go dtí leibhéal managable. Agus nuair a bhfuil Spike tráchta atá caite ... a chur ar an am a tune i gceart! Tags: Apache , Debian , go tapa , Lenny , Linux , memcached , MySQL , feidhmíocht , PHP , slashdot , tiúnadh