php5-fpm invoked oom-killerPHP5-fpm fails to startHow do you restart php-fpm?Nginx + PHP5-FPM repeated cut...
What's the name of the logical fallacy where a debater extends a statement far beyond the original statement to make it true?
Do you waste sorcery points if you try to apply metamagic to a spell from a scroll but fail to cast it?
What is this high flying aircraft over Pennsylvania?
Why would five hundred and five be same as one?
Giving feedback to someone without sounding prejudiced
Why the "ls" command is showing the permissions of files in a FAT32 partition?
In One Punch Man, is King actually weak?
Why do Radio Buttons not fill the entire outer circle?
Mimic lecturing on blackboard, facing audience
Grepping string, but include all non-blank lines following each grep match
Why can't the Brexit deadlock in the UK parliament be solved with a plurality vote?
Proving an identity involving cross products and coplanar vectors
How to make money from a browser who sees 5 seconds into the future of any web page?
Is there anyway, I can have two passwords for my wi-fi
When and why was runway 07/25 at Kai Tak removed?
How do I prevent inappropriate ads from appearing in my game?
How to reduce predictors the right way for a logistic regression model
Should I assume I have passed probation?
Did I make a mistake by ccing email to boss to others?
Review your own paper in Mathematics
Animation: customize bounce interpolation
Do I have to know the General Relativity theory to understand the concept of inertial frame?
The Digit Triangles
Why is the Sun approximated as a black body at ~ 5800 K?
php5-fpm invoked oom-killer
PHP5-fpm fails to startHow do you restart php-fpm?Nginx + PHP5-FPM repeated cut outs 502Nginx + Php5-fpm not rendering php filesNGINX + PHP5-FPM segfaults under high load!php-fpm invokes oom-killerPHP5-FPM and 'ondemand'php5-fpm: server reached pm.max_childrenphp5-fpm pm ondemand and still memory problemsphp5-fpm optimization for long polling
2
One of my server (Ubuntu 12.04, 16CPU, 32GB RAM) runs the app section of my website (Nginx 1.1.19, PHP5-FPM, Php 5.3.10, Symfony2 Web Framework)
Suddenly, users started getting HTTP 5** Errors. When I went on the server, there was huge disk IO and the culprit was [flush].
When I went through the kernel logs using dmesg, it showed the reason for [flush] was php5-fpm invoked oom-killer
Kernel Log entry from the time of Incident:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295736] php5-fpm invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295740] php5-fpm cpuset=/ mems_allowed=0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295743] Pid: 13310, comm: php5-fpm Not tainted 3.2.0-69-virtual #103-Ubuntu
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295744] Call Trace:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295751] [<ffffffff8111b971>] dump_header+0x91/0xe0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295754] [<ffffffff8111bd25>] oom_kill_process+0x85/0xb0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295757] [<ffffffff8111c0ca>] out_of_memory+0xfa/0x220
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295761] [<ffffffff81121aa3>] __alloc_pages_nodemask+0x8c3/0x8e0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295764] [<ffffffff8115a853>] alloc_pages_vma+0xb3/0x190
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295768] [<ffffffff8113ae9c>] do_anonymous_page.isra.38+0x7c/0x2f0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295772] [<ffffffff8113e921>] handle_pte_fault+0x1e1/0x200
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295777] [<ffffffff8165d9be>] ? _raw_spin_lock+0xe/0x20
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295780] [<ffffffff8113c089>] ? __pte_alloc+0xa9/0x160
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295782] [<ffffffff8113fb19>] handle_mm_fault+0x269/0x370
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295785] [<ffffffff81661504>] do_page_fault+0x184/0x550
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295789] [<ffffffff81013728>] ? __switch_to+0x138/0x360
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295793] [<ffffffff8105668d>] ? set_next_entity+0xad/0xd0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295796] [<ffffffff8105682a>] ? finish_task_switch+0x4a/0xf0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295799] [<ffffffff8165b6ec>] ? __schedule+0x3cc/0x6f0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295802] [<ffffffff8165e135>] page_fault+0x25/0x30
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295804] Mem-Info:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295805] Node 0 DMA per-cpu:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295808] CPU 0: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295809] CPU 1: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295810] CPU 2: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295812] CPU 3: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295813] CPU 4: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295814] CPU 5: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295816] CPU 6: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295817] CPU 7: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295819] CPU 8: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295820] CPU 9: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295822] CPU 10: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295823] CPU 11: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295824] CPU 12: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295826] CPU 13: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295827] CPU 14: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295828] CPU 15: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295830] Node 0 DMA32 per-cpu:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295831] CPU 0: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295833] CPU 1: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295834] CPU 2: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295836] CPU 3: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295837] CPU 4: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295838] CPU 5: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295840] CPU 6: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295841] CPU 7: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295842] CPU 8: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295844] CPU 9: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295845] CPU 10: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295846] CPU 11: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295848] CPU 12: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295850] CPU 13: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295851] CPU 14: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295853] CPU 15: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295854] Node 0 Normal per-cpu:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295856] CPU 0: hi: 186, btch: 31 usd: 15
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295857] CPU 1: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295859] CPU 2: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295860] CPU 3: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295861] CPU 4: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295863] CPU 5: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295864] CPU 6: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295865] CPU 7: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295867] CPU 8: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295868] CPU 9: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295870] CPU 10: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295871] CPU 11: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295872] CPU 12: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295874] CPU 13: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295875] CPU 14: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295877] CPU 15: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295880] active_anon:7408050 inactive_anon:436 isolated_anon:0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295881] active_file:21159 inactive_file:33077 isolated_file:419
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295882] unevictable:0 dirty:36283 writeback:0 unstable:0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295882] free:47243 slab_reclaimable:20065 slab_unreclaimable:15299
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295883] mapped:15592 shmem:138 pagetables:28362 bounce:0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295885] Node 0 DMA free:15908kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295892] lowmem_reserve[]: 0 3760 30220 30220
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295895] Node 0 DMA32 free:114216kB min:8404kB low:10504kB high:12604kB active_anon:3651120kB inactive_anon:284kB active_file:7848kB inactive_file:38464kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3850496kB mlocked:0kB dirty:38664kB writeback:0kB mapped:272kB shmem:0kB slab_reclaimable:8444kB slab_unreclaimable:7164kB kernel_stack:208kB pagetables:10056kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:116761 all_unreclaimable? yes
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295902] lowmem_reserve[]: 0 0 26460 26460
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295905] Node 0 Normal free:58848kB min:59144kB low:73928kB high:88716kB active_anon:25981080kB inactive_anon:1460kB active_file:76788kB inactive_file:93844kB unevictable:0kB isolated(anon):0kB isolated(file):1676kB present:27095040kB mlocked:0kB dirty:106468kB writeback:0kB mapped:62096kB shmem:552kB slab_reclaimable:71816kB slab_unreclaimable:54032kB kernel_stack:2352kB pagetables:103392kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:388409 all_unreclaimable? yes
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295912] lowmem_reserve[]: 0 0 0 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295915] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15908kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295922] Node 0 DMA32: 268*4kB 544*8kB 367*16kB 391*32kB 271*64kB 165*128kB 103*256kB 32*512kB 5*1024kB 0*2048kB 1*4096kB = 114240kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295929] Node 0 Normal: 13697*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 58884kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295937] 54555 total pagecache pages
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295938] 0 pages in swap cache
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295939] Swap cache stats: add 0, delete 0, find 0/0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295940] Free swap = 0kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295941] Total swap = 0kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359778] 7864304 pages RAM
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359780] 144336 pages reserved
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359781] 654931 pages shared
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359782] 7618451 pages non-shared
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359783] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359798] [ 445] 0 445 4800 221 4 0 0 rpcbind
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359801] [ 452] 108 452 5376 279 2 0 0 rpc.statd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359804] [ 482] 0 482 4308 94 0 0 0 upstart-udev-br
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359807] [ 494] 0 494 5431 327 1 -17 -1000 udevd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359809] [ 608] 0 608 5430 231 1 -17 -1000 udevd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359812] [ 624] 0 624 5430 233 3 -17 -1000 udevd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359815] [ 706] 0 706 3797 49 3 0 0 upstart-socket-
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359818] [ 882] 0 882 1816 213 6 0 0 dhclient3
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359821] [ 993] 0 993 12509 499 2 -17 -1000 sshd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359824] [ 1018] 101 1018 63430 480 4 0 0 rsyslogd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359827] [ 1040] 102 1040 5954 155 13 0 0 dbus-daemon
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359830] [ 1048] 0 1048 7445 153 9 0 0 rpc.idmapd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359832] [ 1151] 0 1151 3626 190 3 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359835] [ 1157] 0 1157 3626 189 1 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359837] [ 1162] 0 1162 3626 189 5 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359840] [ 1163] 0 1163 3626 188 1 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359842] [ 1165] 0 1165 3626 188 6 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359845] [ 1170] 0 1170 4778 211 4 0 0 cron
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359847] [ 1171] 0 1171 4227 89 12 0 0 atd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359850] [ 1172] 0 1172 1082 149 10 0 0 acpid
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359852] [ 1178] 0 1178 3995 173 0 0 0 irqbalance
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359855] [ 1179] 103 1179 46916 650 10 0 0 whoopsie
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359858] [ 1195] 106 1195 80805 236 5 0 0 memcached
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359861] [ 1228] 109 1228 4199 99 0 0 0 nrsysmond
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359863] [ 1230] 109 1230 62851 1327 0 0 0 nrsysmond
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359866] [ 1340] 0 1340 4164 105 9 0 0 newrelic-daemon
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359869] [ 1341] 0 1341 61114 1948 13 0 0 newrelic-daemon
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359871] [ 1381] 0 1381 3310 100 6 0 0 mdadm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359874] [ 1444] 0 1444 3626 189 0 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359877] [ 1445] 0 1445 3188 177 3 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359879] [ 1890] 0 1890 369259 2517 3 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359882] [ 1916] 0 1916 20056 1040 8 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359885] [ 1917] 33 1917 20473 1994 8 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359887] [ 1918] 33 1918 20399 1927 0 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359890] [ 1919] 33 1919 20454 2020 7 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359893] [ 1920] 33 1920 20201 1775 5 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359896] [ 6544] 33 6544 624267 32944 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359898] [ 7293] 33 7293 618265 25932 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359901] [ 7381] 33 7381 618555 26851 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359904] [ 8280] 33 8280 618008 27712 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359906] [ 8737] 33 8737 619699 27793 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359909] [ 8908] 33 8908 618037 26037 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359912] [ 9093] 33 9093 618471 25848 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359914] [ 9479] 33 9479 688805 75927 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359917] [ 9503] 33 9503 5636713 4312476 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359919] [10392] 33 10392 621196 28967 13 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359922] [10509] 33 10509 618070 26222 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359925] [10510] 33 10510 622084 30147 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359927] [10540] 33 10540 620672 29171 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359930] [10799] 33 10799 670237 78421 4 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359932] [11066] 33 11066 616125 23836 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359935] [11250] 33 11250 619613 27729 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359937] [11254] 33 11254 619605 27515 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359940] [11426] 33 11426 620759 28353 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359943] [11428] 33 11428 618500 24942 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359945] [11429] 33 11429 832300 131639 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359948] [11444] 33 11444 390293 26564 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359951] [12119] 33 12119 389579 26295 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359953] [12542] 33 12542 390336 27483 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359956] [12577] 33 12577 1770706 1177669 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359958] [13310] 33 13310 1559195 1166076 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359961] [13481] 33 13481 729336 121410 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359963] [14074] 33 14074 613987 21729 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359966] [14686] 33 14686 405943 20594 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359968] [14771] 33 14771 1341333 17036 10 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359971] [14814] 33 14814 375034 16151 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359973] [14888] 33 14888 374759 15618 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359976] [14917] 33 14917 382071 16366 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359979] [14919] 33 14919 373976 14146 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359981] [14982] 33 14982 373012 11466 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359984] [14983] 33 14983 372940 10379 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359986] [14984] 33 14984 374522 12917 14 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359989] [14985] 33 14985 372898 9890 7 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359991] [14986] 33 14986 373733 11996 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359994] [14987] 33 14987 373203 12061 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359996] [14988] 33 14988 373368 12404 13 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359999] [14989] 33 14989 372407 8838 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360002] [14990] 33 14990 373491 11722 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360004] [14991] 33 14991 371987 8545 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360007] [14992] 33 14992 372184 9768 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360010] [14993] 33 14993 375221 12528 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360012] [15011] 33 15011 372181 8558 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360015] [15012] 33 15012 374863 13426 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360018] [15013] 33 15013 372497 10278 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360020] [15014] 33 15014 372489 9905 8 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360023] [15015] 33 15015 373200 11024 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360026] [15017] 33 15017 372621 9203 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360028] [15018] 33 15018 371993 9152 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360031] [15019] 33 15019 374082 10949 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360033] [15020] 33 15020 373137 10112 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360036] [15021] 33 15021 377055 14606 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360038] [15027] 33 15027 372243 9148 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360041] [15028] 33 15028 373802 11787 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360044] [15029] 33 15029 372094 7847 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360046] [15030] 33 15030 372093 7475 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360049] [15031] 33 15031 372606 9650 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360071] Out of memory: Kill process 9503 (php5-fpm) score 564 or sacrifice child
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.363179] Killed process 9503 (php5-fpm) total-vm:22546852kB, anon-rss:17203684kB, file-rss:46220kB
Seems like the issue is with php5-fpm overcommiting the memory for some cases.
How do I ensure that such cases, doesn't create HIGH DISK IO / Server Unstable?
php-fpm php5 dmesg
asked Feb 1 '15 at 4:56
user268083user268083
2612
bumped to the homepage by Community♦ 14 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
2
One of my server (Ubuntu 12.04, 16CPU, 32GB RAM) runs the app section of my website (Nginx 1.1.19, PHP5-FPM, Php 5.3.10, Symfony2 Web Framework)
Suddenly, users started getting HTTP 5** Errors. When I went on the server, there was huge disk IO and the culprit was [flush].
When I went through the kernel logs using dmesg, it showed the reason for [flush] was php5-fpm invoked oom-killer
Kernel Log entry from the time of Incident:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295736] php5-fpm invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295740] php5-fpm cpuset=/ mems_allowed=0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295743] Pid: 13310, comm: php5-fpm Not tainted 3.2.0-69-virtual #103-Ubuntu
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295744] Call Trace:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295751] [<ffffffff8111b971>] dump_header+0x91/0xe0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295754] [<ffffffff8111bd25>] oom_kill_process+0x85/0xb0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295757] [<ffffffff8111c0ca>] out_of_memory+0xfa/0x220
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295761] [<ffffffff81121aa3>] __alloc_pages_nodemask+0x8c3/0x8e0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295764] [<ffffffff8115a853>] alloc_pages_vma+0xb3/0x190
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295768] [<ffffffff8113ae9c>] do_anonymous_page.isra.38+0x7c/0x2f0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295772] [<ffffffff8113e921>] handle_pte_fault+0x1e1/0x200
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295777] [<ffffffff8165d9be>] ? _raw_spin_lock+0xe/0x20
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295780] [<ffffffff8113c089>] ? __pte_alloc+0xa9/0x160
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295782] [<ffffffff8113fb19>] handle_mm_fault+0x269/0x370
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295785] [<ffffffff81661504>] do_page_fault+0x184/0x550
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295789] [<ffffffff81013728>] ? __switch_to+0x138/0x360
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295793] [<ffffffff8105668d>] ? set_next_entity+0xad/0xd0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295796] [<ffffffff8105682a>] ? finish_task_switch+0x4a/0xf0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295799] [<ffffffff8165b6ec>] ? __schedule+0x3cc/0x6f0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295802] [<ffffffff8165e135>] page_fault+0x25/0x30
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295804] Mem-Info:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295805] Node 0 DMA per-cpu:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295808] CPU 0: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295809] CPU 1: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295810] CPU 2: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295812] CPU 3: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295813] CPU 4: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295814] CPU 5: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295816] CPU 6: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295817] CPU 7: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295819] CPU 8: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295820] CPU 9: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295822] CPU 10: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295823] CPU 11: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295824] CPU 12: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295826] CPU 13: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295827] CPU 14: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295828] CPU 15: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295830] Node 0 DMA32 per-cpu:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295831] CPU 0: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295833] CPU 1: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295834] CPU 2: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295836] CPU 3: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295837] CPU 4: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295838] CPU 5: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295840] CPU 6: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295841] CPU 7: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295842] CPU 8: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295844] CPU 9: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295845] CPU 10: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295846] CPU 11: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295848] CPU 12: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295850] CPU 13: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295851] CPU 14: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295853] CPU 15: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295854] Node 0 Normal per-cpu:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295856] CPU 0: hi: 186, btch: 31 usd: 15
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295857] CPU 1: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295859] CPU 2: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295860] CPU 3: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295861] CPU 4: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295863] CPU 5: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295864] CPU 6: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295865] CPU 7: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295867] CPU 8: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295868] CPU 9: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295870] CPU 10: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295871] CPU 11: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295872] CPU 12: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295874] CPU 13: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295875] CPU 14: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295877] CPU 15: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295880] active_anon:7408050 inactive_anon:436 isolated_anon:0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295881] active_file:21159 inactive_file:33077 isolated_file:419
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295882] unevictable:0 dirty:36283 writeback:0 unstable:0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295882] free:47243 slab_reclaimable:20065 slab_unreclaimable:15299
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295883] mapped:15592 shmem:138 pagetables:28362 bounce:0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295885] Node 0 DMA free:15908kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295892] lowmem_reserve[]: 0 3760 30220 30220
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295895] Node 0 DMA32 free:114216kB min:8404kB low:10504kB high:12604kB active_anon:3651120kB inactive_anon:284kB active_file:7848kB inactive_file:38464kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3850496kB mlocked:0kB dirty:38664kB writeback:0kB mapped:272kB shmem:0kB slab_reclaimable:8444kB slab_unreclaimable:7164kB kernel_stack:208kB pagetables:10056kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:116761 all_unreclaimable? yes
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295902] lowmem_reserve[]: 0 0 26460 26460
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295905] Node 0 Normal free:58848kB min:59144kB low:73928kB high:88716kB active_anon:25981080kB inactive_anon:1460kB active_file:76788kB inactive_file:93844kB unevictable:0kB isolated(anon):0kB isolated(file):1676kB present:27095040kB mlocked:0kB dirty:106468kB writeback:0kB mapped:62096kB shmem:552kB slab_reclaimable:71816kB slab_unreclaimable:54032kB kernel_stack:2352kB pagetables:103392kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:388409 all_unreclaimable? yes
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295912] lowmem_reserve[]: 0 0 0 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295915] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15908kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295922] Node 0 DMA32: 268*4kB 544*8kB 367*16kB 391*32kB 271*64kB 165*128kB 103*256kB 32*512kB 5*1024kB 0*2048kB 1*4096kB = 114240kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295929] Node 0 Normal: 13697*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 58884kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295937] 54555 total pagecache pages
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295938] 0 pages in swap cache
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295939] Swap cache stats: add 0, delete 0, find 0/0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295940] Free swap = 0kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295941] Total swap = 0kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359778] 7864304 pages RAM
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359780] 144336 pages reserved
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359781] 654931 pages shared
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359782] 7618451 pages non-shared
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359783] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359798] [ 445] 0 445 4800 221 4 0 0 rpcbind
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359801] [ 452] 108 452 5376 279 2 0 0 rpc.statd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359804] [ 482] 0 482 4308 94 0 0 0 upstart-udev-br
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359807] [ 494] 0 494 5431 327 1 -17 -1000 udevd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359809] [ 608] 0 608 5430 231 1 -17 -1000 udevd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359812] [ 624] 0 624 5430 233 3 -17 -1000 udevd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359815] [ 706] 0 706 3797 49 3 0 0 upstart-socket-
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359818] [ 882] 0 882 1816 213 6 0 0 dhclient3
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359821] [ 993] 0 993 12509 499 2 -17 -1000 sshd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359824] [ 1018] 101 1018 63430 480 4 0 0 rsyslogd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359827] [ 1040] 102 1040 5954 155 13 0 0 dbus-daemon
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359830] [ 1048] 0 1048 7445 153 9 0 0 rpc.idmapd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359832] [ 1151] 0 1151 3626 190 3 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359835] [ 1157] 0 1157 3626 189 1 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359837] [ 1162] 0 1162 3626 189 5 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359840] [ 1163] 0 1163 3626 188 1 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359842] [ 1165] 0 1165 3626 188 6 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359845] [ 1170] 0 1170 4778 211 4 0 0 cron
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359847] [ 1171] 0 1171 4227 89 12 0 0 atd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359850] [ 1172] 0 1172 1082 149 10 0 0 acpid
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359852] [ 1178] 0 1178 3995 173 0 0 0 irqbalance
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359855] [ 1179] 103 1179 46916 650 10 0 0 whoopsie
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359858] [ 1195] 106 1195 80805 236 5 0 0 memcached
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359861] [ 1228] 109 1228 4199 99 0 0 0 nrsysmond
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359863] [ 1230] 109 1230 62851 1327 0 0 0 nrsysmond
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359866] [ 1340] 0 1340 4164 105 9 0 0 newrelic-daemon
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359869] [ 1341] 0 1341 61114 1948 13 0 0 newrelic-daemon
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359871] [ 1381] 0 1381 3310 100 6 0 0 mdadm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359874] [ 1444] 0 1444 3626 189 0 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359877] [ 1445] 0 1445 3188 177 3 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359879] [ 1890] 0 1890 369259 2517 3 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359882] [ 1916] 0 1916 20056 1040 8 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359885] [ 1917] 33 1917 20473 1994 8 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359887] [ 1918] 33 1918 20399 1927 0 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359890] [ 1919] 33 1919 20454 2020 7 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359893] [ 1920] 33 1920 20201 1775 5 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359896] [ 6544] 33 6544 624267 32944 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359898] [ 7293] 33 7293 618265 25932 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359901] [ 7381] 33 7381 618555 26851 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359904] [ 8280] 33 8280 618008 27712 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359906] [ 8737] 33 8737 619699 27793 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359909] [ 8908] 33 8908 618037 26037 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359912] [ 9093] 33 9093 618471 25848 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359914] [ 9479] 33 9479 688805 75927 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359917] [ 9503] 33 9503 5636713 4312476 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359919] [10392] 33 10392 621196 28967 13 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359922] [10509] 33 10509 618070 26222 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359925] [10510] 33 10510 622084 30147 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359927] [10540] 33 10540 620672 29171 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359930] [10799] 33 10799 670237 78421 4 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359932] [11066] 33 11066 616125 23836 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359935] [11250] 33 11250 619613 27729 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359937] [11254] 33 11254 619605 27515 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359940] [11426] 33 11426 620759 28353 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359943] [11428] 33 11428 618500 24942 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359945] [11429] 33 11429 832300 131639 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359948] [11444] 33 11444 390293 26564 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359951] [12119] 33 12119 389579 26295 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359953] [12542] 33 12542 390336 27483 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359956] [12577] 33 12577 1770706 1177669 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359958] [13310] 33 13310 1559195 1166076 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359961] [13481] 33 13481 729336 121410 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359963] [14074] 33 14074 613987 21729 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359966] [14686] 33 14686 405943 20594 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359968] [14771] 33 14771 1341333 17036 10 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359971] [14814] 33 14814 375034 16151 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359973] [14888] 33 14888 374759 15618 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359976] [14917] 33 14917 382071 16366 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359979] [14919] 33 14919 373976 14146 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359981] [14982] 33 14982 373012 11466 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359984] [14983] 33 14983 372940 10379 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359986] [14984] 33 14984 374522 12917 14 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359989] [14985] 33 14985 372898 9890 7 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359991] [14986] 33 14986 373733 11996 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359994] [14987] 33 14987 373203 12061 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359996] [14988] 33 14988 373368 12404 13 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359999] [14989] 33 14989 372407 8838 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360002] [14990] 33 14990 373491 11722 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360004] [14991] 33 14991 371987 8545 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360007] [14992] 33 14992 372184 9768 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360010] [14993] 33 14993 375221 12528 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360012] [15011] 33 15011 372181 8558 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360015] [15012] 33 15012 374863 13426 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360018] [15013] 33 15013 372497 10278 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360020] [15014] 33 15014 372489 9905 8 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360023] [15015] 33 15015 373200 11024 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360026] [15017] 33 15017 372621 9203 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360028] [15018] 33 15018 371993 9152 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360031] [15019] 33 15019 374082 10949 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360033] [15020] 33 15020 373137 10112 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360036] [15021] 33 15021 377055 14606 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360038] [15027] 33 15027 372243 9148 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360041] [15028] 33 15028 373802 11787 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360044] [15029] 33 15029 372094 7847 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360046] [15030] 33 15030 372093 7475 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360049] [15031] 33 15031 372606 9650 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360071] Out of memory: Kill process 9503 (php5-fpm) score 564 or sacrifice child
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.363179] Killed process 9503 (php5-fpm) total-vm:22546852kB, anon-rss:17203684kB, file-rss:46220kB
Seems like the issue is with php5-fpm overcommiting the memory for some cases.
How do I ensure that such cases, doesn't create HIGH DISK IO / Server Unstable?
php-fpm php5 dmesg
asked Feb 1 '15 at 4:56
user268083user268083
2612
bumped to the homepage by Community♦ 14 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
2
2
2
1
One of my server (Ubuntu 12.04, 16CPU, 32GB RAM) runs the app section of my website (Nginx 1.1.19, PHP5-FPM, Php 5.3.10, Symfony2 Web Framework)
Suddenly, users started getting HTTP 5** Errors. When I went on the server, there was huge disk IO and the culprit was [flush].
When I went through the kernel logs using dmesg, it showed the reason for [flush] was php5-fpm invoked oom-killer
Kernel Log entry from the time of Incident:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295736] php5-fpm invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295740] php5-fpm cpuset=/ mems_allowed=0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295743] Pid: 13310, comm: php5-fpm Not tainted 3.2.0-69-virtual #103-Ubuntu
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295744] Call Trace:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295751] [<ffffffff8111b971>] dump_header+0x91/0xe0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295754] [<ffffffff8111bd25>] oom_kill_process+0x85/0xb0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295757] [<ffffffff8111c0ca>] out_of_memory+0xfa/0x220
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295761] [<ffffffff81121aa3>] __alloc_pages_nodemask+0x8c3/0x8e0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295764] [<ffffffff8115a853>] alloc_pages_vma+0xb3/0x190
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295768] [<ffffffff8113ae9c>] do_anonymous_page.isra.38+0x7c/0x2f0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295772] [<ffffffff8113e921>] handle_pte_fault+0x1e1/0x200
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295777] [<ffffffff8165d9be>] ? _raw_spin_lock+0xe/0x20
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295780] [<ffffffff8113c089>] ? __pte_alloc+0xa9/0x160
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295782] [<ffffffff8113fb19>] handle_mm_fault+0x269/0x370
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295785] [<ffffffff81661504>] do_page_fault+0x184/0x550
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295789] [<ffffffff81013728>] ? __switch_to+0x138/0x360
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295793] [<ffffffff8105668d>] ? set_next_entity+0xad/0xd0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295796] [<ffffffff8105682a>] ? finish_task_switch+0x4a/0xf0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295799] [<ffffffff8165b6ec>] ? __schedule+0x3cc/0x6f0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295802] [<ffffffff8165e135>] page_fault+0x25/0x30
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295804] Mem-Info:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295805] Node 0 DMA per-cpu:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295808] CPU 0: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295809] CPU 1: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295810] CPU 2: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295812] CPU 3: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295813] CPU 4: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295814] CPU 5: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295816] CPU 6: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295817] CPU 7: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295819] CPU 8: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295820] CPU 9: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295822] CPU 10: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295823] CPU 11: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295824] CPU 12: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295826] CPU 13: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295827] CPU 14: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295828] CPU 15: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295830] Node 0 DMA32 per-cpu:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295831] CPU 0: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295833] CPU 1: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295834] CPU 2: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295836] CPU 3: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295837] CPU 4: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295838] CPU 5: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295840] CPU 6: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295841] CPU 7: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295842] CPU 8: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295844] CPU 9: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295845] CPU 10: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295846] CPU 11: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295848] CPU 12: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295850] CPU 13: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295851] CPU 14: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295853] CPU 15: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295854] Node 0 Normal per-cpu:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295856] CPU 0: hi: 186, btch: 31 usd: 15
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295857] CPU 1: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295859] CPU 2: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295860] CPU 3: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295861] CPU 4: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295863] CPU 5: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295864] CPU 6: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295865] CPU 7: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295867] CPU 8: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295868] CPU 9: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295870] CPU 10: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295871] CPU 11: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295872] CPU 12: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295874] CPU 13: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295875] CPU 14: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295877] CPU 15: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295880] active_anon:7408050 inactive_anon:436 isolated_anon:0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295881] active_file:21159 inactive_file:33077 isolated_file:419
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295882] unevictable:0 dirty:36283 writeback:0 unstable:0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295882] free:47243 slab_reclaimable:20065 slab_unreclaimable:15299
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295883] mapped:15592 shmem:138 pagetables:28362 bounce:0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295885] Node 0 DMA free:15908kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295892] lowmem_reserve[]: 0 3760 30220 30220
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295895] Node 0 DMA32 free:114216kB min:8404kB low:10504kB high:12604kB active_anon:3651120kB inactive_anon:284kB active_file:7848kB inactive_file:38464kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3850496kB mlocked:0kB dirty:38664kB writeback:0kB mapped:272kB shmem:0kB slab_reclaimable:8444kB slab_unreclaimable:7164kB kernel_stack:208kB pagetables:10056kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:116761 all_unreclaimable? yes
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295902] lowmem_reserve[]: 0 0 26460 26460
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295905] Node 0 Normal free:58848kB min:59144kB low:73928kB high:88716kB active_anon:25981080kB inactive_anon:1460kB active_file:76788kB inactive_file:93844kB unevictable:0kB isolated(anon):0kB isolated(file):1676kB present:27095040kB mlocked:0kB dirty:106468kB writeback:0kB mapped:62096kB shmem:552kB slab_reclaimable:71816kB slab_unreclaimable:54032kB kernel_stack:2352kB pagetables:103392kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:388409 all_unreclaimable? yes
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295912] lowmem_reserve[]: 0 0 0 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295915] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15908kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295922] Node 0 DMA32: 268*4kB 544*8kB 367*16kB 391*32kB 271*64kB 165*128kB 103*256kB 32*512kB 5*1024kB 0*2048kB 1*4096kB = 114240kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295929] Node 0 Normal: 13697*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 58884kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295937] 54555 total pagecache pages
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295938] 0 pages in swap cache
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295939] Swap cache stats: add 0, delete 0, find 0/0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295940] Free swap = 0kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295941] Total swap = 0kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359778] 7864304 pages RAM
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359780] 144336 pages reserved
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359781] 654931 pages shared
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359782] 7618451 pages non-shared
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359783] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359798] [ 445] 0 445 4800 221 4 0 0 rpcbind
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359801] [ 452] 108 452 5376 279 2 0 0 rpc.statd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359804] [ 482] 0 482 4308 94 0 0 0 upstart-udev-br
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359807] [ 494] 0 494 5431 327 1 -17 -1000 udevd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359809] [ 608] 0 608 5430 231 1 -17 -1000 udevd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359812] [ 624] 0 624 5430 233 3 -17 -1000 udevd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359815] [ 706] 0 706 3797 49 3 0 0 upstart-socket-
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359818] [ 882] 0 882 1816 213 6 0 0 dhclient3
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359821] [ 993] 0 993 12509 499 2 -17 -1000 sshd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359824] [ 1018] 101 1018 63430 480 4 0 0 rsyslogd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359827] [ 1040] 102 1040 5954 155 13 0 0 dbus-daemon
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359830] [ 1048] 0 1048 7445 153 9 0 0 rpc.idmapd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359832] [ 1151] 0 1151 3626 190 3 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359835] [ 1157] 0 1157 3626 189 1 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359837] [ 1162] 0 1162 3626 189 5 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359840] [ 1163] 0 1163 3626 188 1 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359842] [ 1165] 0 1165 3626 188 6 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359845] [ 1170] 0 1170 4778 211 4 0 0 cron
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359847] [ 1171] 0 1171 4227 89 12 0 0 atd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359850] [ 1172] 0 1172 1082 149 10 0 0 acpid
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359852] [ 1178] 0 1178 3995 173 0 0 0 irqbalance
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359855] [ 1179] 103 1179 46916 650 10 0 0 whoopsie
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359858] [ 1195] 106 1195 80805 236 5 0 0 memcached
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359861] [ 1228] 109 1228 4199 99 0 0 0 nrsysmond
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359863] [ 1230] 109 1230 62851 1327 0 0 0 nrsysmond
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359866] [ 1340] 0 1340 4164 105 9 0 0 newrelic-daemon
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359869] [ 1341] 0 1341 61114 1948 13 0 0 newrelic-daemon
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359871] [ 1381] 0 1381 3310 100 6 0 0 mdadm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359874] [ 1444] 0 1444 3626 189 0 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359877] [ 1445] 0 1445 3188 177 3 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359879] [ 1890] 0 1890 369259 2517 3 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359882] [ 1916] 0 1916 20056 1040 8 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359885] [ 1917] 33 1917 20473 1994 8 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359887] [ 1918] 33 1918 20399 1927 0 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359890] [ 1919] 33 1919 20454 2020 7 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359893] [ 1920] 33 1920 20201 1775 5 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359896] [ 6544] 33 6544 624267 32944 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359898] [ 7293] 33 7293 618265 25932 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359901] [ 7381] 33 7381 618555 26851 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359904] [ 8280] 33 8280 618008 27712 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359906] [ 8737] 33 8737 619699 27793 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359909] [ 8908] 33 8908 618037 26037 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359912] [ 9093] 33 9093 618471 25848 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359914] [ 9479] 33 9479 688805 75927 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359917] [ 9503] 33 9503 5636713 4312476 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359919] [10392] 33 10392 621196 28967 13 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359922] [10509] 33 10509 618070 26222 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359925] [10510] 33 10510 622084 30147 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359927] [10540] 33 10540 620672 29171 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359930] [10799] 33 10799 670237 78421 4 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359932] [11066] 33 11066 616125 23836 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359935] [11250] 33 11250 619613 27729 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359937] [11254] 33 11254 619605 27515 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359940] [11426] 33 11426 620759 28353 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359943] [11428] 33 11428 618500 24942 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359945] [11429] 33 11429 832300 131639 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359948] [11444] 33 11444 390293 26564 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359951] [12119] 33 12119 389579 26295 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359953] [12542] 33 12542 390336 27483 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359956] [12577] 33 12577 1770706 1177669 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359958] [13310] 33 13310 1559195 1166076 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359961] [13481] 33 13481 729336 121410 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359963] [14074] 33 14074 613987 21729 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359966] [14686] 33 14686 405943 20594 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359968] [14771] 33 14771 1341333 17036 10 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359971] [14814] 33 14814 375034 16151 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359973] [14888] 33 14888 374759 15618 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359976] [14917] 33 14917 382071 16366 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359979] [14919] 33 14919 373976 14146 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359981] [14982] 33 14982 373012 11466 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359984] [14983] 33 14983 372940 10379 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359986] [14984] 33 14984 374522 12917 14 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359989] [14985] 33 14985 372898 9890 7 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359991] [14986] 33 14986 373733 11996 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359994] [14987] 33 14987 373203 12061 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359996] [14988] 33 14988 373368 12404 13 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359999] [14989] 33 14989 372407 8838 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360002] [14990] 33 14990 373491 11722 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360004] [14991] 33 14991 371987 8545 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360007] [14992] 33 14992 372184 9768 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360010] [14993] 33 14993 375221 12528 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360012] [15011] 33 15011 372181 8558 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360015] [15012] 33 15012 374863 13426 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360018] [15013] 33 15013 372497 10278 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360020] [15014] 33 15014 372489 9905 8 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360023] [15015] 33 15015 373200 11024 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360026] [15017] 33 15017 372621 9203 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360028] [15018] 33 15018 371993 9152 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360031] [15019] 33 15019 374082 10949 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360033] [15020] 33 15020 373137 10112 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360036] [15021] 33 15021 377055 14606 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360038] [15027] 33 15027 372243 9148 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360041] [15028] 33 15028 373802 11787 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360044] [15029] 33 15029 372094 7847 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360046] [15030] 33 15030 372093 7475 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360049] [15031] 33 15031 372606 9650 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360071] Out of memory: Kill process 9503 (php5-fpm) score 564 or sacrifice child
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.363179] Killed process 9503 (php5-fpm) total-vm:22546852kB, anon-rss:17203684kB, file-rss:46220kB
Seems like the issue is with php5-fpm overcommiting the memory for some cases.
How do I ensure that such cases, doesn't create HIGH DISK IO / Server Unstable?
php-fpm php5 dmesg
asked Feb 1 '15 at 4:56
user268083user268083
2612
One of my server (Ubuntu 12.04, 16CPU, 32GB RAM) runs the app section of my website (Nginx 1.1.19, PHP5-FPM, Php 5.3.10, Symfony2 Web Framework)
Suddenly, users started getting HTTP 5** Errors. When I went on the server, there was huge disk IO and the culprit was [flush].
When I went through the kernel logs using dmesg, it showed the reason for [flush] was php5-fpm invoked oom-killer
Kernel Log entry from the time of Incident:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295736] php5-fpm invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295740] php5-fpm cpuset=/ mems_allowed=0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295743] Pid: 13310, comm: php5-fpm Not tainted 3.2.0-69-virtual #103-Ubuntu
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295744] Call Trace:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295751] [<ffffffff8111b971>] dump_header+0x91/0xe0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295754] [<ffffffff8111bd25>] oom_kill_process+0x85/0xb0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295757] [<ffffffff8111c0ca>] out_of_memory+0xfa/0x220
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295761] [<ffffffff81121aa3>] __alloc_pages_nodemask+0x8c3/0x8e0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295764] [<ffffffff8115a853>] alloc_pages_vma+0xb3/0x190
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295768] [<ffffffff8113ae9c>] do_anonymous_page.isra.38+0x7c/0x2f0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295772] [<ffffffff8113e921>] handle_pte_fault+0x1e1/0x200
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295777] [<ffffffff8165d9be>] ? _raw_spin_lock+0xe/0x20
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295780] [<ffffffff8113c089>] ? __pte_alloc+0xa9/0x160
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295782] [<ffffffff8113fb19>] handle_mm_fault+0x269/0x370
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295785] [<ffffffff81661504>] do_page_fault+0x184/0x550
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295789] [<ffffffff81013728>] ? __switch_to+0x138/0x360
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295793] [<ffffffff8105668d>] ? set_next_entity+0xad/0xd0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295796] [<ffffffff8105682a>] ? finish_task_switch+0x4a/0xf0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295799] [<ffffffff8165b6ec>] ? __schedule+0x3cc/0x6f0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295802] [<ffffffff8165e135>] page_fault+0x25/0x30
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295804] Mem-Info:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295805] Node 0 DMA per-cpu:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295808] CPU 0: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295809] CPU 1: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295810] CPU 2: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295812] CPU 3: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295813] CPU 4: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295814] CPU 5: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295816] CPU 6: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295817] CPU 7: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295819] CPU 8: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295820] CPU 9: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295822] CPU 10: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295823] CPU 11: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295824] CPU 12: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295826] CPU 13: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295827] CPU 14: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295828] CPU 15: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295830] Node 0 DMA32 per-cpu:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295831] CPU 0: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295833] CPU 1: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295834] CPU 2: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295836] CPU 3: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295837] CPU 4: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295838] CPU 5: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295840] CPU 6: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295841] CPU 7: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295842] CPU 8: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295844] CPU 9: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295845] CPU 10: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295846] CPU 11: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295848] CPU 12: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295850] CPU 13: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295851] CPU 14: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295853] CPU 15: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295854] Node 0 Normal per-cpu:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295856] CPU 0: hi: 186, btch: 31 usd: 15
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295857] CPU 1: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295859] CPU 2: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295860] CPU 3: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295861] CPU 4: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295863] CPU 5: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295864] CPU 6: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295865] CPU 7: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295867] CPU 8: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295868] CPU 9: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295870] CPU 10: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295871] CPU 11: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295872] CPU 12: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295874] CPU 13: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295875] CPU 14: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295877] CPU 15: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295880] active_anon:7408050 inactive_anon:436 isolated_anon:0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295881] active_file:21159 inactive_file:33077 isolated_file:419
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295882] unevictable:0 dirty:36283 writeback:0 unstable:0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295882] free:47243 slab_reclaimable:20065 slab_unreclaimable:15299
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295883] mapped:15592 shmem:138 pagetables:28362 bounce:0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295885] Node 0 DMA free:15908kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295892] lowmem_reserve[]: 0 3760 30220 30220
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295895] Node 0 DMA32 free:114216kB min:8404kB low:10504kB high:12604kB active_anon:3651120kB inactive_anon:284kB active_file:7848kB inactive_file:38464kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3850496kB mlocked:0kB dirty:38664kB writeback:0kB mapped:272kB shmem:0kB slab_reclaimable:8444kB slab_unreclaimable:7164kB kernel_stack:208kB pagetables:10056kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:116761 all_unreclaimable? yes
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295902] lowmem_reserve[]: 0 0 26460 26460
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295905] Node 0 Normal free:58848kB min:59144kB low:73928kB high:88716kB active_anon:25981080kB inactive_anon:1460kB active_file:76788kB inactive_file:93844kB unevictable:0kB isolated(anon):0kB isolated(file):1676kB present:27095040kB mlocked:0kB dirty:106468kB writeback:0kB mapped:62096kB shmem:552kB slab_reclaimable:71816kB slab_unreclaimable:54032kB kernel_stack:2352kB pagetables:103392kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:388409 all_unreclaimable? yes
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295912] lowmem_reserve[]: 0 0 0 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295915] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15908kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295922] Node 0 DMA32: 268*4kB 544*8kB 367*16kB 391*32kB 271*64kB 165*128kB 103*256kB 32*512kB 5*1024kB 0*2048kB 1*4096kB = 114240kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295929] Node 0 Normal: 13697*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 58884kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295937] 54555 total pagecache pages
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295938] 0 pages in swap cache
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295939] Swap cache stats: add 0, delete 0, find 0/0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295940] Free swap = 0kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295941] Total swap = 0kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359778] 7864304 pages RAM
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359780] 144336 pages reserved
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359781] 654931 pages shared
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359782] 7618451 pages non-shared
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359783] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359798] [ 445] 0 445 4800 221 4 0 0 rpcbind
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359801] [ 452] 108 452 5376 279 2 0 0 rpc.statd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359804] [ 482] 0 482 4308 94 0 0 0 upstart-udev-br
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359807] [ 494] 0 494 5431 327 1 -17 -1000 udevd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359809] [ 608] 0 608 5430 231 1 -17 -1000 udevd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359812] [ 624] 0 624 5430 233 3 -17 -1000 udevd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359815] [ 706] 0 706 3797 49 3 0 0 upstart-socket-
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359818] [ 882] 0 882 1816 213 6 0 0 dhclient3
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359821] [ 993] 0 993 12509 499 2 -17 -1000 sshd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359824] [ 1018] 101 1018 63430 480 4 0 0 rsyslogd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359827] [ 1040] 102 1040 5954 155 13 0 0 dbus-daemon
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359830] [ 1048] 0 1048 7445 153 9 0 0 rpc.idmapd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359832] [ 1151] 0 1151 3626 190 3 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359835] [ 1157] 0 1157 3626 189 1 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359837] [ 1162] 0 1162 3626 189 5 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359840] [ 1163] 0 1163 3626 188 1 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359842] [ 1165] 0 1165 3626 188 6 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359845] [ 1170] 0 1170 4778 211 4 0 0 cron
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359847] [ 1171] 0 1171 4227 89 12 0 0 atd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359850] [ 1172] 0 1172 1082 149 10 0 0 acpid
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359852] [ 1178] 0 1178 3995 173 0 0 0 irqbalance
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359855] [ 1179] 103 1179 46916 650 10 0 0 whoopsie
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359858] [ 1195] 106 1195 80805 236 5 0 0 memcached
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359861] [ 1228] 109 1228 4199 99 0 0 0 nrsysmond
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359863] [ 1230] 109 1230 62851 1327 0 0 0 nrsysmond
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359866] [ 1340] 0 1340 4164 105 9 0 0 newrelic-daemon
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359869] [ 1341] 0 1341 61114 1948 13 0 0 newrelic-daemon
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359871] [ 1381] 0 1381 3310 100 6 0 0 mdadm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359874] [ 1444] 0 1444 3626 189 0 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359877] [ 1445] 0 1445 3188 177 3 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359879] [ 1890] 0 1890 369259 2517 3 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359882] [ 1916] 0 1916 20056 1040 8 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359885] [ 1917] 33 1917 20473 1994 8 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359887] [ 1918] 33 1918 20399 1927 0 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359890] [ 1919] 33 1919 20454 2020 7 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359893] [ 1920] 33 1920 20201 1775 5 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359896] [ 6544] 33 6544 624267 32944 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359898] [ 7293] 33 7293 618265 25932 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359901] [ 7381] 33 7381 618555 26851 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359904] [ 8280] 33 8280 618008 27712 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359906] [ 8737] 33 8737 619699 27793 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359909] [ 8908] 33 8908 618037 26037 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359912] [ 9093] 33 9093 618471 25848 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359914] [ 9479] 33 9479 688805 75927 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359917] [ 9503] 33 9503 5636713 4312476 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359919] [10392] 33 10392 621196 28967 13 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359922] [10509] 33 10509 618070 26222 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359925] [10510] 33 10510 622084 30147 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359927] [10540] 33 10540 620672 29171 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359930] [10799] 33 10799 670237 78421 4 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359932] [11066] 33 11066 616125 23836 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359935] [11250] 33 11250 619613 27729 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359937] [11254] 33 11254 619605 27515 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359940] [11426] 33 11426 620759 28353 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359943] [11428] 33 11428 618500 24942 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359945] [11429] 33 11429 832300 131639 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359948] [11444] 33 11444 390293 26564 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359951] [12119] 33 12119 389579 26295 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359953] [12542] 33 12542 390336 27483 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359956] [12577] 33 12577 1770706 1177669 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359958] [13310] 33 13310 1559195 1166076 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359961] [13481] 33 13481 729336 121410 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359963] [14074] 33 14074 613987 21729 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359966] [14686] 33 14686 405943 20594 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359968] [14771] 33 14771 1341333 17036 10 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359971] [14814] 33 14814 375034 16151 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359973] [14888] 33 14888 374759 15618 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359976] [14917] 33 14917 382071 16366 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359979] [14919] 33 14919 373976 14146 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359981] [14982] 33 14982 373012 11466 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359984] [14983] 33 14983 372940 10379 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359986] [14984] 33 14984 374522 12917 14 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359989] [14985] 33 14985 372898 9890 7 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359991] [14986] 33 14986 373733 11996 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359994] [14987] 33 14987 373203 12061 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359996] [14988] 33 14988 373368 12404 13 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359999] [14989] 33 14989 372407 8838 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360002] [14990] 33 14990 373491 11722 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360004] [14991] 33 14991 371987 8545 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360007] [14992] 33 14992 372184 9768 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360010] [14993] 33 14993 375221 12528 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360012] [15011] 33 15011 372181 8558 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360015] [15012] 33 15012 374863 13426 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360018] [15013] 33 15013 372497 10278 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360020] [15014] 33 15014 372489 9905 8 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360023] [15015] 33 15015 373200 11024 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360026] [15017] 33 15017 372621 9203 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360028] [15018] 33 15018 371993 9152 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360031] [15019] 33 15019 374082 10949 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360033] [15020] 33 15020 373137 10112 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360036] [15021] 33 15021 377055 14606 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360038] [15027] 33 15027 372243 9148 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360041] [15028] 33 15028 373802 11787 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360044] [15029] 33 15029 372094 7847 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360046] [15030] 33 15030 372093 7475 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360049] [15031] 33 15031 372606 9650 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360071] Out of memory: Kill process 9503 (php5-fpm) score 564 or sacrifice child
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.363179] Killed process 9503 (php5-fpm) total-vm:22546852kB, anon-rss:17203684kB, file-rss:46220kB
Seems like the issue is with php5-fpm overcommiting the memory for some cases.
How do I ensure that such cases, doesn't create HIGH DISK IO / Server Unstable?
php-fpm php5 dmesg
php-fpm php5 dmesg
asked Feb 1 '15 at 4:56
user268083user268083
2612
asked Feb 1 '15 at 4:56
user268083user268083
2612
asked Feb 1 '15 at 4:56
user268083user268083
2612
asked Feb 1 '15 at 4:56
user268083user268083
2612
asked Feb 1 '15 at 4:56
user268083user268083
2612
2612
bumped to the homepage by Community♦ 14 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 14 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
0
You need to make sure that you don't run too many PHP-FPM processes on your server. Look at pm.max_children, pm.min_spare_servers and pm.max_spare_servers settings in your PHP-FPM pool configuration.
answered Feb 1 '15 at 10:54
Tero KilkanenTero Kilkanen
20.5k22644
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "2"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f664062%2fphp5-fpm-invoked-oom-killer%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
0
You need to make sure that you don't run too many PHP-FPM processes on your server. Look at pm.max_children, pm.min_spare_servers and pm.max_spare_servers settings in your PHP-FPM pool configuration.
answered Feb 1 '15 at 10:54
Tero KilkanenTero Kilkanen
20.5k22644
add a comment |
0
You need to make sure that you don't run too many PHP-FPM processes on your server. Look at pm.max_children, pm.min_spare_servers and pm.max_spare_servers settings in your PHP-FPM pool configuration.
answered Feb 1 '15 at 10:54
Tero KilkanenTero Kilkanen
20.5k22644
add a comment |
0
0
0
You need to make sure that you don't run too many PHP-FPM processes on your server. Look at pm.max_children, pm.min_spare_servers and pm.max_spare_servers settings in your PHP-FPM pool configuration.
answered Feb 1 '15 at 10:54
Tero KilkanenTero Kilkanen
20.5k22644
You need to make sure that you don't run too many PHP-FPM processes on your server. Look at pm.max_children, pm.min_spare_servers and pm.max_spare_servers settings in your PHP-FPM pool configuration.
answered Feb 1 '15 at 10:54
Tero KilkanenTero Kilkanen
20.5k22644
answered Feb 1 '15 at 10:54
Tero KilkanenTero Kilkanen
20.5k22644
answered Feb 1 '15 at 10:54
Tero KilkanenTero Kilkanen
20.5k22644
answered Feb 1 '15 at 10:54
Tero KilkanenTero Kilkanen
20.5k22644
20.5k22644
add a comment |
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Server Fault!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f664062%2fphp5-fpm-invoked-oom-killer%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
