apache http 2.2.29: Segmentation faultPHP Causing Segmentation Fault & Apache Blank ResponseApache...
Did Shadowfax go to Valinor?
Watching something be written to a file live with tail
Perform and show arithmetic with LuaLaTeX
Why does Kotter return in Welcome Back Kotter?
Why "Having chlorophyll without photosynthesis is actually very dangerous" and "like living with a bomb"?
Languages that we cannot (dis)prove to be Context-Free
Can a vampire attack twice with their claws using Multiattack?
When a company launches a new product do they "come out" with a new product or do they "come up" with a new product?
Do I have a twin with permutated remainders?
Can an x86 CPU running in real mode be considered to be basically an 8086 CPU?
LWC SFDX source push error TypeError: LWC1009: decl.moveTo is not a function
How does quantile regression compare to logistic regression with the variable split at the quantile?
Is it possible to run Internet Explorer on OS X El Capitan?
Why can't we play rap on piano?
Why is Minecraft giving an OpenGL error?
Can I make popcorn with any corn?
NMaximize is not converging to a solution
How is it possible to have an ability score that is less than 3?
Is it unprofessional to ask if a job posting on GlassDoor is real?
Why doesn't H₄O²⁺ exist?
How to determine what difficulty is right for the game?
Has there ever been an airliner design involving reducing generator load by installing solar panels?
If human space travel is limited by the G force vulnerability, is there a way to counter G forces?
Are the number of citations and number of published articles the most important criteria for a tenure promotion?
apache http 2.2.29: Segmentation fault
PHP Causing Segmentation Fault & Apache Blank ResponseApache segmentation faultApache fails to start due to Segmentation faultApache Segmentation Faultchild pid … exit signal Segmentation fault (11)apache child pid exit signal Segmentation fault (11)child pid xxxx exit signal Segmentation faultApache2 configtest segmentation faultApache access log full of unauthorized and suspicious requests, how to take actionimagemagick segmentation fault after apache graceful restart
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
We are using apache http server 2.2.29 on Linux. Below is the information related to Linux version (as found in /proc/version file)
Linux version 2.6.32-573.el6.x86_64 (mockbuild@x86-027.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed Jul 1 18:23:37 EDT 2015
We have observed that apache is generating many core dumps and getting crashed regularly. When it generates the core dump, below is being printed in the http’s error log:
[Mon Jun 20 01:28:09 2016] [notice] child pid 6520 exit signal Segmentation fault (11), possible coredump in /app/http_instance_name
[Mon Jun 20 03:27:54 2016] [notice] child pid 6462 exit signal Segmentation fault (11), possible coredump in /app/http_instance_name
Below core files had generated at these segmentation faults.
-rw------- 1 http http 65130496 Jun 20 01:28 core.6520
-rw------- 1 http http 66134016 Jun 20 03:27 core.6462
We are running on below Apache http version:
Server version: Apache/2.2.29 (Unix)
Server built: Mar 4 2016 15:21:54
I have tried to analyse the core files using gdb tool , however, i was not able to understand much from it?
Please let me know if further information is required help get more understanding pertinent to this issue.
can this be due to some modules used in httpd.conf file? We are observing this issue across all our environments.
UPDATE: Common Pattern
The most common pattern I am observing when the segmentation fault error being is thrown is , the below error comes for most of the cases just before the segmentation fault appears
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] (-2)Unknown error 18446744073709551614: proxy: error reading status line from remote server rxxxx.xxxx.net:8543, referer: https://sit1-xxxxxx.co.uk:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] proxy: Error reading from remote server returned by /CommonWeb/pages/common/includes/js/dojo.js, referer: https://sit1.xxxxxx:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] (-2)Unknown error 18446744073709551614: proxy: error reading status line from remote server rpovassxxxx.srxxx.net:8543, referer: https://sit1-****:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] proxy: Error reading from remote server returned by /CommonWeb/pages/common/includes/js/resize.js, referer: https://xxxx-xxxx:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:03 2016] [notice] child pid 28308 exit signal Segmentation fault (11), possible coredump in /app/
Below is the output of 'gdb full' during debugging:
#0 0x0000003704a0e7dd in read () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x000000000044f8b7 in ap_mpm_pod_check (pod=<value optimized out>) at pod.c:54
c = <value optimized out>
fd = 6
rc = <value optimized out>
#2 0x000000000044de74 in child_main (child_num_arg=1) at worker.c:1259
threads = 0xc2bab0
rv = <value optimized out>
ts = 0xc079f8
thread_attr = 0xc07a18
start_thread_id = <value optimized out>
#3 0x000000000044e0dd in make_child (s=0xb44870, slot=1) at worker.c:1342
pid = 0
#4 0x000000000044efc0 in perform_idle_server_maintenance (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>)
at worker.c:1568
j = <value optimized out>
free_slots = {1, 5, 6, 7, 11817336, 0, 11784504, 0, -1236161560, 11137, 11817336, 0, 11800920, 0, 11815024, 0, 11784504, 0, -1361903789, 11137,
4337480, 0, -1238262800, 11137, 29, 0, 12768544, 0, 11815024, 0, 12489224, 0}
total_non_dead = <value optimized out>
active_thread_count = <value optimized out>
idle_thread_count = <value optimized out>
free_length = 1
i = <value optimized out>
ps = <value optimized out>
totally_free_length = <value optimized out>
last_non_dead = <value optimized out>
#5 server_main_loop (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at worker.c:1680
status = 11
pid = {pid = -1, in = 0xb45178, out = 0xb41158, err = 0xb3d138}
i = <value optimized out>
child_slot = <value optimized out>
exitwhy = 6
processed_status = <value optimized out>
#6 ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at worker.c:1782
remaining_children_to_start = <value optimized out>
rv = <value optimized out>
#7 0x0000000000425e34 in main (argc=3, argv=0x7ffcafb44eb8) at main.c:753
c = 0 '00'
configtestonly = <value optimized out>
confname = 0x458a64 "conf/httpd.conf"
def_server_root = 0x458a4f "/app/***"
temp_error_log = 0x0
error = <value optimized out>
process = 0xb44870
server_conf = 0xb44870
pglobal = 0xb3b128
pconf = 0xb3d138
plog = 0xb45178
ptemp = 0xb41158
pcommands = 0xb3f148
opt = 0xb3f238
rv = <value optimized out>
mod = <value optimized out>
optarg = 0x0
signal_server = <value optimized out>'
Below modules are being used in httpd.conf file.
`cat httpd.conf | grep LoadMo
# have to place corresponding LoadModule' lines at this location so the
# LoadModule foo_module modules/mod_foo.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_dbd_module modules/mod_authn_dbd.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule dbd_module modules/mod_dbd.so
LoadModule dumpio_module modules/mod_dumpio.so
LoadModule reqtimeout_module modules/mod_reqtimeout.so
LoadModule ext_filter_module modules/mod_ext_filter.so
LoadModule include_module modules/mod_include.so
LoadModule filter_module modules/mod_filter.so
LoadModule substitute_module modules/mod_substitute.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule log_forensic_module modules/mod_log_forensic.so
LoadModule logio_module modules/mod_logio.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule cern_meta_module modules/mod_cern_meta.so
LoadModule expires_module modules/mod_expires.so
LoadModule headers_module modules/mod_headers.so
LoadModule ident_module modules/mod_ident.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule unique_id_module modules/mod_unique_id.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule version_module modules/mod_version.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_scgi_module modules/mod_proxy_scgi.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
#LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule mime_module modules/mod_mime.so
LoadModule dav_module modules/mod_dav.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule info_module modules/mod_info.so
LoadModule cgid_module modules/mod_cgid.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule imagemap_module modules/mod_imagemap.so
LoadModule actions_module modules/mod_actions.so
LoadModule speling_module modules/mod_speling.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule auth_kerb_module modules/mod_auth_kerb.so
LoadModule manager_module modules/mod_manager.so
LoadModule slotmem_module modules/mod_slotmem.so
LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
LoadModule advertise_module modules/mod_advertise.so
Below is the output of httpd -V (capital V)
./httpd -V
Server version: Apache/2.2.29 (Unix)
Server built: Jan 13 2016 15:10:57
Server's Module Magic Number: 20051115:36
Server loaded: APR 1.5.1, APR-Util 1.5.3
Compiled using: APR 1.5.1, APR-Util 1.5.3
Architecture: 64-bit
Server MPM: Worker
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/worker"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/app/****"
-D SUEXEC_BIN="/app/****/bin/suexec"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
**OUTPUT of file * **
httpd.exp: script text executable for .
mod_actions.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_advertise.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_alias.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_auth_basic.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_auth_digest.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_auth_kerb.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_anon.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_dbd.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_dbm.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_default.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_file.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_dbm.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_default.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_groupfile.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_host.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_owner.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_user.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_autoindex.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_cern_meta.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_cgid.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_cgi.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dav_fs.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dav.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dbd.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_deflate.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dir.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dumpio.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_expires.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_ext_filter.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_filter.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_headers.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_ident.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_imagemap.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_include.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_info.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_log_config.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_log_forensic.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_logio.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_manager.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_mime_magic.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_mime.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_negotiation.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_ajp.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_balancer.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_cluster.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_connect.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_ftp.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_http.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_scgi.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_reqtimeout.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_rewrite.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_setenvif.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_slotmem.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_speling.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_ssl.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_status.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_substitute.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_unique_id.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_userdir.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_usertrack.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_version.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_vhost_alias.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
apache-2.2 segmentation-fault
asked Jun 20 '16 at 16:50
user2607367user2607367
65
bumped to the homepage by Community♦ 3 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
|
show 10 more comments
We are using apache http server 2.2.29 on Linux. Below is the information related to Linux version (as found in /proc/version file)
Linux version 2.6.32-573.el6.x86_64 (mockbuild@x86-027.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed Jul 1 18:23:37 EDT 2015
We have observed that apache is generating many core dumps and getting crashed regularly. When it generates the core dump, below is being printed in the http’s error log:
[Mon Jun 20 01:28:09 2016] [notice] child pid 6520 exit signal Segmentation fault (11), possible coredump in /app/http_instance_name
[Mon Jun 20 03:27:54 2016] [notice] child pid 6462 exit signal Segmentation fault (11), possible coredump in /app/http_instance_name
Below core files had generated at these segmentation faults.
-rw------- 1 http http 65130496 Jun 20 01:28 core.6520
-rw------- 1 http http 66134016 Jun 20 03:27 core.6462
We are running on below Apache http version:
Server version: Apache/2.2.29 (Unix)
Server built: Mar 4 2016 15:21:54
I have tried to analyse the core files using gdb tool , however, i was not able to understand much from it?
Please let me know if further information is required help get more understanding pertinent to this issue.
can this be due to some modules used in httpd.conf file? We are observing this issue across all our environments.
UPDATE: Common Pattern
The most common pattern I am observing when the segmentation fault error being is thrown is , the below error comes for most of the cases just before the segmentation fault appears
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] (-2)Unknown error 18446744073709551614: proxy: error reading status line from remote server rxxxx.xxxx.net:8543, referer: https://sit1-xxxxxx.co.uk:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] proxy: Error reading from remote server returned by /CommonWeb/pages/common/includes/js/dojo.js, referer: https://sit1.xxxxxx:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] (-2)Unknown error 18446744073709551614: proxy: error reading status line from remote server rpovassxxxx.srxxx.net:8543, referer: https://sit1-****:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] proxy: Error reading from remote server returned by /CommonWeb/pages/common/includes/js/resize.js, referer: https://xxxx-xxxx:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:03 2016] [notice] child pid 28308 exit signal Segmentation fault (11), possible coredump in /app/
Below is the output of 'gdb full' during debugging:
#0 0x0000003704a0e7dd in read () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x000000000044f8b7 in ap_mpm_pod_check (pod=<value optimized out>) at pod.c:54
c = <value optimized out>
fd = 6
rc = <value optimized out>
#2 0x000000000044de74 in child_main (child_num_arg=1) at worker.c:1259
threads = 0xc2bab0
rv = <value optimized out>
ts = 0xc079f8
thread_attr = 0xc07a18
start_thread_id = <value optimized out>
#3 0x000000000044e0dd in make_child (s=0xb44870, slot=1) at worker.c:1342
pid = 0
#4 0x000000000044efc0 in perform_idle_server_maintenance (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>)
at worker.c:1568
j = <value optimized out>
free_slots = {1, 5, 6, 7, 11817336, 0, 11784504, 0, -1236161560, 11137, 11817336, 0, 11800920, 0, 11815024, 0, 11784504, 0, -1361903789, 11137,
4337480, 0, -1238262800, 11137, 29, 0, 12768544, 0, 11815024, 0, 12489224, 0}
total_non_dead = <value optimized out>
active_thread_count = <value optimized out>
idle_thread_count = <value optimized out>
free_length = 1
i = <value optimized out>
ps = <value optimized out>
totally_free_length = <value optimized out>
last_non_dead = <value optimized out>
#5 server_main_loop (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at worker.c:1680
status = 11
pid = {pid = -1, in = 0xb45178, out = 0xb41158, err = 0xb3d138}
i = <value optimized out>
child_slot = <value optimized out>
exitwhy = 6
processed_status = <value optimized out>
#6 ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at worker.c:1782
remaining_children_to_start = <value optimized out>
rv = <value optimized out>
#7 0x0000000000425e34 in main (argc=3, argv=0x7ffcafb44eb8) at main.c:753
c = 0 '00'
configtestonly = <value optimized out>
confname = 0x458a64 "conf/httpd.conf"
def_server_root = 0x458a4f "/app/***"
temp_error_log = 0x0
error = <value optimized out>
process = 0xb44870
server_conf = 0xb44870
pglobal = 0xb3b128
pconf = 0xb3d138
plog = 0xb45178
ptemp = 0xb41158
pcommands = 0xb3f148
opt = 0xb3f238
rv = <value optimized out>
mod = <value optimized out>
optarg = 0x0
signal_server = <value optimized out>'
Below modules are being used in httpd.conf file.
`cat httpd.conf | grep LoadMo
# have to place corresponding LoadModule' lines at this location so the
# LoadModule foo_module modules/mod_foo.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_dbd_module modules/mod_authn_dbd.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule dbd_module modules/mod_dbd.so
LoadModule dumpio_module modules/mod_dumpio.so
LoadModule reqtimeout_module modules/mod_reqtimeout.so
LoadModule ext_filter_module modules/mod_ext_filter.so
LoadModule include_module modules/mod_include.so
LoadModule filter_module modules/mod_filter.so
LoadModule substitute_module modules/mod_substitute.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule log_forensic_module modules/mod_log_forensic.so
LoadModule logio_module modules/mod_logio.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule cern_meta_module modules/mod_cern_meta.so
LoadModule expires_module modules/mod_expires.so
LoadModule headers_module modules/mod_headers.so
LoadModule ident_module modules/mod_ident.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule unique_id_module modules/mod_unique_id.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule version_module modules/mod_version.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_scgi_module modules/mod_proxy_scgi.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
#LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule mime_module modules/mod_mime.so
LoadModule dav_module modules/mod_dav.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule info_module modules/mod_info.so
LoadModule cgid_module modules/mod_cgid.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule imagemap_module modules/mod_imagemap.so
LoadModule actions_module modules/mod_actions.so
LoadModule speling_module modules/mod_speling.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule auth_kerb_module modules/mod_auth_kerb.so
LoadModule manager_module modules/mod_manager.so
LoadModule slotmem_module modules/mod_slotmem.so
LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
LoadModule advertise_module modules/mod_advertise.so
Below is the output of httpd -V (capital V)
./httpd -V
Server version: Apache/2.2.29 (Unix)
Server built: Jan 13 2016 15:10:57
Server's Module Magic Number: 20051115:36
Server loaded: APR 1.5.1, APR-Util 1.5.3
Compiled using: APR 1.5.1, APR-Util 1.5.3
Architecture: 64-bit
Server MPM: Worker
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/worker"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/app/****"
-D SUEXEC_BIN="/app/****/bin/suexec"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
**OUTPUT of file * **
httpd.exp: script text executable for .
mod_actions.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_advertise.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_alias.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_auth_basic.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_auth_digest.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_auth_kerb.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_anon.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_dbd.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_dbm.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_default.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_file.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_dbm.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_default.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_groupfile.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_host.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_owner.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_user.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_autoindex.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_cern_meta.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_cgid.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_cgi.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dav_fs.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dav.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dbd.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_deflate.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dir.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dumpio.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_expires.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_ext_filter.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_filter.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_headers.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_ident.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_imagemap.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_include.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_info.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_log_config.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_log_forensic.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_logio.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_manager.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_mime_magic.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_mime.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_negotiation.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_ajp.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_balancer.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_cluster.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_connect.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_ftp.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_http.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_scgi.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_reqtimeout.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_rewrite.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_setenvif.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_slotmem.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_speling.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_ssl.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_status.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_substitute.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_unique_id.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_userdir.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_usertrack.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_version.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_vhost_alias.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
apache-2.2 segmentation-fault
asked Jun 20 '16 at 16:50
user2607367user2607367
65
bumped to the homepage by Community♦ 3 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
What modules are you loading and what is the output ofhttpd -Vandldd /path/to/httpd. You might also grab a few lines just prior to the segfault from the access logs from all these servers you are seeing this on and look for common patterns.
– Aaron
Jun 20 '16 at 19:19
I have updated the questions with the modules being used in httpd.conf file. httpd -v gives the same output as mentioned in the question. Server version: Apache/2.2.29 (Unix) Server built: Mar 4 2016 15:21:54
– user2607367
Jun 21 '16 at 9:59
Without a known bug for your software version, then the first thing you should be looking at is your core files, using GDB see also stackoverflow.com/questions/8305866/… (hint: use the command bt to a stack trace)
– symcbean
Jun 21 '16 at 10:43
I am not sure how to use gdb. Can you explain how to use it? I am not getting much help from the forum that you have mentioned.
– user2607367
Jun 21 '16 at 11:14
I have updated a question with a common pattern observed in the error logs when segmentation fault occurs. It seems to be occurring due a bug in mod_proxy.
– user2607367
Jun 21 '16 at 14:03
|
show 10 more comments
We are using apache http server 2.2.29 on Linux. Below is the information related to Linux version (as found in /proc/version file)
Linux version 2.6.32-573.el6.x86_64 (mockbuild@x86-027.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed Jul 1 18:23:37 EDT 2015
We have observed that apache is generating many core dumps and getting crashed regularly. When it generates the core dump, below is being printed in the http’s error log:
[Mon Jun 20 01:28:09 2016] [notice] child pid 6520 exit signal Segmentation fault (11), possible coredump in /app/http_instance_name
[Mon Jun 20 03:27:54 2016] [notice] child pid 6462 exit signal Segmentation fault (11), possible coredump in /app/http_instance_name
Below core files had generated at these segmentation faults.
-rw------- 1 http http 65130496 Jun 20 01:28 core.6520
-rw------- 1 http http 66134016 Jun 20 03:27 core.6462
We are running on below Apache http version:
Server version: Apache/2.2.29 (Unix)
Server built: Mar 4 2016 15:21:54
I have tried to analyse the core files using gdb tool , however, i was not able to understand much from it?
Please let me know if further information is required help get more understanding pertinent to this issue.
can this be due to some modules used in httpd.conf file? We are observing this issue across all our environments.
UPDATE: Common Pattern
The most common pattern I am observing when the segmentation fault error being is thrown is , the below error comes for most of the cases just before the segmentation fault appears
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] (-2)Unknown error 18446744073709551614: proxy: error reading status line from remote server rxxxx.xxxx.net:8543, referer: https://sit1-xxxxxx.co.uk:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] proxy: Error reading from remote server returned by /CommonWeb/pages/common/includes/js/dojo.js, referer: https://sit1.xxxxxx:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] (-2)Unknown error 18446744073709551614: proxy: error reading status line from remote server rpovassxxxx.srxxx.net:8543, referer: https://sit1-****:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] proxy: Error reading from remote server returned by /CommonWeb/pages/common/includes/js/resize.js, referer: https://xxxx-xxxx:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:03 2016] [notice] child pid 28308 exit signal Segmentation fault (11), possible coredump in /app/
Below is the output of 'gdb full' during debugging:
#0 0x0000003704a0e7dd in read () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x000000000044f8b7 in ap_mpm_pod_check (pod=<value optimized out>) at pod.c:54
c = <value optimized out>
fd = 6
rc = <value optimized out>
#2 0x000000000044de74 in child_main (child_num_arg=1) at worker.c:1259
threads = 0xc2bab0
rv = <value optimized out>
ts = 0xc079f8
thread_attr = 0xc07a18
start_thread_id = <value optimized out>
#3 0x000000000044e0dd in make_child (s=0xb44870, slot=1) at worker.c:1342
pid = 0
#4 0x000000000044efc0 in perform_idle_server_maintenance (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>)
at worker.c:1568
j = <value optimized out>
free_slots = {1, 5, 6, 7, 11817336, 0, 11784504, 0, -1236161560, 11137, 11817336, 0, 11800920, 0, 11815024, 0, 11784504, 0, -1361903789, 11137,
4337480, 0, -1238262800, 11137, 29, 0, 12768544, 0, 11815024, 0, 12489224, 0}
total_non_dead = <value optimized out>
active_thread_count = <value optimized out>
idle_thread_count = <value optimized out>
free_length = 1
i = <value optimized out>
ps = <value optimized out>
totally_free_length = <value optimized out>
last_non_dead = <value optimized out>
#5 server_main_loop (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at worker.c:1680
status = 11
pid = {pid = -1, in = 0xb45178, out = 0xb41158, err = 0xb3d138}
i = <value optimized out>
child_slot = <value optimized out>
exitwhy = 6
processed_status = <value optimized out>
#6 ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at worker.c:1782
remaining_children_to_start = <value optimized out>
rv = <value optimized out>
#7 0x0000000000425e34 in main (argc=3, argv=0x7ffcafb44eb8) at main.c:753
c = 0 '00'
configtestonly = <value optimized out>
confname = 0x458a64 "conf/httpd.conf"
def_server_root = 0x458a4f "/app/***"
temp_error_log = 0x0
error = <value optimized out>
process = 0xb44870
server_conf = 0xb44870
pglobal = 0xb3b128
pconf = 0xb3d138
plog = 0xb45178
ptemp = 0xb41158
pcommands = 0xb3f148
opt = 0xb3f238
rv = <value optimized out>
mod = <value optimized out>
optarg = 0x0
signal_server = <value optimized out>'
Below modules are being used in httpd.conf file.
`cat httpd.conf | grep LoadMo
# have to place corresponding LoadModule' lines at this location so the
# LoadModule foo_module modules/mod_foo.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_dbd_module modules/mod_authn_dbd.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule dbd_module modules/mod_dbd.so
LoadModule dumpio_module modules/mod_dumpio.so
LoadModule reqtimeout_module modules/mod_reqtimeout.so
LoadModule ext_filter_module modules/mod_ext_filter.so
LoadModule include_module modules/mod_include.so
LoadModule filter_module modules/mod_filter.so
LoadModule substitute_module modules/mod_substitute.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule log_forensic_module modules/mod_log_forensic.so
LoadModule logio_module modules/mod_logio.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule cern_meta_module modules/mod_cern_meta.so
LoadModule expires_module modules/mod_expires.so
LoadModule headers_module modules/mod_headers.so
LoadModule ident_module modules/mod_ident.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule unique_id_module modules/mod_unique_id.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule version_module modules/mod_version.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_scgi_module modules/mod_proxy_scgi.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
#LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule mime_module modules/mod_mime.so
LoadModule dav_module modules/mod_dav.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule info_module modules/mod_info.so
LoadModule cgid_module modules/mod_cgid.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule imagemap_module modules/mod_imagemap.so
LoadModule actions_module modules/mod_actions.so
LoadModule speling_module modules/mod_speling.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule auth_kerb_module modules/mod_auth_kerb.so
LoadModule manager_module modules/mod_manager.so
LoadModule slotmem_module modules/mod_slotmem.so
LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
LoadModule advertise_module modules/mod_advertise.so
Below is the output of httpd -V (capital V)
./httpd -V
Server version: Apache/2.2.29 (Unix)
Server built: Jan 13 2016 15:10:57
Server's Module Magic Number: 20051115:36
Server loaded: APR 1.5.1, APR-Util 1.5.3
Compiled using: APR 1.5.1, APR-Util 1.5.3
Architecture: 64-bit
Server MPM: Worker
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/worker"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/app/****"
-D SUEXEC_BIN="/app/****/bin/suexec"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
**OUTPUT of file * **
httpd.exp: script text executable for .
mod_actions.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_advertise.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_alias.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_auth_basic.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_auth_digest.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_auth_kerb.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_anon.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_dbd.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_dbm.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_default.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_file.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_dbm.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_default.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_groupfile.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_host.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_owner.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_user.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_autoindex.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_cern_meta.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_cgid.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_cgi.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dav_fs.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dav.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dbd.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_deflate.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dir.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dumpio.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_expires.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_ext_filter.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_filter.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_headers.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_ident.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_imagemap.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_include.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_info.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_log_config.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_log_forensic.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_logio.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_manager.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_mime_magic.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_mime.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_negotiation.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_ajp.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_balancer.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_cluster.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_connect.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_ftp.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_http.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_scgi.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_reqtimeout.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_rewrite.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_setenvif.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_slotmem.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_speling.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_ssl.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_status.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_substitute.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_unique_id.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_userdir.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_usertrack.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_version.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_vhost_alias.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
apache-2.2 segmentation-fault
asked Jun 20 '16 at 16:50
user2607367user2607367
65
We are using apache http server 2.2.29 on Linux. Below is the information related to Linux version (as found in /proc/version file)
Linux version 2.6.32-573.el6.x86_64 (mockbuild@x86-027.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed Jul 1 18:23:37 EDT 2015
We have observed that apache is generating many core dumps and getting crashed regularly. When it generates the core dump, below is being printed in the http’s error log:
[Mon Jun 20 01:28:09 2016] [notice] child pid 6520 exit signal Segmentation fault (11), possible coredump in /app/http_instance_name
[Mon Jun 20 03:27:54 2016] [notice] child pid 6462 exit signal Segmentation fault (11), possible coredump in /app/http_instance_name
Below core files had generated at these segmentation faults.
-rw------- 1 http http 65130496 Jun 20 01:28 core.6520
-rw------- 1 http http 66134016 Jun 20 03:27 core.6462
We are running on below Apache http version:
Server version: Apache/2.2.29 (Unix)
Server built: Mar 4 2016 15:21:54
I have tried to analyse the core files using gdb tool , however, i was not able to understand much from it?
Please let me know if further information is required help get more understanding pertinent to this issue.
can this be due to some modules used in httpd.conf file? We are observing this issue across all our environments.
UPDATE: Common Pattern
The most common pattern I am observing when the segmentation fault error being is thrown is , the below error comes for most of the cases just before the segmentation fault appears
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] (-2)Unknown error 18446744073709551614: proxy: error reading status line from remote server rxxxx.xxxx.net:8543, referer: https://sit1-xxxxxx.co.uk:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] proxy: Error reading from remote server returned by /CommonWeb/pages/common/includes/js/dojo.js, referer: https://sit1.xxxxxx:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] (-2)Unknown error 18446744073709551614: proxy: error reading status line from remote server rpovassxxxx.srxxx.net:8543, referer: https://sit1-****:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] proxy: Error reading from remote server returned by /CommonWeb/pages/common/includes/js/resize.js, referer: https://xxxx-xxxx:8443/CommonWeb/pages/common/jsp/login.jsp
[Mon May 16 16:08:03 2016] [notice] child pid 28308 exit signal Segmentation fault (11), possible coredump in /app/
Below is the output of 'gdb full' during debugging:
#0 0x0000003704a0e7dd in read () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x000000000044f8b7 in ap_mpm_pod_check (pod=<value optimized out>) at pod.c:54
c = <value optimized out>
fd = 6
rc = <value optimized out>
#2 0x000000000044de74 in child_main (child_num_arg=1) at worker.c:1259
threads = 0xc2bab0
rv = <value optimized out>
ts = 0xc079f8
thread_attr = 0xc07a18
start_thread_id = <value optimized out>
#3 0x000000000044e0dd in make_child (s=0xb44870, slot=1) at worker.c:1342
pid = 0
#4 0x000000000044efc0 in perform_idle_server_maintenance (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>)
at worker.c:1568
j = <value optimized out>
free_slots = {1, 5, 6, 7, 11817336, 0, 11784504, 0, -1236161560, 11137, 11817336, 0, 11800920, 0, 11815024, 0, 11784504, 0, -1361903789, 11137,
4337480, 0, -1238262800, 11137, 29, 0, 12768544, 0, 11815024, 0, 12489224, 0}
total_non_dead = <value optimized out>
active_thread_count = <value optimized out>
idle_thread_count = <value optimized out>
free_length = 1
i = <value optimized out>
ps = <value optimized out>
totally_free_length = <value optimized out>
last_non_dead = <value optimized out>
#5 server_main_loop (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at worker.c:1680
status = 11
pid = {pid = -1, in = 0xb45178, out = 0xb41158, err = 0xb3d138}
i = <value optimized out>
child_slot = <value optimized out>
exitwhy = 6
processed_status = <value optimized out>
#6 ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at worker.c:1782
remaining_children_to_start = <value optimized out>
rv = <value optimized out>
#7 0x0000000000425e34 in main (argc=3, argv=0x7ffcafb44eb8) at main.c:753
c = 0 '00'
configtestonly = <value optimized out>
confname = 0x458a64 "conf/httpd.conf"
def_server_root = 0x458a4f "/app/***"
temp_error_log = 0x0
error = <value optimized out>
process = 0xb44870
server_conf = 0xb44870
pglobal = 0xb3b128
pconf = 0xb3d138
plog = 0xb45178
ptemp = 0xb41158
pcommands = 0xb3f148
opt = 0xb3f238
rv = <value optimized out>
mod = <value optimized out>
optarg = 0x0
signal_server = <value optimized out>'
Below modules are being used in httpd.conf file.
`cat httpd.conf | grep LoadMo
# have to place corresponding LoadModule' lines at this location so the
# LoadModule foo_module modules/mod_foo.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_dbd_module modules/mod_authn_dbd.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule dbd_module modules/mod_dbd.so
LoadModule dumpio_module modules/mod_dumpio.so
LoadModule reqtimeout_module modules/mod_reqtimeout.so
LoadModule ext_filter_module modules/mod_ext_filter.so
LoadModule include_module modules/mod_include.so
LoadModule filter_module modules/mod_filter.so
LoadModule substitute_module modules/mod_substitute.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule log_forensic_module modules/mod_log_forensic.so
LoadModule logio_module modules/mod_logio.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule cern_meta_module modules/mod_cern_meta.so
LoadModule expires_module modules/mod_expires.so
LoadModule headers_module modules/mod_headers.so
LoadModule ident_module modules/mod_ident.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule unique_id_module modules/mod_unique_id.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule version_module modules/mod_version.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_scgi_module modules/mod_proxy_scgi.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
#LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule mime_module modules/mod_mime.so
LoadModule dav_module modules/mod_dav.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule info_module modules/mod_info.so
LoadModule cgid_module modules/mod_cgid.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule imagemap_module modules/mod_imagemap.so
LoadModule actions_module modules/mod_actions.so
LoadModule speling_module modules/mod_speling.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule auth_kerb_module modules/mod_auth_kerb.so
LoadModule manager_module modules/mod_manager.so
LoadModule slotmem_module modules/mod_slotmem.so
LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
LoadModule advertise_module modules/mod_advertise.so
Below is the output of httpd -V (capital V)
./httpd -V
Server version: Apache/2.2.29 (Unix)
Server built: Jan 13 2016 15:10:57
Server's Module Magic Number: 20051115:36
Server loaded: APR 1.5.1, APR-Util 1.5.3
Compiled using: APR 1.5.1, APR-Util 1.5.3
Architecture: 64-bit
Server MPM: Worker
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/worker"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/app/****"
-D SUEXEC_BIN="/app/****/bin/suexec"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
**OUTPUT of file * **
httpd.exp: script text executable for .
mod_actions.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_advertise.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_alias.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_auth_basic.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_auth_digest.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_auth_kerb.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_anon.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_dbd.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_dbm.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_default.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authn_file.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_dbm.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_default.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_groupfile.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_host.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_owner.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_authz_user.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_autoindex.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_cern_meta.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_cgid.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_cgi.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dav_fs.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dav.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dbd.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_deflate.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dir.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_dumpio.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_expires.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_ext_filter.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_filter.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_headers.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_ident.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_imagemap.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_include.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_info.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_log_config.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_log_forensic.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_logio.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_manager.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_mime_magic.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_mime.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_negotiation.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_ajp.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_balancer.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_cluster.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_connect.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_ftp.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_http.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy_scgi.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_proxy.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_reqtimeout.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_rewrite.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_setenvif.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_slotmem.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_speling.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_ssl.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_status.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_substitute.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_unique_id.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_userdir.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_usertrack.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_version.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
mod_vhost_alias.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
apache-2.2 segmentation-fault
apache-2.2 segmentation-fault
asked Jun 20 '16 at 16:50
user2607367user2607367
65
asked Jun 20 '16 at 16:50
user2607367user2607367
65
edited Jun 22 '16 at 16:28
user2607367
asked Jun 20 '16 at 16:50
user2607367user2607367
65
asked Jun 20 '16 at 16:50
user2607367user2607367
65
asked Jun 20 '16 at 16:50
user2607367user2607367
65
65
bumped to the homepage by Community♦ 3 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 3 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
What modules are you loading and what is the output ofhttpd -Vandldd /path/to/httpd. You might also grab a few lines just prior to the segfault from the access logs from all these servers you are seeing this on and look for common patterns.
– Aaron
Jun 20 '16 at 19:19
I have updated the questions with the modules being used in httpd.conf file. httpd -v gives the same output as mentioned in the question. Server version: Apache/2.2.29 (Unix) Server built: Mar 4 2016 15:21:54
– user2607367
Jun 21 '16 at 9:59
Without a known bug for your software version, then the first thing you should be looking at is your core files, using GDB see also stackoverflow.com/questions/8305866/… (hint: use the command bt to a stack trace)
– symcbean
Jun 21 '16 at 10:43
I am not sure how to use gdb. Can you explain how to use it? I am not getting much help from the forum that you have mentioned.
– user2607367
Jun 21 '16 at 11:14
I have updated a question with a common pattern observed in the error logs when segmentation fault occurs. It seems to be occurring due a bug in mod_proxy.
– user2607367
Jun 21 '16 at 14:03
|
show 10 more comments
What modules are you loading and what is the output ofhttpd -Vandldd /path/to/httpd. You might also grab a few lines just prior to the segfault from the access logs from all these servers you are seeing this on and look for common patterns.
– Aaron
Jun 20 '16 at 19:19
I have updated the questions with the modules being used in httpd.conf file. httpd -v gives the same output as mentioned in the question. Server version: Apache/2.2.29 (Unix) Server built: Mar 4 2016 15:21:54
– user2607367
Jun 21 '16 at 9:59
Without a known bug for your software version, then the first thing you should be looking at is your core files, using GDB see also stackoverflow.com/questions/8305866/… (hint: use the command bt to a stack trace)
– symcbean
Jun 21 '16 at 10:43
I am not sure how to use gdb. Can you explain how to use it? I am not getting much help from the forum that you have mentioned.
– user2607367
Jun 21 '16 at 11:14
I have updated a question with a common pattern observed in the error logs when segmentation fault occurs. It seems to be occurring due a bug in mod_proxy.
– user2607367
Jun 21 '16 at 14:03
What modules are you loading and what is the output of
httpd -V and ldd /path/to/httpd. You might also grab a few lines just prior to the segfault from the access logs from all these servers you are seeing this on and look for common patterns.– Aaron
Jun 20 '16 at 19:19
What modules are you loading and what is the output of
httpd -V and ldd /path/to/httpd. You might also grab a few lines just prior to the segfault from the access logs from all these servers you are seeing this on and look for common patterns.– Aaron
Jun 20 '16 at 19:19
I have updated the questions with the modules being used in httpd.conf file. httpd -v gives the same output as mentioned in the question. Server version: Apache/2.2.29 (Unix) Server built: Mar 4 2016 15:21:54
– user2607367
Jun 21 '16 at 9:59
I have updated the questions with the modules being used in httpd.conf file. httpd -v gives the same output as mentioned in the question. Server version: Apache/2.2.29 (Unix) Server built: Mar 4 2016 15:21:54
– user2607367
Jun 21 '16 at 9:59
Without a known bug for your software version, then the first thing you should be looking at is your core files, using GDB see also stackoverflow.com/questions/8305866/… (hint: use the command bt to a stack trace)
– symcbean
Jun 21 '16 at 10:43
Without a known bug for your software version, then the first thing you should be looking at is your core files, using GDB see also stackoverflow.com/questions/8305866/… (hint: use the command bt to a stack trace)
– symcbean
Jun 21 '16 at 10:43
I am not sure how to use gdb. Can you explain how to use it? I am not getting much help from the forum that you have mentioned.
– user2607367
Jun 21 '16 at 11:14
I am not sure how to use gdb. Can you explain how to use it? I am not getting much help from the forum that you have mentioned.
– user2607367
Jun 21 '16 at 11:14
I have updated a question with a common pattern observed in the error logs when segmentation fault occurs. It seems to be occurring due a bug in mod_proxy.
– user2607367
Jun 21 '16 at 14:03
I have updated a question with a common pattern observed in the error logs when segmentation fault occurs. It seems to be occurring due a bug in mod_proxy.
– user2607367
Jun 21 '16 at 14:03
|
show 10 more comments
1 Answer
1
active
oldest
votes
For GDB use https://httpd.apache.org/dev/debugging.html.
You would be able to trackdown the actual cause of the error.
If the crash is hard to reproduce it may be a good idea to configure Apache to only use one child processes for handling requests. The config is something like this:
StartServers 1
MinSpareServers 1
MaxSpareServers 1
Check your loaded configuration file- php.ini parameters.Configure it as per the server configuration(H/W resources)
Check if any page caching technique used like APC or memcached, if yes then allocate resources accordingly.
answered Jun 21 '16 at 12:36
Sachin SinghSachin Singh
1568
I have updated a question with a common pattern observed in the error logs when segmentation fault occurs. It seems to be occurring due a bug in mod_proxy.
– user2607367
Jun 21 '16 at 14:03
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "2"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f785078%2fapache-http-2-2-29-segmentation-fault%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
For GDB use https://httpd.apache.org/dev/debugging.html.
You would be able to trackdown the actual cause of the error.
If the crash is hard to reproduce it may be a good idea to configure Apache to only use one child processes for handling requests. The config is something like this:
StartServers 1
MinSpareServers 1
MaxSpareServers 1
Check your loaded configuration file- php.ini parameters.Configure it as per the server configuration(H/W resources)
Check if any page caching technique used like APC or memcached, if yes then allocate resources accordingly.
answered Jun 21 '16 at 12:36
Sachin SinghSachin Singh
1568
I have updated a question with a common pattern observed in the error logs when segmentation fault occurs. It seems to be occurring due a bug in mod_proxy.
– user2607367
Jun 21 '16 at 14:03
add a comment |
For GDB use https://httpd.apache.org/dev/debugging.html.
You would be able to trackdown the actual cause of the error.
If the crash is hard to reproduce it may be a good idea to configure Apache to only use one child processes for handling requests. The config is something like this:
StartServers 1
MinSpareServers 1
MaxSpareServers 1
Check your loaded configuration file- php.ini parameters.Configure it as per the server configuration(H/W resources)
Check if any page caching technique used like APC or memcached, if yes then allocate resources accordingly.
answered Jun 21 '16 at 12:36
Sachin SinghSachin Singh
1568
I have updated a question with a common pattern observed in the error logs when segmentation fault occurs. It seems to be occurring due a bug in mod_proxy.
– user2607367
Jun 21 '16 at 14:03
add a comment |
For GDB use https://httpd.apache.org/dev/debugging.html.
You would be able to trackdown the actual cause of the error.
If the crash is hard to reproduce it may be a good idea to configure Apache to only use one child processes for handling requests. The config is something like this:
StartServers 1
MinSpareServers 1
MaxSpareServers 1
Check your loaded configuration file- php.ini parameters.Configure it as per the server configuration(H/W resources)
Check if any page caching technique used like APC or memcached, if yes then allocate resources accordingly.
answered Jun 21 '16 at 12:36
Sachin SinghSachin Singh
1568
For GDB use https://httpd.apache.org/dev/debugging.html.
You would be able to trackdown the actual cause of the error.
If the crash is hard to reproduce it may be a good idea to configure Apache to only use one child processes for handling requests. The config is something like this:
StartServers 1
MinSpareServers 1
MaxSpareServers 1
Check your loaded configuration file- php.ini parameters.Configure it as per the server configuration(H/W resources)
Check if any page caching technique used like APC or memcached, if yes then allocate resources accordingly.
answered Jun 21 '16 at 12:36
Sachin SinghSachin Singh
1568
answered Jun 21 '16 at 12:36
Sachin SinghSachin Singh
1568
answered Jun 21 '16 at 12:36
Sachin SinghSachin Singh
1568
answered Jun 21 '16 at 12:36
Sachin SinghSachin Singh
1568
1568
I have updated a question with a common pattern observed in the error logs when segmentation fault occurs. It seems to be occurring due a bug in mod_proxy.
– user2607367
Jun 21 '16 at 14:03
add a comment |
I have updated a question with a common pattern observed in the error logs when segmentation fault occurs. It seems to be occurring due a bug in mod_proxy.
– user2607367
Jun 21 '16 at 14:03
I have updated a question with a common pattern observed in the error logs when segmentation fault occurs. It seems to be occurring due a bug in mod_proxy.
– user2607367
Jun 21 '16 at 14:03
I have updated a question with a common pattern observed in the error logs when segmentation fault occurs. It seems to be occurring due a bug in mod_proxy.
– user2607367
Jun 21 '16 at 14:03
add a comment |
Thanks for contributing an answer to Server Fault!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f785078%2fapache-http-2-2-29-segmentation-fault%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What modules are you loading and what is the output of
httpd -Vandldd /path/to/httpd. You might also grab a few lines just prior to the segfault from the access logs from all these servers you are seeing this on and look for common patterns.– Aaron
Jun 20 '16 at 19:19
I have updated the questions with the modules being used in httpd.conf file. httpd -v gives the same output as mentioned in the question. Server version: Apache/2.2.29 (Unix) Server built: Mar 4 2016 15:21:54
– user2607367
Jun 21 '16 at 9:59
Without a known bug for your software version, then the first thing you should be looking at is your core files, using GDB see also stackoverflow.com/questions/8305866/… (hint: use the command bt to a stack trace)
– symcbean
Jun 21 '16 at 10:43
I am not sure how to use gdb. Can you explain how to use it? I am not getting much help from the forum that you have mentioned.
– user2607367
Jun 21 '16 at 11:14
I have updated a question with a common pattern observed in the error logs when segmentation fault occurs. It seems to be occurring due a bug in mod_proxy.
– user2607367
Jun 21 '16 at 14:03