Ryfujk

How could a scammer know the apps on my phone / iTunes account?

Check this translation of Amores 1.3.26

Rejected in 4th interview round citing insufficient years of experience

Using "wallow" verb with object

Font with correct density?

Why did it take so long to abandon sail after steamships were demonstrated?

Theorems like the Lovász Local Lemma?

How to explain that I do not want to visit a country due to personal safety concern?

Meaning of "SEVERA INDEOVI VAS" from 3rd Century slab

Is having access to past exams cheating and, if yes, could it be proven just by a good grade?

Provisioning profile doesn't include the application-identifier and keychain-access-groups entitlements

How Did the Space Junk Stay in Orbit in Wall-E?

How to deal with a cynical class?

An Accountant Seeks the Help of a Mathematician

I need to drive a 7/16" nut but am unsure how to use the socket I bought for my screwdriver

Can hydraulic brake levers get hot when brakes overheat?

Humanity loses the vast majority of its technology, information, and population in the year 2122. How long does it take to rebuild itself?

Good allowance savings plan?

At what level can a dragon innately cast its spells?

Professor being mistaken for a grad student

How could a female member of a species produce eggs unto death?

Replacing Windows 7 security updates with anti-virus?

PTIJ: Who should pay for Uber rides: the child or the parent?

Bash: What does "masking return values" mean?

DoS - Apache threads hanging when not receiving TCP ACKs after sending HTTP response

How to best defend against a “slowloris” DOS attack against an Apache web server?Apache service hung and incoming requests not acceptedApache mod status is showing ..readingApache processes at 100% with no traffic - how can I see what's going on?Icinga HTTP on Apache server does not return 200 intermitentlyLoad Balancing not working as expected in Apache in round-robin modeApache is running… but not serving, after reinstalling PHPapachelog shows http requests but not tcp connectionsBlock traffic behind AWS ELBApache stopped working permanently after error “Server ran out of threads to serve requests.”

2

2

In summary, I had a case where all Apache threads were hung because all of them were waiting for a TCP ACK from clients after having sent the HTTP page, and because of that, the Apache threads were waiting 300s (Timeout value of conf) before going to the next request. And then the same thing happened.

The things happened as is: there was a sudden peak of traffic, apache & db servers load goes up as expected, and at some point, every apache thread goes into this state, and the apache & db load goes to 0. And it stayed this way for hours. After a apache restart, pages are processed normally again, load goes up, and again this happen and load goes to 0.
Now, if this was an attack or a consequence of some bad soft/hardware is yet to be known.

To the details: When everything is hung, you go see the server-status page of apaches, and all threads are marked as "W" (working), and you see the timer SS (time the request is being processed) going up up to 300s, and then go next to the next HTTP request and start again to 300s.

In the socket part via netstat, we see all the sockets from these Apache thread hung in CLOSE_WAIT with a high Send-Q value (packets sent not acknowledged).
Using strace, we indeed see Apache doing a poll() of 300s on the socket, waiting for the packets to be acknowleged.

Now, whether it is an attack or some bad network configuration that lost the packets, my question is: how do we prevent this? It seems to be a particular nasty kind of attack.

I am aware about the slow loris attack, when you make a HTTP request very slowly, and this can be mitigated if you have a CDN, a reverse proxy, ...
But for this particular case, I am not seeing something that can prevent that?

How would you prevent this to happen?

Thanks!

apache-2.2 denial-of-service hanging

share|improve this question

edited Dec 5 '11 at 13:43

pauska

18.2k44975

asked Dec 5 '11 at 11:30

nandnand

112

bumped to the homepage by Community♦ 11 mins ago

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Were you able to figure out what was causing this? I have a server intermittently doing the exact same thing, with many connections in CLOSE_WAIT and a high Send-Q value.
  
  – Eric Mason
  Feb 7 '15 at 9:09
  

  
  
  
  

add a comment | 

2

2

In summary, I had a case where all Apache threads were hung because all of them were waiting for a TCP ACK from clients after having sent the HTTP page, and because of that, the Apache threads were waiting 300s (Timeout value of conf) before going to the next request. And then the same thing happened.

The things happened as is: there was a sudden peak of traffic, apache & db servers load goes up as expected, and at some point, every apache thread goes into this state, and the apache & db load goes to 0. And it stayed this way for hours. After a apache restart, pages are processed normally again, load goes up, and again this happen and load goes to 0.
Now, if this was an attack or a consequence of some bad soft/hardware is yet to be known.

To the details: When everything is hung, you go see the server-status page of apaches, and all threads are marked as "W" (working), and you see the timer SS (time the request is being processed) going up up to 300s, and then go next to the next HTTP request and start again to 300s.

In the socket part via netstat, we see all the sockets from these Apache thread hung in CLOSE_WAIT with a high Send-Q value (packets sent not acknowledged).
Using strace, we indeed see Apache doing a poll() of 300s on the socket, waiting for the packets to be acknowleged.

Now, whether it is an attack or some bad network configuration that lost the packets, my question is: how do we prevent this? It seems to be a particular nasty kind of attack.

I am aware about the slow loris attack, when you make a HTTP request very slowly, and this can be mitigated if you have a CDN, a reverse proxy, ...
But for this particular case, I am not seeing something that can prevent that?

How would you prevent this to happen?

Thanks!

apache-2.2 denial-of-service hanging

share|improve this question

edited Dec 5 '11 at 13:43

pauska

18.2k44975

asked Dec 5 '11 at 11:30

nandnand

112

bumped to the homepage by Community♦ 11 mins ago

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Were you able to figure out what was causing this? I have a server intermittently doing the exact same thing, with many connections in CLOSE_WAIT and a high Send-Q value.
  
  – Eric Mason
  Feb 7 '15 at 9:09
  

  
  
  
  

add a comment | 

In summary, I had a case where all Apache threads were hung because all of them were waiting for a TCP ACK from clients after having sent the HTTP page, and because of that, the Apache threads were waiting 300s (Timeout value of conf) before going to the next request. And then the same thing happened.

The things happened as is: there was a sudden peak of traffic, apache & db servers load goes up as expected, and at some point, every apache thread goes into this state, and the apache & db load goes to 0. And it stayed this way for hours. After a apache restart, pages are processed normally again, load goes up, and again this happen and load goes to 0.
Now, if this was an attack or a consequence of some bad soft/hardware is yet to be known.

To the details: When everything is hung, you go see the server-status page of apaches, and all threads are marked as "W" (working), and you see the timer SS (time the request is being processed) going up up to 300s, and then go next to the next HTTP request and start again to 300s.

In the socket part via netstat, we see all the sockets from these Apache thread hung in CLOSE_WAIT with a high Send-Q value (packets sent not acknowledged).
Using strace, we indeed see Apache doing a poll() of 300s on the socket, waiting for the packets to be acknowleged.

Now, whether it is an attack or some bad network configuration that lost the packets, my question is: how do we prevent this? It seems to be a particular nasty kind of attack.

I am aware about the slow loris attack, when you make a HTTP request very slowly, and this can be mitigated if you have a CDN, a reverse proxy, ...
But for this particular case, I am not seeing something that can prevent that?

How would you prevent this to happen?

Thanks!

apache-2.2 denial-of-service hanging

share|improve this question

edited Dec 5 '11 at 13:43

pauska

18.2k44975

asked Dec 5 '11 at 11:30

nandnand

112

share|improve this question

edited Dec 5 '11 at 13:43

pauska

18.2k44975

asked Dec 5 '11 at 11:30

nandnand

112

share|improve this question

edited Dec 5 '11 at 13:43

pauska

18.2k44975

asked Dec 5 '11 at 11:30

nandnand

112

edited Dec 5 '11 at 13:43

pauska

18.2k44975

edited Dec 5 '11 at 13:43

pauska

18.2k44975

asked Dec 5 '11 at 11:30

nandnand

112

asked Dec 5 '11 at 11:30

nandnand

112

1 Answer
1

active

oldest

votes

0

put a litespeed infront of apache. That can solve your problem. That is exactly what we do for slow request, reflection and regular ddos problems. Solve means at least at some scale i meant.

Edited Section:

Sure,

Litespeed is a web server/load balancer a type of reverse proxy also rate base limiting aviable ddos attack mitigation software. We had the same type of attacks you meantioned and much more worse close to 400.000 http request per second based attacks. We install a litespeed server with 24 core 128 gb ram litespeeds runs in mounted ram and operation system runs in raid0 striped ssd disks. And that solved and stop this attack.

Also via configuration and setup you can stop the slow loris attack. We still have a load balancer, but the incoming traffic passes through first the litespeed then proxeyed to our hw loadbalancer.

share|improve this answer

edited Dec 5 '11 at 14:15

answered Dec 5 '11 at 11:52

x0rx0r

313

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Can you elaborate on your answer? The wording is a bit confusing.
  
  – Bart Silverstrim
  Dec 5 '11 at 11:59
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Almost any proxy/firewall can prevent slow loris because it will wait for the whole HTTP request before forwarding it to Apache. In this case, when Apache send back the HTTP answer, it may be forwarded by the proxy/firewall, but the proxy/firewall won't "simulate" an ACK for Apache. Apache has to wait for the actual ACK from the client. And if it never comes, 300s timeout. So I'm not sure your solution is adressing this specific point?
  
  – nand
  Dec 5 '11 at 16:56
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I meant Load balancer working as l7 proxy. Apache's client is only the load balancer. Cause of the proxy only setup can be done on the load balancer no on apaches. You can play around all timeout or protocol specified stuff from load balancer. Load balancer can simulate ack can be work as proxy arp / similar like junipers mib.
  
  – x0r
  Dec 8 '11 at 9:25
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "2"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f337694%2fdos-apache-threads-hanging-when-not-receiving-tcp-acks-after-sending-http-resp%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

0

put a litespeed infront of apache. That can solve your problem. That is exactly what we do for slow request, reflection and regular ddos problems. Solve means at least at some scale i meant.

Edited Section:

Sure,

Litespeed is a web server/load balancer a type of reverse proxy also rate base limiting aviable ddos attack mitigation software. We had the same type of attacks you meantioned and much more worse close to 400.000 http request per second based attacks. We install a litespeed server with 24 core 128 gb ram litespeeds runs in mounted ram and operation system runs in raid0 striped ssd disks. And that solved and stop this attack.

Also via configuration and setup you can stop the slow loris attack. We still have a load balancer, but the incoming traffic passes through first the litespeed then proxeyed to our hw loadbalancer.

share|improve this answer

edited Dec 5 '11 at 14:15

answered Dec 5 '11 at 11:52

x0rx0r

313

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Can you elaborate on your answer? The wording is a bit confusing.
  
  – Bart Silverstrim
  Dec 5 '11 at 11:59
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Almost any proxy/firewall can prevent slow loris because it will wait for the whole HTTP request before forwarding it to Apache. In this case, when Apache send back the HTTP answer, it may be forwarded by the proxy/firewall, but the proxy/firewall won't "simulate" an ACK for Apache. Apache has to wait for the actual ACK from the client. And if it never comes, 300s timeout. So I'm not sure your solution is adressing this specific point?
  
  – nand
  Dec 5 '11 at 16:56
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I meant Load balancer working as l7 proxy. Apache's client is only the load balancer. Cause of the proxy only setup can be done on the load balancer no on apaches. You can play around all timeout or protocol specified stuff from load balancer. Load balancer can simulate ack can be work as proxy arp / similar like junipers mib.
  
  – x0r
  Dec 8 '11 at 9:25
  

  
  
  
  

add a comment | 

0

put a litespeed infront of apache. That can solve your problem. That is exactly what we do for slow request, reflection and regular ddos problems. Solve means at least at some scale i meant.

Edited Section:

Sure,

Litespeed is a web server/load balancer a type of reverse proxy also rate base limiting aviable ddos attack mitigation software. We had the same type of attacks you meantioned and much more worse close to 400.000 http request per second based attacks. We install a litespeed server with 24 core 128 gb ram litespeeds runs in mounted ram and operation system runs in raid0 striped ssd disks. And that solved and stop this attack.

Also via configuration and setup you can stop the slow loris attack. We still have a load balancer, but the incoming traffic passes through first the litespeed then proxeyed to our hw loadbalancer.

share|improve this answer

edited Dec 5 '11 at 14:15

answered Dec 5 '11 at 11:52

x0rx0r

313

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Can you elaborate on your answer? The wording is a bit confusing.
  
  – Bart Silverstrim
  Dec 5 '11 at 11:59
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Almost any proxy/firewall can prevent slow loris because it will wait for the whole HTTP request before forwarding it to Apache. In this case, when Apache send back the HTTP answer, it may be forwarded by the proxy/firewall, but the proxy/firewall won't "simulate" an ACK for Apache. Apache has to wait for the actual ACK from the client. And if it never comes, 300s timeout. So I'm not sure your solution is adressing this specific point?
  
  – nand
  Dec 5 '11 at 16:56
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I meant Load balancer working as l7 proxy. Apache's client is only the load balancer. Cause of the proxy only setup can be done on the load balancer no on apaches. You can play around all timeout or protocol specified stuff from load balancer. Load balancer can simulate ack can be work as proxy arp / similar like junipers mib.
  
  – x0r
  Dec 8 '11 at 9:25
  

  
  
  
  

add a comment | 

put a litespeed infront of apache. That can solve your problem. That is exactly what we do for slow request, reflection and regular ddos problems. Solve means at least at some scale i meant.

Edited Section:

Sure,

Litespeed is a web server/load balancer a type of reverse proxy also rate base limiting aviable ddos attack mitigation software. We had the same type of attacks you meantioned and much more worse close to 400.000 http request per second based attacks. We install a litespeed server with 24 core 128 gb ram litespeeds runs in mounted ram and operation system runs in raid0 striped ssd disks. And that solved and stop this attack.

Also via configuration and setup you can stop the slow loris attack. We still have a load balancer, but the incoming traffic passes through first the litespeed then proxeyed to our hw loadbalancer.

share|improve this answer

edited Dec 5 '11 at 14:15

answered Dec 5 '11 at 11:52

x0rx0r

313

share|improve this answer

edited Dec 5 '11 at 14:15

answered Dec 5 '11 at 11:52

x0rx0r

313

answered Dec 5 '11 at 11:52

x0rx0r

313

answered Dec 5 '11 at 11:52

x0rx0r

313