Ryfujk

Co-worker team leader wants to inject his friend's awful software into our development. What should I say to our common boss?

Be in awe of my brilliance!

Counting certain elements in lists

PTIJ: is Mi Yodeya found in the Torah codes?

Font with correct density?

Did CPM support custom hardware using device drivers?

Russian cases: A few examples, I'm really confused

Can hydraulic brake levers get hot when brakes overheat?

Fill color and outline color with the same value

My adviser wants to be the first author

How could a female member of a species produce eggs unto death?

Old race car problem/puzzle

Is it possible that AIC = BIC?

Can anyone tell me why this program fails?

What is the greatest age difference between a married couple in Tanach?

How do anti-virus programs start at Windows boot?

Welcoming 2019 Pi day: How to draw the letter π?

An Accountant Seeks the Help of a Mathematician

Why using two cd commands in bash script does not execute the second command

Brexit - No Deal Rejection

Can elves maintain concentration in a trance?

Does this AnyDice function accurately calculate the number of ogres you make unconcious with three 4th-level castings of Sleep?

Bash replace string at multiple places in a file from command line

Pinhole Camera with Instant Film

HAProxy stats says all Thrift servers are down

HaProxy + IIS pages gradually get slowerHaProxy - Http and SSL pass through configHAProxy, health checking multiple servers with different host namesHaproxy logging not workHAProxy 1.5.4 Performance TuningHA-Proxy 301 re-direct: https to https://wwwHaProxy giving - 503 Service UnavailableHAProxy not logging all requestsRunning Hudson, Sonar and HA Proxy on the same serveropenldap with haproxy - (ldap_result() failed: Can't contact LDAP server)

0

I am trying to setup HAProxy to load balance a group of Thrift servers. For some reason the HAProxy stats page says the servers are all down.

Here is the current HAProxy config I am trying.

global

    log 127.0.0.1   local0

    log 127.0.0.1   local1 notice

    chroot      /var/lib/haproxy

    pidfile     /var/run/haproxy.pid

    maxconn     60000

    user        haproxy

    group       haproxy



defaults

    log     global

    option  dontlognull

    option redispatch

    retries 3

    maxconn 2000

    contimeout      5000

    clitimeout      50000

    srvtimeout      50000



listen stats :5000

    mode http

    stats enable

    stats hide-version

    stats realm Haproxy Statistics

    stats uri /

    stats auth user:pass



listen metrix :5002

    mode tcp

    option tcplog

    balance leastconn

    server m1 127.0.0.1:9000 check

    server m2 127.0.0.1:9001 check

    server m3 127.0.0.1:9002 check

    server m4 127.0.0.1:9003 check

    server m5 127.0.0.1:9004 check

    server m6 127.0.0.1:9005 check

    server m7 127.0.0.1:9006 check

    server m8 127.0.0.1:9007 check

One other things. The Thrift servers are running in Supervisor which I have noticed some weirdness with. But I have tried just running the Thrift server not in supervisor and it still doesn't work.

I have tried all of the Thrift servers (TSimpleServer, TNonBlockingServer, TThreadedServer) all of them have the same problem.

Update

Running tcpdump port 9000

15:12:31.878502 IP ip-00.00.00.00.ec2.internal.cslistener > ip-11.11.11.11.ec2.internal.36206: Flags [R.], seq 0, ack 3433673377, win 0, length 0

15:12:33.878425 IP ip-11.11.11.11.ec2.internal.36207 > ip-00.00.00.00.ec2.internal.cslistener: Flags [S], seq 3459211721, win 5840, options [mss 1460,sackOK,TS val 440815982 ecr 0,nop,wscale 10], length 0

Where 00.00.00.00 is the server running the Thrift server and 11.11.11.11 is the server running HAProxy.
These 2 lines repeat continuously.

Running netstat -tlnp contains the following:

tcp        0      0 127.0.0.1:9000              0.0.0.0:*                   LISTEN      19472/python

So the Thrift server is listening on the right port.

haproxy supervisord

share|improve this question

edited Feb 25 '13 at 23:53

user897929

asked Feb 25 '13 at 20:49

user897929user897929

1011

bumped to the homepage by Community♦ 6 mins ago

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  What do you see when you tcpdump on 127.0.0.1 port 9000?
  
  – Ladadadada
  Feb 25 '13 at 21:44
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  A lot of this: 15:12:31.878502 IP ip-00.00.00.00.ec2.internal.cslistener > ip-11.11.11.11.ec2.internal.36206: Flags [R.], seq 0, ack 3433673377, win 0, length 0 15:12:33.878425 IP ip-11.11.11.11.ec2.internal.36207 > ip-00.00.00.00.ec2.internal.cslistener: Flags [S], seq 3459211721, win 5840, options [mss 1460,sackOK,TS val 440815982 ecr 0,nop,wscale 10], length 0 Where 00.00.00.00 is the server running the Thrift server and 11.11.11.11 is the server running HAProxy.
  
  – user897929
  Feb 25 '13 at 23:15
  

  
  
  
  

add a comment | 

0

I am trying to setup HAProxy to load balance a group of Thrift servers. For some reason the HAProxy stats page says the servers are all down.

Here is the current HAProxy config I am trying.

global

    log 127.0.0.1   local0

    log 127.0.0.1   local1 notice

    chroot      /var/lib/haproxy

    pidfile     /var/run/haproxy.pid

    maxconn     60000

    user        haproxy

    group       haproxy



defaults

    log     global

    option  dontlognull

    option redispatch

    retries 3

    maxconn 2000

    contimeout      5000

    clitimeout      50000

    srvtimeout      50000



listen stats :5000

    mode http

    stats enable

    stats hide-version

    stats realm Haproxy Statistics

    stats uri /

    stats auth user:pass



listen metrix :5002

    mode tcp

    option tcplog

    balance leastconn

    server m1 127.0.0.1:9000 check

    server m2 127.0.0.1:9001 check

    server m3 127.0.0.1:9002 check

    server m4 127.0.0.1:9003 check

    server m5 127.0.0.1:9004 check

    server m6 127.0.0.1:9005 check

    server m7 127.0.0.1:9006 check

    server m8 127.0.0.1:9007 check

One other things. The Thrift servers are running in Supervisor which I have noticed some weirdness with. But I have tried just running the Thrift server not in supervisor and it still doesn't work.

I have tried all of the Thrift servers (TSimpleServer, TNonBlockingServer, TThreadedServer) all of them have the same problem.

Update

Running tcpdump port 9000

15:12:31.878502 IP ip-00.00.00.00.ec2.internal.cslistener > ip-11.11.11.11.ec2.internal.36206: Flags [R.], seq 0, ack 3433673377, win 0, length 0

15:12:33.878425 IP ip-11.11.11.11.ec2.internal.36207 > ip-00.00.00.00.ec2.internal.cslistener: Flags [S], seq 3459211721, win 5840, options [mss 1460,sackOK,TS val 440815982 ecr 0,nop,wscale 10], length 0

Where 00.00.00.00 is the server running the Thrift server and 11.11.11.11 is the server running HAProxy.
These 2 lines repeat continuously.

Running netstat -tlnp contains the following:

tcp        0      0 127.0.0.1:9000              0.0.0.0:*                   LISTEN      19472/python

So the Thrift server is listening on the right port.

haproxy supervisord

share|improve this question

edited Feb 25 '13 at 23:53

user897929

asked Feb 25 '13 at 20:49

user897929user897929

1011

bumped to the homepage by Community♦ 6 mins ago

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  What do you see when you tcpdump on 127.0.0.1 port 9000?
  
  – Ladadadada
  Feb 25 '13 at 21:44
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  A lot of this: 15:12:31.878502 IP ip-00.00.00.00.ec2.internal.cslistener > ip-11.11.11.11.ec2.internal.36206: Flags [R.], seq 0, ack 3433673377, win 0, length 0 15:12:33.878425 IP ip-11.11.11.11.ec2.internal.36207 > ip-00.00.00.00.ec2.internal.cslistener: Flags [S], seq 3459211721, win 5840, options [mss 1460,sackOK,TS val 440815982 ecr 0,nop,wscale 10], length 0 Where 00.00.00.00 is the server running the Thrift server and 11.11.11.11 is the server running HAProxy.
  
  – user897929
  Feb 25 '13 at 23:15
  

  
  
  
  

add a comment | 

I am trying to setup HAProxy to load balance a group of Thrift servers. For some reason the HAProxy stats page says the servers are all down.

Here is the current HAProxy config I am trying.

global

    log 127.0.0.1   local0

    log 127.0.0.1   local1 notice

    chroot      /var/lib/haproxy

    pidfile     /var/run/haproxy.pid

    maxconn     60000

    user        haproxy

    group       haproxy



defaults

    log     global

    option  dontlognull

    option redispatch

    retries 3

    maxconn 2000

    contimeout      5000

    clitimeout      50000

    srvtimeout      50000



listen stats :5000

    mode http

    stats enable

    stats hide-version

    stats realm Haproxy Statistics

    stats uri /

    stats auth user:pass



listen metrix :5002

    mode tcp

    option tcplog

    balance leastconn

    server m1 127.0.0.1:9000 check

    server m2 127.0.0.1:9001 check

    server m3 127.0.0.1:9002 check

    server m4 127.0.0.1:9003 check

    server m5 127.0.0.1:9004 check

    server m6 127.0.0.1:9005 check

    server m7 127.0.0.1:9006 check

    server m8 127.0.0.1:9007 check

One other things. The Thrift servers are running in Supervisor which I have noticed some weirdness with. But I have tried just running the Thrift server not in supervisor and it still doesn't work.

I have tried all of the Thrift servers (TSimpleServer, TNonBlockingServer, TThreadedServer) all of them have the same problem.

Update

Running tcpdump port 9000

15:12:31.878502 IP ip-00.00.00.00.ec2.internal.cslistener > ip-11.11.11.11.ec2.internal.36206: Flags [R.], seq 0, ack 3433673377, win 0, length 0

15:12:33.878425 IP ip-11.11.11.11.ec2.internal.36207 > ip-00.00.00.00.ec2.internal.cslistener: Flags [S], seq 3459211721, win 5840, options [mss 1460,sackOK,TS val 440815982 ecr 0,nop,wscale 10], length 0

Where 00.00.00.00 is the server running the Thrift server and 11.11.11.11 is the server running HAProxy.
These 2 lines repeat continuously.

Running netstat -tlnp contains the following:

tcp        0      0 127.0.0.1:9000              0.0.0.0:*                   LISTEN      19472/python

So the Thrift server is listening on the right port.

haproxy supervisord

share|improve this question

edited Feb 25 '13 at 23:53

user897929

asked Feb 25 '13 at 20:49

user897929user897929

1011

global

    log 127.0.0.1   local0

    log 127.0.0.1   local1 notice

    chroot      /var/lib/haproxy

    pidfile     /var/run/haproxy.pid

    maxconn     60000

    user        haproxy

    group       haproxy



defaults

    log     global

    option  dontlognull

    option redispatch

    retries 3

    maxconn 2000

    contimeout      5000

    clitimeout      50000

    srvtimeout      50000



listen stats :5000

    mode http

    stats enable

    stats hide-version

    stats realm Haproxy Statistics

    stats uri /

    stats auth user:pass



listen metrix :5002

    mode tcp

    option tcplog

    balance leastconn

    server m1 127.0.0.1:9000 check

    server m2 127.0.0.1:9001 check

    server m3 127.0.0.1:9002 check

    server m4 127.0.0.1:9003 check

    server m5 127.0.0.1:9004 check

    server m6 127.0.0.1:9005 check

    server m7 127.0.0.1:9006 check

    server m8 127.0.0.1:9007 check

15:12:31.878502 IP ip-00.00.00.00.ec2.internal.cslistener > ip-11.11.11.11.ec2.internal.36206: Flags [R.], seq 0, ack 3433673377, win 0, length 0

15:12:33.878425 IP ip-11.11.11.11.ec2.internal.36207 > ip-00.00.00.00.ec2.internal.cslistener: Flags [S], seq 3459211721, win 5840, options [mss 1460,sackOK,TS val 440815982 ecr 0,nop,wscale 10], length 0

tcp        0      0 127.0.0.1:9000              0.0.0.0:*                   LISTEN      19472/python

share|improve this question

edited Feb 25 '13 at 23:53

user897929

asked Feb 25 '13 at 20:49

user897929user897929

1011

share|improve this question

edited Feb 25 '13 at 23:53

user897929

asked Feb 25 '13 at 20:49

user897929user897929

1011

asked Feb 25 '13 at 20:49

user897929user897929

1011

asked Feb 25 '13 at 20:49

user897929user897929

1011

1 Answer
1

active

oldest

votes

1

There's not enough in that tcpdump to be sure but it looks like your health check is sending a SYN packet and is getting a RST packet in return. (Feel free to post more of it in your actual question where you can format it properly.)

I suspect there's nothing listening on 127.0.0.1:9000 (or any of the other ports).

You can check this with sudo netstat -tlnp.

Either the Thrift servers are listening on different IP addresses and/or ports or they aren't listening at all.

share|improve this answer

answered Feb 25 '13 at 23:26

LadadadadaLadadadada

21.9k54881

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I updated the question with the formatted out of tcpdump and netstat.
  
  – user897929
  Feb 25 '13 at 23:54
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "2"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f482395%2fhaproxy-stats-says-all-thrift-servers-are-down%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

1

There's not enough in that tcpdump to be sure but it looks like your health check is sending a SYN packet and is getting a RST packet in return. (Feel free to post more of it in your actual question where you can format it properly.)

I suspect there's nothing listening on 127.0.0.1:9000 (or any of the other ports).

You can check this with sudo netstat -tlnp.

Either the Thrift servers are listening on different IP addresses and/or ports or they aren't listening at all.

share|improve this answer

answered Feb 25 '13 at 23:26

LadadadadaLadadadada

21.9k54881

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I updated the question with the formatted out of tcpdump and netstat.
  
  – user897929
  Feb 25 '13 at 23:54
  

  
  
  
  

add a comment | 

1

There's not enough in that tcpdump to be sure but it looks like your health check is sending a SYN packet and is getting a RST packet in return. (Feel free to post more of it in your actual question where you can format it properly.)

I suspect there's nothing listening on 127.0.0.1:9000 (or any of the other ports).

You can check this with sudo netstat -tlnp.

Either the Thrift servers are listening on different IP addresses and/or ports or they aren't listening at all.

share|improve this answer

answered Feb 25 '13 at 23:26

LadadadadaLadadadada

21.9k54881

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I updated the question with the formatted out of tcpdump and netstat.
  
  – user897929
  Feb 25 '13 at 23:54
  

  
  
  
  

add a comment | 

There's not enough in that tcpdump to be sure but it looks like your health check is sending a SYN packet and is getting a RST packet in return. (Feel free to post more of it in your actual question where you can format it properly.)

I suspect there's nothing listening on 127.0.0.1:9000 (or any of the other ports).

You can check this with sudo netstat -tlnp.

Either the Thrift servers are listening on different IP addresses and/or ports or they aren't listening at all.

share|improve this answer

answered Feb 25 '13 at 23:26

LadadadadaLadadadada

21.9k54881

share|improve this answer

answered Feb 25 '13 at 23:26

LadadadadaLadadadada

21.9k54881

answered Feb 25 '13 at 23:26

LadadadadaLadadadada

21.9k54881

answered Feb 25 '13 at 23:26

LadadadadaLadadadada

21.9k54881