Ryfujk

I was hoping that perhaps one of you guys have an idea about what is happening. For your convenience, I summarized the issue I have been experiencing below including everything we have tried so far.

Summary of issue

Brief summary

• On occasion, some users may experience the site indefinitely loading
  when navigating to our domain.


• While this indefinite loading happens, other users within the same network or outside of it will be able to access the site with no
  issues whatsoever


• This issue seems to occur only (as of now) when there are multiple computers in the same network under one public IP connecting to the
  server (the router's IP). One of these devices on occasion will
  experience the indefinite load


• This indefinite load can occur in the SSH port (1035), or the SSL port (443). It can happen in the browser or terminal.


• As said above, other devices in the network will not experience this issue


• Once a user experiences this indefinite load, any browser they use or ssh will not work. Something is temporarily preventing them from
  reaching the server through any of the publicly opened ports


• Newrelic and ntop report no CPU spikes, memory or HD issues. The server has plenty of resources available


• This issue is reproducible in our staging environment as well


• The issue is not easy to reproduce however, it is more likely to happen as more people access our site from the same network

Server hardware

• Hosting: Softlayer (IBM)


• 2 X 2.4GHz Intel Xeon-Haswell (E5-2620-V3-HexCore)


• Total of 16 cores


• 8x8GB Micron 8GB DDR4 1Rx4


• SuperMicro X10DRU-i+ Motherboard


• OS: Ubuntu16.04-64 Minimal for Bare Metal

Server applications

• The server is running Magento 1.9.3 using php-fpm, and nginx


• Sessions are managed using memcached


• Other caches are managed using redis


• The server is running elastic search


• We use a CDN hosted on AWS (CloudFront) with the server as an origin


• The above applications run on the same server in order to decrease network latency


• The server interfaces with a Percona MySQL server (db1.viyet.com)


• Only ports 80, 443 and 1035 (our ssh port) are open to the public


• The server uses IPTables for security


• We use no DDOS throttling system such as CloudFlare

What we have tried:

• We were able to reproduce this issue in various different networks (home, office, ..) with multiple devices


• We switch to a new, much more powerful server (described above) only to experience the same issue


• We increased the network speed of the site and DB from 100mbps to 1Gbps


• We temporarily turned off the following but were still experiencing the issue: IP tables., elastic search, Memcached


• When the user experiences the indefinite load issue, we tried restarting/stopping PHP, flusing redis cache and flushing memcached with no luck


• After migrating to a new server and increasing network speed we found that the issue happens just as frequently however, the timeout window is reduced. Used to be about 2 mins once a user experiences the issue. Now it's about 1.5 minutes

What "seems" to be happening
It seems like something is temporarily preventing access to the ports described above when many requests come from different devices under the same IP. It's as if there is a limitation.

Do you have any leads as to what could be causing this or any ideas how to get to the bottom of this issue?