So you have a cluster and are using or considering using the to forward outside traffic to in-cluster services. Fast-forward a few months, helm nginx ingress external traffic for all environments dev, staging, production was going through the ingress servers. We all know how it happens. First, you get excited about that shiny new thing. Then, eventually, some shit happens. My First Ingress Outage Let me start by saying that if you are not alerting onwell, you should. What happens when some pod fails to respond to the liveness probes. What are the lessons learned from this incident. Will it spawn one or two worker processes. Now take the listen directive; it does not specify the backlog parameter which is 511 by default on Linux. In other words, make sure your config is in tune with your kernel. Do this thought helm nginx ingress to every line of the generated config. Kernel Params Using ingress or not, make sure to always review and tune the kernel params of your nodes according to the expected workloads. This is a rather complex subject on its own, so I have no intention of covering everything in this post; take a look at the section helm nginx ingress more pointers in this area. As the values of different conntrack params need to be set in conformance with each other ie. However, bear in mind that increasing these params will result inso be gentle. The trade-off is that this limits the things we can do to our cluster. For instance, if we decide to run a load test on a staging service, we need to be really careful or we risk affecting production services running in the same cluster. Even though the level helm nginx ingress isolation provided by containers is generally good, they still that are subject to abuse. Ingress Reloads Gone Wrong At this point, we were already running a dedicated ingress controller for the production environment. Everything was running pretty smoothly until we decided to migrate a WebSocket application to Kubernetes + ingress. Shortly after the migration, I started noticing a strange trend in memory usage for the production ingress pods. What helm nginx ingress hell is happening. Why was the memory consumption skyrocketing like this. Once the master process receives the signal to reload configuration, it checks the syntax validity of the new configuration file and tries to apply the configuration provided in it. If this is a success, the master process starts new worker processes and sends messages to old worker processes, requesting them to shut down. Otherwise, the master process rolls back the changes and continues to work with the old configuration. Old worker processes, receiving a command to shut down, stop accepting new connections and continue to service current requests until all such requests are serviced. After that, the old worker processes exit. Remember we are proxying WebSocket connections, which are long-running by nature; a WebSocket connection might take hours, or even days to close depending on the application. If we have that many workers in that state, this means the ingress configuration got reloaded many times, and workers were unable to terminate due to the long-running connections. Number of unnecessary reloads went down to zero after deploying a fixed version. Thanks to for his assistance in finding and fixing this bug. This is why we decided to create a specific ingress controller deployment just for proxying these long-running connections. Thus, if you are observing frequent autoscaling events for your applications during normal load, it might be a sign that your HorizontalPodAutoscalers need adjustment. Horizontal pod autoscaler in action during peak hours. Thus, in case your application really experiences an increased load, it might take ~4 minutes 3m from the autoscaler back-off + ~1m from the metrics sync for the autoscaler to react to the increased load, which might be just enough time for your service to degrade.
