Preserve Source IP In AWS Classic Load-Balancer And Istio’s Envoy Using Proxy Protocol

Preserving Source IP address is an important factor in a live environment because the IP address is one of the things which enables you to do some advanced stuff like:

Security: Security is an important factor which we cannot ignore. With the Source IP you can white list the access to the applications which are behind the internet-facing load balancer.

Rate Limiting: limiting the number of requests per second to your application based on IP address provides DDoS Protection‎ from Bots.

Logging and Visualizing: Logging based on country or cities is very useful if you want to track which country had most users for your application.

Access Control: Running 2 or more versions of applications? You can segregate premium and normal user based on their IPs.

And many more …

I’m starting this blog series, throughout this I will cover functionalities:
1. Preserving Source IP address of the client.
2. Apply IP Whitelisting on Kubernetes microservices.
3. Create Fluentd docker image with GeoIP plugin.
4. Create a kibana dashboard for IP logs using EFK.

Environment where implementing this:

1. Kubernetes Environment (Kubernetes v-1.15.3)
2. Service Mesh using Istio.

Note: I will refer load balancer as reverse-proxy interchangeably.

So, before jumping on to preserving source IP through proxy-protocol on reverse-proxies, let me tell you what issue I faced with reverse-proxies that brought about the need to dig deeper on workings of proxy-protocol and reverse proxy.

In short, a sweet answer to question what is a reverse proxy?

Reverse-Proxy

A Reverse-proxy is a server which gets connected on upstream servers on behalf of users. The upstream server can be either an application server, a load balancer or another proxy/reverse-proxy. More on Reverse proxy.

Issue In Reverse-Proxy

The main issue of reverse-proxy is that it will remove the user IP.

Here is the flow of the requests and responses:

1. The client gets connected through the firewall to the reverse-proxy and sends it’s request.
2. The Reverse-Proxy validates the request, analyses it to choose the right farm. then forwards it to the load balancer in the LAN, through the firewall.
3. The load balancer chooses a server in the farm and forwards the request to it.
4. The server processes the request then answers to the load balancer.
5. The load balancer forwards the response to the reverse-proxy.
6. The reverse-proxy forward the response to the client.

Basically, the source IP is modified twice in this kind of architecture: during the steps 2 and 3.
And of course, the more you chain load balancer and reverse proxies, the more the source IP will be changed.

Exploring on reverse-proxy unusual behaviour for few hours lead to me the concept “Proxy protocol” !

TCP Proxy Protocol — Limited to TCP traffic on L4.

Now jump onto how Proxy protocol become a saviour.

Proxy Protocol

In 10000-foot view below, we can see that the need of Proxy protocol is just to tell reverse-proxy(Classic loadbalancer in my case) to add another header in packets. In that header value we put the user details which may be useful to server applications. It looks like this:

PROXY TCP4 192.168.0.1 192.168.0.11 56324 80rn

Means our proxy protocol works between two or more reverse proxies or between reverse-proxy and our server application.

I assume that till this point I have given a basic idea of what we are going to implement.

Moving To Our Practical :

Our Practical contains two steps:

1. Enable Proxy protocol on load balancer listeners. Yes, you are right, to tell reverse-proxy to create another header. I’m taking two methods to implement this on load balancer :
i. When using load balancer with “Kubernetes Load-Balancer service”.
ii. When using load balancer with “EC2 instance directly“.

2. Parse proxy protocol using “Listener Filters”.

LoadBalancer Connected With ‘Kubernetes Load-Balancer service‘ :

Generate service manifest for ingress-gateway:

kubectl get svc istio-ingressgateway -n istio-system -o yaml > istio-ingressgateway.yaml

In ingress-gateway service do the following configuration.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
  labels:
    app: istio-ingressgateway
    istio: ingressgateway
    operator.istio.io/component: IngressGateway
    operator.istio.io/managed: Reconcile
    operator.istio.io/version: 1.4.3
    release: istio
  name: istio-ingressgateway
  namespace: istio-system
spec:
  externalTrafficPolicy: Local
  ports:
  ...
  ...

1. service.beta.kubernetes.io/aws-load-balancer-proxy-protocol:’*’
This annotation is used to setup proxy protocol on packet arriving on every listener of Load Balancer.

2. externalTrafficPolicy:Local  To know about externalTrafficPolicy 

LoadBalancer Connected With EC2 Instance

Follow here

You can use the following command to verify whether Proxy Protocol is enabled on Classic loadbalancer ports or not:

aws elb describe-load-balancers –load-balancer-name <load-balancer-name>

Until here we are all set with main user info which required by application server or by envoy/nginx proxy, but wait, how our application or any other following reverse-proxy can read this additional field in the packet?

To do that, we need to parse the extra proxy-protocol header using Listener filters.

Parse Proxy Protocol Using Http Listener Filters

Listener filter checks proxy header to retrieve the connection information. I’m using Envoy proxy in Istio, so I apply apply Http Listener filters using EnvoyFilter Policy resource . More on Listener Filter

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: proxy-protocol
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: LISTENER
    patch:
      operation: MERGE
      value:
        listener_filters:
        - name: envoy.listener.proxy_protocol
        - name: envoy.listener.tls_inspector

For Http Listener filter in Nginx, follow this

Now Check the logs of the Nginx, Envoy, or whichever you are using and we got Source IP ! Awesome ?

Note: Enable Envoy proxy to print logs to stdout using:

–set meshConfig.accessLogFile=”/dev/stdout”

A journey of thousand miles begins with a single step, similarly, we are done with our initial step of getting source IP and we are ready to do awesome stuff with it.

What’s next?

In the coming posts, we will see “IP Whitelisting using Istio Policy on Kubernetes Microservices” and “kibana dashboard for IP logs using EFK”.

More on Proxy protocol, Reverse proxy?

Listener filters
Accepting the PROXY Protocol in Nginx
Configure Proxy Protocol Support for Your Classic Load Balancer
Forward and Reverse Proxy