Preserving Source IP address is an important factor in a live environment because the IP address is one of the things which enables you to do some advanced stuff like:
Security: Security is an important factor which we cannot ignore. With the Source IP you can white list the access to the applications which are behind the internet-facing load balancer.
Rate Limiting: limiting the number of requests per second to your application based on IP address provides DDoS Protection from Bots.
Logging and Visualizing: Logging based on country or cities is very useful if you want to track which country had most users for your application.
Access Control: Running 2 or more versions of applications? You can segregate premium and normal user based on their IPs.
And many more …
I’m starting this blog series, throughout this I will cover functionalities:
1. Preserving Source IP address of the client.
2. Apply IP Whitelisting on Kubernetes microservices.
3. Create Fluentd docker image with GeoIP plugin.
4. Create a kibana dashboard for IP logs using EFK.
Environment where implementing this:
1. Kubernetes Environment (Kubernetes v-1.15.3)
2. Service Mesh using Istio.
Note: I will refer load balancer as reverse-proxy interchangeably.
So, before jumping on to preserving source IP through proxy-protocol on reverse-proxies, let me tell you what issue I faced with reverse-proxies that brought about the need to dig deeper on workings of proxy-protocol and reverse proxy.
In short, a sweet answer to question what is a reverse proxy?
A Reverse-proxy is a server which gets connected on upstream servers on behalf of users. The upstream server can be either an application server, a load balancer or another proxy/reverse-proxy. More on Reverse proxy.
Issue In Reverse-Proxy
The main issue of reverse-proxy is that it will remove the user IP.
Here is the flow of the requests and responses:
1. The client gets connected through the firewall to the reverse-proxy and sends it’s request.
2. The Reverse-Proxy validates the request, analyses it to choose the right farm. then forwards it to the load balancer in the LAN, through the firewall.
3. The load balancer chooses a server in the farm and forwards the request to it.
4. The server processes the request then answers to the load balancer.
5. The load balancer forwards the response to the reverse-proxy.
6. The reverse-proxy forward the response to the client.
Basically, the source IP is modified twice in this kind of architecture: during the steps 2 and 3.
And of course, the more you chain load balancer and reverse proxies, the more the source IP will be changed.
Exploring on reverse-proxy unusual behaviour for few hours lead to me the concept “Proxy protocol” !
TCP Proxy Protocol — Limited to TCP traffic on L4.
Now jump onto how Proxy protocol become a saviour.
In 10000-foot view below, we can see that the need of Proxy protocol is just to tell reverse-proxy(Classic loadbalancer in my case) to add another header in packets. In that header value we put the user details which may be useful to server applications. It looks like this:
PROXY TCP4 192.168.0.1 192.168.0.11 56324 80rn
Means our proxy protocol works between two or more reverse proxies or between reverse-proxy and our server application.
I assume that till this point I have given a basic idea of what we are going to implement.
Moving To Our Practical :
Our Practical contains two steps:
1. Enable Proxy protocol on load balancer listeners. Yes, you are right, to tell reverse-proxy to create another header. I’m taking two methods to implement this on load balancer :
i. When using load balancer with “Kubernetes Load-Balancer service”.
ii. When using load balancer with “EC2 instance directly“.
2. Parse proxy protocol using “Listener Filters”.
LoadBalancer Connected With ‘Kubernetes Load-Balancer service‘ :
Generate service manifest for ingress-gateway:
kubectl get svc istio-ingressgateway -n istio-system -o yaml > istio-ingressgateway.yaml
In ingress-gateway service do the following configuration.
apiVersion: v1 kind: Service metadata: annotations: service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*' labels: app: istio-ingressgateway istio: ingressgateway operator.istio.io/component: IngressGateway operator.istio.io/managed: Reconcile operator.istio.io/version: 1.4.3 release: istio name: istio-ingressgateway namespace: istio-system spec: externalTrafficPolicy: Local ports: ... ...
This annotation is used to setup proxy protocol on packet arriving on every listener of Load Balancer.
2. externalTrafficPolicy:Local To know about externalTrafficPolicy
LoadBalancer Connected With EC2 Instance
You can use the following command to verify whether Proxy Protocol is enabled on Classic loadbalancer ports or not:
aws elb describe-load-balancers –load-balancer-name <load-balancer-name>
Until here we are all set with main user info which required by application server or by envoy/nginx proxy, but wait, how our application or any other following reverse-proxy can read this additional field in the packet?
To do that, we need to parse the extra proxy-protocol header using Listener filters.
Parse Proxy Protocol Using Http Listener Filters
Listener filter checks proxy header to retrieve the connection information. I’m using Envoy proxy in Istio, so I apply apply Http Listener filters using EnvoyFilter Policy resource . More on Listener Filter
apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: proxy-protocol namespace: istio-system spec: workloadSelector: labels: istio: ingressgateway configPatches: - applyTo: LISTENER patch: operation: MERGE value: listener_filters: - name: envoy.listener.proxy_protocol - name: envoy.listener.tls_inspector
For Http Listener filter in Nginx, follow this
Now Check the logs of the Nginx, Envoy, or whichever you are using and we got Source IP ! Awesome ?
Note: Enable Envoy proxy to print logs to stdout using:
A journey of thousand miles begins with a single step, similarly, we are done with our initial step of getting source IP and we are ready to do awesome stuff with it.
In the coming posts, we will see “IP Whitelisting using Istio Policy on Kubernetes Microservices” and “kibana dashboard for IP logs using EFK”.
More on Proxy protocol, Reverse proxy?
Accepting the PROXY Protocol in Nginx
Configure Proxy Protocol Support for Your Classic Load Balancer
Forward and Reverse Proxy
4 thoughts on “Preserve Source IP In AWS Classic Load-Balancer And Istio’s Envoy Using Proxy Protocol”
Hello Arpeet, Which version of the envoy proxy you used? We want to the implement the proxy protocol on the LB (AWS classic LB), later on want to extract the proxy header on the Istio sidecar and then a add them into custom header of our software. Reason being our software don’t support proxy headers to want to utilize the istio sidecar to do that handling. Do you think it is possible and on which version of envoy proxy?
Que-Which version of the envoy proxy you used?
Solution- I’m using Istio version 1.4.3 which uses envoy version 1.12.0
Que-later on want to extract the proxy header on the Istio sidecar and then add them into the custom header of our software.
Solution-You can extract proxy header using Envoyfilter’s Lua script.
For example, the IP address in the proxy protocol store in “X-Forwarded-For” Header.
Then can Extract that value in that header and store it in the new custom header, like I have created custom header “my-custom-header”.
local xff_header = request_handle:headers():get(“X-Forwarded-For”)
local first_ip = string.gmatch(xff_header, “(%d+.%d+.%d+.%d+)”)();
Thanks a lot Arpeet!! 🙂
LikeLiked by 2 people