Juan Manuel Rey bio photo

Juan Manuel Rey

Unix Geek. Sysadmin by heart turned cloud architect. Working for Microsoft.

Twitter Google+ LinkedIn Github Stackoverflow

In the article about Kubernetes on ACS I briefly touched the topic of Kubernetes Ingress, originally I was going to made a post about Ingress however I thought it woud be better to explain the different methods to expose a Kubernetes based app and how are they implemented on Azure.

The way to expose your app is by using a Kubernetes Service. There are three types of services, or ServiceTypes.

  • ClusterIP
  • NodePort
  • LoadBalancer

And there is also another special way of exposing your workloads known as Kubernetes Ingress. I will briefly touch the first two and go deeper into LoadBalancer and Ingress since those are more relevant for Azure.

ClusterIP ServiceType

ClusterIP is the default Kubernetes ServiceType. When a service is set to ClsuterIP Kubernetes wil expose the service on an cluster internal IP and the service will only be reacheable within the cluster. All the Kuebrnetes services realted to the cluster are exposed as ClusterIP types.

Lets a see a quick example, we will deploy an NGINX server and exposed it with a CluterIP type service.

apiVersion: v1
kind: Service
metadata:
  name: my-nginx
  labels:
    run: my-nginx
spec:
  ports:
  - port: 80
    protocol: TCP
  selector:
    run: my-nginx
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: my-nginx
spec:
  selector:
    matchLabels:
      run: my-nginx
  replicas: 2
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx
        ports:
        - containerPort: 80

Deploy it with kubectl and check the pods and service created.

$ kubectl get pods
NAME                       READY     STATUS    RESTARTS   AGE
my-nginx-5d69b5ff7-4mdcq   1/1       Running   0          1m
my-nginx-5d69b5ff7-c9xxc   1/1       Running   0          1m
$ kubectl get service
NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.0.0.1      <none>        443/TCP   131d
my-nginx     ClusterIP   10.0.199.84   <none>        80/TCP    1m

In an Azure deployed cluster this type of service will be used by inter-cluster service communication like in any other cluster and it will not be reacheable from the outside even if we use the Azure CNI plugin for container networking.

NodePort ServiceType

The NodePort service is no more than a port open in every node of the cluster, Kubernetes will take care of routing the incoming traffic to the service. The port will be allocated by the master(s) from a cluster configured pool, the defaul pool is 30000-32676. Take a look at the below daigram

Of course NodePort services are supported in any Kubernetes cluster running on Azure, including those deployed with ACS, ACS Engine or AKS. However since the nodes aren’t exposed directly to the internet the functionality is very limited, also usign a NodePort service for a production workload is not the best choice due to its limitations, however it can be used as building block for Ingress or a LoadBalancer service.

LoadBalancer ServiceType

The LoadBalancer service is the standard and most common way of exposing your services to the outside world from a Kubernetes cluster running ontop a cloud infrastructure like Microsoft Azure. This ServiceType will leverage the cloud provider built-in mechanism to provision and configrue a load balancer. On Azure this will provision an Azure Load Balancer and configure the load balncing rules, health probes, backend pools and frontend IPs.

The diagram below illustrates this example with two different apps being exposed to the outside via a LoadBalancer service that provisions an Azure Load Balancer with its corresponding public IP address.

The load balancer configuration can also be verified in the Azure portal.

This doesn’t mean that a new Azure LB is created for each service, as shown in the example AKS will deploy an Azure Load Balancer and publish all the services through it until it reaches the maximum allowed vaulues in terms of front’end IPs, rules, etc, and then it will provision a new Azure Load Balancer instance. For example in one of my AKS clusters I have three services exposed through the same load balancer instance.

$ az network lb list
Location       Name                    ProvisioningState    ResourceGroup                           ResourceGuid
-------------  ----------------------  -------------------  --------------------------------------  ------------------------------------
westeurope     k8s-master-lb-73809577  Succeeded            k8s-aad-demo                            a5b483c6-0e16-49ef-b5ac-db2b60bcc4a6
westeurope     vmss1lb                 Succeeded            vmssrgtest                              234ac3b4-1bd1-43f8-92fd-c46399163aa8
canadacentral  kubernetes              Succeeded            MC_aksrg-ready_aks-ready_canadacentral  84686a77-b527-48d3-bfcf-1b51490966ff
$ az network lb frontend-ip list -g MC_aksrg-ready_aks-ready_canadacentral --lb-name kubernetes
Name                              PrivateIpAllocationMethod    ProvisioningState    ResourceGroup
--------------------------------  ---------------------------  -------------------  --------------------------------------
ad0fbc671079611e88e9c0a58ac1f0b1  Dynamic                      Succeeded            MC_aksrg-ready_aks-ready_canadacentral
ad0fe9226079611e88e9c0a58ac1f0b1  Dynamic                      Succeeded            MC_aksrg-ready_aks-ready_canadacentral
a6852cdb8079811e88e9c0a58ac1f0b1  Dynamic                      Succeeded            MC_aksrg-ready_aks-ready_canadacentral
$ az network lb rule list -g MC_aksrg-ready_aks-ready_canadacentral --lb-name kubernetes
  BackendPort  EnableFloatingIp      FrontendPort    IdleTimeoutInMinutes  LoadDistribution    Name                                       Protocol    ProvisioningState    ResourceGroup
-------------  ------------------  --------------  ----------------------  ------------------  -----------------------------------------  ----------  -------------------  --------------------------------------
           80  True                            80                       4  Default             ad0fbc671079611e88e9c0a58ac1f0b1-TCP-80    Tcp         Succeeded            MC_aksrg-ready_aks-ready_canadacentral
         7744  True                          7744                       4  Default             ad0fe9226079611e88e9c0a58ac1f0b1-TCP-7744  Tcp         Succeeded            MC_aksrg-ready_aks-ready_canadacentral
           80  True                            80                       4  Default             a6852cdb8079811e88e9c0a58ac1f0b1-TCP-80    Tcp         Succeeded            MC_aksrg-ready_aks-ready_canadacentral
$ kubectl get service
NAME                    TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)          AGE
brigade-brigade-api     ClusterIP      10.0.210.50    <none>          7745/TCP         5d
brigade-brigade-cr      LoadBalancer   10.0.151.0     52.228.64.137   80:32139/TCP     5d
brigade-brigade-gw      LoadBalancer   10.0.214.191   52.228.26.152   7744:31483/TCP   5d
example-python-python   ClusterIP      10.0.53.32     <none>          80/TCP           5d
kashti-kashti           LoadBalancer   10.0.83.80     52.237.35.43    80:32137/TCP     5d
kubernetes              ClusterIP      10.0.0.1       <none>          443/TCP          5d

The same procedure will be repeated for each app or api that needs external access, for a few of them it can work but in a public cloud provider the public IP addresses are not free and also you will probably exhaust the maximum allowed number public IP addresses, in Azure for example the default limit is 20 public IP addresses per subscription. There is another technical restriction in Azure, the standard Azure Load Balancer is not able to do SSL termination. If you are going to run just a few services then LoadBalancer is pefect solution, and tighly integrated on Azure, however for more complex deployments the best option is to use Kubernetes Ingress.

Internal Load Balancer

By default a LoadBalancer service will provision a public Azure Load Balancer, however an internal one can also be provisioned in order to expose the service to other internal workloads running on you Azure subcription(s). Keep in mind that although this configuration will work with any Kubernetes installation on Azure is only useful with manually or ACS Engine deployed clusters since at this moment custom VNET and VNET Peering are not supported with AKS.

To provision an internal LoadBalancer we will need to add the following annotations to the service.

metadata:
  name: my-nginx
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"

Lets create the example NGINX deployment with LoadBalancer service and the internal annotations.

$ kubectl create -f nginx-svc-ilb.yaml
service "my-nginx" created
deployment "my-nginx" created
$ kubectl get pods
NAME                       READY     STATUS    RESTARTS   AGE
my-nginx-5d69b5ff7-wwcvp   1/1       Running   0          7m
my-nginx-5d69b5ff7-z5xfp   1/1       Running   0          7m
$ kubectl get service
NAME         TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)        AGE
kubernetes   ClusterIP      10.0.0.1      <none>         443/TCP        143d
my-nginx     LoadBalancer   10.0.39.204   10.240.0.127   80:31052/TCP   7m

The field EXTERNAL-IP shows an internal IP address belonging to the same subnet as the nodes of the cluster. Take a look into the Azure portal and verify the new internal load balancer has been created.

Kubernetes Ingress

Ingress is by far the most interesting and powerful of all the methods availabe in Kubernetes to expose your services. In the Kubernetes official documentation Ingress is defined as An API object that manages external access to the services in a cluster, typically HTTP. Ingress can provide load balancing, SSL termination and name-based virtual hosting. However in the real sense Ingress is not a service but a construct that sits on top of your services as an entry point to the cluster, providing simple host and URL based HTTP routing capabilities.

Ingress has two main components:

  • Ingress Resource
  • Ingress Controller

Defined by namespace, the Ingress resource represents the configured routing rules that will govern how the services are accessed from the external user. Using the example from the diagram, if a request for contoso.com reaches the cluster then ingress will route the traffic to Service but if the request is for contoso.com/accounting then ingress will route the traffic to Service D. Following is a common Ingress resource definition.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - host: contoso.com
    http:
      paths:
      - backend:
          serviceName: contoso-service
          servicePort: 80
  - host: fabrikam.net
    http:
      paths:
      - path:
        backend:
          serviceName: fabrikam-service
          servicePort: 80

Deployed as a pod the Ingress Controller will route the traffic to the corresponding service following the rules defined on the Ingress Resouce. On top of Ingress a LoadBalancer service needs to be deployed in order to expose the Ingress controller to the outside, this will be the main entry point into the cluster.

There are several Ingress Controllers available to be used on Azure.

The most common, and almost the de-facto standard in Kubernetes, is NGINX. It is also the recommended one right now for AKS. NGINX is the only one I’ve used, but I am dying to try the other two specially Contour so expect a future blog post on it. For Traefik I found this blog post describing how to use with Azure Container Service.

Deploying Ingress with NGINX Ingress Controller

The installation of Ingress is very well described on AKS documentation but I will briefly describe the process for the sake of completion of the article. Thanks to Helm deploying an Ingress Controller on your Kubernetes cluster is is as easy as:

helm install stable/nginx-ingress

The installation will deploy two pods for the NGINX contoller and two services, one of them of LoadBalancer type.

$ kubectl get pods -o wide
NAME                                                 READY     STATUS             RESTARTS   AGE       IP            NODE
nginx-ingress-controller-5b4b998b67-mlkqj            1/1       Running            0          30d       10.244.5.23   aks-nodepool1-68968121-1
nginx-ingress-default-backend-58bf6f478b-zmj79       1/1       Running            0          30d       10.244.3.16   aks-nodepool1-68968121-3
$ kubectl get service -o wide
NAME                            TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE       SELECTOR
kubernetes                      ClusterIP      10.0.0.1       <none>          443/TCP                      41d       <none>
nginx-ingress-controller        LoadBalancer   10.0.126.42    52.237.39.132   80:30930/TCP,443:31914/TCP   30d       app=nginx-ingress,component=controller,release=nginx-ingress
nginx-ingress-default-backend   ClusterIP      10.0.56.240    <none>          80/TCP                       30d       app=nginx-ingress,component=default-backend,release=nginx-ingress

Create now the Ingress Resource, I am using the above minimal example.

$ kubectl create -f ingress-example.yml
ingress "my-ingress" created
$ kubectl get ingress
NAME         HOSTS                          ADDRESS   PORTS     AGE
my-ingress   www.contoso.com,fabrikam.net             80        9s
$
$ kubectl describe ingress my-ingress
Name:             my-ingress
Namespace:        default
Address:
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host             Path  Backends
  ----             ----  --------
  www.contoso.com
                      contoso-service:80 (<none>)
  fabrikam.net
                      fabrikam-service:80 (<none>)
Annotations:
Events:
  Type    Reason  Age   From                Message
  ----    ------  ----  ----                -------
  Normal  CREATE  25s   ingress-controller  Ingress default/my-ingress

If you want to learn more about Ingress I highly recommend you to read the official Kubernetes Ingress documentation and the docs section of nginx-ingress GitHub repository.

We are done for now, in a future post I will touch the topic of deploying and using Contour on AKS.

– Juanma