Direct encrypted traffic from IBM Cloud Kubernetes Service Ingress to Istio Ingress Gateway

Configure the IBM Cloud Kubernetes Service Application Load Balancer to direct traffic to the Istio Ingress gateway with mutual TLS

By Vadim Eisenberg (IBM) | May 15, 2020 (updated on July 17, 2020) | 10 minute read

In this blog post I show how to configure the Ingress Application Load Balancer (ALB) on IBM Cloud Kubernetes Service (IKS) to direct traffic to the Istio ingress gateway, while securing the traffic between them using mutual TLS authentication.

When you use IKS without Istio, you may control your ingress traffic using the provided ALB. This ingress-traffic routing is configured using a Kubernetes Ingress resource with ALB-specific annotations. IKS provides a DNS domain name, a TLS certificate that matches the domain, and a private key for the certificate. IKS stores the certificates and the private key in a Kubernetes secret.

When you start using Istio in your IKS cluster, the recommended method to send traffic to your Istio enabled workloads is by using the Istio Ingress Gateway instead of using the Kubernetes Ingress. One of the main reasons to use the Istio ingress gateway is the fact the ALB provided by IKS will not be able to communicate directly with the services inside the mesh when you enable STRICT mutual TLS. During your transition to having only Istio ingress gateway as your main entry point, you can continue to use the traditional Ingress for non-Istio services while using the Istio ingress gateway for services that are part of the mesh.

IKS provides a convenient way for clients to access Istio ingress gateway by letting you register a new DNS subdomain for the Istio gateway’s IP with an IKS command. The domain is in the following format: <cluster_name>-<globally_unique_account_HASH>-0001.<region>.containers.appdomain.cloud, for example mycluster-a1b2cdef345678g9hi012j3kl4567890-0001.us-south.containers.appdomain.cloud. In the same way as for the ALB domain, IKS provides a certificate and a private key, storing them in another Kubernetes secret.

This blog describes how you can chain together the IKS Ingress ALB and the Istio ingress gateway to send traffic to your Istio enabled workloads while being able to continue using the ALB specific features and the ALB subdomain name. You configure the IKS Ingress ALB to direct traffic to the services inside an Istio service mesh through the Istio ingress gateway, while using mutual TLS authentication between the ALB and the gateway. For the mutual TLS authentication, you will configure the ALB and the Istio ingress gateway to use the certificates and keys provided by IKS for the ALB and NLB subdomains. Using certificates provided by IKS saves you the overhead of managing your own certificates for the connection between the ALB and the Istio ingress gateway.

You will use the NLB subdomain certificate as the server certificate for the Istio ingress gateway as intended. The NLB subdomain certificate represents the identity of the server that serves a particular NLB subdomain, in this case, the ingress gateway.

You will use the ALB subdomain certificate as the client certificate in mutual TLS authentication between the ALB and the Istio Ingress. When ALB acts as a server it presents the ALB certificate to the clients so the clients can authenticate the ALB. When ALB acts as a client of the Istio ingress gateway, it presents the same certificate to the Istio ingress gateway, so the Istio ingress gateway could authenticate the ALB.

Traffic to the services without an Istio sidecar can continue to flow as before directly from the ALB.

The diagram below exemplifies the described setting. It shows two services in the cluster, service A and service B. service A has an Istio sidecar injected and requires mutual TLS. service B has no Istio sidecar. service B can be accessed by clients through the ALB, which directly communicates with service B. service A can be also accessed by clients through the ALB, but in this case the traffic must pass through the Istio ingress gateway. Mutual TLS authentication between the ALB and the gateway is based on the certificates provided by IKS. The clients can also access the Istio ingress gateway directly. IKS registers different DNS domains for the ALB and for the ingress gateway.

A cluster with the ALB and the Istio ingress gateway

Initial setting

Create the httptools namespace and enable Istio sidecar injection:

$ kubectl create namespace httptools
$ kubectl label namespace httptools istio-injection=enabled
namespace/httptools created
namespace/httptools labeled

Deploy the httpbin sample to httptools:

$ kubectl apply -f @samples/httpbin/httpbin.yaml@ -n httptools
service/httpbin created
deployment.apps/httpbin created

Create secrets for the ALB and the Istio ingress gateway

IKS generates a TLS certificate and a private key and stores them as a secret in the default namespace when you register a DNS domain for an external IP by using the ibmcloud ks nlb-dns-create command. IKS stores the ALB’s certificate and private key also as a secret in the default namespace. You need these credentials to establish the identities that the ALB and the Istio ingress gateway will present during the mutual TLS authentication between them. You will configure the ALB and the Istio ingress gateway to exchange these certificates, to trust the certificates of one another, and to use their private keys to encrypt and sign the traffic.

Store the name of your cluster in the CLUSTER_NAME environment variable:
```
$ export CLUSTER_NAME=<your cluster name>
```

Store the domain name of your ALB in the ALB_INGRESS_DOMAIN environment variable:

$ ibmcloud ks cluster get --cluster $CLUSTER_NAME | grep Ingress
Ingress Subdomain:              <your ALB ingress domain>
Ingress Secret:                 <your ALB secret>

$ export ALB_INGRESS_DOMAIN=<your ALB ingress domain>
$ export ALB_SECRET=<your ALB secret>

Store the external IP of your istio-ingressgateway service in an environment variable.

$ export INGRESS_GATEWAY_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
$ echo INGRESS_GATEWAY_IP = $INGRESS_GATEWAY_IP

Create a DNS domain and certificates for the IP of the Istio Ingress Gateway service:

$ ibmcloud ks nlb-dns create classic --cluster $CLUSTER_NAME --ip $INGRESS_GATEWAY_IP --secret-namespace istio-system
Host name subdomain is created as <some domain>

Store the domain name from the previous command in an environment variable:
```
$ export INGRESS_GATEWAY_DOMAIN=<the domain from the previous command>
```

List the registered domain names:

$ ibmcloud ks nlb-dnss --cluster $CLUSTER_NAME
Retrieving host names, certificates, IPs, and health check monitors for network load balancer (NLB) pods in cluster <your cluster>...
OK
Hostname                          IP(s)                       Health Monitor   SSL Cert Status   SSL Cert Secret Name                          Secret Namespace
<your ingress gateway hostname>   <your ingress gateway IP>   None             created           <the matching secret name>           istio-system
...

Wait until the status of the certificate (the fourth field) of the new domain name becomes enabled (initially it is pending).

Store the name of the secret of the new domain name:

$ export INGRESS_GATEWAY_SECRET=<the secret's name as shown in the SSL Cert Secret Name column>

Extract the certificate and the key from the secret provided for the ALB:

$ mkdir alb_certs
$ kubectl get secret $ALB_SECRET --namespace=default -o yaml | grep 'tls.key:' | cut -f2 -d: | base64 --decode > alb_certs/client.key
$ kubectl get secret $ALB_SECRET --namespace=default -o yaml | grep 'tls.crt:' | cut -f2 -d: | base64 --decode > alb_certs/client.crt
$ ls -al alb_certs
-rw-r--r--   1 user  staff  3738 Sep 11 07:57 client.crt
-rw-r--r--   1 user  staff  1675 Sep 11 07:57 client.key

Download the issuer certificate of the Let’s Encrypt certificate, which is the issuer of the certificates provided by IKS. You specify this certificate as the certificate of a certificate authority to trust, for both the ALB and the Istio ingress gateway.
```
$ curl https://letsencrypt.org/certs/trustid-x3-root.pem --output trusted.crt
```
Create a Kubernetes secret to be used by the ALB to establish mutual TLS connection.
The certificates provided by IKS expire every 90 days and are automatically renewed by IKS 37 days before they expire. You will have to recreate the secrets by rerunning the instructions of this section every time the secrets provided by IKS are updated. You may want to use scripts or operators to automate this and keep the secrets in sync.
```
$ kubectl create secret generic alb-certs -n istio-system --from-file=trusted.crt --from-file=alb_certs/client.crt --from-file=alb_certs/client.key
secret "alb-certs" created
```

For mutual TLS, a separate Secret named <tls-cert-secret>-cacert with a cacert key is needed for the ingress gateway.

$ kubectl create -n istio-system secret generic $INGRESS_GATEWAY_SECRET-cacert --from-file=ca.crt=trusted.crt
secret/cluster_name-hash-XXXX-cacert created

Configure a mutual TLS ingress gateway

In this section you configure the Istio ingress gateway to perform mutual TLS between external clients and the gateway. You use the certificates and the keys provided to you for the ingress gateway and the ALB.

Define a Gateway to allow access on port 443 only, with mutual TLS:

$ kubectl apply -n httptools -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: default-ingress-gateway
spec:
  selector:
    istio: ingressgateway # use istio default ingress gateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: MUTUAL
      credentialName: $INGRESS_GATEWAY_SECRET
    hosts:
    - "$INGRESS_GATEWAY_DOMAIN"
    - "httpbin.$ALB_INGRESS_DOMAIN"
EOF

Configure routes for traffic entering via the Gateway:

$ kubectl apply -n httptools -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: default-ingress
spec:
  hosts:
  - "$INGRESS_GATEWAY_DOMAIN"
  - "httpbin.$ALB_INGRESS_DOMAIN"
  gateways:
  - default-ingress-gateway
  http:
  - match:
    - uri:
        prefix: /status
    route:
    - destination:
        port:
          number: 8000
        host: httpbin.httptools.svc.cluster.local
EOF

Send a request to httpbin by curl, passing as parameters the client certificate (the --cert option) and the private key (the --key option):

$ curl https://$INGRESS_GATEWAY_DOMAIN/status/418 --cert alb_certs/client.crt  --key alb_certs/client.key

-=[ teapot ]=-

   _...._
 .'  _ _ `.
| ."` ^ `". _,
\_;`"---"`|//
  |       ;/
  \_     _/
    `"""`

Remove the directories with the ALB and ingress gateway certificates and keys.
```
$ rm -r alb_certs trusted.crt
```

Configure the ALB

You need to configure your Ingress resource to direct traffic to the Istio ingress gateway while using the certificate stored in the alb-certs secret. Normally, the ALB decrypts HTTPS requests before forwarding traffic to your apps. You can configure the ALB to re-encrypt the traffic before it is forwarded to the Istio ingress gateway by using the ssl-services annotation on the Ingress resource. This annotation also allows you to specify the certificate stored in the alb-certs secret, required for mutual TLS.

Configure the Ingress resource for the ALB. You must create the Ingress resource in the istio-system namespace in order to forward the traffic to the Istio ingress gateway.

$ kubectl apply -f - <<EOF
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: alb-ingress
  namespace: istio-system
  annotations:
    ingress.bluemix.net/ssl-services: "ssl-service=istio-ingressgateway ssl-secret=alb-certs proxy-ssl-name=$INGRESS_GATEWAY_DOMAIN"
spec:
  tls:
  - hosts:
    - httpbin.$ALB_INGRESS_DOMAIN
    secretName: $ALB_SECRET
  rules:
  - host: httpbin.$ALB_INGRESS_DOMAIN
    http:
      paths:
      - path: /status
        backend:
          serviceName: istio-ingressgateway
          servicePort: 443
EOF

Test the ALB ingress:

$ curl https://httpbin.$ALB_INGRESS_DOMAIN/status/418

-=[ teapot ]=-

   _...._
 .'  _ _ `.
| ."` ^ `". _,
\_;`"---"`|//
  |       ;/
  \_     _/
    `"""`

Congratulations! You configured the IKS Ingress ALB to send encrypted traffic to the Istio ingress gateway. You allocated a host name and certificate for your Istio ingress gateway and used that certificate as the server certificate for Istio ingress gateway. As the client certificate of the ALB you used the certificate provided by IKS for the ALB. Once you had the certificates deployed as Kubernetes secrets, you directed the ingress traffic from the ALB to the Istio ingress gateway for some specific paths and used the certificates for mutual TLS authentication between the ALB and the Istio ingress gateway.

Cleanup

Delete the Gateway configuration, the VirtualService, and the secrets:

$ kubectl delete ingress alb-ingress -n istio-system
$ kubectl delete virtualservice default-ingress -n httptools
$ kubectl delete gateway default-ingress-gateway -n httptools
$ kubectl delete secrets alb-certs -n istio-system
$ rm -rf alb_certs trusted.crt
$ unset CLUSTER_NAME ALB_INGRESS_DOMAIN ALB_SECRET INGRESS_GATEWAY_DOMAIN INGRESS_GATEWAY_SECRET

Shutdown the httpbin service:

$ kubectl delete -f @samples/httpbin/httpbin.yaml@ -n httptools

Delete the httptools namespace:
```
$ kubectl delete namespace httptools
```