Metrics and Logs FAQ

You can collect telemetry data about Istio using Prometheus. And then use Prometheus’s HTTP API to query that data.

In-proxy telemetry (aka v2) reduces resource cost and improves proxy performance as compared to the Mixer-based telemetry (aka v1) approach, and is the preferred mechanism for surfacing telemetry in Istio. However, there are few differences in reported telemetry between v1 and v2 which are listed below:

• Missing labels for out-of-mesh traffic In-proxy telemetry relies on metadata exchange between Envoy proxies to gather information like peer workload name, namespace and labels. In Mixer-based telemetry this functionality was performed by Mixer as part of combining request attributes with the platform data. This metadata exchange is performed by the Envoy proxies by adding a specific HTTP header for HTTP protocol or augmenting ALPN protocol for TCP protocol as described here. This requires Envoy proxies to be injected at both the client & server workloads, implying that the telemetry reported when one peer is not in the mesh will be missing peer attributes like workload name, namespace and labels. However, if both peers have proxies injected all the labels mentioned here are available in the generated metrics.

• TCP metadata exchange requires mTLS TCP metadata exchange relies on the Istio ALPN protocol which requires mutual TLS (mTLS) to be enabled for the Envoy proxies to exchange metadata successfully. This implies that if mTLS is not enabled in your cluster, telemetry for TCP protocol will not include peer information like workload name, namespace and labels.

• No mechanism for configuring custom buckets for histogram metrics Mixer-based telemetry supported customizing buckets for histogram type metrics like request duration and TCP byte sizes. In-proxy telemetry has no such available mechanism. Additionally, the buckets available for latency metrics in in-proxy telemetry are in milliseconds as compared to seconds in Mixer-based telemetry. However, more buckets are available by default in in-proxy telemetry for latency metrics at the lower latency levels.

• No metric expiration for short-lived metrics Mixer-based telemetry supported metric expiration whereby metrics which were not generated for a configurable amount of time were de-registered for collection by Prometheus. This is useful in scenarios where short-lived jobs surface telemetry only for a short amount of time, and de-registering the metrics prevents reporting of metrics which would no longer change in the future, thereby reducing network traffic and storage in Prometheus. This expiration mechanism is not available in in-proxy telemetry.

It is sometimes useful to exclude access to specific URLs from being reported. For example, you might wish to exclude some health-checking URLs. You can skip telemetry reports specific to certain URLs by excluding them using a match clause of the telemetry configuration such as:

match: source.name != "health"

You can use docker-compose to install Prometheus. Also, without the Kubernetes API server, components such as Mixer will need local configuration for rules/handlers/instances.

You can enable tracing to determine the flow of a request in Istio.

Additionally, you can use the following commands to know more about the state of the mesh:

• istioctl proxy-config: Retrieve information about proxy configuration when running in Kubernetes:

  # Retrieve information about bootstrap configuration for the Envoy instance in the specified pod.
  $ istioctl proxy-config bootstrap productpage-v1-bb8d5cbc7-k7qbm
  
  # Retrieve information about cluster configuration for the Envoy instance in the specified pod.
  $ istioctl proxy-config cluster productpage-v1-bb8d5cbc7-k7qbm
  
  # Retrieve information about listener configuration for the Envoy instance in the specified pod.
  $ istioctl proxy-config listener productpage-v1-bb8d5cbc7-k7qbm
  
  # Retrieve information about route configuration for the Envoy instance in the specified pod.
  $ istioctl proxy-config route productpage-v1-bb8d5cbc7-k7qbm
  
  # Retrieve information about endpoint configuration for the Envoy instance in the specified pod.
  $ istioctl proxy-config endpoints productpage-v1-bb8d5cbc7-k7qbm
  
  # Try the following to discover more proxy-config commands
  $ istioctl proxy-config --help
  

• kubectl get: Gets information about different resources in the mesh along with routing configuration:

  # List all virtual services
  $ kubectl get virtualservices
  

• Mixer access logs: Mixer writes access logs that contain information about requests. You can get them with:

  # Fill <istio namespace> with the namespace of your istio mesh. Ex: istio-system
  $ TELEMETRY_POD=`kubectl get po -n <istio namespace> | grep istio-telemetry | awk '{print $1;}'`
  $ kubectl logs $TELEMETRY_POD -c mixer  -n istio-system  | grep accesslog
  

Yes. Istio ships with configuration for Prometheus that enables collection of application metrics when mutual TLS is enabled or disabled.

The kubernetes-pods job collects application metrics from pods in environments without mutual TLS. The kubernetes-pods-istio-secure job collects metrics from application pods when mutual TLS is enabled for Istio.

Both jobs require that the following annotations are added to any deployments from which application metric collection is desired:

• prometheus.io/scrape: "true"
• prometheus.io/path: "<metrics path>"
• prometheus.io/port: "<metrics port>"

A few notes:

• If the Prometheus pod started before the Istio Citadel pod could generate the required certificates and distribute them to Prometheus, the Prometheus pod will need to be restarted in order to collect from mutual TLS-protected targets.
• If your application exposes Prometheus metrics on a dedicated port, that port should be added to the service and deployment specifications.