2022.08.30

Hello boot / test metrics

This application is part of an exercise to comprehend how metrics merging works in Istio.

Description of metrics merging (as I understand it)

The istio-agent process in the sidecar exposes the scrape endpoint to prometheus.

This scrape endpoint is implemented by merging (or aggregating) the metrics from two sources:

  • the application’s scrape endpoint
  • the envoy sidecar’s scrape endpoint, which collects its own networking-related metrics.

Related

Writeup by Rob Salmond, which is a little obfuscated, but worth a read.

His blog entry references the metrics merging design doc.

The design doc mentions the construction of an “internal” environment variable ISTIO_PROMETHEUS_ANNOTATIONS derived from the information in the annotations, that tells the agent the url of the app/workload’s scrape endpoint.

How to communicate the url of the app scrape endpoint

The convention appears to be as follows:

On the app workload pod manifest, add the annotations:

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"
  prometheus.io/path: "/actuator/prometheus/"

The istio-agent will use that information to locate the application’s scrape endpoint, and take care to aggregate the metrics together with envoy’s collected metrics/scrape endpoint.

Building the app

  1. Produce a local image named test-metrics:<version> (the version is specified in the gradle build file), tag it and push it to a local registry (assumption: using a local k8s cluster configured with a local registry exposed to k8s on port 5000 and exposed on localhost on <port>, see how to do this with k3d).

    gradle bootBuildImage --imageName=localhost:<port>/test-metrics:<version> --publishImage

To test this

  1. Create a hello-world style spring boot app with the micrometer prometheus dependency and check that you can hit that endpoint (this project).

  2. Deploy the Istio Prometheus add-on.

    k apply -f ${ISTIO_DIR}/samples/addons/prometheus.yaml

  3. Construct a k8s deployment manifest for this app (see manifests/hello-boot.yaml), and deploy the app to an istio-enabled k8s cluster.

    k apply -f manifests/hello-boot.yaml

  4. Verify that the workload metrics endpoint indeed functions.

    k exec -it helloboot-v1-<tab> -c istio-proxy -- curl localhost:8080/actuator/prometheus

  5. Check envoy’s prometheus scrape endpoint.

    k exec -it helloboot-v1-<tab> -c istio-proxy -- curl localhost:15090/stats/prometheus

  6. Locate and check the value of the ISTIO_PROMETHEUS_ANNOTATIONS environment variable inside the sidecar container.

    k exec -it helloboot-v1-<tab> -c istio-proxy -- /bin/sh
echo $ISTIO_PROMETHEUS_ANNOTATIONS

    The output should look like this:

    {"scrape":"true","path":"/actuator/prometheus","port":"8080"}

  7. Verify that the aggregated endpoint response includes both envoy and app metrics

    k exec -it helloboot-v1-585487d5f7-9d27j -c istio-proxy -- curl localhost:15020/stats/prometheus

  8. Give it a couple of minutes, and check the prometheus dashboard for the presence of app workload metrics, e.g. http_server_requests_seconds_count or jvm_memory_max_bytes (or any custom metrics you choose to expose in your app via micrometer).

    istioctl dashboard prometheus

GitHub

View Github