Skip to main content

Glasnostic for Istio

caution

Help us test Glasnostic for Istio. Request early access by sending an email to info@glasnostic.com

Requirements

Kubernetes cluster and tools

You need a running Kubernetes cluster, using any Kubernetes version 1.16 or higher. Glasnostic officially supports Kubernetes deployments on AWS EKS, Azure AKS and Rancher.

To successfully deploy and explore Glasnostic, your clusters worker node(s) should have at least 4 vCPU, 16GB of memory and 50GB of storage. See the Systems requirements for additional information for recommended sizing.

Istio

Glasnostic/Istio currently supports the following Istio versions:

  • 1.14
  • 1.13
  • 1.12

Older releases may work but are not actively tested by Glasnostic anymore. Glasnostic/Istio is known to work with the minimal Istio profile. To install Istio, follow the instructions on the Istio website.

You need to run Istio version 1.12 or higher, older versions are not supported. Refer to the Istio documentation for setup and configuration options.

Install Glasnostic

Preparation

To connect your Glasnostic installation to the Glastnostic platform you need to provide the network ID. Set the key in the file glasnostic-istio.yaml by replacing <PUT_YOUR_KEY_HERE> with the key's value at the top of the file.

Configure Istio sidecar injection

First, install the Istio component of Glasnostic, which is built as a WebAssembly module that get's loaded into the istio-proxy/Envoy sidecar. Initially this WASM module needs to be downloaded and mounted as a Kubernetes volume.

Run the following commands to load Glasnostic/Istio onto your cluster.

kubectl create namespace glasnostic-system
kubectl apply -f wasm-cache.yaml

Verify the successful deployment by running

kubectl get ds -n glasnostic-system

Next modify the Istio sidecar injector configuration to load the WASM module automatically. We provide a small script which will update the configuration and append the userVolume setting.

Ensure kubectl points to the right configuration and context and then run

scripts/patch.sh

This commands creates a backup of the existing Istio injection configuration and store it in a file named inject-config.yaml.bak. Make sure to keep this file if you want to restore the injector setup to it's previous state.

Note that the configuration is not applied to existing pods - they need to be restarted to have the changes take effect.

Install Glasnostic Data Plane

The final step is to install Glasnostic/Istio daemon. Make sure you have set the right Network ID and then run:

kubectl apply -f glasnostic-istio.yaml

Glasnostic is now added to your Istio installation. Log in to the Glasnostic Console to access the data.

External Communication

Using Glasnostic on Istio with Kubernetes, we're dealing with the following external traffic:

The pods of the glasnostic-istio deployment must be allowed to connect via TCP port 443 to glasnostic.com and commander.glasnostic.com to send metrics data and to be able to use the control plane running at glasnostic.com.

Additionally, if you're using an HTTP proxy, you'll have to configure the HTTPS_PROXY variable to point to your HTTP proxy. You have to add this variable to the config map glasnostic-config found in the file glasnostic-istio.yaml.

Troubleshooting

Setting the cluster domain

Glasnostic on Istio is automatically retrieving the cluster's domain (per default, that is svc.cluster.local.) by querying the DNS name of the API service (kubernetes.default). If this probing is not successful, Glasnostic on Istio will exit with an error message. In that case, it's possible to set the cluster's domain by setting the environment variable CLUSTER_DOMAIN (it's in FQDN format, e.g. svc.cluster.local.).

Traffic is shown with label Passthrough Cluster

The label Passthrough Cluster is usually used to mark ingress or egress traffic. However, under heavy load, the Istio sidecar proxy might not distinguish such traffic from regular in-cluster traffic. Hence, traffic can be shown with the label Passthrough Cluster.

The background is that Istio is checking for each connection the protocol type and is either using a TCP or an HTTP filter. TCP is used as a fallback, and HTTP filters are not called if this protocol detection times out. If now one sidecar is using TCP and the other one HTTP, we see this artifact.

To mitigate the problem, we can increase the protocolDetectionTimeout setting of Istio or disable the timeout by setting it to 0s (the default for current Istio versions).

To do so, edit the value in Istio's configmap by calling:

kubectl --namespace istio-system edit configmap istio

Uninstall Glasnostic

The follwoing instructions will help you to remove Glasnostic/Istio. The underlying Istio installation will not be removed. Run the following commands to uninstall:

kubectl delete ns glasnostic-system
kubectl -n istio-system delete envoyfilters.networking.istio.io/glasnostic-wasm
scripts/restore.sh

This will uninstall all Glasnostic components. It will also restore and reload the modified Istio injection configuration and clean up any dangling Envoy filter configurations.