Skip to main content

Glasnostic for Kubernetes - Troubleshooting

Logging

Glasnosticd log level

To ease troubleshooting potential issues within the glasnosticd containers you can change the log level. The log level is configured via an environment variable set in a ConfigMap. The config map will look similar to the one below and you can use standard Kubernetes tooling to edit the config map and change the log level.

apiVersion: v1
kind: ConfigMap
metadata:
name: glasnosticd-config
data:
NETWORK_KEY: "yournetworkkey"
IGNORE_PASSTHROUGH_REPORT: "true"
LOG_LEVEL: "debug"

Apply the change configuration and restart all glasnosticd pods in your cluster so they pick up the new configuration. Observe the log output and see more details that will help you identify problems.

Sidecar log level

To change the log level in the sidecar, injected into your Pods automatically, you need to configure your Pod or Deployment directly. To do this set an annotation on the Pod or Deployment directly. This will change Envoy's log level directly in the sidecar to enable developers to see more log messages while troubleshooting.

To set the annotation on a single Pod, apply the below annotation to your pod definition.

apiVersion: v1
kind: Pod
metadata:
name: my-pod
annotations:
sidecar.glasnostic.io/logLevel: debug
...

For pods created by a Deployment, the annotation could be added to spec.template.metadata:

apiVersion: apps/v1
kind: Deployment
metadata:
name: my-deployment
spec:
template:
metadata:
annotations:
sidecar.glasnostic.io/logLevel: debug
...

After applying the defintion to your cluster every new Pod will have the annotation set and you will see more log output in the respective container log file. To apply this immediately to any Deployment remove any running Pod within the Deployment and have Kubernetes re-create the Pod automatically.