Custom Logs on AKS & Azure Monitor


Let’s look at a concrete problem:

  • I have containers deployed in AKS
  • Those container log into custom files
  • I want to analyse those logs using Azure Monitor (Log Analytics)

We’ll look at how to do that.

We leverage Azure Monitor for containers.

Custom logs in Log Analytics also is interesting. That component allows us to collect files on VMs and parse them given a schema. Unfortunately, to this date (end-of-January 2019), there are no integration between that feature and Azure Monitor for containers. This makes it harder but not impossible.

As usual, the code used here is available on GitHub.

Kubernetes’ recommendations

Let’s first look at what Kubernetes online documentation has to say.

The Logging Architecture page gives a few options on how to integrate a logging agent.

By default, Kubernetes considers logs as something coming out of the standard output and standard error of a container. This is what the documentation refers to as basic logging. Those logs can be consumed with kubectl logs.

One of the documented recommendation is to use an node-agent. This is what Azure Monitor for containers does: it deploys an agent on each node with a Daemon set:

$ kubectl get ds --all-namespaces=true
NAMESPACE     NAME                DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
kube-system   kube-proxy          3         3         3       3            3           beta.kubernetes.io/os=linux   4h
kube-system   kube-svc-redirect   3         3         3       3            3           beta.kubernetes.io/os=linux   4h
kube-system   omsagent            3         3         3       3            3           beta.kubernetes.io/os=linux   4h

We see the omsagent as the last daemon set.

Azure Monitor for containers will gather different Kubernetes metrics and will gather container basic logs, i.e. stdout / stderr.

This doesn’t help us directly with custom files.

Another approach suggested by Kubernetes documentation is to use a sidecar container to expose the logs. Basically, the sidecar reads the files and output them in its standard output. The logs can then be read by a node agent.

Another approach is to deploy a container agent in a sidecar container. Unfortunately, to this date (end of January 2019), Azure Monitor doesn’t support this scenario.

Finally, the last approach is to have the application push the logs to the logging back end. Although this is supported via Azure Monitor API, it is the most difficult to implement. We won’t cover it here.

Using Azure Monitor / Log Analytics

Now let’s look at how to implement those suggestions using Azure Monitor / Log Analytics.

We’ll go from the easiest to the hardest scenario to implement.

All those examples are based on a cluster where the monitoring add on has been activated.

Ingesting standard output

The simplest case is where the logs are outputted to the standard output.

Let’s deploy a pod that does just that:

apiVersion: v1
kind: Pod
metadata:
  name: standard-output-pod
spec:
  containers:
  - name: main-container
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "$i: $(date) dog";
        i=$((i+1));
        sleep 1;
      done

We can check the logs it produces:

$ kubectl apply -f https://raw.githubusercontent.com/vplauzon/aks/master/custom-logs/standard-output.yaml
$ kubectl get pods
NAME                  READY   STATUS    RESTARTS   AGE
standard-output-pod   1/1     Running   0          22m

$ kubectl logs standard-output-pod
0: Mon Jan 28 20:54:47 UTC 2019 dog
1: Mon Jan 28 20:54:48 UTC 2019 dog
2: Mon Jan 28 20:54:49 UTC 2019 dog
3: Mon Jan 28 20:54:50 UTC 2019 dog
...

If we give a few minutes for Azure Monitor to gather the logs, we can use the following query:

Log query

Hence this solution fits nicely with Azure monitor for Containers.

Ingesting a single file via a sidecar

Now, let’s consider a scenario where we have a container logging to a single file.

As suggested in
Kubernetes documentation, we can use a sidecar container to expose the logs.

Let’s simulate it this way:

apiVersion: v1
kind: Pod
metadata:
  name: single-file-to-output-pod
spec:
  containers:
  - name: main-container
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "$i: $(date) dog" >> /var/log/mylogs/log.log;
        i=$((i+1));
        sleep 1;
      done
    volumeMounts:
    - name: logs
      mountPath: /var/log/mylogs
  - name:  log-display
    image: busybox
    args: [/bin/sh, -c, 'tail -n+1 -f /var/log/mylogs/log.log']
    volumeMounts:
    - name: logs
      mountPath: /var/log/mylogs
  volumes:
  - name: logs
    emptyDir: {}

Here we have a multi-container pod. The two pods share a mounted volume of type emptyDir. That volume is bound to the pod and gets deleted with it.

The first container writes in a file while the second one display (using tail) that file.

$ kubectl apply -f https://raw.githubusercontent.com/vplauzon/aks/master/custom-logs/single-file-to-output.yaml
$ kubectl logs single-file-to-output-pod log-display
0: Mon Jan 28 22:21:45 UTC 2019 dog
1: Mon Jan 28 22:21:46 UTC 2019 dog
2: Mon Jan 28 22:21:47 UTC 2019 dog
...

Similarly, those logs get ingested by Azure Monitor for Containers.

Ingesting multiple files via a sidecar

Now, rarely does an application logs on the same file forever. Typically, applications log on a file for a while (e.g. an hour) before creating a new file.

We can use the same method than in the previous section and simulate it this way:

apiVersion: v1
kind: Pod
metadata:
  name: multiple-file-to-output-pod
spec:
  containers:
  - name: main-container
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "$i: $(date) dog" >> /var/log/mylogs/1.log;
        echo "$i: $(date) cat" >> /var/log/mylogs/2.log;
        i=$((i+1));
        sleep 1;
      done
    volumeMounts:
    - name: logs
      mountPath: /var/log/mylogs
  - name:  log-display
    image: busybox
    args: [/bin/sh, -c, 'tail -n+1 -f /var/log/mylogs/*.log']
    volumeMounts:
    - name: logs
      mountPath: /var/log/mylogs
  volumes:
  - name: logs
    emptyDir: {}

Here we write to 2 files all the time.

We can test it:

$ kubectl apply -f https://raw.githubusercontent.com/vplauzon/aks/master/custom-logs/multiple-file-to-output.yaml
$ kubectl logs multiple-file-to-output-pod log-display
==> /var/log/mylogs/1.log <==
0: Mon Jan 28 22:53:00 UTC 2019 dog

==> /var/log/mylogs/2.log <==
0: Mon Jan 28 22:53:00 UTC 2019 cat

==> /var/log/mylogs/1.log <==
1: Mon Jan 28 22:53:01 UTC 2019 dog

==> /var/log/mylogs/2.log <==
1: Mon Jan 28 22:53:01 UTC 2019 cat
...

Again, this will be ingested by Azure Monitor for containers.

On first level it might seem that there is a lot of noise in those logs since we keep switching files. The pattern is more likely to be that a single file is going to be appended to for an hour before a new file takes over. So, there would be very little of this “==>” noise.

This noise could be filtered by piping the output into a shell command filtering. It could also be filtered against in queries.

Ingesting multiple files on the host

Another angle to this problem is to use Custom logs in Log Analytics. This is typically used for gathering log files on VMs.

What we can do is to mount an hostPath volume on our main container. A host path volume is a path on the host of the pod. So basically, we would log directly on the host running the pod and then have the agent running on the node gathering those files.

This does work but we do not recommend this approach for multiple reasons:

  1. Logging directly on the host could eventually saturate the host file system. We could paliate to that by using logrotate in a daemon set ; but that adds considerable complexity.
  2. By default, the monitoring addon on AKS doesn’t enlist log ingestion on the host, only on the platform (e.g. Kubernetes metrics) and containers. We need to manually enlist the hosts in a Log Analytics workspace or rely on the “default workspace” enforced by Azure Security Center.
  3. If we apply that to muliple solutions, we would need a different host-folder for each solution in order for them not to clash. This means we would replicate the pod-model on the host which is a little awkward.

For both those reasons, that solution is more complex. There could also be complex edge cases. For instance, if 2 pods get schedule on the same node, would they start writing on the same files on the host?

In general, we advise against touching the hosts.

Nevertheless, if we want to do it, we could simulate it like this:

apiVersion: v1
kind: Pod
metadata:
  name: multiple-file-to-host-pod
spec:
  containers:
  - name: main-container
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "$i: $(date) dog" >> /var/log/mylogs/1.log;
        echo "$i: $(date) cat" >> /var/log/mylogs/2.log;
        i=$((i+1));
        sleep 1;
      done
    volumeMounts:
    - name: logs
      mountPath: /var/log/mylogs
  volumes:
  - name: logs
    hostPath:
      # directory location on host
      path: /var/log/my-app-logs
      type: Directory

We would then follow the custom logs procedure.

Summary

We showed different way to ingest custom logs from AKS to Azure Monitor.

The easiest path is to leverage Azure Monitor for Containers. This requires us to output the logs to the standard output of a container somehow. Typically, we use a side car container to display the files written by another container.

We also discussed of using custom logs ingestion on the host. We do not encourage using that solution.


One thought on “Custom Logs on AKS & Azure Monitor

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s