Loki eventually running out of disk space

[0x6675636b796f75676974687562]

Our Grafana instance is configured to use Loki as a data source:

Yet, if the deployment has been already running for a while, an attempt to read any data from http://loki-headless:3100 may fail:

$ kubectl port-forward svc/loki-headless 3100:3100
Forwarding from 127.0.0.1:3100 -> 3100
Forwarding from [::1]:3100 -> 3100
Handling connection for 3100
Handling connection for 3100
E1028 19:15:59.198902 1075296 portforward.go:406] an error occurred forwarding 3100 -> 3100: error forwarding port 3100 to pod 2444f44c3633d6be7250dcbe0aa36181a37aaed5433fe380a7616fcb384d
f8f0, uid : failed to execute portforward in network namespace "/var/run/netns/cni-40f89c3a-8386-d8f6-d6d7-832f0b8fea0e": failed to dial 3100: dial tcp4 127.0.0.1:3100: connect: connection refused
E1028 19:15:59.194892 1075296 portforward.go:406] an error occurred forwarding 3100 -> 3100: error forwarding port 3100 to pod 2444f44c3633d6be7250dcbe0aa36181a37aaed5433fe380a7616fcb384d
f8f0, uid : failed to execute portforward in network namespace "/var/run/netns/cni-40f89c3a-8386-d8f6-d6d7-832f0b8fea0e": failed to dial 3100: dial tcp4 127.0.0.1:3100: connect: connection refused
E1028 19:15:59.282113 1075296 portforward.go:234] lost connection to pod

This may be caused by the Loki instance having run out of disk space:

$ kubectl exec -it po/save-cloud-loki-0 -- /bin/sh
/ $ du -hs /var/loki/chunks
9.6G    /var/loki/chunks
/ $ df -h /var/loki/chunks
Filesystem                Size      Used Available Use% Mounted on
/dev/sdb                  9.8G      9.8G         0 100% /var/loki

The effective config of the running instance is located at /etc/loki/config/config.yaml. According to the documentation, for a non-default retention, a compactor section needs to be added to the YAML, e.g.:

compactor:
  working_directory: /data/retention
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  delete_request_store: gcs

[0x6675636b796f75676974687562]

After Loki has been running for ~1 day:

$ kubectl exec -it po/save-cloud-loki-0 -- /bin/sh
/ $ du -hs /var/loki/chunks
8.3M    /var/loki/chunks
/ $ df -h /var/loki/chunks
Filesystem                Size      Used Available Use% Mounted on
/dev/sda                  9.8G     50.0M      9.7G   1% /var/loki