====== Introduction ======

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

Compared to other log aggregation systems, Loki:

  * does not do full text indexing on logs. By storing compressed, unstructured logs and only indexing metadata, Loki is simpler to operate and cheaper to run. <
  * indexes and groups log streams using the same labels you’re already using with Prometheus, enabling you to seamlessly switch between metrics and logs using the same labels that you’re already using with Prometheus. <
  * is an especially good fit for storing Kubernetes Pod logs. Metadata such as Pod labels is automatically scraped and indexed. <
  * has native support in Grafana (needs Grafana v6.0). <

A Loki-based logging stack consists of 3 components:

  * promtail is the agent, responsible for gathering logs and sending them to Loki. <
  * loki is the main server, responsible for storing logs and processing queries. <
  * Grafana for querying and displaying the logs. <

Loki is like Prometheus, but for logs: we prefer a multidimensional label-based approach to indexing, and want a single-binary, easy to operate system with no dependencies. Loki differs from Prometheus by focusing on logs instead of metrics, and delivering logs via push, instead of pull.

====== Links ======

===== General =====

[[https://grafana.com/oss/loki/]]\\
[[https://github.com/grafana/loki/|Loki GitHub repository]]\\
[[https://github.com/grafana/loki/releases|Loki binary releases]]\\

===== Grafana Dashboards =====

[[sw:grafana:start|See Grafana for the dashboards regarding Grafana Loki]]

====== Docs ======

[[https://github.com/grafana/loki/tree/master/docs]]\\
[[https://grafana.com/docs/loki/latest/operations/storage/retention/|Operations - Storage - Retention]]\\
[[https://grafana.com/docs/loki/latest/operations/storage/table-manager/#retention|Operations - Storage - Table Manager - Retention]]\\

====== Files and Directories ======

===== General Files and Directories =====

^ File or Directory                                 ^ Description                                                                            ^
| ''/etc/loki/''                                                               | The //Loki// configuration directory.                     |
| ''/etc/promtail/''                                                       | The //Promtail//  configuration directory.                     |
| ''/etc/promtail/conf.d/''                                            | The //Promtail// drop-in configuration directory.                     |
| ''/usr/sbin/loki''                                                        | The //Loki// server binary.                                     |
| ''/usr/sbin/logcli''                                                     | The ''logcli'' CLI binary.                                         |
| ''/usr/sbin/promtail''                                                | The //Promtail// daemon process binary.                |
| ''/usr/lib/systemd/system/loki.service''                 | The //Loki// systemd unit file.                                |
| ''/usr/lib/systemd/system/promtail.service''         | The //Promtail// systemd unit file.                           |
| ''/srv/loki/''                                                                | The //Loki// server data directory.                         |
| ''/var/lib/promtail/''                                                 | The //Promtail// status directory.                     |

===== Configuration Files =====

^ File or Directory                          ^ Description                                                                           ^
| ''/etc/default/loki''                           | The configuration file with environment variables for the //Loki// server.                 |
| ''/etc/default/promtail''                   | The configuration file with environment variables for //Promtail//.                 |
| ''/etc/loki/loki.yml''                          | The //Loki// server configuration file.                 |
| ''/etc/promtail/promtail.yml''         | The //Promtail// daemon process configuration file.                 |
| ''/etc/promtail/conf.d/*.yml''           | The //Promtail// drop-in //scrape// configuration files.                 |

====== Getting Help ======

//Loki// help:

<cli>
user@host:~$ /usr/sbin/loki -h | less -S
</cli>

//Promtail// help:

<cli>
user@host:~$ /usr/sbin/promtail -h | less -S
</cli>

''logcli'' help:

<cli>
user@host:~$ /usr/sbin/logcli --help 2>&1| less -S
</cli>

====== Install ======

Install [[sw/grafana/start|Grafana]] as a prerequisite.

Download the //Loki// binary releases for ''loki'', ''logcli'' and ''promtail'':

<cli>
root@host:~$ cd /tmp
root@host:~$ wget https://github.com/grafana/loki/releases/download/v1.4.1/loki-linux-arm.zip
root@host:~$ wget https://github.com/grafana/loki/releases/download/v1.4.1/loki-canary-linux-arm.zip
root@host:~$ wget https://github.com/grafana/loki/releases/download/v1.4.1/logcli-linux-arm.zip
root@host:~$ wget https://github.com/grafana/loki/releases/download/v1.4.1/promtail-linux-arm.zip
</cli>

To install //Loki// on Debian: 

<cli>
root@host:~$ cd /tmp
root@host:~$ for FL in {loki,logcli,promtail}*.zip; do unzip -x $FL; done
root@host:~$ rm {loki,logcli,promtail}*.zip
root@host:~$ chmod 755 *-linux-*
root@host:~$ chown root:root *-linux-*
root@host:~$ for FL in *-linux-*; do echo mv -i $FL ${FL%%-li*} ; done
root@host:~$ mv -i logcli loki loki-canary promtail /usr/sbin/
</cli>

Create the //Loki// startup configuration:

<cli> 
root@host:~$ vi /etc/default/loki
</cli> 

File contents:

<file config /etc/default/loki>
#ARGS="-config.file=/etc/loki/loki.yml -log.level=debug"
ARGS="-config.file=/etc/loki/loki.yml"
</file>

Create the //Promtail// startup configuration:

<cli>
root@host:~$ vi /etc/default/promtail
</cli>

File contents:

<file config /etc/default/promtail>
#ARGS="-config.file=/etc/promtail/promtail.yml -log.level=debug"
ARGS="-config.file=/etc/promtail/promtail.yml"
</file>

Create the //Loki// and //Promtail// ''rsyslog'' configuration files:

<cli>
root@host:~$ vi /etc/rsyslog.d/loki.conf
</cli>

File contents:

<file config /etc/rsyslog.d/loki.conf>
template(name="LOKI_TEMPLATE" type="string" string="/var/log/loki/%programname%.log")
if ($programname startswith 'loki') then {
   action(
        type="omfile"
        dynaFile="LOKI_TEMPLATE"
        fileCreateMode="0640"
        fileOwner="grafana"
        fileGroup="grafana"
        dirCreateMode="0750"
        dirOwner="grafana"
        dirGroup="grafana"
        ioBufferSize="64k"
    )
   stop
}
</file>

<cli>
root@host:~$ vi /etc/rsyslog.d/promtail.conf
</cli>

File contents:

<file config /etc/rsyslog.d/promtail.conf>
template(name="PROMTAIL_TEMPLATE" type="string" string="/var/log/promtail/%programname%.log")
if ($programname startswith 'promtail') then {
   action(
        type="omfile"
        dynaFile="PROMTAIL_TEMPLATE"
        fileCreateMode="0640"
        fileOwner="root"
        fileGroup="grafana"
        dirCreateMode="0750"
        dirOwner="root"
        dirGroup="grafana"
        ioBufferSize="64k"
    )
   stop
}
</file>

Restart the ''rsyslog'' daemon:

<cli>
root@host:~$ systemctl restart rsyslog.service
</cli>

Create the //Loki// systemd unit file:

<cli>
root@host:~$ vi /usr/lib/systemd/system/loki.service
</cli> 

File contents:

<file config /usr/lib/systemd/system/loki.service>
[Unit]
Description=Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system
Documentation=https://github.com/grafana/loki
Wants=network-online.target
After=network-online.target

[Service]
User=grafana
Group=grafana
EnvironmentFile=-/etc/default/loki
Restart=on-failure
ExecStart=/usr/sbin/loki $ARGS
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=20s
SendSIGKILL=no

[Install]
WantedBy=multi-user.target
</file>

Create the //Promtail// systemd unit file:

<cli>
root@host:~$ vi /usr/lib/systemd/system/promtail.service
</cli>

File contents:

<file config /usr/lib/systemd/system/promtail.service>
[Unit]
Description=promtail is the agent responsible for gathering logs and sending them to Loki.
Documentation=https://github.com/grafana/loki/blob/master/docs/promtail.md
Wants=network-online.target
After=network-online.target

[Service]
User=root
Group=grafana
EnvironmentFile=-/etc/default/promtail
Restart=on-failure
ExecStart=/usr/sbin/promtail $ARGS
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=20s
SendSIGKILL=no

[Install]
WantedBy=multi-user.target
</file>

Reload the systemd process:

<cli> 
root@host:~$ systemctl daemon-reload
</cli> 

Download the //Loki// and //Promtail// default configuration files:

<cli>
root@host:~$ mkdir -p /etc/{loki,promtail}/
root@host:~$ cd /etc/loki
root@host:~$ wget -O /etc/loki/loki.yml https://raw.githubusercontent.com/grafana/loki/master/cmd/loki/loki-local-config.yaml
root@host:~$ cd /etc/promtail
root@host:~$ wget -O /etc/promtail/promtail.yml https://raw.githubusercontent.com/grafana/loki/master/cmd/promtail/promtail-local-config.yaml
root@host:~$ chown root:root /etc/{loki,promtail}/*.yml
root@host:~$ chmod 644 /etc/{loki,promtail}/*.yml
</cli>

Create the //Loki// and //Promtail// data directories:

<cli>
root@host:~$ mkdir -p /srv/loki/{chunks,index,wal}
root@host:~$ chown -R grafana:grafana /srv/loki
root@host:~$ mkdir -p /var/lib/promtail/
</cli>

If a local IPTables firewall is active on the system running //Promtail// and the embedded server of //Promtail// should be used and accessible for debugging purposes (see configuration below) the TCP port ''9080'' needs to be opened for access to the //Promtail// WebUI:

<file config iptables.conf>
# Allow access to Promtail from local networks
-A INPUT -p tcp -s <YOUR-NETWORK> --dport 9080 -j ACCEPT
</file>

and reload the IPTables rules:

<cli>
root@host:~$ iptables-restore < iptables.conf
</cli>

====== Configuration ======

===== Loki =====

==== Initial ====

Configure the //Loki// data directories:

<cli>
root@host:~$ vi /etc/loki/loki.yml
</cli>

File contents:

<file config /etc/loki/loki.yml>
[...]
ingester:
  [...]
    wal:
    dir: /srv/loki/wal

[...]
storage_config:
  boltdb:
    directory: /srv/loki/index

  filesystem:
    directory: /srv/loki/chunks
[...]
</file>

==== Retention ====

Configure the //Loki// data retention (i.e. how long log index and actual log data are kept). See [[https://grafana.com/docs/loki/latest/operations/storage/retention/|Operations - Storage - Retention]] for details.

Create the //Loki// data directories for ''boltdb-shipper'':

<cli>
root@host:~$ mkdir -p /srv/loki/{boltdb_shipper/{active,cache},compactor}
root@host:~$ chown -R grafana:grafana /srv/loki/{boltdb_shipper,compactor}
</cli>

Configure the //Loki// data retention:

<cli>
root@host:~$ vi /etc/loki/loki.yml
</cli>

File contents:

<file config /etc/loki/loki.yml>
[...]
schema_config:
  configs:
    - from: 2018-04-15
      store: boltdb-shipper
      object_store: filesystem
      index:
        period: 24h
[...]
storage_config:
  boltdb_shipper:
    active_index_directory: "/srv/loki/boltdb_shipper/active"
    cache_location: "/srv/loki/boltdb_shipper/cache"
    cache_ttl: 24h 
    shared_store: filesystem
[...]
limits_config:
  retention_period: 8904h
[...]
compactor:
  retention_enabled: true
  shared_store: filesystem
  working_directory: /srv/loki/compactor/
[...]
</file>

==== Retention - Legacy ====

Configure the //Loki// data retention (i.e. how long log index and actual log data are kept). See [[https://grafana.com/docs/loki/latest/operations/storage/table-manager/#retention|Operations - Storage - Table Manager - Retention]] for details.

<note important>
The behaviour of //Loki// with regard to data retention changed in version 2.4.0 and later. It will no longer run the ''table-manager'' thread in a Single binary deployment (i.e. ''target=all''). See [[https://grafana.com/docs/loki/latest/upgrading/#240|Upgrading Grafana Loki - v2.4.0]] for details.
</note>

As a workaround, instruct //Loki// explicitly to start the ''table-manager'' thread by editing the startup configuration:

<cli> 
root@host:~$ vi /etc/default/loki
</cli> 

File contents:

<file config /etc/default/loki>
ARGS="-target=all,table-manager -config.file=/etc/loki/loki.yml"
</file>

<cli>
root@host:~$ vi /etc/loki/loki.yml
</cli>

File contents:

<file config /etc/loki/loki.yml>
[...]
table_manager:
  retention_deletes_enabled: true
  retention_period: 8904h # e.g. keep log index and log data for 53 weeks (~1 year).
[...]
</file>

<note important>
The value of ''table_manager.retention_period'' *must* be a whole number multiple of the value of ''schema_config.configs.<config>.index.period''!
</note>

===== Promtail =====

Disable the //Promtail// embedded server or change the configuration in oder to only let it listen to ''localhost'' IP adresses:

<cli>
root@host:~$ vi /etc/promtail/promtail.yml
</cli>

File contents:

<file config /etc/promtail/promtail.yml>
[...]
server:
  # Enable for debugging only.
  disable: true
  http_listen_address: 127.0.0.1
  http_listen_port: 9080
  grpc_listen_address: 127.0.0.1
  grpc_listen_port: 0
[...]
</file>

Configure the //Promtail// data file storing the file position for each tailed logfile:

<cli>
root@host:~$ vi /etc/promtail/promtail.yml
</cli>

File contents:

<file config /etc/promtail/promtail.yml>
[...]
positions:
  filename: /var/lib/promtail/positions.yml
[...]
</file>

Configure the //Loki// servers to send the logfile data to:

<note important>
If multiple //Loki// servers are configured here, they are tried in the given order using only a single processing thread. In case of a connection error or a slow connection with one or more //Loki// servers, this will impact the whole log shipping process. It is thus recommended to instead run multiple parallel instances of //Promtail// if log shipping to multiple //Loki// servers is required.
</note>

<cli>
root@host:~$ vi /etc/promtail/promtail.yml
</cli>

File contents:

<file config /etc/promtail/promtail.yml>
[...]
clients:
  - url: http://<LOKI_SERVER_FQDN>:<LOKI_SERVER_PORT>/loki/api/v1/push
[...]
</file>

Configure the logfiles which //Promtail// should monitor and //scrape// the log data from:

<cli>
root@host:~$ vi /etc/promtail/promtail.yml
</cli>

File contents:

<file config /etc/promtail/promtail.yml>
[...]
scrape_configs:
  - job_name: system
    static_configs:
    - targets: [ localhost ]
      labels:
        job: varlogs
        host: <LOCAL_HOSTNAME>
        <MORE_LABLES>
        __path__: /var/log/{*log,dmesg,mail.{err,info,warn},messages}
[...]
</file>

Configure the //Promtail// drop-in configuration directory in which individual //scrape// configuration files will be placed. This directory is monitored by //Promtail// for changes and re-read in a default interval of 5 minutes:

<cli>
root@host:~$ vi /etc/promtail/promtail.yml
</cli>

File contents:

<file config /etc/promtail/promtail.yml>
[...]
scrape_configs:
  [...]
  - job_name: confd
    file_sd_configs:
      - files:
        - /etc/promtail/conf.d/*.yml
[...]
</file>

Example of an individual //Promtail// //scrape// configuration file. In this example here for the //Promtail// console output logged into a file by the above ''rsyslog'' configuration:

<cli>
root@host:~$ vi /etc/promtail/conf.d/promtail.yml
</cli>

File contents:

<file config /etc/promtail/conf.d/promtail.yml>
[
  {
    "targets":[ "localhost" ],
    "labels": {
      "__path__": "/var/log/promtail/*log",
      "job": "promtail",
      "host": "<LOCAL_HOSTNAME>",
      <MORE_LABLES>
    }
  },
]
</file>

====== Usage ======

===== Start =====

To start //Loki// and //Promtail//:

<cli>
root@host:~$ systemctl enable loki.service promtail.service
root@host:~$ systemctl start loki.service promtail.service
</cli>

===== Stop =====

To stop //Loki// and //Promtail//:

<cli>
root@host:~$ systemctl stop loki.service promtail.service
</cli>

===== Status Check =====

To check the status of //Loki// and //Promtail//:

<cli>
root@host:~$ systemctl status loki.service promtail.service
</cli>

===== Configuration check =====

To check a //Loki// configuration file for validity:

<cli>
root@host:~$ /usr/sbin/loki -verify-config -config.file=/etc/loki/loki.yml
</cli>

====== Recipies ======

FIXME

<nspages .:recipies -subns -pagesInNs -simpleList -title -hideNoSubns -hideNoPages -textNS="" -exclude:start>

===== Log streams =====

To check which log streams have been defined in //Loki//:

<cli>
root@host:~$ /usr/sbin/logcli series '{}'
</cli>

===== Log labels =====

To check which log labels have been defined in //Loki// and what their cardinality with regard to the log streams is:

<cli>
root@host:~$ /usr/sbin/logcli series '{}' --analyze-labels
</cli>

====== Known Issues ======

====== Data retention not working ======

Old log data and log index data is not properly cleaned up altough retention is properly configured. See [[#retention_-_legacy|Retention - Legacy]] and [[https://grafana.com/docs/loki/latest/upgrading/#240|Upgrading Grafana Loki - v2.4.0]] for details.