Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

Compared to other log aggregation systems, Loki:

• does not do full text indexing on logs. By storing compressed, unstructured logs and only indexing metadata, Loki is simpler to operate and cheaper to run.

• indexes and groups log streams using the same labels you’re already using with Prometheus, enabling you to seamlessly switch between metrics and logs using the same labels that you’re already using with Prometheus.

• is an especially good fit for storing Kubernetes Pod logs. Metadata such as Pod labels is automatically scraped and indexed.

• has native support in Grafana (needs Grafana v6.0).

A Loki-based logging stack consists of 3 components:

• promtail is the agent, responsible for gathering logs and sending them to Loki.

• loki is the main server, responsible for storing logs and processing queries.

• Grafana for querying and displaying the logs.

Loki is like Prometheus, but for logs: we prefer a multidimensional label-based approach to indexing, and want a single-binary, easy to operate system with no dependencies. Loki differs from Prometheus by focusing on logs instead of metrics, and delivering logs via push, instead of pull.

https://grafana.com/oss/loki/
Loki GitHub repository
Loki binary releases

See Grafana for the dashboards regarding Grafana Loki

https://github.com/grafana/loki/tree/master/docs
Operations - Storage - Retention
Operations - Storage - Table Manager - Retention

Loki help:

user@host:~$ /usr/sbin/loki -h | less -S

Promtail help:

user@host:~$ /usr/sbin/promtail -h | less -S

logcli help:

user@host:~$ /usr/sbin/logcli --help 2>&1| less -S

Install Grafana as a prerequisite.

Download the Loki binary releases for loki, logcli and promtail:

root@host:~$ cd /tmp
root@host:~$ wget https://github.com/grafana/loki/releases/download/v1.4.1/loki-linux-arm.zip
root@host:~$ wget https://github.com/grafana/loki/releases/download/v1.4.1/loki-canary-linux-arm.zip
root@host:~$ wget https://github.com/grafana/loki/releases/download/v1.4.1/logcli-linux-arm.zip
root@host:~$ wget https://github.com/grafana/loki/releases/download/v1.4.1/promtail-linux-arm.zip

To install Loki on Debian:

root@host:~$ cd /tmp
root@host:~$ for FL in {loki,logcli,promtail}*.zip; do unzip -x $FL; done
root@host:~$ rm {loki,logcli,promtail}*.zip
root@host:~$ chmod 755 *-linux-*
root@host:~$ chown root:root *-linux-*
root@host:~$ for FL in *-linux-*; do echo mv -i $FL ${FL%%-li*} ; done
root@host:~$ mv -i logcli loki loki-canary promtail /usr/sbin/

Create the Loki startup configuration:

root@host:~$ vi /etc/default/loki

File contents:

/etc/default/loki

#ARGS="-config.file=/etc/loki/loki.yml -log.level=debug"
ARGS="-config.file=/etc/loki/loki.yml"

Create the Promtail startup configuration:

root@host:~$ vi /etc/default/promtail

File contents:

/etc/default/promtail

#ARGS="-config.file=/etc/promtail/promtail.yml -log.level=debug"
ARGS="-config.file=/etc/promtail/promtail.yml"

Create the Loki and Promtail rsyslog configuration files:

root@host:~$ vi /etc/rsyslog.d/loki.conf

File contents:

/etc/rsyslog.d/loki.conf

template(name="LOKI_TEMPLATE" type="string" string="/var/log/loki/%programname%.log")
if ($programname startswith 'loki') then {
   action(
        type="omfile"
        dynaFile="LOKI_TEMPLATE"
        fileCreateMode="0640"
        fileOwner="grafana"
        fileGroup="grafana"
        dirCreateMode="0750"
        dirOwner="grafana"
        dirGroup="grafana"
        ioBufferSize="64k"
    )
   stop
}

root@host:~$ vi /etc/rsyslog.d/promtail.conf

File contents:

/etc/rsyslog.d/promtail.conf

template(name="PROMTAIL_TEMPLATE" type="string" string="/var/log/promtail/%programname%.log")
if ($programname startswith 'promtail') then {
   action(
        type="omfile"
        dynaFile="PROMTAIL_TEMPLATE"
        fileCreateMode="0640"
        fileOwner="root"
        fileGroup="grafana"
        dirCreateMode="0750"
        dirOwner="root"
        dirGroup="grafana"
        ioBufferSize="64k"
    )
   stop
}

Restart the rsyslog daemon:

root@host:~$ systemctl restart rsyslog.service

Create the Loki systemd unit file:

root@host:~$ vi /usr/lib/systemd/system/loki.service

File contents:

/usr/lib/systemd/system/loki.service

[Unit]
Description=Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system
Documentation=https://github.com/grafana/loki
Wants=network-online.target
After=network-online.target
 
[Service]
User=grafana
Group=grafana
EnvironmentFile=-/etc/default/loki
Restart=on-failure
ExecStart=/usr/sbin/loki $ARGS
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=20s
SendSIGKILL=no
 
[Install]
WantedBy=multi-user.target

Create the Promtail systemd unit file:

root@host:~$ vi /usr/lib/systemd/system/promtail.service

File contents:

/usr/lib/systemd/system/promtail.service

[Unit]
Description=promtail is the agent responsible for gathering logs and sending them to Loki.
Documentation=https://github.com/grafana/loki/blob/master/docs/promtail.md
Wants=network-online.target
After=network-online.target
 
[Service]
User=root
Group=grafana
EnvironmentFile=-/etc/default/promtail
Restart=on-failure
ExecStart=/usr/sbin/promtail $ARGS
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=20s
SendSIGKILL=no
 
[Install]
WantedBy=multi-user.target

Reload the systemd process:

root@host:~$ systemctl daemon-reload

Download the Loki and Promtail default configuration files:

root@host:~$ mkdir -p /etc/{loki,promtail}/
root@host:~$ cd /etc/loki
root@host:~$ wget -O /etc/loki/loki.yml https://raw.githubusercontent.com/grafana/loki/master/cmd/loki/loki-local-config.yaml
root@host:~$ cd /etc/promtail
root@host:~$ wget -O /etc/promtail/promtail.yml https://raw.githubusercontent.com/grafana/loki/master/cmd/promtail/promtail-local-config.yaml
root@host:~$ chown root:root /etc/{loki,promtail}/*.yml
root@host:~$ chmod 644 /etc/{loki,promtail}/*.yml

Create the Loki and Promtail data directories:

root@host:~$ mkdir -p /srv/loki/{chunks,index,wal}
root@host:~$ chown -R grafana:grafana /srv/loki
root@host:~$ mkdir -p /var/lib/promtail/

If a local IPTables firewall is active on the system running Promtail and the embedded server of Promtail should be used and accessible for debugging purposes (see configuration below) the TCP port 9080 needs to be opened for access to the Promtail WebUI:

iptables.conf

# Allow access to Promtail from local networks
-A INPUT -p tcp -s <YOUR-NETWORK> --dport 9080 -j ACCEPT

and reload the IPTables rules:

root@host:~$ iptables-restore < iptables.conf

Configure the Loki data directories:

root@host:~$ vi /etc/loki/loki.yml

File contents:

/etc/loki/loki.yml

[...]
ingester:
  [...]
    wal:
    dir: /srv/loki/wal
 
[...]
storage_config:
  boltdb:
    directory: /srv/loki/index
 
  filesystem:
    directory: /srv/loki/chunks
[...]

Configure the Loki data retention (i.e. how long log index and actual log data are kept). See Operations - Storage - Retention for details.

Create the Loki data directories for boltdb-shipper:

root@host:~$ mkdir -p /srv/loki/{boltdb_shipper/{active,cache},compactor}
root@host:~$ chown -R grafana:grafana /srv/loki/{boltdb_shipper,compactor}

Configure the Loki data retention:

root@host:~$ vi /etc/loki/loki.yml

File contents:

/etc/loki/loki.yml

[...]
schema_config:
  configs:
    - from: 2018-04-15
      store: boltdb-shipper
      object_store: filesystem
      index:
        period: 24h
[...]
storage_config:
  boltdb_shipper:
    active_index_directory: "/srv/loki/boltdb_shipper/active"
    cache_location: "/srv/loki/boltdb_shipper/cache"
    cache_ttl: 24h 
    shared_store: filesystem
[...]
limits_config:
  retention_period: 8904h
[...]
compactor:
  retention_enabled: true
  shared_store: filesystem
  working_directory: /srv/loki/compactor/
[...]

Configure the Loki data retention (i.e. how long log index and actual log data are kept). See Operations - Storage - Table Manager - Retention for details.

As a workaround, instruct Loki explicitly to start the table-manager thread by editing the startup configuration:

root@host:~$ vi /etc/default/loki

File contents:

/etc/default/loki

ARGS="-target=all,table-manager -config.file=/etc/loki/loki.yml"

root@host:~$ vi /etc/loki/loki.yml

File contents:

/etc/loki/loki.yml

[...]
table_manager:
  retention_deletes_enabled: true
  retention_period: 8904h # e.g. keep log index and log data for 53 weeks (~1 year).
[...]

Disable the Promtail embedded server or change the configuration in oder to only let it listen to localhost IP adresses:

root@host:~$ vi /etc/promtail/promtail.yml

File contents:

/etc/promtail/promtail.yml

[...]
server:
  # Enable for debugging only.
  disable: true
  http_listen_address: 127.0.0.1
  http_listen_port: 9080
  grpc_listen_address: 127.0.0.1
  grpc_listen_port: 0
[...]

Configure the Promtail data file storing the file position for each tailed logfile:

root@host:~$ vi /etc/promtail/promtail.yml

File contents:

/etc/promtail/promtail.yml

[...]
positions:
  filename: /var/lib/promtail/positions.yml
[...]

Configure the Loki servers to send the logfile data to:

root@host:~$ vi /etc/promtail/promtail.yml

File contents:

/etc/promtail/promtail.yml

[...]
clients:
  - url: http://<LOKI_SERVER_FQDN>:<LOKI_SERVER_PORT>/loki/api/v1/push
[...]

Configure the logfiles which Promtail should monitor and scrape the log data from:

root@host:~$ vi /etc/promtail/promtail.yml

File contents:

/etc/promtail/promtail.yml

[...]
scrape_configs:
  - job_name: system
    static_configs:
    - targets: [ localhost ]
      labels:
        job: varlogs
        host: <LOCAL_HOSTNAME>
        <MORE_LABLES>
        __path__: /var/log/{*log,dmesg,mail.{err,info,warn},messages}
[...]

Configure the Promtail drop-in configuration directory in which individual scrape configuration files will be placed. This directory is monitored by Promtail for changes and re-read in a default interval of 5 minutes:

root@host:~$ vi /etc/promtail/promtail.yml

File contents:

/etc/promtail/promtail.yml

[...]
scrape_configs:
  [...]
  - job_name: confd
    file_sd_configs:
      - files:
        - /etc/promtail/conf.d/*.yml
[...]

Example of an individual Promtail scrape configuration file. In this example here for the Promtail console output logged into a file by the above rsyslog configuration:

root@host:~$ vi /etc/promtail/conf.d/promtail.yml

File contents:

/etc/promtail/conf.d/promtail.yml

[
  {
    "targets":[ "localhost" ],
    "labels": {
      "__path__": "/var/log/promtail/*log",
      "job": "promtail",
      "host": "<LOCAL_HOSTNAME>",
      <MORE_LABLES>
    }
  },
]

To start Loki and Promtail:

root@host:~$ systemctl enable loki.service promtail.service
root@host:~$ systemctl start loki.service promtail.service

To stop Loki and Promtail:

root@host:~$ systemctl stop loki.service promtail.service

To check the status of Loki and Promtail:

root@host:~$ systemctl status loki.service promtail.service

To check a Loki configuration file for validity:

root@host:~$ /usr/sbin/loki -verify-config -config.file=/etc/loki/loki.yml

To check which log streams have been defined in Loki:

root@host:~$ /usr/sbin/logcli series '{}'

To check which log labels have been defined in Loki and what their cardinality with regard to the log streams is:

root@host:~$ /usr/sbin/logcli series '{}' --analyze-labels

Old log data and log index data is not properly cleaned up altough retention is properly configured. See Retention - Legacy and Upgrading Grafana Loki - v2.4.0 for details.