====== Introduction ======

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. It is now a standalone open source project and maintained independently of any company. To emphasize this, and to clarify the project's governance structure, Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes.

Prometheus collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels.

Prometheus's main features are:

  * a multi-dimensional data model with time series data identified by metric name and key/value pairs <
  * PromQL, a flexible query language to leverage this dimensionality <
  * no reliance on distributed storage; single server nodes are autonomous <
  * time series collection happens via a pull model over HTTP <
  * pushing time series is supported via an intermediary gateway <
  * targets are discovered via service discovery or static configuration <
  * multiple modes of graphing and dashboarding support <

====== Links ======


===== General =====

[[https://prometheus.io/]]\\

====== Docs ======

[[https://prometheus.io/docs/]]\\

====== Files and Directories ======

===== General Files and Directories =====

^ File or Directory                                 ^ Description                                                                            ^
| ''/etc/prometheus/''                                                 | The //Prometheus// configuration directory.                     |
| ''/usr/bin/prometheus''                                           | The //Prometheus// server binary.                                     |
| ''/usr/bin/promtool''                                                | The ''promtool'' CLI binary.                                         |
| ''/usr/lib/systemd/system/prometheus.service''   | The //Prometheus// systemd unit file.                                |
| ''/srv/prometheus/''                                                 | The //Prometheus// server data directory.                         |

===== Configuration Files =====

^ File or Directory                          ^ Description                                                                           ^
| ''/etc/default/prometheus''                           | The configuration file with environment variables for the //Prometheus// server.                 |
| ''/etc/prometheus/prometheus.yml''            | The //Prometheus// server configuration file.                 |

====== Getting Help ======

//Prometheus// Manual pages: 

<cli>
user@host:~$ man prometheus
</cli>

''promtool'' Manual pages: 

<cli>
user@host:~$ man promtool
</cli>

====== Install ======

To install //Prometheus// on Debian: 

<cli>
root@host:~$ apt-get update
root@host:~$ apt-get install prometheus
</cli>

Create the //Prometheus// ''rsyslog'' configuration files:

<cli>
root@host:~$ vi /etc/rsyslog.d/prometheus.conf
</cli>

File contents:

<file config /etc/rsyslog.d/prometheus.conf>
template(name="PROMETHEUS_TEMPLATE" type="string" string="/var/log/prometheus/%programname%.log")
if ($programname startswith 'prometheus') then {
   action(
        type="omfile"
        dynaFile="PROMETHEUS_TEMPLATE"
        fileCreateMode="0640"
        fileOwner="prometheus"
        fileGroup="prometheus"
        dirCreateMode="0750"
        dirOwner="prometheus"
        dirGroup="prometheus"
        ioBufferSize="64k"
    )
   stop
}
</file>

Restart the ''rsyslog'' daemon:

<cli>
root@host:~$ systemctl restart rsyslog.service
</cli>

Create the //Prometheus// data directories:

<cli>
root@host:~$ mkdir -p /srv/prometheus/
root@host:~$ chown -R prometheus:prometheus /srv/prometheus
</cli>

If a local IPTables firewall is active on the system running //Prometheus// and the embedded server of //Prometheus// should be used and accessible the TCP port ''9090'' needs to be opened for access to the //Prometheus// WebUI:

<file config iptables.conf>
# Allow access to Prometheus from local networks
-A INPUT -p tcp -s <YOUR-NETWORK> --dport 9090 -j ACCEPT
</file>

and reload the IPTables rules:

<cli>
root@host:~$ iptables-restore < iptables.conf
</cli>

====== Configuration ======

===== Initial =====

Configure the //Prometheus// data directory:

<cli>
root@host:~$ vi /etc/default/prometheus
</cli>

File contents:

<file config /etc/default/prometheus>
[...]
ARGS="--storage.tsdb.path='/srv/prometheus/metrics2/'"
[...]
</file>

===== Adding Prometheus to Grafana =====

See [[sw:grafana:intro#adding_prometheus_to_grafana|Adding Prometheus to Grafana]]

===== Scraping metrics with Prometheus =====

Configure //Prometheus// to //scrape// metrics from various sources:

==== Loki ====

<cli>
root@host:~$ vi /etc/prometheus/prometheus.yml
</cli>

File contents:

<file config /etc/prometheus/prometheus.yml>
[...]
scrape_configs:
  [...]
  - job_name: loki
    static_configs:
      - targets: ['localhost:3100']
  [...]
[...]
</file>


==== Promtail ====

<cli>
root@host:~$ vi /etc/prometheus/prometheus.yml
</cli>

File contents:

<file config /etc/prometheus/prometheus.yml>
[...]
scrape_configs:
  [...]
  - job_name: promtail
    static_configs:
      - targets: ['localhost:9080']
  [...]
[...]
</file>

====== Usage ======

===== Start =====

To start //Prometheus//:

<cli>
root@host:~$ systemctl enable prometheus.service
root@host:~$ systemctl start prometheus.service
</cli>

===== Stop =====

To stop //Prometheus//:

<cli>
root@host:~$ systemctl stop prometheus.service
</cli>

===== Status Check =====

To check the status of //Prometheus//: 

<cli>
root@host:~$ systemctl status prometheus.service
</cli>

====== Recipies ======

FIXME

<nspages .:recipies -subns -pagesInNs -simpleList -title -hideNoSubns -hideNoPages -textNS="" -exclude:start>

====== Known Issues ======

FIXME