Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. It is now a standalone open source project and maintained independently of any company. To emphasize this, and to clarify the project's governance structure, Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes.

Prometheus collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels.

Prometheus's main features are:

• a multi-dimensional data model with time series data identified by metric name and key/value pairs

• PromQL, a flexible query language to leverage this dimensionality

• no reliance on distributed storage; single server nodes are autonomous

• time series collection happens via a pull model over HTTP

• pushing time series is supported via an intermediary gateway

• targets are discovered via service discovery or static configuration

• multiple modes of graphing and dashboarding support

https://prometheus.io/

https://prometheus.io/docs/

Prometheus Manual pages:

user@host:~$ man prometheus

promtool Manual pages:

user@host:~$ man promtool

To install Prometheus on Debian:

root@host:~$ apt-get update
root@host:~$ apt-get install prometheus

Create the Prometheus rsyslog configuration files:

root@host:~$ vi /etc/rsyslog.d/prometheus.conf

File contents:

/etc/rsyslog.d/prometheus.conf

template(name="PROMETHEUS_TEMPLATE" type="string" string="/var/log/prometheus/%programname%.log")
if ($programname startswith 'prometheus') then {
   action(
        type="omfile"
        dynaFile="PROMETHEUS_TEMPLATE"
        fileCreateMode="0640"
        fileOwner="prometheus"
        fileGroup="prometheus"
        dirCreateMode="0750"
        dirOwner="prometheus"
        dirGroup="prometheus"
        ioBufferSize="64k"
    )
   stop
}

Restart the rsyslog daemon:

root@host:~$ systemctl restart rsyslog.service

Create the Prometheus data directories:

root@host:~$ mkdir -p /srv/prometheus/
root@host:~$ chown -R prometheus:prometheus /srv/prometheus

If a local IPTables firewall is active on the system running Prometheus and the embedded server of Prometheus should be used and accessible the TCP port 9090 needs to be opened for access to the Prometheus WebUI:

iptables.conf

# Allow access to Prometheus from local networks
-A INPUT -p tcp -s <YOUR-NETWORK> --dport 9090 -j ACCEPT

and reload the IPTables rules:

root@host:~$ iptables-restore < iptables.conf

Configure the Prometheus data directory:

root@host:~$ vi /etc/default/prometheus

File contents:

/etc/default/prometheus

[...]
ARGS="--storage.tsdb.path='/srv/prometheus/metrics2/'"
[...]

See Adding Prometheus to Grafana

Configure Prometheus to scrape metrics from various sources:

root@host:~$ vi /etc/prometheus/prometheus.yml

File contents:

/etc/prometheus/prometheus.yml

[...]
scrape_configs:
  [...]
  - job_name: loki
    static_configs:
      - targets: ['localhost:3100']
  [...]
[...]

root@host:~$ vi /etc/prometheus/prometheus.yml

File contents:

/etc/prometheus/prometheus.yml

[...]
scrape_configs:
  [...]
  - job_name: promtail
    static_configs:
      - targets: ['localhost:9080']
  [...]
[...]

To start Prometheus:

root@host:~$ systemctl enable prometheus.service
root@host:~$ systemctl start prometheus.service

To stop Prometheus:

root@host:~$ systemctl stop prometheus.service

To check the status of Prometheus:

root@host:~$ systemctl status prometheus.service