bityard Blog

This is an old revision of the document!


// Check_MK Monitoring - SAP Cloud Connector

The SAP Cloud Connector provides a service which is used to connect on-premise systems – non-SAP, SAP ECC and SAP HANA – with applications running on the SAP Cloud Platform. This article introduces a new Check_MK agent and several service checks to monitor the status of the SAP Cloud Connector and its connections to the SAP Cloud Platform.

For the impatient and TL;DR here is the Check_MK package of the SAP Cloud Connector monitoring checks:

SAP Cloud Connector monitoring checks (Compatible with Check_MK versions 1.4.0p19 and later)

The sources are to be found in my Check_MK repository on GitHub


Monitoring the SAP Cloud Connector can be done in two different, not mutually exclusive, ways. The first approach uses the traditional application monitoring features, already built into Check_MK, like:

  • the presence and count of the application processes

  • the reachability of the applications TCP ports

  • the validity of the SSL Certificates

  • queries to the applications health-check URL

The first approach is covered by the section Built-in Check_MK Monitoring below.

The second approach uses a new Check_MK agent and several service checks, dedicated to monitor the internal status of the SAP Cloud Connector and its connections to the SAP Cloud Platform. The new Check_MK agent uses the monitoring API provided by the SAP Cloud Connector in order the monitor the application specific states and metrics. The monitoring endpoints on the SAP Cloud Connector currently used by this Check_MK agent are the:

  • “List of Subaccounts” (URL: https://<scchost>:<sccport>/api/monitoring/subaccounts)

  • “List of Open Connections” (URL: https://<scchost>:<sccport>/api/monitoring/connections/backends)

  • “Performance Monitor Data” (URL: https://<scchost>:<sccport>/api/monitoring/performance/backends)

  • “Top Time Consumers” (URL: https://<scchost>:<sccport>/api/monitoring/performance/toptimeconsumers)

At the time of writing, there unfortunately is no monitoring endpoint on the SAP Cloud Connector for the Most Recent Requests metric. This metric is currently only available via the SAP Cloud Connectors WebUI. The Most Recent Requests metric would be a much more interesting and useful metric than the currently available Top Time Consumers or Performance Monitor Data, both of which have limitations. The application requests covered by the Top Time Consumers metric need a manual acknowledgement inside the SAP Cloud Connector in order to reset events with the longest request runtime, which limits the metrics usability for external monitoring tools. The Performance Monitor Data metric aggregates the application requests into buckets based on their overall runtime. By itself this can be useful for external monitoring tools and is in fact used by the Check_MK agent covered in this article. In the process of runtime bucket aggregation though, the Performance Monitor Data metric hides the much more useful breakdown of each request into runtime subsections (“External (Back-end)”, “Open Connection”, “Internal (SCC)”, “SSO Handling” and “Latency Effects”). Hopefully the Most Recent Requests metric will in the future also be exposed via the monitoring API provided by the SAP Cloud Connector. The new Check_MK agent can then be extended to use the newly exposed metric in order to gain a more fine grained insight into the runtime of application requests through the SAP Cloud Connector.

The second approach is covered by the section SAP Cloud Connector Agent below.

Built-in Check_MK Monitoring

Application Processes

To monitor the SAP Cloud Connector process, use the standard Check_MK check “State and count of processes”. This can be found in the WATO WebUI under:

-> Manual Checks
   -> Applications, Processes & Services
      -> State and count of processes
         -> Create rule in folder ...
            -> Rule Options
               Description: Process monitoring of the SAP Cloud Connector
               Checktype: [ps - State and Count of Processes]
               Process Name: SAP Cloud Connector

            -> Parameters
               [x] Process Matching
               [Exact name of the process without arguments]
               [/opt/sapjvm_8/bin/java]

               [x] Name of the operating system user
               [Exact name of the operating system user]
               [sccadmin]

               [x] Levels for process count
               Critical below [1] processes
               Warning below [1] processes
               Warning above [1] processes
               Critical above [2] processes

            -> Conditions
               Folder [The folder containing the SAP Cloud Connector systems]
               and/or
               Explicit hosts [x]
               Specify explicit host names [SAP Cloud Connector systems]

Application Health Check

To implement a rudimentary monitoring of the SAP Cloud Connector application health, use the standard Check_MK check “Check HTTP service” to query the Health Check endpoint of the monitoring API provided by the SAP Cloud Connector. The “Check HTTP service” can be found in the WATO WebUI under:

-> Host & Service Parameters
   -> Active checks (HTTP, TCP, etc.)
      -> Check HTTP service
         -> Create rule in folder ...
            -> Rule Options
               Description: SAP Cloud Connector (SCC)

            -> Check HTTP service
               Name: SAP SCC
               [x] Check the URL
               [x] URI to fetch (default is /)
               [/exposed?action=ping]

               [x] TCP Port
               [8443]

               [x] Use SSL/HTTPS for the connection:
               [Use SSL with auto negotiation]

            -> Conditions
               Folder [The folder containing the SAP Cloud Connector systems]
               and/or
               Explicit hosts [x]
               Specify explicit host names [SAP Cloud Connector systems]

SSL Certificates

To monitor the validity of the SSL certificate of the SAP Cloud Connector WebUI, use the standard Check_MK check “Check HTTP service”. The “Check HTTP service” can be found in the WATO WebUI under:

-> Host & Service Parameters
   -> Active checks (HTTP, TCP, etc.)
      -> Check HTTP service
         -> Create rule in folder ...
            -> Rule Options
               Description: SAP Cloud Connector (SCC)

            -> Check HTTP service
               Name: SAP SCC Certificate
               [x] Check SSL Certificate Age
               Warning at or below [30] days
               Critical at or below [60] days

               [x] TCP Port
               [8443]

            -> Conditions
               Folder [The folder containing the SAP Cloud Connector systems]
               and/or
               Explicit hosts [x]
               Specify explicit host names [SAP Cloud Connector systems]

SAP Cloud Connector Agent

The new Check_MK package to monitor the status of the SAP Cloud Connector and its connections to the SAP Cloud Platform consists of three major parts – an agent plugin, two check plugins and several auxiliary files and plugins (WATO plugins, Perf-o-meter plugins, metrics plugins and man pages).

Prerequisites

The following prerequisites are necessary in order for the SAP Cloud Connector agent to work properly:

  • A SAP Cloud Connector application user must be created for the Check_MK agent to be able to authenticate against the SAP Cloud Connector and gain access to the protected monitoring API endpoints. See the article SAP Cloud Connector - Configuring Multiple Local Administrative Users on how to create a new application user.

  • A DNS alias or an additional IP address for the SAP Cloud Connector service.

  • An additional host in Check_MK for the SAP Cloud Connector service with the previously created DNS alias or IP address.

  • Installation of the Python requests library on the Check_MK server. This library is used in the Check_MK agent plugin agent_sapcc to perform the authentication and the HTTP requests against the monitoring API of the SAP Cloud Connector. On e.g. RHEL based systems it can be installed with:

    root@host:# yum install python-requests
    
  • Installation of the new Check_MK package for the SAP Cloud Connector monitoring checks on the Check_MK server.

SAP Cloud Connector Agent Plugin

The Check_MK agent plugin agent_sapcc is responsible for querying the endpoints of the monitoring API on the SAP Cloud Connector, which are described above. It transforms the data returned from the monitoring endpoints into a format digestible by Check_MK. The following example shows the – anonymized and abbreviated – agent plugin output for a SAP Cloud Connector system:

<<<check_mk>>>
Version: 0.1

<<<sapcc_connections_backends:sep(59)>>>
subaccounts,abcdefghi,locationID;Test Location
subaccounts,abcdefghi,regionHost;hana.ondemand.com
subaccounts,abcdefghi,subaccount;abcdefghi

<<<sapcc_performance_backends:sep(59)>>>
subaccounts,abcdefghi,backendPerformance,PROTOCOL/sapecc.example.com:PORT,buckets,1,minimumCallDurationMs;10
subaccounts,abcdefghi,backendPerformance,PROTOCOL/sapecc.example.com:PORT,buckets,1,numberOfCalls;1
subaccounts,abcdefghi,backendPerformance,PROTOCOL/sapecc.example.com:PORT,buckets,2,minimumCallDurationMs;20
subaccounts,abcdefghi,backendPerformance,PROTOCOL/sapecc.example.com:PORT,buckets,2,numberOfCalls;36
[...]
subaccounts,abcdefghi,backendPerformance,PROTOCOL/sapecc.example.com:PORT,buckets,20,minimumCallDurationMs;3000
subaccounts,abcdefghi,backendPerformance,PROTOCOL/sapecc.example.com:PORT,buckets,21,minimumCallDurationMs;4000
subaccounts,abcdefghi,backendPerformance,PROTOCOL/sapecc.example.com:PORT,buckets,22,minimumCallDurationMs;5000
subaccounts,abcdefghi,backendPerformance,PROTOCOL/sapecc.example.com:PORT,name;PROTOCOL/sapecc.example.com:44300
subaccounts,abcdefghi,backendPerformance,PROTOCOL/sapecc.example.com:PORT,protocol;PROTOCOL
subaccounts,abcdefghi,backendPerformance,PROTOCOL/sapecc.example.com:PORT,virtualHost;sapecc.example.com
subaccounts,abcdefghi,backendPerformance,PROTOCOL/sapecc.example.com:PORT,virtualPort;44300
subaccounts,abcdefghi,locationID;Test Location
subaccounts,abcdefghi,regionHost;hana.ondemand.com
subaccounts,abcdefghi,sinceTime;2019-02-13T08:05:36.084 +0100
subaccounts,abcdefghi,subaccount;abcdefghi

<<<sapcc_performance_toptimeconsumers:sep(59)>>>
subaccounts,abcdefghi,locationID;Test Location
subaccounts,abcdefghi,regionHost;hana.ondemand.com
subaccounts,abcdefghi,requests,0,externalTime;373
subaccounts,abcdefghi,requests,0,id;932284302
subaccounts,abcdefghi,requests,0,internalBackend;sapecc.example.com:PORT
subaccounts,abcdefghi,requests,0,openRemoteTime;121
subaccounts,abcdefghi,requests,0,protocol;PROTOCOL
subaccounts,abcdefghi,requests,0,receivedBytes;264
subaccounts,abcdefghi,requests,0,resource;/sap-webservice-url/
subaccounts,abcdefghi,requests,0,sentBytes;4650
subaccounts,abcdefghi,requests,0,startTime;2019-02-13T11:31:59.113 +0100
subaccounts,abcdefghi,requests,0,totalTime;536
subaccounts,abcdefghi,requests,0,user;RFC_USER
subaccounts,abcdefghi,requests,0,virtualBackend;sapecc.example.com:PORT
subaccounts,abcdefghi,requests,1,externalTime;290
subaccounts,abcdefghi,requests,1,id;1882731830
subaccounts,abcdefghi,requests,1,internalBackend;sapecc.example.com:PORT
subaccounts,abcdefghi,requests,1,latencyTime;77
subaccounts,abcdefghi,requests,1,openRemoteTime;129
subaccounts,abcdefghi,requests,1,protocol;PROTOCOL
subaccounts,abcdefghi,requests,1,receivedBytes;264
subaccounts,abcdefghi,requests,1,resource;/sap-webservice-url/
subaccounts,abcdefghi,requests,1,sentBytes;4639
subaccounts,abcdefghi,requests,1,startTime;2019-02-13T11:31:59.114 +0100
subaccounts,abcdefghi,requests,1,totalTime;532
subaccounts,abcdefghi,requests,1,user;RFC_USER
subaccounts,abcdefghi,requests,1,virtualBackend;sapecc.example.com:PORT
[...]
subaccounts,abcdefghi,requests,49,externalTime;128
subaccounts,abcdefghi,requests,49,id;1774317106
subaccounts,abcdefghi,requests,49,internalBackend;sapecc.example.com:PORT
subaccounts,abcdefghi,requests,49,protocol;PROTOCOL
subaccounts,abcdefghi,requests,49,receivedBytes;263
subaccounts,abcdefghi,requests,49,resource;/sap-webservice-url/
subaccounts,abcdefghi,requests,49,sentBytes;4660
subaccounts,abcdefghi,requests,49,startTime;2019-02-16T11:32:09.352 +0100
subaccounts,abcdefghi,requests,49,totalTime;130
subaccounts,abcdefghi,requests,49,user;RFC_USER
subaccounts,abcdefghi,requests,49,virtualBackend;sapecc.example.com:PORT
subaccounts,abcdefghi,sinceTime;2019-02-13T08:05:36.085 +0100
subaccounts,abcdefghi,subaccount;abcdefghi

<<<sapcc_subaccounts:sep(59)>>>
subaccounts,abcdefghi,displayName;Test Application
subaccounts,abcdefghi,locationID;Test Location
subaccounts,abcdefghi,regionHost;hana.ondemand.com
subaccounts,abcdefghi,subaccount;abcdefghi
subaccounts,abcdefghi,tunnel,applicationConnections,abcdefg:hijklmnopqr,connectionCount;8
subaccounts,abcdefghi,tunnel,applicationConnections,abcdefg:hijklmnopqr,name;abcdefg:hijklmnopqr
subaccounts,abcdefghi,tunnel,applicationConnections,abcdefg:hijklmnopqr,type;JAVA
subaccounts,abcdefghi,tunnel,connectedSince;2019-02-14T10:11:00.630 +0100
subaccounts,abcdefghi,tunnel,connections;8
subaccounts,abcdefghi,tunnel,state;Connected
subaccounts,abcdefghi,tunnel,user;P123456

The agent plugin comes with a Check_MK check plugin of the same name, which is solely responsible for the construction of the command line arguments from the WATO configuration and passing it to the Check_MK agent plugin.

With the additional WATO plugin sapcc_agent.py it is possible to configure the username and password for the SAP Cloud Connector application user which is used to connect to the monitoring API. It is also possible to configure the TCP port and the connection timeout for the connection to the monitoring API through the WATO WebUI and thus override the default values. The default value for the TCP port is 8443, the default value for the connection timeout is 30 seconds. The configuration options for the Check_MK agent plugin agent_sapcc can be found in the WATO WebUI under:

-> Host & Service Parameters
   -> Datasource Programs
      -> SAP Cloud Connector systems
         -> Create rule in folder ...
            -> Rule Options
               Description: SAP Cloud Connector (SCC)

            -> SAP Cloud Connector systems
               SAP Cloud Connector user name: [username]
               SAP Cloud Connector password: [password]
               SAP Cloud Connector TCP port: [8443]

            -> Conditions
               Folder [The folder containing the SAP Cloud Connector systems]
               and/or
               Explicit hosts [x]
               Specify explicit host names [SAP Cloud Connector systems]

After saving the new rule, restarting Check_MK and doing an inventory on the additional host for the SAP Cloud Connector service in Check_MK, several new services starting with the name prefix SAP CC should appear.

The following image shows a status output example from the WATO WebUI with the service checks HTTP SAP SCC TLS and HTTP SAP SCC TLS Certificate from the Built-in Check_MK Monitoring described above. In addition to those, the example also shows the service checks based on the data from the SAP Cloud Connector Agent. The service checks SAP CC Application Connection, SAP CC Subaccount and SAP CC Tunnel are provided by the check plugin sapcc_subaccounts, the service check SAP CC Perf Backend is provided by the plugin sapcc_performance_backends:

Status output example for the complete monitoring of the SAP Cloud Connector

SAP Cloud Connector Subaccount

The check plugin sapcc_subaccounts implements the three sub-checks sapcc_subaccounts.app_conn, sapcc_subaccounts.info and sapcc_subaccounts.tunnel.

Info

The sub-check sapcc_subaccounts.info just gathers information on several configuration options for each subaccount on the SAP Cloud Connector and displays them in the status details of the check. These configuration options are the:

  • subaccount name on the SAP Cloud Platform to which the connection is made.

  • display name of the subaccount.

  • location ID of the subaccount.

  • the region host of the SAP Cloud Platform to which the SAP Cloud Connector establishes a connection.

The sub-check sapcc_subaccounts.info always returns an OK status. No performance data is currently reported by this check.

Tunnel

The sub-check sapcc_subaccounts.tunnel is responsible for the monitoring of each tunnel connection for each subaccount on the SAP Cloud Connector. Upon inventory this sub-check creates a service check for each tunnel connection found on the SAP Cloud Connector. During normal check execution, the status of the tunnel connection is determined for each inventorized item. If the tunnel connection is not in the Connected state, an alarm is raised accordingly. Additionally, the number of currently active connections over a tunnel as well as the elapsed time in seconds since the tunnel connection was established are determined for each inventorized item. If either the value of the currently active connections or the number of seconds since the connection was established are above or below the configured warning and critical threshold values, an alarm is raised accordingly. For both values, performance data is reported by the check.

With the additional WATO plugin sapcc_subaccounts.py it is possible to configure the warning and critical levels for the sub-check sapcc_subaccounts.tunnel through the WATO WebUI and thus override the following default values:

Metric Warning Low Threshold Critical Low Threshold Warning High Threshold Critical High Threshold
Number of connections 0 0 30 40
Connection duration 0 sec 0 sec 284012568 sec 315569520 sec

The configuration options for the tunnel connection levels can be found in the WATO WebUI under:

-> Host & Service Parameters
   -> Parameters for discovered services
      -> Applications, Processes & Services
         -> SAP Cloud Connector Subaccounts
            -> Create Rule in Folder ...
               -> Rule Options
                  Description: SAP Cloud Connector Subaccounts

               -> Parameters
                  [x] Number of tunnel connections
                      Warning if equal or below [0] connections
                      Critical if equal or below [0] connections
                      Warning if equal or above [30] connections
                      Critical if equal or above [40] connections
                  [x] Connection time of tunnel connections 
                      Warning if equal or below [0] seconds
                      Critical if equal or below [0] seconds
                      Warning if equal or above [284012568] seconds
                      Critical if equal or above [315569520] seconds

               -> Conditions
                  Folder [The folder containing the SAP Cloud Connector systems]
                  and/or
                  Explicit hosts [x]
                  Specify explicit host names [SAP Cloud Connector systems]
                  and/or
                  Application Or Tunnel Name [x]
                  Specify explicit values [Tunnel name]

The above image with a status output example from the WATO WebUI shows one sapcc_subaccounts.tunnel service check as the last of the displayed items. The service name is prefixed by the string SAP CC Tunnel and followed by the subaccount name, which in this example is anonymized. For each tunnel connection the connection state, the overall number of application connections currently active over the tunnel, the time when the tunnel connection was established and the number of seconds elapsed since establishing the connection are shown. The overall number of currently active application connections is also visualized in the perf-o-meter, with a logarithmic scale growing from the left to the right.

The following image shows an example of the two metric graphs for a single sapcc_subaccounts.tunnel service check:

Example metric graph for a single sapcc_subaccounts.tunnel service check

The upper graph shows the time elapsed since the tunnel connection was established. The lower graph shows the overall number of application connections currently active over the tunnel connection. Both graphs would show warning and critical thresholds values, which in this example are currently outside the displayed range of values for the y-axis.

Application Connection

The sub-check sapcc_subaccounts.app_conn is responsible for the monitoring of each applications connection through each tunnel connection for each subaccount on the SAP Cloud Connector. Upon inventory this sub-check creates a service check for each application connection found on the SAP Cloud Connector. During normal check execution, the number of currently active connections for each application is determined for each inventorized item. If the value of the currently active connections is above or below the configured warning and critical threshold values, an alarm is raised accordingly. For the number of currently active connections, performance data is reported by the check.

With the additional WATO plugin sapcc_subaccounts.py it is possible to configure the warning and critical levels for the sub-check sapcc_subaccounts.app_conn through the WATO WebUI and thus override the following default values:

Metric Warning Low Threshold Critical Low Threshold Warning High Threshold Critical High Threshold
Number of connections 0 0 30 40

The configuration options for the tunnel connection levels can be found in the WATO WebUI under:

-> Host & Service Parameters
   -> Parameters for discovered services
      -> Applications, Processes & Services
         -> SAP Cloud Connector Subaccounts
            -> Create Rule in Folder ...
               -> Rule Options
                  Description: SAP Cloud Connector Subaccounts

               -> Parameters
                  [x] Number of application connections
                      Warning if equal or below [0] connections
                      Critical if equal or below [0] connections
                      Warning if equal or above [30] connections
                      Critical if equal or above [40] connections

               -> Conditions
                  Folder [The folder containing the SAP Cloud Connector systems]
                  and/or
                  Explicit hosts [x]
                  Specify explicit host names [SAP Cloud Connector systems]
                  and/or
                  Application Or Tunnel Name [x]
                  Specify explicit values [Application name]

The above image with a status output example from the WATO WebUI shows one sapcc_subaccounts.app_conn service check as the 5th item from top of the displayed items. The service name is prefixed by the string SAP CC Application Connection and followed by the application name, which in this example is anonymized. For each application connection the number of currently active connections and the connection type are shown. The number of currently active application connections is also visualized in the perf-o-meter, with a logarithmic scale growing from the left to the right.

The following image shows an example of the metric graph for a single sapcc_subaccounts.app_conn service check:

Example metric graph for a single sapcc_subaccounts.app_conn service check

The graph shows the number of currently active application connections. The graph would show warning and critical thresholds values, which in this example are currently outside the displayed range of values for the y-axis.

SAP Cloud Connector Performance Backends

The check sapcc_performance_backends is responsible for the monitoring of the performance of each (on-premise) backend system connected to the SAP Cloud Connector. Upon inventory this check creates a service check for each backend connection found on the SAP Cloud Connector. During normal check execution, the number of requests to the backend system, categorized in one of the 22 runtime buckets is determined for each inventorized item. From these raw values, the request rate in requests per second is derived for each of the 22 runtime buckets. Also from the raw values, the following four additional metrics are derived:

  • calls_total: the total request rate over all of the 22 runtime buckets.

  • calls_pct_ok: the relative number of requests in percent with a runtime below a given runtime warning threshold.

  • calls_pct_warn: the relative number of requests in percent with a runtime above a given runtime warning threshold.

  • calls_pct_crit: the relative number of requests in percent with a runtime above a given runtime critical threshold.

If the relative number of requests is above the configured warning and critical threshold values, an alarm is raised accordingly. For each of the 22 runtime buckets, the total number of requests and the relative number of requests (calls_pct_ok, calls_pct_warn, calls_pct_crit), performance data is reported by the check.

With the additional WATO plugin sapcc_performance_backends.py it is possible to configure the warning and critical levels for the check sapcc_performance_backends through the WATO WebUI and thus override the following default values:

Metric Warning Threshold Critical Threshold
Request runtime 500 msec 1000 msec
Percentage of requests over request runtime thresholds 10% 5%

The configuration options for the backend performance levels can be found in the WATO WebUI under:

-> Host & Service Parameters
   -> Parameters for discovered services
      -> Applications, Processes & Services
         -> SAP Cloud Connector Backend Performance
            -> Create Rule in Folder ...
               -> Rule Options
                  Description: SAP Cloud Connector Backend Performance

               -> Parameters
                  [x] Runtime bucket definition and calls per bucket in percent
                      Warning if percentage of calls in warning bucket equal or above [10.00] %
                      Assign calls to warning bucket if runtime equal or above [500] milliseconds
                      Critical if percentage of calls in critical bucket equal or above [5.00] %
                      Assign calls to critical bucket if runtime equal or above [1000] milliseconds

               -> Conditions
                  Folder [The folder containing the SAP Cloud Connector systems]
                  and/or
                  Explicit hosts [x]
                  Specify explicit host names [SAP Cloud Connector systems]
                  and/or
                  Backend Name [x]
                  Specify explicit values [Backend name]

The above image with a status output example from the WATO WebUI shows one sapcc_performance_backends service check as the 6th item from top of the displayed items. The service name is prefixed by the string SAP CC Perf Backend and followed by a string concatenated from the protocol, FQDN and TCP port of the backend system, which in this example is anonymized. For each backend connection the total number of requests, the total request rate, the percentage of requests below the runtime warning threshold, the percentage of requests above the runtime warning threshold and the percentage of requests above the runtime critical threshold are shown. The relative number of requests in percent are also visualized in the perf-o-meter.

The following image shows an example of the metric graph for the total request rate from the sapcc_performance_backends service check:

Example metric graph for the total request rate from the sapcc_performance_backends service check

The following image shows an example of the metric graph for the relative number of requests from the sapcc_performance_backends service check:

Example metric graph for the relative number of requests from the sapcc_performance_backends service check

The graph shows the percentage of requests below the runtime warning threshold in the color green at the bottom, the percentage of requests above the runtime warning threshold in the color yellow stacked above and the percentage of requests above the runtime critical threshold in the color red stacked at the top.

The following image shows an example of the combined metric graphs for the request rates to a single backend system in each of the 22 runtime buckets from the sapcc_performance_backends service check:

Example combined metric graphs for the request rates to a single backend system in each of the 22 runtime buckets from the sapcc_performance_backends service check

To provide a better overview, the individual metrics are grouped together into three graphs. The first graph shows the request rate in the runtime buckets >=10ms, >=20ms, >=30ms, >=40ms, >=50ms, >=75ms and >=100ms. The second graph shows the request rate in the runtime buckets >=125ms, >=150ms, >=200ms, >=300ms, >=400ms, >=500ms, >=750ms and >=1000ms. The third and last graph shows the request rate in the runtime buckets >=1250ms, >=1500ms, >=2000ms, >=2500ms, >=3000ms, >=4000ms and >=5000ms.

The following image shows an example of the individual metric graphs for the request rates to a single backend system in each of the 22 runtime buckets from the sapcc_performance_backends service check:

Example individual metric graphs for the request rates to a single backend system in each of the 22 runtime buckets from the sapcc_performance_backends service check

Each of the metric graphs shows exactly the same data as the previously show combined graphs. The combined metric graphs are actually based on the individual metric graphs for the request rates to a single backend system.

Conclusion

The newly introduced checks for the SAP Cloud Connector enables you to monitor several application specific aspects of the SAP Cloud Connector with your Check_MK Server. The combination of built-in Check_MK monitoring facilities and a new agent plugin for the SAP Cloud Connector complement each other in this regard. While the new SAP Cloud Connector agent plugin for Check_MK utilizes most of the data provided by the monitoring endpoints on the SAP Cloud Connector, a more in-depth monitoring could be achieved if the data from the Most Recent Requests metric would also be exposed over the monitoring API of SAP Cloud Connector. It hope this will be the case in a future release of the SAP Cloud Connector.

I hope you find the provided new check useful and enjoyed reading this blog post. Please don't hesitate to drop me a note if you have any suggestions or run into any issues with the provided checks.

// Installing Anitya Upstream Release Monitoring on RHEL 7

Anitya is a release monitoring project developed and provided by the Fedora (Fedora) project. It is used to monitor upstream software projects for new releases in a automated fashion. It is thus similar to the uscan command from the Debian (Debian) devscripts package. Unlike the uscan command, Anitya is written in Python (Python) and comes with an integrated WebUI. Also unlike uscan, Anitya uses the FedMsg (Federated Message Bus) in order to publish messages indicating a detected upstream release event. These messages can in turn be consumed by other systems in order to trigger various actions upon a new upstream release. Anitya can be used “as a service” through https://release-monitoring.org/ or be installed as a dedicated instance. This article describes the necessary steps to set up an own, dedicated Anitya instance on a RHEL 7 system.


According to the official Anitya installation and configuration documentation there are only two steps – pip install anitya and editing /etc/anitya/anitya.toml – needed to set up Anitya. I have found this a bit too optimistic in case of a RHEL 7 system which was the installation target in my case. Here is a more detailed description of the necessary steps to get Anitya up and running on RHEL 7:

  1. Install some basic Anitya dependencies like Python, the Python installer, the Apache webserver, the Apache WSGI module and some development files:

    root@host:# yum install python httpd mod_wsgi python2-pip python-devel zeromq-devel
    
  2. Install Anitya and its dependencies:

    root@host:# pip2 install anitya
    

    Unfortunately, the Python 2 environment is necessary since the Apache WSGI module in RHEL 7 is compiled against Python 2.

  3. Adjust the import of the Anitya classes in the WGSI application. Open the file /usr/lib/python2.7/site-packages/anitya/wsgi.py in an editor of your choice:

    root@host:# vi /usr/lib/python2.7/site-packages/anitya/wsgi.py
    

    Alter its contents like shown in the following patch:

    wsgi.py
    --- /usr/lib/python2.7/site-packages/anitya/wsgi.py.orig        2018-12-05 16:59:26.000000000 +0100
    +++ /usr/lib/python2.7/site-packages/anitya/wsgi.py     2018-12-05 16:40:40.000000000 +0100
    @@ -14,7 +14,7 @@
     # You should have received a copy of the GNU General Public License
     # along with this program; if not, write to the Free Software
     # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
    -from .app import create
    +from anitya.app import create
     
     
     application = create()
  4. Create and adjust the basic Anitya configuration.

    Create the Anitya configuration directory:

    root@host:# mkdir /etc/anitya
    

    and download the sample configuration file from the Anitya GitHub repository:

    root@host:# wget -O /etc/anitya/anitya.toml https://raw.githubusercontent.com/release-monitoring/anitya/master/files/anitya.toml.sample
    

    Generate a secret key with e.g. the pwgen command:

    root@host:# pwgen -sncB 32 1
    

    Adjust the several settings in the Anitya configuration file. Open the file /etc/anitya/anitya.toml in an editor of your choice:

    root@host:# vi /etc/anitya/anitya.toml
    

    Alter the four configuration entries admin_email, secret_key, db_url and default_regex like shown in the following example:

    /etc/anitya/anitya.toml
    admin_email = "<YOUR-EMAIL-ADDRESS>"
    secret_key = "<SECRET-KEY>"
    db_url = "sqlite:////var/www/html/anitya.sqlite"
    default_regex = '''(?i)%(name)s(?:[-_]?(?:minsrc|src|source))?[-_]([^-/_\s]+?)(?:[-_]
                       (?:minsrc|src|source|asc|release))?\.(?:tar|t[bglx]z|tbz2|zip)'''

    Substitute the placeholder <YOUR-EMAIL-ADDRESS> with the email address at which you want to receive emails about errors of the Anitya application. This address will also be used as part of the request headers which identify the Anitya HTTP user agent. Substitute the placeholder <SECRET-KEY> with the secret key generated above.

    In case a SQLite database is used, make sure the value of the db_url entry points to a location within the filesystem that is reachable from the context of the webserver the Anitya application is running in. If e.g. the Apache webserver is used, the SQLite database should be located somewhere within the DocumentRoot of the Apache VirtualHost.

    Be careful with the syntax of the configuration file and the configuration entries in there. It seems that the configuration parsing part of Anitya is not very robust. A single invalid or syntactically incorrect entry will cause Anitya to skip all other entries as well and fall back to its default values. It took me a while to figure out that the Anitya database was generated and searched in the default location sqlite:////var/tmp/anitya-dev.sqlite no matter which value was given for the db_url configuration entry, just because another configuration entry – specifically the default_regex – had a syntactically incorrect value.

  5. Download the Anitya cron job from the Anitya GitHub repository:

    root@host:# cd /usr/lib/python2.7/site-packages/anitya
    root@host:# wget https://raw.githubusercontent.com/release-monitoring/anitya/<RELEASE>/files/anitya_cron.py
    

    The Anitya cron job should match the installed Anitya release. To ensure this, substitute the placeholder <RELEASE> in the above URL with the installed Anitya version. E.g. for the Anitya version 0.13.2:

    root@host:# wget https://raw.githubusercontent.com/release-monitoring/anitya/0.13.2/files/anitya_cron.py
    

    Make the Anitya cron job executeable:

    root@host:# chmod 755 /usr/lib/python2.7/site-packages/anitya/anitya_cron.py
    

    and adjust the Python interpreter to the RHEL 7 standard by opening the file /usr/lib/python2.7/site-packages/anitya/anitya_cron.py in an editor of your choice:

    root@host:# vi /usr/lib/python2.7/site-packages/anitya/anitya_cron.py
    

    Alter its contents like shown in the following patch:

    anitya_cron.py.patch
    # diff -uwi anitya_cron.py.orig anitya_cron.py
    --- anitya_cron.py.orig 2018-12-23 19:52:31.000000000 +0100
    +++ anitya_cron.py      2018-12-23 19:52:37.000000000 +0100
    @@ -1,4 +1,4 @@
    -#!/usr/bin/env python3
    +#!/usr/bin/env python2
     # -*- coding: utf-8 -*-
     
     import sys
  6. Download the Anitya database creation and initialization script from the Anitya GitHub repository:

    root@host:# cd /usr/lib/python2.7/site-packages/anitya
    root@host:# wget https://raw.githubusercontent.com/release-monitoring/anitya/master/createdb.py
    

    Run the downloaded script in order to create and initialize the Anitya database:

    root@host:# python createdb.py
    

    To grant access to the webserver the Anitya application is running in, adjust the ownership and the permissions of the Anitya database file:

    root@host:# chown apache:apache /var/www/html/anitya.sqlite
    root@host:# chmod 644 /var/www/html/anitya.sqlite
    
  7. Reapply changes to the database from a missing database version.

    Configure Alembic by opening the file /usr/lib/python2.7/site-packages/anitya/alembic.ini in an editor of your choice:

    root@host:# cd /usr/lib/python2.7/site-packages/anitya
    root@host:# vi alembic.ini
    

    Insert the contents like shown in the following configuration example:

    alembic.ini
    # A generic, single database configuration.
     
    [alembic]
    # path to migration scripts
    script_location = /usr/lib/python2.7/site-packages/anitya/db/migrations
     
    # template used to generate migration files
    # file_template = %%(rev)s_%%(slug)s
     
    # max length of characters to apply to the
    # "slug" field
    #truncate_slug_length = 40
     
    # set to 'true' to run the environment during
    # the 'revision' command, regardless of autogenerate
    # revision_environment = false
     
    # set to 'true' to allow .pyc and .pyo files without
    # a source .py file to be detected as revisions in the
    # versions/ directory
    # sourceless = false
    sqlalchemy.url = sqlite:////var/www/html/anitya.sqlite
     
    # Logging configuration
    [loggers]
    keys = root,sqlalchemy,alembic
     
    [handlers]
    keys = console
     
    [formatters]
    keys = generic
     
    [logger_root]
    level = WARN
    handlers = console
    qualname =
     
    [logger_sqlalchemy]
    level = WARN
    handlers =
    qualname = sqlalchemy.engine
    [logger_alembic]
    level = INFO
    handlers =
    qualname = alembic
     
    [handler_console]
    class = StreamHandler
    args = (sys.stderr,)
    level = NOTSET
    formatter = generic
     
    [formatter_generic]
    format = %(levelname)-5.5s [%(name)s] %(message)s
    datefmt = %H:%M:%S

    Initialize the Alembic table in the Anitya database and show the current database version:

    root@host:# alembic current
    7a8c4aa92678 (head)
    

    Manually change the database version known to Alembic in order to trick it into again applying the missing change to the database:

    root@host:# sqlite3 /var/www/html/anitya.sqlite
    
    sqlite> INSERT INTO alembic_version (version_num) VALUES ('a52d2fe99d4f');
    sqlite> .quit
    

    Reapply the change to the database from the missing database version feeaa70ead67:

    root@host:# alembic upgrade feeaa70ead67
    

    Determine the current “head” database version and manually change the database version known to Alembic:

    root@host:# alembic history
    540bdcf7edbc -> 7a8c4aa92678 (head), Add missing GitHub owner/project pairs
    27342bce1d0f -> 540bdcf7edbc, Convert GitHub URL to owner/project
    [...]
    
    root@host:# sqlite3 /var/www/html/anitya.sqlite
    
    sqlite> UPDATE alembic_version SET version_num = '7a8c4aa92678';
    sqlite> .quit
    
  8. Setup the Anitya cron job with the previously downloaded script by opening the file /etc/cron.d/anitya in an editor of your choice:

    root@host:# vi /etc/cron.d/anitya
    

    Insert the contents like shown in the following example:

    /etc/cron.d/anitya
    */5 * * * * root /usr/lib/python2.7/site-packages/anitya/anitya_cron.py 1>>/var/log/anitya.log 2>&1

    Adjust the execution interval according to your environment and needs.

  9. Create a VirtualHost configuration for e.g. the Apache webserver in whose context the Anitya application will be running in and start the Apache webserver.

    Create the VirtualHost configuration by opening the file /etc/httpd/conf.d/anitya.conf in an editor of your choice:

    root@host:# vi /etc/httpd/conf.d/anitya.conf
    

    Insert the contents like shown in the following configuration example:

    /etc/httpd/conf.d/anitya.conf
    <VirtualHost <IP>:<PORT>>
      DocumentRoot "/var/www/html/"
      ServerName <FQDN>
      <Directory />
        Options FollowSymLinks
        AllowOverride None
      </Directory>
     
      WSGIDaemonProcess anitya user=apache group=apache processes=2 threads=25 python-path=/usr/lib/python2.7/site-packages
      WSGIProcessGroup anitya
      WSGIScriptAlias /anitya /usr/lib/python2.7/site-packages/anitya/wsgi.py process-group=anitya
     
      <Directory "/usr/lib/python2.7/site-packages/anitya">
        <Files wsgi.py>
          Order deny,allow
          Allow from all
          Require all granted
        </Files>
     
        Order allow,deny
        Options Indexes FollowSymLinks
        Allow from all
      </Directory>
     
      Alias "/anitya/static" "/usr/lib/python2.7/site-packages/anitya/static"
      <Directory "/usr/lib/python2.7/site-packages/anitya/static">
        Order allow,deny
        Options Indexes FollowSymLinks
        Allow from all
        Require all granted
      </Directory>
    </VirtualHost>

    Substitute the placeholders <IP> and <PORT> with the IP address and TCP port on which the virtual host should be listening. Substitute the placeholder <FQDN> with the fully qualified domain name under which you want to reach the Anitya application with your browser.

    Enable and start the Apache webserver:

    root@host:# systemctl enable httpd
    root@host:# systemctl start httpd
    

After performing the steps outlined above, your own, dedicated Anitya instance on a RHEL 7 system should be ready. The WebUI of the Anitya instance will be reachable with a browser under the following URL http://<FQDN>/anitya, where the placeholder <FQDN> has to be substituted with the fully qualified domain name given in the above Apache VirtualHost configuration.

After logging into Anitya and adding an upstream software project, the next run of the Anitya cron job should discover the initial and current versions of the upstream software project. This should be visible in two places, firstly in the logfile of the Anitya cron job /var/log/anitya.log and secondly on the projects page of the Anitya WebUI. If this is not the case after the next execution interval of the Anitya cron job, review the logfile /var/log/anitya.log for possible errors.

I hope you find the provided Anitya installation and configuration guide for a RHEL 7 target system useful and enjoyed reading this blog post. Please don't hesitate to drop me a note if you have any suggestions or run into any issues with this guide. In future articles i will – hopefully soon – write about the neccessary steps to set up the FedMsg (Federated Message Bus) and connect Anitya to it, as well as an enhancement to Anitya itself.

This website uses cookies for visitor traffic analysis. By using the website, you agree with storing the cookies on your computer.More information