Monitoring HTTP server status

The aim of this document is to provide information about how to monitor the status of HTTP servers with which your machine interacts with and how to make handling decisions based on their status using lib_srv_qualifier.js (http://yate.null.ro/websvn/filedetails.php?repname=yate-common&path=%2Ftrunk%2Fscripts%2Flib_srv_qualifier.js) from yate-common (http://yate.null.ro/websvn/listing.php?repname=yate-common&path=%2Ftrunk%2F#path_trunk) .

Enabling functionality

For this, in your Yate machine, you need to have enabled a script that loads and configures lib_srv_qualifier.js from SVN/yate-common. The script should look like this:

conf_name = “my_http_qual”;

module_name = “my_http_qual”;

Server.type = “HTTP”;

#require “lib_srv_qualifier.js”

qualifier_init(true);

/* vi: set ts=8 sw=4 sts=4 noet: */

where:

  • conf_name specifies the file to use for configuration.
  • module_name specifies the name of the module and it is used in debug messages and in filtering Yate messages.
  • Server.type specifies what kind of monitoring you want. Options are SIP and HTTP. For this case, HTTP is the needed option.

How it works

In order to determine the availability status of a HTTP server, the script generates at the configured interval of time a http.request message. In order for this to work, a module handling http.request messages must be loaded in YATE (e.g. HTTP client module or http_req.sh script) . A HTTP module will generate a HTTP request on the wire and provide back the answer.

The server for which the request is generates is picked using round-robin, so every number_of_servers * check_interval seconds, that server is checked. The server is considered to be available if no error was reported by the HTTP client module and the response to HTTP request is in the 2xx range.

Upon receiving the response to the HTTP request, if there is a transition from one state to another (e.g up → down / down → up) the script will generate a notification message that looks like this:

Sniffed ‘http.qualify’ time=1604330971.439216

  thread=0x3062400 ‘Engine Worker’

  data=(nil)

  retval='(null)’

  param[‘server’] = ‘http://192.168.168.119/test/’

  param[‘name’] = ‘test_http’

  param[‘autoblock’] = ‘true’

  param[‘available’] = ‘true’

  param[‘code’] = ‘200’

where:

  • server is the checked HTTP URL
  • name is the name of the configuration section for this server
  • available denotes the status of the server: true for up, false otherwise.
  • autoblock specifies if the script will block HTTP requests towards that URL if the server is down
  • code is the result of the HTTP request
  • error is the error reported by the HTTP module for failing to send the request

This message can then be caught by other modules in Yate that are interested in the state of HTTP servers.

Configuration

Once the script in loaded, it will read as configuration the file with the name set from conf_name.

This configuration file specifies what servers should be monitored.

The configuration file for HTTP monitoring should look like this:

[general]

autoblock=yes

check_interval=1-

timeout=5

;http_req_prio=85

 

[server test]

url=http://192.168.168.119/test/

timeout=10

autoblock=no

 

[server http://192.168.168.241/test/%5D

;enable=no

 

[server http://10.10.10.10/%5D

The general section configures:

  • check_interval: interval in seconds at which the script picks the server to check
  • autoblock: true if the script should automatically block all HTTP request for a server that is detected as being down. False otherwise
  • timeout: time in seconds to set as a timeout for HTTP requests. This is passed on to the module actually making the request.
  • http_req_prio: the priority of the script handler for http.request. This is needed for when the script will block request to down servers. This handler should always have a higher priority than other http.request handling module in order for autoblock to work.

The values in the section from the example are the default values

Every sections that starts with server configures a server to be monitored. The configuration parameters for each section are:

  • enable: boolean that enables or disables the checking of this server. Default value is false.
  • url: the HTTP URL towards a HTTP request should be make. If not provided, URL will be taken to be the name of the configuration sections without the ‘server ‘ prefix
  • timeout: this has the same significance as the parameter from the general configuration section, but it’s set to override the value from there.
  • autoblock: it has the same significance as the parameter from the general configuration section, but it’s set to override the value from there.

Changes of parameters are applied on reload.

Uses of HTTP monitoring

Auto-blocking requests

If a HTTP server is detected as being down by the monitoring script and the server has autoblock parameter set to true, then the script will block the HTTP requests made towards URLs that start with the one of the server.

Below is an example of a piece of log showing a HTTP request being blocked after the server was detected as being down.

Example of server being detected as being down by the monitoring script:

Sniffed ‘http.request’ time=1604333069.843392

  thread=0x7ff0d00fa5e0 ‘JsScheduler’

  data=(nil)

  retval='(null)’

  param[‘module’] = ‘http_srv_qualifier’

  param[‘url’] = ‘http://10.10.10.10/’

  param[‘wait’] = ‘true’

  param[‘timeout’] = ‘5000’

2020-11-02_16:04:34.854000 <httpclient/145:NOTE> Failed state=Starting reason=timeout error=’CURLE:28 Timeout was reached’ [0x7ff0d4004e70]

Returned true ‘http.request’ delay=5.010741

  thread=0x7ff0d00fa5e0 ‘JsScheduler’

  data=(nil)

  retval='(null)’

  param[‘module’] = ‘http_srv_qualifier’

  param[‘url’] = ‘http://10.10.10.10/&#8217;

  param[‘wait’] = ‘true’

  param[‘timeout’] = ‘5000’

  param[‘handlers’] = ‘httpclient:90’

  param[‘reason’] = ‘timeout’

  param[‘error’] = ‘CURLE:28 Timeout was reached’

2020-11-02_16:04:34.854321 Server http://10.10.10.10/ is unavailable

Sniffed ‘http.qualify’ time=1604333074.854514

  thread=0x3061bf0 ‘Engine Worker’

  data=(nil)

  retval='(null)’

  param[‘server’] = ‘http://10.10.10.10/&#8217;

  param[‘name’] = ‘http://10.10.10.10/&#8217;

  param[‘autoblock’] = ‘true’

  param[‘available’] = ‘false’

  param[‘error’] = ‘CURLE:28 Timeout was reached’

Returned false ‘http.qualify’ delay=0.001526

  thread=0x3061bf0 ‘Engine Worker’

  data=(nil)

  retval='(null)’

  param[‘server’] =  ‘http://10.10.10.10/&#8217;

  param[‘name’] = ‘http://10.10.10.10/&#8217;

  param[‘autoblock’] = ‘true’

  param[‘available’] = ‘false’

  param[‘error’] = ‘CURLE:28 Timeout was reached’

Below is the same script blocking a http.request message (and implicitly the HTTP request that would have been made by a HTTP module) towards a server that it knows is not available. It sets as response a general failure:

Sniffed ‘http.request’ time=1604332878.100992

  thread=0x3062060 ‘Engine Worker’

  data=(nil)

  retval='(null)’

  param[‘url’] = ‘http://10.10.10.10/test/test.php?msisdn=&imsi=&data=04069121&#8217;

  param[‘accept’] = ‘text/plain;charset=UTF-8’

  param[‘multiline’] = ‘false’

  param[‘agent’] = ‘YATE/6.2.1’

  param[‘wait’] = ‘true’

Returned true ‘http.request’ delay=0.002839

  thread=0x3062060 ‘Engine Worker’

  data=(nil)

  retval='(null)’

  param[‘url’] = ‘http://10.10.10.10/test/test.php?msisdn=&imsi=&data=04069121&#8217;

  param[‘accept’] = ‘text/plain;charset=UTF-8’

  param[‘multiline’] = ‘false’

  param[‘agent’] = ‘YATE/6.2.1’

  param[‘wait’] = ‘true’

  param[‘handlers’] = ‘http_srv_qualifier:85’

  param[‘error’] = ‘failure’

Handling http.qualify messages

Because the script emits a http.qualify message when the state of the HTTP server changes, interested parties can catch the message an do some decisions based on that.

Below there is an example of script that would periodically submit some data to a server, but stops when the server is down:

Engine.debugName(“http_ping”);

Message.trackName(“http_ping”);

 

url = “http://10.10.10.10/ping=&#8221;;

send = true;

function onHTTPQual(msg)

{

    if (url.startsWith(msg.server))

        send = !!msg.available;

    var str = “activated”;

    if (!send)

        str = “deactivated”;

    Engine.debug(Engine.DebugInfo,”Sending ping is ” + str);

    return false; // let others see the message too

}

function onInterval()

{

    if (send) {

        Engine.debug(Engine.DebugAll,”Sending ping”);

        var m = new Message(“http.request”);

        m.module = “http_ping”;

        m.url = url + Date.now();

        m.wait = false;

        m.timeout = 5000;

        m.enqueue();

    }

    else

        Engine.debug(Engine.DebugNote,”Cannot set ping, server is down”);

    }

Message.install(onHTTPQual,”http.qualify”,50);

Engine.setInterval(onInterval,5000);

The log shows how the script worked upon detecting that the server is down:

2020-11-02_16:41:16.384731 Sending ping

Sniffed ‘http.request’ time=1604335276.384834

  thread=0x3061960 ‘Engine Worker’

  data=(nil)

  retval='(null)’

  param[‘module’] = ‘http_ping’

  param[‘url’] = ‘http://10.10.10.10/ping=1604335276384&#8217;

  param[‘wait’] = ‘false’

  param[‘timeout’] = ‘5000’

Returned true ‘http.request’ delay=0.000193

  thread=0x3061960 ‘Engine Worker’

  data=(nil)

  retval='(null)’

  param[‘module’] = ‘http_ping’

  param[‘url’] = ‘http://10.10.10.10/ping=1604335276384&#8217;

  param[‘wait’] = ‘false’

  param[‘timeout’] = ‘5000’

  param[‘handlers’] = ‘httpclient:90’

Sniffed ‘http.qualify’ time=1604335279.992129

  thread=0x30616b0 ‘Engine Worker’

  data=(nil)

  retval='(null)’

  param[‘server’] = ‘http://10.10.10.10/&#8217;

  param[‘name’] = ‘http://10.10.10.10/&#8217;

  param[‘autoblock’] = ‘true’

  param[‘available’] = ‘false’

  param[‘error’] = ‘CURLE:28 Timeout was reached’

2020-11-02_16:41:19.994135 Sending ping is deactivated

Returned false ‘http.qualify’ delay=0.002239

  thread=0x30616b0 ‘Engine Worker’

  data=(nil)

  retval='(null)’

  param[‘server’] = ‘http://10.10.10.10/&#8217;

  param[‘name’] = ‘http://10.10.10.10/&#8217;

  param[‘autoblock’] = ‘true’

  param[‘available’] = ‘false’

  param[‘error’] = ‘CURLE:28 Timeout was reached’

  param[‘handlers’] = ‘http_ping:50’

2020-11-02_16:41:21.390599 Cannot set ping, server is down

2020-11-02_16:41:26.409782 Cannot set ping, server is down