Skip to main content

Sumo Logic App for HAProxy

Thumbnail icon

HAProxy is open source software that provides a high availability load balancer and proxy server for TCP and HTTP-based applications that spreads requests across multiple servers.

The Sumo Logic App for HAProxy is a unified logs and metrics app that helps you monitor the availability, performance, and health of your HAProxy cluster. Preconfigured dashboards provide insights into active servers, visitor locations, sessions, errors, response time, and throughput.

HAProxy Log Types​

The app supports Logs and Metrics from the open source version of HAProxy. The App is tested on the 2.3.9 version of HAProxy.

The HAProxy logs are generated in files as configured in the configuration file /etc/haproxy/haproxy.cfg (learn more).

The Sumo Logic App for HAProxy supports metrics generated by the HAProxy plugin for Telegraf. The app assumes prometheus format Metrics.

Sample Log Messages​

{
  "timestamp": 1620894084572,
  "log": "<150>May 13 08:21:20 haproxy[9]: 127.0.0.1:38718 stats stats/<STATS> 0/0/0/0/0 200 3926 - - LR-- 1/1/0/0/0 0/0 \"GET /haproxy?stats/;csv HTTP/1.1\"",
  "stream": "stdout",
  "time": "2021-05-13T08:21:20.005706219Z"
}

Sample Query​

This query example is from the HAProxy - Overview dashboard > Top 5 Clients causing Errors panels.

<HAProxy_Cluster_Filter> proxy_system=haproxy component=proxy
| json "log" as _rawlog nodrop
| if (isEmpty(_rawlog), _raw, _rawlog) as haproxy_log_message
| parse regex field=haproxy_log_message "(?<syslog_host>[\w\-\.]+) (?<ps>\w+)\[(?<pid>\d+)\]: (?<c_ip>[\w\.]+):(?<c_port>\d+) (?<rs>[^\"]*) (?<status_code>\d+) (?<bytes>\d+) (?<req_cookie>\S+) (?<res_cookie>\S+) (?<t_state>[\w-]+) (?<actconn>\d+)/(?<feconn>\d+)/(?<beconn>\d+)\/(?<srv_conn>\d+)/(?<retries>\d+) (?<srv_queue>\d+)/(?<backend_queue>\d+) \"(?<method>\w+) (?<request>[^\"]*) (?<http_version>\w+)"
| where status_code matches "4*" or status_code matches "5*" and  !isEmpty(c_ip)
| count by c_ip
| sort by _count
| limit 5

Collecting Logs and Metrics for HAProxy​

This section provides instructions for configuring logs and metrics collection for the Sumo Logic App for HAProxy.

Configuring log and metric collection for the HAProxy App includes the following tasks:

Step 1: Configure Fields in Sumo Logic​

Create the following Fields in Sumo Logic prior to configuring collection. This ensures that your logs and metrics are tagged with relevant metadata, which is required by the app dashboards. For information on setting up fields, see Sumo Logic Fields.

If you're using HAProxy in a Kubernetes environment, create the fields:

  • pod_labels_component
  • pod_labels_environment
  • pod_labels_proxy_system
  • pod_labels_proxy_cluster

Step 2: Configure Collection for HAProxy​

Sumo Logic supports collection of logs and metrics data from HAProxy in both Kubernetes and non-Kubernetes environments.

In Kubernetes environments, we use the Telegraf Operator, which is packaged with our Kubernetes collection (learn more). The diagram below illustrates how data is collected from HAProxy in Kubernetes environments. In the architecture shown below, there are four services that make up the metric collection pipeline: Telegraf, Prometheus, Fluentd and FluentBit.

haproxy k8s flow

The first service in the pipeline is Telegraf. Telegraf collects metrics from HAProxy. Note that we’re running Telegraf in each pod we want to collect metrics from as a sidecar deployment for example, Telegraf runs in the same pod as the containers it monitors. Telegraf uses the HAProxy input plugin to obtain metrics. (For simplicity, the diagram doesn’t show the input plugins.) The injection of the Telegraf sidecar container is done by the Telegraf Operator. We also have Fluentbit that collects logs written to standard out and forwards them to FluentD, which in turn sends all the logs and metrics data to a Sumo Logic HTTP Source.

Prerequisites

It’s assumed that you are using the latest helm chart version. If not, upgrade using the instructions here.

Configure Metrics Collection​

This section explains the steps to collect HAProxy metrics from a Kubernetes environment. In Kubernetes environments, we use the Telegraf Operator, which is packaged with our Kubernetes collection. You can learn more on this here. Follow the steps listed below to collect metrics from a Kubernetes environment.

  1. On your HAProxy Pods, add the following annotations:
annotations:
    telegraf.influxdata.com/class: sumologic-prometheus
    prometheus.io/scrape: "true"
    prometheus.io/port: "9273"
    telegraf.influxdata.com/inputs: |+
[[inputs.haproxy]]
  servers = ["http://127.0.0.1:1024/stats"]
  [inputs.haproxy.tags]
    environment="dev_CHANGEME"
    component="proxy"
    proxy_system="haproxy"
    proxy_cluster="haproxy_on_k8s_CHANGEME"

  1. Enter in values for the following parameters (marked CHANGEME above):

    • telegraf.influxdata.com/inputs. This contains the required configuration for the Telegraf HAProxy Input plugin. See this doc for more information on configuring the HAProxy input plugin for Telegraf. As telegraf will be run as a sidecar, the host should always be localhost.

    • In the input plugins section:

      • servers. The URL to the HAProxy server. This can be a comma-separated list to connect to multiple HAProxy servers. See this doc for more information on additional parameters for configuring the HAProxy input plugin for Telegraf.
      • In the tags section [inputs.haproxy.tags]
        • environment. This is the deployment environment where the HAProxy cluster identified by the value of servers resides. For example: dev, prod or qa. While this value is optional we highly recommend setting it.
        • proxy_cluster. Enter a name to identify this HAProxy cluster. This cluster name will be shown in the Sumo Logic dashboards.
      Do not modify the other values

      Modifying these values will cause the Sumo Logic apps to function incorrectly

      • telegraf.influxdata.com/class: sumologic-prometheus - Instructs the Telegraf operator what output to use.
      • prometheus.io/scrape: "true" - Ensures our Prometheus will scrape the metrics.
      • prometheus.io/port: "9273" - Tells prometheus what ports to scrape on.
      • telegraf.influxdata.com/inputs - In the tags section, for example: [inputs.haproxy.tags]
      • component: “proxy” - Used by Sumo Logic apps to identify application components.
      • proxy_system: “haproxy” - Identifies the proxy system.

    For all other parameters, please see this doc for more parameters that can be configured in the Telegraf agent globally.

  2. Kubernetes collection will start collecting metrics from the pods having the labels and annotations defined in the previous step.

  3. Verify metrics in Sumo Logic.

Configure Logs Collection​

This section explains the steps to collect HAProxy logs from a Kubernetes environment.

  1. Add labels on your HAProxy pods to capture logs from standard output (stdout) on Kubernetes:

    labels:
      environment: "prod"
      component: "proxy"
      proxy_system: "haproxy"    
          proxy_cluster: "haproxy_prod_cluster01"

  2. Enter in values for the following parameters:

    • environment. This is the deployment environment where the HAProxy cluster identified by the value of servers resides. For example: dev, prod or qa. While this value is optional we highly recommend setting it.
    • proxy_cluster. Enter a name to identify this HAProxy cluster. This cluster name will be shown in the Sumo Logic dashboards.
    Do not modify the other values

    Do not modify the other values Modifying these values will cause the Sumo Logic apps to function incorrectly

    • component: “proxy”. This value is used by Sumo Logic apps to identify application components.
    • proxy_system: “haproxy”. This value identifies the proxy system.

    For all other parameters see this doc for more parameters that can be configured in the Telegraf agent globally.

  3. Collecting HAProxy Logs from a Log File (Optional). Follow the steps below to capture HAProxy logs from a log file on Kubernetes.

    1. Determine the location of the HAProxy log file on Kubernetes. This can be determined from the HAProxy.conf for your HAProxy cluster along with the mounts on the HAProxy pods.
    2. Install the Sumo Logic tailing sidecar operator.
    3. Add the following annotation in addition to the existing annotations.
    annotations:
      tailing-sidecar: sidecarconfig;<mount>:<path_of_Haproxy_log_file>/<Haproxy_log_file_name>

    Example:

    annotations:
      tailing-sidecar: sidecarconfig;data:/var/log//haproxy.log
    1. Make sure that the HAProxy pods are running and annotations are applied by using the command:
    kubectl describe pod <haproxy_pod_name>

  4. Sumo Logic Kubernetes collection will automatically start collecting logs from the pods having the annotations defined above.

  5. Add an FER to normalize the fields in Kubernetes environments. Labels created in Kubernetes environments automatically are prefixed with pod_labels. To normalize these for our app to work, we need to create a Field Extraction Rule if not already created for Proxy Application Components. To do so:

    1. Go to Manage Data > Logs > Field Extraction Rules.
    2. Click the + Add button on the top right of the table.
    3. The Add Field Extraction Rule form will appear.
    4. Enter the following options:
    • Rule Name. Enter the name as App Observability - Proxy.
    • Applied At. Choose Ingest Time
    • Scope. Select Specific Data
    • Scope: Enter the following keyword search expression: 
      pod_labels_environment=* pod_labels_component=proxy pod_labels_proxy_system=* pod_labels_proxy_cluster=*
    • Parse Expression. Enter the following parse expression:
      | if (!isEmpty(pod_labels_environment), pod_labels_environment, "") as environment
      | pod_labels_component as component
      | pod_labels_proxy_system as proxy_system
      | pod_labels_proxy_cluster as proxy_cluster
    1. Click Save to create the rule.

Verify logs are flowing into Sumo Logic by running the following logs query:

component="proxy" proxy_cluster="<Your-HAProxy-Server>" proxy_system="haproxy"

Installing the HAProxy Monitors​

Sumo Logic has provided pre-packaged alerts available through Sumo Logic monitors to help you proactively determine if a HAProxy cluster is available and performing as expected. These monitors are based on metric and log data and include pre-set thresholds that reflect industry best practices and recommendations. For more information about individual alerts, see HAProxy Alerts.

To install these monitors, you must have the Manage Monitors role capability. You can install by importing a JSON file or using a Terraform script.

note

There are limits to how many alerts can be enabled. For more information, see Monitors for details.

Method A: Import a JSON file​

Download the JSON file that describes the monitors. The JSON contains the alerts that are based on Sumo Logic searches that do not have any scope filters and therefore will be applicable to all HAProxy clusters, the data for which has been collected via the instructions in the previous sections.

However, if you would like to restrict these alerts to specific clusters or environments, update the JSON file by replacing the text proxy_cluster=* with <Your Custom Filter>. Custom filter examples:

  • For alerts applicable only to a specific cluster, your custom filter would be: proxy_cluster=dev-haproxy01
  • For alerts applicable to all clusters that start with haproxy-prod: proxy_cluster=haproxy-prod*
  • For alerts applicable to a specific cluster within a production environment: proxy_cluster=dev-haproxy01 AND environment=prod. This assumes you have set the optional environment tag while configuring collection.
  1. Go to Manage Data > Alerts > Monitors.
  2. Click Add.
  3. Click Import.
  4. On the Import Content popup, enter HAProxy in the Name field, paste in the JSON into the the popup, and click Import.
  5. The monitors are created in a "HAProxy" folder. The monitors are disabled by default. See the Monitors topic for information about enabling monitors and configuring notifications or connections.

Method 2: Use a Terraform script​

  1. Generate an access key and access ID for a user that has the Manage Monitors role capability. For instructions see Access Keys.
  2. Download Terraform 0.13 or later, and install it.
  3. Download the Sumo Logic Terraform package for HAProxy monitors. The alerts package is available in the Sumo Logic github repository. You can either download it using the git clone command or as a zip file.
  4. Alert Configuration. After extracting the package, navigate to the terraform-sumologic-sumo-logic-monitor/monitor_packages/haproxy/ directory. Edit the haproxy.auto.tfvars file and add the Sumo Logic Access Key and Access ID from Step 1 and your Sumo Logic deployment. If you're not sure of your deployment, see Sumo Logic Endpoints and Firewall Security.
access_id   = "<SUMOLOGIC ACCESS ID>"
access_key  = "<SUMOLOGIC ACCESS KEY>"
environment = "<SUMOLOGIC DEPLOYMENT>"

The Terraform script installs the alerts without any scope filters, if you would like to restrict the alerts to specific clusters or environments, update the haproxy_data_source variable. For example:

  • To configure alerts for A specific cluster, set haproxy_data_source to something like proxy_cluster=haproxy.prod.01
  • To configure alerts for All clusters in an environment, set haproxy_data_source to something like environment=prod
  • To configure alerts for Multiple clusters using a wildcard, set haproxy_data_source to something like proxy_cluster=haproxy-prod*
  • To configure alerts for A specific cluster within a specific environment, set haproxy_data_source to something like proxy_cluster=haproxy-1 and environment=prod. This assumes you have configured and applied Fields as described in Step 1: Configure Fields of the Sumo Logic of the Collect Logs and Metrics for HAProxy.

All monitors are disabled by default on installation. To enable all of the monitors, set the monitors_disabled parameter to false.

By default, the monitors will be located in a "HAProxy" folder on the Monitors page. To change the name of the folder, update the monitor folder name in the folder variable in the haproxy.auto.tfvars file.

  1. If you want the alerts to send email or connection notifications, edit the haproxy_notifications.auto.tfvars file to populate the connection_notifications and email_notifications sections. Examples are provided below.

In the variable definition below, replace <CONNECTION_ID> with the connection ID of the Webhook connection. You can obtain the Webhook connection ID by calling the Monitors API.

Pagerduty connection example
connection_notifications = [
    {
      connection_type       = "PagerDuty",
      connection_id         = "<CONNECTION_ID>",
      payload_override      = "{\"service_key\": \"your_pagerduty_api_integration_key\",\"event_type\": \"trigger\",\"description\": \"Alert: Triggered {{TriggerType}} for Monitor {{Name}}\",\"client\": \"Sumo Logic\",\"client_url\": \"{{QueryUrl}}\"}",
      run_for_trigger_types = ["Critical", "ResolvedCritical"]
    },
    {
      connection_type       = "Webhook",
      connection_id         = "<CONNECTION_ID>",
      payload_override      = "",
      run_for_trigger_types = ["Critical", "ResolvedCritical"]
    }
  ]

For information about overriding the payload for different connection types, see Set Up Webhook Connections.

Email notifications example
email_notifications = [
    {
      connection_type       = "Email",
      recipients            = ["abc@example.com"],
      subject               = "Monitor Alert: {{TriggerType}} on {{Name}}",
      time_zone             = "PST",
      message_body          = "Triggered {{TriggerType}} Alert on {{Name}}: {{QueryURL}}",
      run_for_trigger_types = ["Critical", "ResolvedCritical"]
    }
  ]
  1. Installing Monitors:
    1. Navigate to the terraform-sumologic-sumo-logic-monitor/monitor_packages/haproxy/ directory and run terraform init. This will initialize Terraform and download the required components.
    2. Run terraform plan to view the monitors that Terraform will create or modify.
    3. Run terraform apply.

Installing the HAProxy App​

Now that you have set up collection for HAProxy, you can install the HAProxy App to use the pre-configured searches and dashboard that provide insight into your data.

To install the app, do the following:

Locate and install the app you need from the App Catalog. If you want to see a preview of the dashboards included with the app before installing, click Preview Dashboards.

  1. From the App Catalog, search for and select the app.
  2. Select the version of the service you're using and click Add to Library. Version selection is applicable only to a few apps currently. For more information, see the Install the Apps from the Library.
  3. To install the app, complete the following fields.
    1. App Name. You can retain the existing name, or enter a name of your choice for the app.

    2. Data Source. Select either of these options for the data source.

      • Choose Source Category, and select a source category from the list.

      • Choose Enter a Custom Data Filter, and enter a custom source category beginning with an underscore. Example: (_sourceCategory=MyCategory).

    3. Advanced. Select the Location in Library (the default is the Personal folder in the library), or click New Folder to add a new folder.
  4. Click Add to Library.

Once an app is installed, it will appear in your Personal folder, or other folder that you specified. From here, you can share it with your organization.

Panels will start to fill automatically. It's important to note that each panel slowly fills with data matching the time range query and received since the panel was created. Results won't immediately be available, but with a bit of time, you'll see full graphs and maps.

Viewing HAProxy Dashboards​

Overview​

The HAProxy - Overview dashboard provides an at-a-glance view of HAProxy Backend and Frontend HTTP error codes percentage, visitor location, URLs and Clients causing errors.

  • Identify Frontend and Backend Sessions percentage usage to understand active sessions. This can help you increase the HAProxy session limit.
  • Gain insights into originated traffic location by region. This can help you allocate computer resources to different regions according to their needs.
  • Gain insights into Client, Server Responses on HAProxy Server. This helps you identify errors in HAProxy Server.
  • Gain insights into Network traffic for the Frontend and Backend system of your HAProxy server.
test

Backend​

The HAProxy - Backend dashboard provides an at-a-glance view for the number of backend active servers, backend weight, respond code from backend and throughput http.

Backend dashboard

Frontend​

The HAProxy - Backend dashboard provides an at-a-glance view detail of HAProxy Frontend. It provides information such as number request to frontend, number of error requests,s, and current session.

test

Server​

The HAProxy - Backend dashboard provides an at-a-glance view detail of HAProxy Server. This dashboard helps you monitoring uptime, and error request by proxy.

test

Error Log Analysis​

The HAProxy - Error Logs Analysis dashboard provides a high-level view of log level breakdowns, comparisons, and trends. The panels also show the geographic locations of clients and clients with critical messages, new connections and outliers, client requests, request trends, and request outliers.

Use this dashboard to:

  • Track requests from clients. A request is a message asking for a resource, such as a page or an image.
  • To track and view client geographic locations generating errors.
  • Track critical alerts and emergency error alerts.
test

Outlier Analysis​

The HAProxy - Outlier Analysis dashboard provides a high-level view of HAProxy server outlier metrics for bytes served, number of visitors, and server errors. You can select the time interval over which outliers are aggregated, then hover the cursor over the graph to display detailed information for that point in time.

Use this dashboard to:

  • Detect outliers in your infrastructure with Sumo Logic’s machine learning algorithm.
  • To identify outliers in incoming traffic and the number of errors encountered by your servers.
test

Threat Analysis​

The HAProxy - Threat Intel dashboard provides an at-a-glance view of threats to HAProxy servers on your network. Dashboard panels display the threat count over a selected time period, geographic locations where threats occurred, source breakdown, actors responsible for threats, severity, and a correlation of IP addresses, method, and status code of threats.

Use this dashboard to:

  • To gain insights and understand threats in incoming traffic and discover potential IOCs. Incoming traffic requests are analyzed using the Sumo - Crowdstrikes threat feed.
test

The HAProxy - Trends dashboard provides an at-a-glance view of traffic to HAProxy servers on your network. Dashboard panels display the traffic count over one day time period, locations where traffic trends for visits by country one days time.

test

Visitor Access Types​

The HAProxy - Visitor Access Types dashboard provides insights into visitor platform types, browsers, and operating systems, as well as the most popular mobile devices, PC and Mac versions used.

Use this dashboard to:

  • Understand which platform and browsers are used to gain access to your infrastructure. These insights can be useful for planning in which browsers, platforms, and operating systems (OS) should be supported by different software services.
test

Visitor Locations​

The HAProxy - Visitor Locations dashboard provides a high-level view of HAProxy visitor geographic locations both worldwide and in the United States. Dashboard panels also show graphic trends for visits by country over time and visits by US region over time.

Use this dashboard to:

  • Gain insights into geographic locations of your user base. This is useful for resource planning in different regions across the globe.
test

Visitor Traffic Insight​

The HAProxy - Visitor Traffic Insight dashboard provides detailed information on the top documents accessed, top referrers, top search terms from popular search engines, and the media types served.

Use this dashboard to:

  • To understand the type of content that is frequently requested by users.
  • It helps in allocating IT resources according to the content types.
test

Web Server Operations​

The HAProxy - Web Server Operations dashboard provides a high-level view combined with detailed information on the top ten bots, geographic locations, and data for clients with high error rates, server errors over time, and non 200 response code status codes. Dashboard panels also show information on server error logs, error log levels, error responses by a server, and the top URIs responsible for 404 responses.

Use this dashboard to:

  • Gain insights into Client, Server Responses on HAProxy Server. This helps you identify errors in HAProxy Server.
  • To identify geo locations of all Client errors. This helps you identify client location causing errors and helps you to block client IPs.
test

HAProxy Alerts​

Alert Type (Metrics/Logs)Alert NameAlert DescriptionTrigger Type (Critical / Warning)Alert ConditionRecover Condition
LogsHAProxy - Access from Highly Malicious SourcesThis alert fires when an HAProxy is accessed from highly malicious IP addresses.Critical > 0 < = 0
LogsHAProxy - High Client (HTTP 4xx) Error RateThis alert fires when there are too many HTTP requests (>5%) with a response status of 4xx.Critical > 00
LogsHAProxy - High Server (HTTP 5xx) Error RateThis alert fires when there are too many HTTP requests (>5%) with a response status of 5xx.Critical > 00
LogsHAProxy - Backend ErrorThis alert fires when we detect backend server errors.Critical >0 < = 0
LogsHAProxy - Backend Server DownThis alert fires when we detect a backend server for a given HAProxy server is down.Critical >0 < = 0
MetricsHAProxy - High Active Backend Server Sessionswhen the percent of backend server connections are high.Warning >80 < = 80
MetricsHAProxy - Frontend Security Blocked RequestsHAProxy is blocking requests for security reasonsWarning >10 < = 10
MetricsHAProxy - Has No Alive BackendsHAProxy has no alive active or backup backend serversCritical >0 < = 0
MetricsHAProxy - Slow Response Timethe HAProxy response times are greater than one second.Critical >1 < = 1
MetricsHAProxy - Pending RequestsHAProxy requests are pendingWarning >0 < = 0
MetricsHAProxy - Retry Highthere is a high retry rateWarning >0 < = 0
MetricsHAProxy - High Server Connection Errorsthere are too many connection errors to backend servers.Warning >100 < = 100
MetricsHAProxy - Server Healthcheck Failureserver healthchecks are failing.Warning >0 < = 0
Sumo Logic YouTubeSumo Logic Twitter
Legal
Privacy Statement
Terms of Use

Copyright © 2022 by Sumo Logic, Inc.