Skip to main content

Sumo Logic App for Apache

Thumbnail icon

The Apache app is a unified logs and metrics app that helps you monitor the availability, performance, health and resource utilization of Apache web server farms. Preconfigured dashboards and searches provide visibility into your environment for real-time or historical analysis: visitor locations, visitor access types, traffic patterns, errors, web server operations, resource utilization and access from known malicious sources.

Log Types and Metrics​

The Sumo Logic App for Apache assumes:

  • The NCSA extended/combined log file format has been configured for Apache access logs and the default error log format for Apache Access logs and Apache Error logs. For a list of metrics that are collected and used by the app, see Apache Metrics.

  • The Apache - Overview dashboard is based on both Apache logs and metrics.

  • Dashboards in the Metrics folder are based on Apache metrics alone.

  • Dashboards and searches in the Logs folder are based on Apache access and error logs

Sample Log Messages​

Access Logs
{
    "timestamp":1620630466883,
    "log":"192.168.29.177 - - [10/May/2021:07:07:44 +0000] \"GET / HTTP/1.1\" 200 45",
    "stream":"stdout",
    "time":"2021-05-10T07:07:44.649858568Z"
}
Error Logs
{
    "timestamp":1620125665927,
    "log":"[Tue May 04 10:54:25.460469 2021] [ssl:error] [pid 53] [client 192.168.85.135:52327] AH02042: rejecting client initiated renegotiation",
    "stream":"stderr",
    "time":"2021-05-04T10:54:25.460664201Z"
}

Sample Query​

This sample Query is from the Top 5 Clients Causing 4xx Errors panel of the Apache - Web server Operations dashboard.

Query String
webserver_system=apache webserver_farm=* HTTP (40* OR 41* OR 42* OR 43* OR 44* or 45* or 49*)
| json "log" nodrop | if (_raw matches "{*", log, _raw) as mesg
| parse regex field=mesg "^(?<src_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" nodrop
| parse regex field=mesg "(?<method>[A-Z]+)\s(?<url>\S+)\sHTTP\/[\d\.]+[\\n]*\"\s(?<status_code>\d+)\s(?<size>[\d-]+)" nodrop
| parse regex field=mesg "(?<method>[A-Z]+)\s(?<url>\S+)\sHTTP\/[\d\.]+[\\n]*\"\s(?<status_code>\d+)\s(?<size>[\d-]+)\s\"(?<referrer>.*?)\"\s\"(?<user_agent>.+?)\".*" nodrop
| where status_code matches "4*"
| count as count by src_ip
| sort count, src_ip asc
| limit 5

Apache App Searches​

The predefined searches in the Apache app are based on the Apache Access logs and Apache Error logs.

Searches based on Apache Access logs​

  • Apache - All HTTP Response codes with their count
  • Apache - Client Errors (4xx response codes) per day
  • Apache - HTTP status code summary over time
  • Apache - Malicious URL requests
  • Apache - Robots
  • Apache - Slowest URLs by average time
  • Apache - Time taken to serve requests
  • Apache - Top 404 referrers
  • Apache - Top browsers
  • Apache - Top clients
  • Apache - Top clients causing errors responses
  • Apache - Top URLs by bytes served
  • Apache - Traffic volume and bytes served per day

Search based on Apache Error logs​

  • Apache - Critical log messages
  • Apache - Log Level counts
  • Apache - Server start and stop events
  • Apache - Server stops and starts over time
  • Apache - Top error reasons
  • Apache - Top files causing errors
  • Apache - Top Referrers causing errors

Collecting Logs and Metrics for Apache​

This section provides instructions for configuring log and metrics collection for the Sumo Logic App for Apache.

Step 1: Configure Fields in Sumo Logic​

Create the following Fields in Sumo Logic prior to configuring collection. This ensures that your logs and metrics are tagged with relevant metadata, which is required by the app dashboards. For information on setting up fields, see Sumo Logic Fields.

If you're using Apache in a Kubernetes environment, create the fields:

  • pod_labels_component
  • pod_labels_environment
  • pod_labels_webserver_system
  • pod_labels_webserver_farm

Step 2: Configure Your Environment for Apache Logs and Metrics Collection​

Sumo Logic supports collection of logs and metrics data from Apache in both Kubernetes and non-Kubernetes environments. Please click on the appropriate link below based on the environment where your Apache farms are hosted.

In Kubernetes environments, we use the Telegraf Operator, which is packaged with our Kubernetes collection. You can learn more about it here.The diagram below illustrates how data is collected from Apache in Kubernetes environments. In the architecture shown below, there are four services that make up the metric collection pipeline: Telegraf, Prometheus, Fluentd and FluentBit.

Apache

The first service in the pipeline is Telegraf. Telegraf collects metrics from Apache. Note that we’re running Telegraf in each pod we want to collect metrics from as a sidecar deployment, meaning, Telegraf runs in the same pod as the containers it monitors.

Telegraf uses the Apache input plugin to obtain metrics. For simplicity, the diagram doesn’t show the input plugins. The injection of the Telegraf sidecar container is done by the Telegraf Operator. We also have Fluentbit that collects logs written to standard out and standard error and forwards them to FluentD, which in turn sends all the logs and metrics data to a Sumo Logic HTTP Source.

Prerequisites

Ensure that you are monitoring your Kubernetes clusters with the Telegraf operator. If you're not, see these instructions to do so.

Configure Metrics Collection​

Follow the steps below to collect metrics from a Kubernetes environment:

  1. Add configuration to enable metrics on Apache pods:
configuration: |-
ServerName localhost:8080
  <IfModule status_module>
    ExtendedStatus On
    <Location /server-status>
      Sethandler server-status
      order deny,allow
      allow from all
    </Location>
  </IfModule>
  1. Add annotations on your Apache pods:
annotations:
 telegraf.influxdata.com/class: sumologic-prometheus
  prometheus.io/scrape: "true"
  prometheus.io/port: "9273"
  telegraf.influxdata.com/inputs: |+
    [[inputs.apache]]
    < urls = ["http://localhost:8080/server-status?auto"] >
      [inputs.apache.tags]
        environment = "<prod_CHANGE_ME>"
        component = "webserver"
        webserver_system = "apache"
        webserver_farm = "<app1apacheeks_CHANGE_ME>"

  1. Enter in values for the parameters marked in brackets (< >) above.

    • telegraf.influxdata.com/inputs: This contains the required configuration for the Telegraf Apache Input plugin. Please refer to this doc for more information on configuring the Apache input plugin for Telegraf. Note: As telegraf will be run as a sidecar the host should always be localhost.
      • In the input plugins section:
        • urls: The URL to the Apache server
        • In the tags section [inputs.apache.tags]
        • environment: This is the deployment environment where the Apache webserver farm identified by the value of urls resides. For example: dev, prod or qa. While this value is optional we highly recommend setting it.
        • webserver_farm: Enter a name to uniquely identify this Apache Webserver farm. This Apache webserver farm name will be shown in the Sumo Logic dashboards.
    Do not modify the other values

    Modifying these values will cause the Sumo Logic apps to function incorrectly

    • telegraf.influxdata.com/class: sumologic-prometheus: Instructs the Telegraf operator what output to use.
    • prometheus.io/scrape: "true": Ensures our Prometheus will scrape the metrics.
    • prometheus.io/port: "9273": Tells prometheus what ports to scrape on.
    • telegraf.influxdata.com/inputs
      • In the tags section [inputs.apache.tags]
      • component: “webserver”: Used by Sumo Logic apps to identify application components.
      • webserver_system: “apache”: Identifies the webserver system.
    • For more information on other parameters and properties that you can configure in the Telegraf agent globally, see Configuring_Telegraf.
    • For more information on configuring the Apache input plugin for Telegraf, see this doc.

  2. Sumo Logic Kubernetes collection will automatically start collecting metrics from the pods having the configuration and annotations defined in the previous step.

  3. Verify metrics in Sumo Logic by running the following metrics query:

    webserver_farm=<your_apache_webserver_farmname> \
    component="webserver" and webserver_system="apache"

Configure Logs Collection​

This section explains the steps to collect Apache logs from a Kubernetes environment.

  1. Collect Apache logs written to standard output and standard error. If your Apache helm chart/pod is writing the logs to standard output or standard error then follow the steps listed below to collect the logs:

    1. On your Apache Pods, add the following pod labels:
    environment: "<prod_CHANGE_ME>"
    component: "webserver"
    webserver_system: "apache"
    webserver_farm: "<app1apacheeks_CHANGE_ME>"
    1. Enter in values for the parameters marked in brackets (< >) above.
      • environment: This is the deployment environment where the Apache webserver farm identified by the value of urls resides. For example: dev, prod or qa. While this value is optional we highly recommend setting it.
      • webserver_farm: Enter a name to identify this Apache webserver farm. This Apache webserver farm name will be shown in the Sumo Logic dashboards.
    Do not modify the other values

    Modifying these values will cause the Sumo Logic apps to function incorrectly

    • component: “webserver”: This value is used by Sumo Logic apps to identify application components.
    • webserver_system: “apache”: This value identifies the webserver system.
    • For all other parameters, please see this doc for more parameters that can be configured in the Telegraf agent globally.
    1. Make sure that the Apache pods are running and annotations are applied by using the command:
    kubectl describe pod <apache_pod_name>

    The Sumo Logic Kubernetes Collection process will automatically capture the logs from stdout/stderr and will send the logs to Sumo Logic. For more information on deploying the Sumo Logic-Kubernetes-Collection, please see this page.

  2. Add an FER to normalize the fields in Kubernetes environments. Labels created in Kubernetes environments automatically are prefixed with pod_labels. To normalize these for our app to work, we need to create a Field Extraction Rule if not already created for Web Server Application Components. To do so:

    1. Go to Manage Data > Logs > Field Extraction Rules.
    2. Click the + Add button on the top right of the table.
    3. The Add Field Extraction Rule form will appear. Enter the following options:
    • Rule Name: Enter the name as App Observability - Webserver
    • Applied At: Choose Ingest Time
    • Scope: Select Specific Data and enter the following keyword search expression:
      pod_labels_environment=* pod_labels_component=webserver pod_labels_webserver_system=* pod_labels_webserver_farm=*
    • Parse Expression: Enter the following parse expression:
      if (!isEmpty(pod_labels_environment),  pod_labels_environment, "") as environment
      | pod_labels_component as component
      | pod_labels_webserver_system as webserver_system
      | pod_labels_webserver_farm as webserver_farm
    1. Click Save to create the rule.
    2. Verify logs are flowing into Sumo Logic by running the following logs query:
      component=webserver webserver_system=apache \
      webserver_farm=<your_apache_webserver_farmname>

Installing Apache Monitors​

This section provides instructions for installing the Sumo Logic Monitors for Apache. These instructions assume you have already set up collection as described in the Collecting Logs and Metrics for Apache page.

Sumo Logic has provided out-of-the-box alerts available through Sumo Logic monitors to help you proactively monitor your Apache Web servers and farms. These monitors are built based on metrics and logs datasets and include pre-set thresholds based on industry best practices and recommendations.

For details about individual alerts, see this page. To install these alerts, you need to have the Manage Monitors role capability. There are limits to how many alerts can be enabled. For more information, see Monitors for details.

You can install monitors by importing a JSON file or using a Terraform script.

Method A: Importing a JSON file​

  1. Download the JSON file that describes the monitors.
  2. In the json file, replace $$apache_data_source with a custom source filter like webserver_farm=dev-apache for setting up alerts for a specific web server farm. If you want to configure this for all your web server farms you can find and replace $$apache_data_source with blank “”.
  3. Go to Manage Data > Alerts > Monitors.
  4. Click Add.
  5. Click Import and then copy paste the above JSON to import monitors. Name will be the folder name.

The monitors are disabled by default. Once you have installed the alerts using this method, navigate to the Apache folder under Monitors to configure them. See this document to enable monitors to send notifications to teams or connections. Please see the instructions detailed in Step 4 of this document.

Method B: Using a Terraform script​

  1. Generate an access key and access ID for a user that has the Manage Monitors role capability in Sumo Logic using these instructions. Please identify which deployment your Sumo Logic account is in, using this link.

  2. Download and install Terraform 0.13 or later.

  3. Download the Sumo Logic Terraform package for Apache alerts. The alerts package is available in the Sumo Logic github repository. You can either download it through the “git clone” command or as a zip file.

  4. Alert Configuration. After the package has been extracted, navigate to the package directory terraform-sumologic-sumo-logic-monitor/monitor_packages/apache/.

    1. Edit the apache.auto.tfvars file and add the Sumo Logic Access Key, Access Id and Deployment from Step 1.
      access_id   = "<SUMOLOGIC ACCESS ID>"
      access_key  = "<SUMOLOGIC ACCESS KEY>"
      environment = "<SUMOLOGIC DEPLOYMENT>"
    2. The Terraform script installs the alerts without any scope filters, if you would like to restrict the alerts to specific web server farm or environments, update the variable apache_data_source. Custom filter examples:
      • A specific web server farm webserver_farm=apache.prod.01
      • All web server farms in an environment environment=prod
      • For alerts applicable to all web server farms that start with apache-prod: webserver_farm=apache-prod*
      • For alerts applicable to a specific web server farm within a production environment: webserver_farm=apache-1 and environment=prod. This assumes you have set the optional environment tag while configuring collection.

    All monitors are disabled by default on installation, if you would like to enable all the monitors, set the parameter monitors_disabled to false in apache.auto.tfvars file.

    By default, the monitors are configured in a monitor folder called “Apache”, if you would like to change the name of the folder, update the monitor folder name in “folder” key at apache.auto.tfvars file.

  5. If you would like the alerts to send email or connection notifications, modify the file apache_notifications.auto.tfvars and populate connection_notifications and email_notifications as per below examples.

Email Notifications Example
connection_notifications = [
    {
      connection_type       = "PagerDuty",
      connection_id         = "<CONNECTION_ID>",
      payload_override      = "{\"service_key\": \"your_pagerduty_api_integration_key\",\"event_type\": \"trigger\",\"description\": \"Alert: Triggered {{TriggerType}} for Monitor {{Name}}\",\"client\": \"Sumo Logic\",\"client_url\": \"{{QueryUrl}}\"}",
      run_for_trigger_types = ["Critical", "ResolvedCritical"]
    },
    {
      connection_type       = "Webhook",
      connection_id         = "<CONNECTION_ID>",
      payload_override      = "",
      run_for_trigger_types = ["Critical", "ResolvedCritical"]
    }
  ]

Replace <CONNECTION_ID> with the connection id of the webhook connection. The webhook connection id can be retrieved by calling the Monitors API.

For overriding payload for different connection types, refer to this document.

Email Notifications Example
email_notifications = [
    {
      connection_type       = "Email",
      recipients            = ["abc@example.com"],
      subject               = "Monitor Alert: {{TriggerType}} on {{Name}}",
      time_zone             = "PST",
      message_body          = "Triggered {{TriggerType}} Alert on {{Name}}: {{QueryURL}}",
      run_for_trigger_types = ["Critical", "ResolvedCritical"]
    }
  ]
  1. Install the Alerts.
    1. Navigate to the package directory terraform-sumologic-sumo-logic-monitor/monitor_packages/apache/ and run terraform init. This will initialize Terraform and will download the required components.
    2. Run terraform plan to view the monitors which will be created/modified by Terraform.
    3. Run terraform apply.

Installing the Apache App​

Now that you have set up logs and metric collections for Apache, you can install the Sumo Logic App for Apache to use the pre-configured Searches and dashboards.

To install the app, do the following:

  1. Locate and install the app you need from the App Catalog. If you want to see a preview of the dashboards included with the app before installing, click Preview Dashboards.
  2. From the App Catalog, search for and select the app.
  3. Select the version of the service you're using and click Add to Library.
  4. To install the app, complete the following fields.
    • App Name. You can retain the existing name, or enter a name of your choice for the app.

    • Data Source.
 Choose Enter a Custom Data Filter, and enter a custom filter. Examples:
      • For all Apache web server farms: webserver_system=apache webserver_farm=*
      • For a specific web server farm: webserver_system=apache webserver_farm=apache.dev.01
      • Advanced. Select the Location in Library (the default is the Personal folder in the library), or click New Folder to add a new folder.
  5. Click Add to Library.

Version selection is applicable only to a few apps currently. For more information, see the Install the Apps from the Library.

Once an app is installed, it will appear in your Personal folder, or other folder that you specified. From here, you can share it with your organization.

Panels will start to fill automatically. It's important to note that each panel slowly fills with data matching the time range query and received since the panel was created. Results won't immediately be available, but with a bit of time, you'll see full graphs and maps.

Viewing Apache Dashboards​

This section provides descriptions of each of the app dashboards.

Filter with template variables

Template variables provide dynamic dashboards that can rescope data on the fly. As you apply variables to troubleshoot through your dashboard, you view dynamic changes to the data for a quicker resolution to the root cause. You can use template variables to drill down and examine the data on a granular level. For more information, see Filter with template variables.

Overview​

The Apache - Overview Dashboard provides an at-a-glance view of the activity and health of the Apache web server farms, and servers by monitoring uptime, requests, response, traffic, visitor geographic locations, and critical error messages.

Use this dashboard to:

  • Get an at-a-glance view of the state of all your Apache web servers
  • Identify the top URLs causing errors
  • Identify the top critical error messages
test

Error Log Analysis​

The Apache - Error Log Analysis dashboard provides a high-level view of error log levels, clients causing errors, critical error messages and trends.

Use this dashboard to:

  • Quickly identify critical errors affecting your Apache web servers
  • Analyze types and patterns of log messages in your Apache web servers
  • Identify clients causing the most errors
  • Monitor trends in error logs and identify outliers
test

The Apache - Trends dashboard provides trends around HTTP responses, server hits, visitor locations, traffic volume and distribution.

Use this dashboard to:

  • Monitor trends and identify outliers
test

Outlier Analysis​

The Apache - Outlier Analysis dashboard helps you quickly identify outliers for key Apache metrics such bytes served, number of visitors, server errors, and client errors.

Use this dashboard to:

  • Automatically detect outliers in the operations of your Apache web servers and take corrective actions if needed
test

Threat Analysis​

The Apache - Threat Intel dashboard provides an at-a-glance view of incoming threats to your Apache servers based on known malicious IP addresses.

Dashboard panels show threat counts, geographic locations, actors, threat severity, URLS accessed.

Use this dashboard to:

  • Identify threats from incoming traffic based on incoming client IP addresses and discover potential IOCs
test

Visitor Locations​

The Apache - Visitor Locations dashboard provides a high-level view of Apache visitor geographic locations both worldwide and in the United States.

Use this dashboard to:

  • Get insights into geographic locations of your user base
test

Visitor Access Types​

The Apache - Visitor Access Types dashboard provides insights into visitor platform types, browsers, device types, and operating systems.

Use this dashboard to:

  • Understand which platform and browsers are being used to access your applications
test

Visitor Traffic Insight​

The Apache - Visitor Traffic Insight dashboard provides summarized information on the top URLs, referrers, search terms, and media types served.

Use this dashboard to:

  • To understand content types of content that are frequently requested by users.
test

Web Server Operations​

The Apache - Web Server Operations Dashboard provides an at-a-glance view of the operations of your Apache web servers. Dashboard panels show information on bots, geographic locations, errors and URLs.

Use this dashboard to:

  • Get insights into client locations, bots and response codes
test

Request State Analysis​

The Apache - Request State Analysis dashboard shows trends around the state of incoming requests to your Apache web servers.

Use this dashboard to:

  • Monitor the state of requests being handled by worker threads over time and take remedial actions to optimize your web servers if needed
test

Server Resource Utilization​

The Apache - Server Resource Utilization dashboard shows the CPU resource utilization and load across threads and CPU of your Apache web servers.

Use this dashboard to:

  • Monitor CPU utilization and load on your Apache web servers
  • Monitor the number of worker and idle threads
test

Server Status​

The Apache - Server Status dashboard shows information related to the state of your Apache server and includes information such as requests and bytes served and latency. information on the number of requests served, time taken to serve the request, and bytes served.

Use this dashboard to:

  • Monitor server uptime
  • Monitor web server performance
test

Apache Alerts​

Sumo Logic provides out-of-the-box alerts available via Sumo Logic monitors. These alerts are built based on logs and metrics datasets and have preset thresholds based on industry best practices and recommendations.

Click to expand. Here are the alerts available for Apache.
Alert NameAlert DescriptionAlert ConditionRecover Condition
Apache - Critical Error MessagesThis alert fires when we detect critical error messages for a given Apache server.> 00
Apache - Access from Highly Malicious SourcesThis alert fires when an Apache is accessed from highly malicious IP addresses.> 00
Apache - High Client (HTTP 4xx) Error RateThis alert fires when there are too many HTTP requests (>5%) with a response status of 4xx. > 00
Apache - High Server (HTTP 5xx) Error RateThis alert fires when there are too many HTTP requests (>5%) with a response status of 5xx. >00
Apache - High CPU UtilizationThis alert fires when the average CPU utilization within a 5 minute interval for an Apache Webserver farm instance is high (>=85%). >=85 <85
Apache - Server RestartedThis alert fires when we detect low uptime (< = 10 minutes) for a given Apache server within a 5 minute interval. <=600 >600

Apache Metrics​

Click to expand. Here are the metrics available for Apache.
List of Apache Telegraf metrics
apache_BusyWorkers
apache_BytesPerReq
apache_BytesPerSec
apache_CPUChildrenSystem
apache_CPUChildrenUser
apache_CPULoad
apache_CPUSystem
apache_CPUUser
apache_DurationPerReq
apache_IdleWorkers
apache_Load1
apache_Load5
apache_Load15
apache_ParentServerConfigGeneration
apache_ParentServerMPMGeneration
apache_ReqPerSec
apache_ServerUptimeSeconds
apache_TotalAccesses
apache_TotalDuration
apache_TotalkBytes
apache_Uptime
apache_scboard_closing
apache_scboard_dnslookup
apache_scboard_finishing
apache_scboard_idle_cleanup
apache_scboard_keepalive
apache_scboard_logging
apache_scboard_open
apache_scboard_reading
apache_scboard_sending
apache_scboard_starting
apache_scboard_waiting
Sumo Logic YouTubeSumo Logic Twitter
Legal
Privacy Statement
Terms of Use

Copyright © 2022 by Sumo Logic, Inc.