Skip to main content

Sumo Logic App for Memcached

Thumbnail icon

The Memcached app is a unified logs and metrics app that helps you monitor the availability, performance, health, and resource utilization of your Memcached clusters. Preconfigured dashboards provide insight into uptime, operational metrics, cache performance, resource utilization, errors, warnings, and commands executed.

The Sumo Logic App for Memcached is tested for Version: 1.4.15.

Sample Log Messages​

{
    "timestamp":1626248704386,
    "log":"28: going from conn_closing to conn_closed",
    "stream":"stderr",
    "time":"2021-07-14T07:45:01.839243101Z"
}

Collecting Logs and Metrics for Memcached​

Configuring log and metric collection for the Memcached App includes the following tasks.

Step 1: Configure Fields in Sumo Logic​

Create the following Fields in Sumo Logic before configuring the collection. This ensures that your logs and metrics are tagged with relevant metadata required by the app dashboards. For information on setting up fields, see Sumo Logic Fields.

If you're using Memcached in a Kubernetes environment, create the fields:

  • pod_labels_component
  • pod_labels_environment
  • pod_labels_db_system
  • pod_labels_db_cluster

Step 2: Configure Logs and Metrics Collection for Memcached​

In Kubernetes environments, we use the Telegraf Operator, which is packaged with our Kubernetes collection. You can learn more about it here.

The diagram below illustrates how data is collected from Memcached in a Kubernetes environment. In the architecture shown below, there are four services that make up the metric collection pipeline: Telegraf, Prometheus, Fluentd, and FluentBit.memcached

The first service in the pipeline is Telegraf. Telegraf collects metrics from Memcached. Note that we’re running Telegraf in each pod we want to collect metrics from as a sidecar deployment, for example, Telegraf runs in the same pod as the containers it monitors. Telegraf uses the Memcached input plugin to obtain metrics. (For simplicity, the diagram doesn’t show the input plugins.) The injection of the Telegraf sidecar container is done by the Telegraf Operator. We also have Fluentbit that collects logs written to standard out and forwards them to FluentD, which in turn sends all the logs and metrics data to a Sumo Logic HTTP Source.

Prerequisites

It’s assumed that you are using the latest helm chart version. If not, upgrade using the instructions here.

Configure Metrics Collection​

Follow the steps listed below to collect Memcached metrics from a Kubernetes environment.

  1. Set up Kubernetes Collection with the Telegraf Operator.
  2. On your Memcached Pods, add the following annotations:
 annotations:
    telegraf.influxdata.com/class: sumologic-prometheus
    prometheus.io/scrape: "true"
    prometheus.io/port: "9273"
    telegraf.influxdata.com/inputs: |+
  servers = ["localhost:11211"]
  [inputs.memcached.tags]
  environment="dev_ENV_TO_BE_CHANGED"
  component="database"
  db_system="memcached"
  db_cluster="memcached_on_k8s_ENV_TO_BE_CHANGED"
  db_cluster_address = "ENV_TO_BE_CHANGED"
  db_cluster_port = "ENV_TO_BE_CHANGED"
  1. Enter in values for the following parameters (marked ENV_TO_BE_CHANGED above):
    • telegraf.influxdata.com/inputs - This contains the required configuration for the Telegraf Memcached Input plugin. Please refer to this doc for more information on configuring the Memcached input plugin for Telegraf. Note: As telegraf will be run as a sidecar the host should always be localhost.
    • In the input plugins section ([[inputs.memcached]]):
      • servers - An array of addresses to gather stats about. Specify an IP on the hostname. Please see this doc for more information on additional parameters for configuring the Memcached input plugin for Telegraf.
    • In the tags section ([inputs.memcached.tags]):
        * `environment` - This is the deployment environment where the Memcached cluster identified by the value of servers resides. For example: dev, prod or qa. While this value is optional we highly recommend setting it.
        * `db_cluster` - Enter a name to identify this Memcached cluster. This cluster name will be shown in the Sumo Logic dashboards.
        * `db_cluster_address` - Enter the cluster hostname or ip address that is used by the application to connect to the database. It could also be the load balancer or proxy endpoint.
        * `db_cluster_port` - Enter the database port. If not provided, a default port will be used.
      note

      db_cluster_address and db_cluster_port should reflect the exact configuration of DB client configuration in your application, especially if you instrument it with OT tracing. The values of these fields should match exactly the connection string used by the database client (reported as values for net.peer.name and net.peer.port metadata fields).

For example, if your application uses “memcached-prod.sumologic.com:3306” as the connection string, the field values should be set as follows: db_cluster_address=memcached-prod.sumologic.com db_cluster_port=3306.

If your application connects directly to a given Memcached node, rather than the whole cluster, use the application connection string to override the value of the “host” field in the Telegraf configuration: host=memcached-prod.sumologic.com

Pivoting to Tracing data from Entity Inspector is possible only for “Memcached address” Entities. :::

  • Here’s an explanation for additional values set by this configuration that we request you do not modify as they will cause the Sumo Logic apps to not function correctly.
    • telegraf.influxdata.com/class: sumologic-prometheus - This instructs the Telegraf operator what output to use. This should not be changed.
    • prometheus.io/scrape: "true" - This ensures our Prometheus will scrape the metrics.
    • prometheus.io/port: "9273" - This tells prometheus what ports to scrape on. This should not be changed.
    • telegraf.influxdata.com/inputs
    • In the tags section, [inputs.memcached.tags]:
      • component: “database” - This value is used by Sumo Logic apps to identify application components.
      • db_system: “memcached” - This value identifies the database system.
  • For all other parameters, see this doc for more parameters that can be configured in the Telegraf agent globally.
  1. Sumo Logic Kubernetes collection will automatically start collecting metrics from the pods having the labels and annotations defined in the previous step.
  2. Verify metrics in Sumo Logic.

Configure Logs Collection​

This section explains the steps to collect Memcached logs from a Kubernetes environment.

  1. Add labels on your Memcached pods to capture logs from standard output (recommended method).

    1. Apply the following labels to the Memcached pods:
      environment: "prod_ENV_TO_BE_CHANGED"
      component: "database"
      db_system: "memcached"
      db_cluster: "memcached_on_k8s_ENV_TO_BE_CHANGED"
      db_cluster_address = "ENV_TO_BE_CHANGED"
      db_cluster_port = "ENV_TO_BE_CHANGED"
    2. Enter in values for the following parameters:

    • environment - This is the deployment environment where the Memcached cluster identified by the value of servers resides. For example dev, prod, or QA. While this value is optional we highly recommend setting it.

    • db_cluster - Enter a name to identify this Memcached cluster. This cluster name will be shown in the Sumo Logic dashboards.

    • db_cluster_address - Enter the cluster hostname or ip address that is used by the application to connect to the database. It could also be the load balancer or proxy endpoint.

    • db_cluster_port - Enter the database port. If not provided, a default port will be used.

      note

      db_cluster_address and db_cluster_port should reflect the exact configuration of DB client configuration in your application, especially if you instrument it with OT tracing. The values of these fields should match exactly the connection string used by the database client (reported as values for net.peer.name and net.peer.port metadata fields).

      For example, if your application uses “memcached-prod.sumologic.com:3306” as the connection string, the field values should be set as follows: db_cluster_address=memcached-prod.sumologic.com db_cluster_port=3306.

      If your application connects directly to a given Memcached node, rather than the whole cluster, use the application connection string to override the value of the “host” field in the Telegraf configuration: host=memcached-prod.sumologic.com.

      Pivoting to Tracing data from Entity Inspector is possible only for “Memcached address” Entities.

    • Here’s an explanation for additional values set by this configuration that we request you do not modify as they will cause the Sumo Logic apps to not function correctly.

      • component: “database” - This value is used by Sumo Logic apps to identify application components.
      • db_system: “memcached” - This value identifies the database system.
      • See this doc for more parameters that can be configured in the Telegraf agent globally.
    1. The Sumologic-Kubernetes-Collection will automatically capture the logs from stdout and will send the logs to Sumologic. For more information on deploying Sumologic-Kubernetes-Collection, visit here.
    2. Verify logs in Sumo Logic.

  2. Collecting Memcached Logs from a Log File (Optional). If your Memcached chart/pod is writing its logs to log files, you can use a sidecar to send log files to standard out. To do this:

    1. Install the Sumo Logic tailing sidecar operator.
    2. Add the following annotation in addition to the existing annotations.
      annotations:
        tailing-sidecar: sidecarconfig;<mount>:<path_of_Memcached_log_file>/<Memcached_log_file_name>
      Example:
      annotations:
          tailing-sidecar: sidecarconfig;data:/var/bitnami/memcached/logs/memcached.log
    3. Make sure that the Memcached pods are running and annotations are applied by using the command:
      kubectl describe pod <Memcached_pod_name>
    4. Sumo Logic Kubernetes collection will automatically start collecting logs from the pods having the annotations defined above.

  3. Add a FER to normalize the fields in Kubernetes environments. This step is not needed if one is using application components solution terraform script. Labels created in Kubernetes environments automatically are prefixed with pod_labels. To normalize these for our app to work, we need to create a Field Extraction Rule if not already created for Proxy Application Components. To do so:

    1. Go to Manage Data > Logs > Field Extraction Rules.
    2. Click the + Add button on the top right of the table.
    3. The Add Field Extraction Rule form will appear:
    4. Enter the following options:
    • Rule Name. Enter the name as App Observability - Database.
    • Applied At. Choose Ingest Time
    • Scope. Select Specific Data
    • Scope: Enter the following keyword search expression:
      pod_labels_environment=* pod_labels_component=database \
      pod_labels_db_system=* pod_labels_db_cluster=*
    • Parse Expression. Enter the following parse expression:
      if (!isEmpty(pod_labels_environment), pod_labels_environment, "") as environment
      | pod_labels_component as component
      | pod_labels_db_system as db_system
      | if (!isEmpty(pod_labels_db_cluster), pod_labels_db_cluster, null) as db_cluster
    1. Click Save to create the rule.

Installing Memcached Monitors​

Sumo Logic has provided pre-packaged alerts available through Sumo Logic monitors to help you proactively determine if a Memcached cluster is available and performing as expected. These monitors are based on metric and log data and include pre-set thresholds that reflect industry best practices and recommendations. For more information about individual alerts, see Memcached Alerts.

To install these monitors, you must have the Manage Monitors role capability.

You can install monitors by importing a JSON file or using a Terraform script.

There are limits to how many alerts can be enabled. For more information, see Monitors for details.

Method A: Importing a JSON file​

  1. Download the JSON file that describes the monitors.
  2. The JSON contains alerts from Sumo Logic searches that do not have any scope filters and, therefore, will apply to all Memcached clusters, the data for which has been collected via the instructions in the previous sections. However, if you would like to restrict these alerts to specific clusters or environments, update the JSON file by replacing the text db_cluster=* with <Your Custom Filter>. Custom filter examples:
    • For alerts applicable only to a specific cluster, your custom filter would be: db_cluster=dev-memcached-01
    • For alerts applicable to all clusters that start with memcached-prod, your custom filter would be: db_cluster=memcachedt-prod*
    • For alerts applicable to specific clusters within a production environment, your custom filter would be: db_cluster=dev-memcached-01 AND environment=prod. This assumes you have set the optional environment tag while configuring collection.
  3. Go to Manage Data > Alerts > Monitors.
  4. Click Add.
  5. Click Import.
  6. On the Import Content popup, enter Memcached in the Name field, paste the JSON into the popup, and click Import.
  7. The monitors are created in "Memcached" folder. The monitors are disabled by default. See the Monitors topic for information about enabling monitors and configuring notifications or connections.

Method B: Using a Terraform script​

  1. Generate a Sumo Logic access key and ID for a user that has the Manage Monitors role capability. For instructions, see Access Keys.

  2. Download Terraform 0.13 or later, and install.

  3. Download the Sumo Logic Terraform package for Memcached monitors. The alerts package is available in the Sumo Logic GitHub repository. You can either download it using the git clone command or as a zip file.

  4. Alert Configuration. After extracting the package, navigate to the terraform-sumologic-sumo-logic-monitor/monitor_packages/Memcached/ directory.

  5. Edit the Memcached.auto.tfvars file and add the Sumo Logic Access Key and Access ID from Step 1 and your Sumo Logic deployment. If you're not sure of your deployment, see Sumo Logic Endpoints and Firewall Security.

    access_id   = "<SUMOLOGIC ACCESS ID>"
    access_key  = "<SUMOLOGIC ACCESS KEY>"
    environment = "<SUMOLOGIC DEPLOYMENT>"

  6. The Terraform script installs the alerts without any scope filters. If you would like to restrict the alerts to specific clusters or environments, update the memcached_data_source variable. For example:

    • To configure alerts for a specific clusters, set memcached_data_source to something like: db_cluster=memcached.prod.01
    • To configure alerts for All clusters in an environment, set memcached_data_source to something like: environment=prod
    • To configure alerts for Multiple clusters using a wildcard, set memcached_data_source to something like: db_cluster=memcached-prod*
    • To configure alerts for specific clusters within a specific environment, set memcached_data_source to something like: db_cluster=memcached-1 and environment=prod. This assumes you have configured and applied Fields as described in Step 1: Configure Fields of the Sumo Logic of the Collect Logs and Metrics for Memcached topic.

    All monitors are disabled by default on installation. To enable all of the monitors, set the monitors_disabled parameter to false.

    By default, the monitors will be located in a "Memcached" folder on the Monitors page. To change the name of the folder, update the monitor folder name in the folder variable in the Memcached.auto.tfvars file.

  7. If you want the alerts to send email or connection notifications, edit the Memcached_notifications.auto.tfvars file to populate the connection_notifications and email_notifications sections. Examples are provided below.

    • In the variable definition below, replace <CONNECTION_ID> with the connection ID of the Webhook connection. You can obtain the Webhook connection ID by calling the Monitors API.
Pagerduty connection example
connection_notifications = [
    {
      connection_type       = "PagerDuty",
      connection_id         = "<CONNECTION_ID>",
      payload_override      = "{\"service_key\": \"your_pagerduty_api_integration_key\",\"event_type\": \"trigger\",\"description\": \"Alert: Triggered {{TriggerType}} for Monitor {{Name}}\",\"client\": \"Sumo Logic\",\"client_url\": \"{{QueryUrl}}\"}",
      run_for_trigger_types = ["Critical", "ResolvedCritical"]
    },
    {
      connection_type       = "Webhook",
      connection_id         = "<CONNECTION_ID>",
      payload_override      = "",
      run_for_trigger_types = ["Critical", "ResolvedCritical"]
    }
  ]

For information about overriding the payload for different connection types, see Set Up Webhook Connections.

Email notifications example
email_notifications = [
    {
      connection_type       = "Email",
      recipients            = ["abc@example.com"],
      subject               = "Monitor Alert: {{TriggerType}} on {{Name}}",
      time_zone             = "PST",
      message_body          = "Triggered {{TriggerType}} Alert on {{Name}}: {{QueryURL}}",
      run_for_trigger_types = ["Critical", "ResolvedCritical"]
    }
  ]
  1. Install Monitors.
    1. Navigate to the terraform-sumologic-sumo-logic-monitor/monitor_packages/Memcached/ directory and run terraform init. This will initialize Terraform and download the required components.
    2. Run terraform plan to view the monitors that Terraform will create or modify.
    3. Run terraform apply.

Installing the Memcached App​

This section demonstrates how to install the Memcached App.

Locate and install the app you need from the App Catalog. If you want to see a preview of the dashboards included with the app before installing, click Preview Dashboards.

  1. From the App Catalog, search for and select the app.
  2. Select the version of the service you're using and click Add to Library. Version selection applies only to a few apps currently. For more information, see the Install the Apps from the Library.
  3. To install the app, complete the following fields.
    • App Name. You can retain the existing name or enter a name of your choice.

    • Advanced. Select the Location in Library (the default is the Personal folder in the library), or click New Folder to add a new folder.
  4. Click Add to Library.

Once an app is installed, it will appear in your Personal folder or another folder that you specified. From here, you can share it with your organization.

Panels will start to fill automatically. It's important to note that each panel slowly fills with data matching the time range query and received since the panel was created. Results won't immediately be available, but you'll see full graphs and maps in a bit of time.

Viewing Memcached Dashboards​

Filter with template variables

Template variables provide dynamic dashboards that can rescope data on the fly. As you apply variables to troubleshoot through your dashboard, you view dynamic changes to the data for a quicker resolution to the root cause. You can use template variables to drill down and examine the data on a granular level. For more information, see Filter with template variables.

Overview​

The Memcached - Overview dashboard provides an at-a-glance view of the Memcached server status, error logs, and database metrics.

Memcached dashboards

Operations​

The Memcached - Operations Dashboard provides detailed analysis on connections, thread requested, network bytes, hash expansion size, table size.

Memcached dashboards

Command Stats​

The Memcached - Command Stats dashboard provides detailed insights into the number of commands being performed.

Memcached dashboards

Cache Information​

The Memcached - Cache Information dashboard provides insight into cache states, cache hit, and miss rate over time.

Memcached dashboards

Logs​

The Memcached - Logs dashboard helps you quickly analyze your Memcached error logs, commands executed, and objects stored.

Memcached dashboards

Memcached Alerts​

Sumo Logic has provided out-of-the-box alerts available via Sumo Logic monitors to help you quickly determine if the Memcached database cluster is available and performing as expected.

Alert NameAlert DescriptionTrigger Type (Critical / Warning)Alert ConditionRecover Condition
Memcached - Commands ErrorThis alert fires when we detect command errors.Critical > 0 <= 0
Memcached - Authentication ErrorThis alert fires when we detect authentication errors continuously for 5 minsWarning >0 <= 0
Memcached - Connection YieldsThis alert fires when we detect yielded connections continuously for 5 minsWarning >5 <= 5
Memcached - High Memory UsageThis alert fires when the memory usage is more than 80%.Warning >80 <= 80
Memcached - Listen DisabledThis alert fires when new queued connections per minute > 5Warning >5 <=5
Memcached - Cache Hit RatioThe hit rate is one of the most important indicators of Memcached performance. A high hit rate means faster responses to your users. If the hit rate is falling, you need quick visibility into why. This alert gets fired low cache hit ratio is less than 50%Critical <=0.5 >0.5
Memcached - Current ConnectionsThis alert gets fired when number of connected clients are 0. If current connections are none then something is wrong.Critical <=0 >0
Memcached - UptimeThis alert gets fires when uptime is < 180. You can use this to detect respawns.Critical <=180 >180
Sumo Logic YouTubeSumo Logic Twitter
Legal
Privacy Statement
Terms of Use

Copyright © 2022 by Sumo Logic, Inc.