Skip to main content

Sumo Logic App for Couchbase

Thumbnail icon

Couchbase, a modern database for enterprise applications, is a distributed document database with a powerful search engine and in-built operational and analytical capabilities. It brings the power of NoSQL to the edge and provides fast, efficient bidirectional synchronization of data between the edge and the cloud.

The Sumo Logic app for Couchbase helps you monitor activity in Couchbase. The pre-configured dashboards provide insight into the Health of the Cluster, the Status of the Buckets, I/O of Reading/Writing, Errors, the Events of Couchbase Servers that help you understand your clusters.

This App has been tested with the following Couchbase with Telegraf versions:

  • Kubernetes: Couchbase version: 7.0.2 - enterprise with Telegraf version 1.21.1
  • Non-Kubernetes: Couchbase version: 7.0.2 - enterprise with Telegraf version 1.21.1
note

Telegraf 1.14 default of Kubernetes Collection will not work.

Collecting Logs and Metrics for the Couchbase App​

This section provides instructions for configuring log and metric collection for the Sumo Logic App for Couchbase.

Step 1: Configure Fields in Sumo Logic​

Create the following Fields in Sumo Logic prior to configuring the collection. This ensures that your logs and metrics are tagged with relevant metadata, which is required by the app dashboards. For information on setting up fields, see Sumo Logic Fields.

If you're using Couchbase in a Kubernetes environment, create the fields:

  • pod_labels_component
  • pod_labels_environment
  • pod_labels_db_system
  • pod_labels_db_cluster

Step 2: Configure Collection for Couchbase​

Sumo Logic supports the collection of logs and metrics data from Couchbase in both Kubernetes and non-Kubernetes environments. Click on the appropriate tab below based on the environment where your Couchbase clusters are hosted.

The following diagram illustrates how data is collected from Couchbase in Kubernetes environments. There are four services that make up the metric collection pipeline: Telegraf, Prometheus, Fluentd, and FluentBit.couchbase1

The first service in the pipeline is Telegraf. Telegraf collects metrics from Couchbase. Note that we’re running Telegraf in each pod we want to collect metrics from as a sidecar deployment that is Telegraf runs in the same pod as the containers it monitors. Telegraf uses the Couchbase input plugin to obtain metrics. For simplicity, the diagram doesn’t show the input plugins. The injection of the Telegraf sidecar container is done by the Telegraf Operator. We also have Fluentbit that collects logs written to standard out and forwards them to FluentD, which in turn sends all the logs and metrics data to a Sumo Logic HTTP Source.

Prerequisites

It’s assumed that you are using the latest helm chart version if not, upgrade using the instructions here. When you upgrade the helm chart, you must upgrade telegraf version to 1.21.1 by adding the statement below in the upgrade command helm chart:

--set telegraf-operator.image.sidecarImage=telegraf:1.21.1

Configure Metrics Collection​

To collect Couchbase metrics from a Kubernetes environment, we use the Telegraf Operator, which is packaged with our Kubernetes collection. You can learn more about this here.

  1. Set up Kubernetes Collection with the Telegraf Operator.
  2. On your Couchbase Pods, add the following annotations:
annotations:
   telegraf.influxdata.com/class: sumologic-prometheus
   prometheus.io/scrape: "true"
   prometheus.io/port: "9273"
   telegraf.influxdata.com/inputs: |+
[[inputs.couchbase]]
  servers = ["http://<USER_TO_BE_CHANGED>:<PASS_TO_BE_CHANGED>@A@localhost:8091"]
  bucket_stats_included = ["*"]
[inputs.couchbase.tags]
  db_cluster="ENV_TO_BE_CHANGED"--If you haven’t defined a cluster in Couchbase, enter `default`
  component="database"
  environment="ENV_TO_BE_CHANGED"
  db_system="couchbase"
  db_cluster_address = "ENV_TO_BE_CHANGED"
  db_cluster_port = "ENV_TO_BE_CHANGED"
  1. Enter in values for the following parameters (marked ENV_TO_BE_CHANGED above):

  • telegraf.influxdata.com/inputs - This contains the required configuration for the Telegraf Couchbase Input plugin. Refer to this doc for more information on configuring the Couchbase input plugin for Telegraf. Note: As telegraf will be run as a sidecar, the host should always be localhost.

  • In the Input plugins section ([[inputs.couchbase]]):

    • servers: This is the endpoint of the management portal of couchbase server. For detail, see this doc .

  • In the tags section ([inputs.couchbase.tags]):

    • environment - This is the deployment environment where the Couchbase cluster identified by the value of servers resides. For example: dev, prod or qa. While this value is optional we highly recommend setting it.

    • db_cluster - Enter a name to identify this Couchbase cluster. This cluster name will be shown in the Sumo Logic dashboards.

    • db_cluster_address - Enter the cluster hostname or ip address that is used by the application to connect to the database. It could also be the load balancer or proxy endpoint.

    • db_cluster_port - Enter the database port. If not provided, a default port will be used.

      note

      db_cluster_address and db_cluster_port should reflect exact configuration of DB client configuration in your application, especially if you instrument it with OT tracing. The values of these fields should match exactly the connection string used by the database client (reported as values for net.peer.name and net.peer.port metadata fields). For example if your application uses “couchbase-prod.sumologic.com:3306” as the connection string, the field values should be set as follows: db_cluster_address=couchbase-prod.sumologic.com db_cluster_port=3306

      If your application connects directly to a given Couchbase node, rather than the whole cluster, use the application connection string to override the value of the “host” field in the Telegraf configuration: host=couchbase-prod.sumologic.com

      Pivoting to Tracing data from Entity Inspector is possible only for “Couchbase address” Entities.

  • Do not modify the following values as it will cause the Sumo Logic apps to not function correctly.

    • telegraf.influxdata.com/class: sumologic-prometheus - This instructs the Telegraf operator what output to use. This should not be changed.
    • prometheus.io/scrape: "true" - This ensures our Prometheus will scrape the metrics.
    • prometheus.io/port: "9273" - This tells prometheus what ports to scrape on. This should not be changed.
    • telegraf.influxdata.com/inputs -- In the tags section ([inputs.couchbase.tags]):
      • component: “database” - This value is used by Sumo Logic apps to identify application components.
      • db_system: “couchbase” - This value identifies the database system.

  • See this doc for more parameters that can be configured in the Telegraf agent globally.

  1. Sumo Logic Kubernetes collection will automatically start collecting metrics from the pods having the labels and annotations defined in the previous step.
  2. Verify metrics in Sumo Logic.

Configure Logs Collection​

This section explains the steps to collect Couchbase logs from a Kubernetes environment.

  1. Add labels on your Couchbase pods to capture logs from standard output on Kubernetes (recommended).

    1. Apply following labels to the Couchbase pod:
      environment="prod_CHANGEME"
      component="database"
      db_system="couchbase"
      db_cluster="<cluster_CHANGEME>"
      db_cluster_address: <your cluster’s hostname or ip address or service endpoint>
      db_cluster_port: <database port>
    2. Enter in values for the following parameters (marked CHANGE_ME above):

    • environment - This is the deployment environment where the Couchbase cluster identified by the value of servers resides. For example:- dev, prod, or QA. While this value is optional we highly recommend setting it.

    • db_cluster - Enter a name to identify this Couchbase cluster. This cluster name will be shown in the Sumo Logic dashboards. If you haven’t defined a cluster in Couchbase, then enter default for db_cluster.

    • db_cluster_address - Enter the cluster hostname or ip address that is used by the application to connect to the database. It could also be the load balancer or proxy endpoint.

    • db_cluster_port - Enter the database port. If not provided, a default port will be used.

      note

      db_cluster_address and db_cluster_port should reflect exact configuration of DB client configuration in your application, especially if you instrument it with OT tracing. The values of these fields should match exactly the connection string used by the database client (reported as values for net.peer.name and net.peer.port metadata fields). For example if your application uses “couchbase-prod.sumologic.com:3306” as the connection string, the field values should be set as follows: db_cluster_address=couchbase-prod.sumologic.com db_cluster_port=3306.

      If your application connects directly to a given Couchbase node, rather than the whole cluster, use the application connection string to override the value of the “host” field in the Telegraf configuration: host=couchbase-prod.sumologic.com.

      Pivoting to Tracing data from Entity Inspector is possible only for “Couchbase address” Entities.

    • Do not modify the following values as it will cause the Sumo Logic apps to not function correctly.

    • component: “database” - This value is used by Sumo Logic apps to identify application components.

    • db_system: “couchbase” - This value identifies the database system. See this doc for more parameters that can be configured in the Telegraf agent globally.

    1. The Sumologic-Kubernetes-Collection will automatically capture the logs from stdout and will send the logs to Sumologic. For more information on deploying Sumologic-Kubernetes-Collection, visit here.
    2. Verify logs in Sumo Logic.

  2. Collecting Couchbase Logs from a Log File on Kubernetes (optional).

    1. Determine the location of the Couchbase log file on Kubernetes. This can be determined from the config file /opt/couchbase/etc/couchbase/static_config squid.conf for your Couchbase cluster along with the mounts on the Couchbase pods.
    2. Install the Sumo Logic tailing sidecar operator.
    3. Add the following annotation in addition to the existing annotations.
      annotations:
        tailing-sidecar: sidecarconfig;<mount>:<path_of_Couchbase_log_file>/<Couchbase_log_file_name>
      Example:
      annotations:
        tailing-sidecar: sidecarconfig;data:/opt/couchbase/var/lib/couchbase/logs/audit.log
    4. Make sure that the Couchbase pods are running and annotations are applied by using the command:
      kubectl describe pod <Couchbase_pod_name>
    5. Sumo Logic Kubernetes collection will automatically start collecting logs from the pods having the annotations defined above.
    6. Verify logs in Sumo Logic.

  3. Add a FER to normalize the fields in Kubernetes environments. This step is not needed if using application components solution terraform script. Labels created in Kubernetes environments automatically are prefixed with pod_labels. To normalize these for our app to work, we need to create a Field Extraction Rule if not already created for Proxy Application Components. To do so:

    1. Go to Manage Data > Logs > Field Extraction Rules.
    2. Click the + Add button on the top right of the table.
    3. The Add Field Extraction Rule form will appear:
    4. Enter the following options:
      • Rule Name. Enter the name as App Observability - Proxy.
      • Applied At. Choose Ingest Time
      • Scope. Select Specific Data
      • Scope: Enter the following keyword search expression: 
      pod_labels_environment=* pod_labels_component=database \
      pod_labels_db_cluster=* pod_labels_db_system=*
      • Parse Expression. Enter the following parse expression:
      if (!isEmpty(pod_labels_environment), pod_labels_environment, "") as environment
      | pod_labels_component as component
      | pod_labels_db_system as db_system
      | pod_labels_db_cluster as db_cluster
      | if (!isEmpty(pod_labels_db_cluster), pod_labels_db_cluster, null) as db_cluster
    5. Click Save to create the rule.

Installing Couchbase Monitors​

The next sections provides instructions for installing the Couchbase App, as well as examples of each of the App dashboards. These instructions assume you have already set up the collection as described in the Collecting Logs and Metrics for the Couchbase App section.

Pre-Packaged Alerts​

Sumo Logic has provided out-of-the-box alerts available through Sumo Logic monitors to help you monitor your Couchbase clusters. These alerts are built based on metrics and logs datasets and include preset thresholds based on industry best practices and recommendations.

For details on the individual alerts, see this page.

  • To install these alerts, you need to have the Manage Monitors role capability.
  • Alerts can be installed by either importing a JSON file or a Terraform script.

There are limits to how many alerts can be enabled - see the Alerts FAQ for details.

Method A: Importing a JSON file​

  1. Download the JSON file that describes the monitors.
  2. The JSON contains the alerts that are based on Sumo Logic searches that do not have any scope filters and therefore will be applicable to all Couchbase clusters, the data for which has been collected via the instructions in the previous sections. However, if you would like to restrict these alerts to specific clusters or environments, update the JSON file by replacing the text db_system=couchbase with <Your Custom Filter>. Custom filter examples:
    1. For alerts applicable only to a specific cluster, your custom filter would be, 'db_cluster=couchbase-standalone.01'.
    2. For alerts applicable to all cluster that start with couchbase-standalone, your custom filter would be,db_cluster=couchbase-standalone*.
    3. For alerts applicable to a specific cluster within a production environment, your custom filter would be,db_cluster=couchbase-1 and environment=standalone (This assumes you have set the optional environment tag while configuring collection).
  3. Go to Manage Data > Alerts > Monitors.
  4. Click Add:
  5. Click Import and then copy-paste the above JSON to import monitors. 6.The monitors are disabled by default. Once you have installed the alerts using this method, navigate to the Couchbase folder under Monitors to configure them. See this document to enable monitors to send notifications to teams or connections. See the instructions detailed in Step 4 of this document.

Method B: Using a Terraform script method​

  1. Generate a Sumo Logic access key and ID Generate an access key and access ID for a user that has the Manage Monitors role capability in Sumo Logic using these instructions. Identify which deployment your Sumo Logic account is in using this link.
  2. Download and install Terraform 0.13 or later.
  3. Download the Sumo Logic Terraform package for Couchbase alerts. The alerts package is available in the Sumo Logic GitHub repository. You can either download it through the “git clone” command or as a zip file.
  4. Alert Configuration After the package has been extracted, navigate to the package directory terraform-sumologic-sumo-logic-monitor/monitor_packages/Couchbase/
  5. Edit the couchbase.auto.tfvars** file and add the Sumo Logic Access Key, Access Id, and Deployment from Step 1.
access_id   = "<SUMOLOGIC ACCESS ID>"
access_key  = "<SUMOLOGIC ACCESS KEY>"
environment = "<SUMOLOGIC DEPLOYMENT>"

The Terraform script installs the alerts without any scope filters, if you would like to restrict the alerts to specific farms or environments, update the variable couchbase_data_source. Custom filter examples:

  • A specific cluster db_cluster=couchbase.standalone.01.
  • All clusters in an environment environment=standalone.
  • For alerts applicable to all clusters that start with couchbase-standalone, your custom filter would be: db_cluster=couchbase-standalone.
  • For alerts applicable to a specific cluster within a production environment, your custom filter would be: db_system=couchbase and environment=standalone. This assumes you have set the optional environment tag while configuring collection.

All monitors are disabled by default on installation. If you would like to enable all the monitors, set the parameter monitors_disabled to false in this file.

By default, the monitors are configured in a monitor folder called “Couchbase”. If you would like to change the name of the folder, update the monitor folder name in “folder” key at couchbase.auto.tfvars file.

If you would like the alerts to send email or connection notifications, configure these in the file couchbase_notifications.auto.tfvars. For configuration examples, refer to the next section.

  1. Email and Connection Notification Configuration Examples. Modify the file couchbase_notifications.auto.tfvars and populate connection_notifications and email_notifications as per below examples.
Pagerduty Connection Example
connection_notifications = [
    {
      connection_type       = "PagerDuty",
      connection_id         = "<CONNECTION_ID>",
      payload_override      = "{\"service_key\": \"your_pagerduty_api_integration_key\",\"event_type\": \"trigger\",\"description\": \"Alert: Triggered {{TriggerType}} for Monitor {{Name}}\",\"client\": \"Sumo Logic\",\"client_url\": \"{{QueryUrl}}\"}",
      run_for_trigger_types = ["Critical", "ResolvedCritical"]
    },
    {
      connection_type       = "Webhook",
      connection_id         = "<CONNECTION_ID>",
      payload_override      = "",
      run_for_trigger_types = ["Critical", "ResolvedCritical"]
    }
  ]

Replace <CONNECTION_ID> with the connection id of the webhook connection. The webhook connection id can be retrieved by calling the Monitors API.

For overriding payload for different connection types, refer to this document.

Email Notifications Example
email_notifications = [
    {
      connection_type       = "Email",
      recipients            = ["abc@example.com"],
      subject               = "Monitor Alert: {{TriggerType}} on {{Name}}",
      time_zone             = "PST",
      message_body          = "Triggered {{TriggerType}} Alert on {{Name}}: {{QueryURL}}",
      run_for_trigger_types = ["Critical", "ResolvedCritical"]
    }
  ]
  1. Install the Alerts: navigate to the package directory terraform-sumologic-sumo-logic-monitor/monitor_packages/Couchbase/ and run terraform init. This will initialize Terraform and will download the required components.
  2. Run terraform plan to view the monitors which will be created/modified by Terraform.
  3. Run terraform apply.
  4. Post Installation. If you haven’t enabled alerts and/or configured notifications through the Terraform procedure outlined above, we highly recommend enabling alerts of interest and configuring each enabled alert to send notifications to other users or services. This is detailed in Step 4 of this document. There are limits to how many alerts can be enabled - see the Alerts FAQ.

Installing the Couchbase App​

This section demonstrates how to install the Couchbase App.

Locate and install the app you need from the App Catalog. If you want to see a preview of the dashboards included with the app before installing, click Preview Dashboards.

  1. From the App Catalog, search for and select the app.
  2. Select the version of the service you're using and click Add to Library. Version selection is applicable only to a few apps currently. For more information, see the Install the Apps from the Library.
  3. To install the app, complete the following fields.
    1. App Name. You can retain the existing name, or enter a name of your choice for the app.

    2. Data Source.
      • Choose Enter a Custom Data Filter, and enter a custom Couchbase cluster filter. Examples:
        1. For all Couchbase clusters db_cluster=*
        2. For a specific cluster: db_cluster=couchbase.dev.01
        3. Clusters within a specific environment: db_cluster=couchbase.dev.01 and environment=prod. This assumes you have set the optional environment tag while configuring collection.
    3. Advanced. Select the Location in the Library (the default is the Personal folder in the library), or click New Folder to add a new folder.
  4. Click Add to Library.

Once an app is installed, it will appear in your Personal folder, or another folder that you specified. From here, you can share it with your organization.

Panels will start to fill automatically. It's important to note that each panel slowly fills with data matching the time range query and received since the panel was created. Results won't immediately be available, but with a bit of time, you'll see full graphs and maps.

Viewing Couchbase Dashboards​

Filter with template variables

Template variables provide dynamic dashboards that can rescope data on the fly. As you apply variables to troubleshoot through your dashboard, you view dynamic changes to the data for a quicker resolution to the root cause. You can use template variables to drill down and examine the data on a granular level. For more information, see Filter with template variables.

Overview​

The Couchbase - Overview dashboard provides an at-a-glance view of the health of the Couchbase clusters and servers, performance, and problems causing errors.

Use this dashboard to:

  • Gain insights into information about the number of nodes, number of buckets, connections, number items, total bytes transferred.
  • Determine errors in clusters: enjections, out of memory errors and error queries.
  • Gain insights into information about the workload of the cluster: percent of used memory, percent of used CPU.
Cassandra dashboards

Bucket I/O​

The Couchbase - Bucket I/O dashboard provides an insight into the operators of buckets in clusters: the number of getting operations, the number of set operations, the number of delete operations, the bytes read/write.

Use this dashboard to:

  • Get insights into information about the total amount of operations in buckets per second; the number of delete misses operations, get operations, set operations, update operations in buckets per second.
  • Get insights into information about the number of bytes read, bytes written over time.
Cassandra dashboards

Cluster Resources​

The Couchbase - Cluster Resources dashboard provides an insight into the resources of clusters: the memory resource usage, the CPU resource usage, the disk resource usage.

Use this dashboard to:

  • Gain insights into the workload of Couchbase clusters such as the percent of CPU used, the percent of Memory used, the High Low watermark.
  • Gain insights into the used resources of Couchbase clusters such as the Disk usage, the Swap space usage, the Memory available.
  • Gain insights into the rate requests, rate of streaming requests on the management port.
Cassandra dashboards

DCP Queues​

The Couchbase - DCP Queues dashboard provides an insight into the DCP queues of buckets in couchbase clusters: the number of DCP connections, DCP senders, the number of items in DCP Queues.

Use this dashboard to:

  • Gain insights into the operations of DCP queues. This helps you identify the performance of your clusters when your cluster rebalance
Cassandra dashboards

Disk Queues​

The Couchbase - Disk Queues dashboard provides an insight into the DCP queues of buckets in couchbase clusters: the number of active items waiting to be written to disk, the number of items being put to disk queue, the average age of items in queues.

Use this dashboard to:

  • Gain insights into the operations of disk queues. This helps you identify performance about read/write of your clusters.
Cassandra dashboards

vBucket​

The Couchbase - vBucket dashboard provides insights into the state of vBucket of buckets in couchbase clusters: the number of vBucket of buckets, the number items in vBuckets, the state of vBuckets.

Use this dashboard to:

  • To determine the number and status of vBucket in your clusters.
Cassandra dashboards

XDCR​

The Couchbase - XDCR dashboard provides insights into replicate operations of buckets cross-cluster: the number of XDCR connections, the number of XDCR items remaining, the number of read-set-delete operations for XDCR.

Use this dashboard to:

  • Gain insights into replicate operations of buckets cross-cluster
Cassandra dashboards

Errors​

The Couchbase - Errors dashboard provides insights into errors from error logs in couchbase servers and couchbase clusters: buckets not ready, nodes not responding, node down, error queries, last error logs.

Use this dashboard to:

  • Quickly identify critical errors affecting your couchbase servers.
  • Identify SQL error queries from clients.
Cassandra dashboards

Events​

The Couchbase - Events dashboard provides insights into events from couchbase servers and couchbase clusters: the number of login failure, login success from clients, add/remove node events, add/remove bucket events, rebalance events.

Use this dashboard to:

  • To audit the activities happening in the cluster. This helps to determine what activities have occurred in the system, helping to control system security.
Cassandra dashboards

HTTP Access​

The Couchbase - HTTP Access dashboard provides insights into HTTP Rest API requests from clients to couchbase servers and couchbase clusters: the latency, HTTP codes, client agents, IP clients, errors with 4XX 5XX response code.

Use this dashboard to:

  • To understand user behavior accessing clusters and servers through Rest API.
Cassandra dashboards

Couchbase Alerts​

Sumo Logic has provided out-of-the-box alerts available via Sumo Logic monitors to help you quickly determine if the Couchbase database cluster is available and performing as expected.

Alert Type (Metrics/Logs)Alert NameAlert DescriptionTrigger Type (Critical / Warning)Alert ConditionRecover Condition
LogsCouchbase - Bucket Not ReadyThis alert fires when a bucket in the Couchbase cluster is not ready.Critical < 0< =0
LogsCouchbase - Node DownThis alert fires when a node in the Couchbase cluster is down.Critical>< =0
LogsCouchbase - Node Not RespondThis alert fires when a node in the Couchbase cluster does not respond too many times.Critical> = 10< 10
LogsCouchbase - Too Many Error Queries on BucketsThis alert fires when there are too many errors queries on a bucket in a Couchbase cluster.Critical> = 1000< 1000
LogsCouchbase - Too Many Login FailuresThis alert fires when there are too many login failures to a node in a Couchbase cluster.Critical> = 1000< 1000
MetricsCouchbase - High CPU UsageThis alert fires when CPU usage on a node in a Couchbase cluster is high.Critical> = 80< 80
MetricsCouchbase - High Memory UsageThis alert fires when memory usage on a node in a Couchbase cluster is high.Critical> = 80< 80
Sumo Logic YouTubeSumo Logic Twitter
Legal
Privacy Statement
Terms of Use

Copyright © 2022 by Sumo Logic, Inc.