Skip to main content

Sumo Logic App for ActiveMQ

icon

The ActiveMQ app is a unified logs and metrics app that helps you monitor the availability, performance, health, and resource utilization of your ActiveMQ messaging clusters. Preconfigured dashboards provide insight into cluster status, nodes, producers, consumers, destinations, resource utilization, message rates, and error logs.

Sample Log Message​

{
  timestamp:1624348918179,
  log:"2021-06-22 08:01:57,993 | DEBUG | Publishing: tcp://activemq-2:61616 for broker transport URI: tcp://activemq-2:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport: tcp:///10.32.0.1:16932@61616",
  stream:"stdout",
  time:"2021-06-22T08:01:58.177654533Z"
}

Collecting Logs and Metrics for ActiveMQ​

This App has been tested with following ActiveMQ versions:

  • 5.16.2.

Configuring log and metric collection for the ActiveMQ App includes the following tasks:

Step 1: Configure Fields in Sumo Logic​

Create the following Fields in Sumo Logic prior to configuring collection. This ensures that your logs and metrics are tagged with relevant metadata, which is required by the app dashboards. For information on setting up fields, see Sumo Logic Fields.

If you're using ActiveMQ in a Kubernetes environment, create the fields:

  • pod_labels_component
  • pod_labels_environment
  • pod_labels_messaging_system
  • pod_labels_messaging_cluster

Step 2: Configure ActiveMQ Logs and Metrics Collection​

Choose your environment:

In Kubernetes environments, we use the Telegraf Operator, which is packaged with our Kubernetes collection (learn more). The diagram below illustrates how data is collected from ActiveMQ in a Kubernetes environment. In the architecture shown below, there are four services that make up the metric collection pipeline: Telegraf, Prometheus, Fluentd and FluentBit.

k8s-diagram

The first service in the pipeline is Telegraf. Telegraf collects metrics from ActiveMQ. Note that we’re running Telegraf in each pod we want to collect metrics from as a sidecar deployment for example, Telegraf runs in the same pod as the containers it monitors. Telegraf uses the Jolokia2 input plugin to obtain metrics. (For simplicity, the diagram doesn’t show the input plugins.) The injection of the Telegraf sidecar container is done by the Telegraf Operator. We also have Fluentbit that collects logs written to standard out and forwards them to FluentD, which in turn sends all the logs and metrics data to a Sumo Logic HTTP Source.

Prerequisites

It’s assumed that you are using the latest helm chart version. If not, upgrade using the instructions here.

Configure Metrics Collection​

This section explains the steps to collect ActiveMQ metrics from a Kubernetes environment.

  1. Set up Kubernetes Collection with the Telegraf Operator.
  2. Configure ActiveMQ Image: To enable Telegraf sidecar to get metrics from ActiveMQ Container, you must enable read metrics from ActiveMQ Container via the JMX MBeans and Disable strict-checking.
    1. Enable reads metrics from ActiveMQ Container via the JMX MBeans. While building the ActiveMQ docker image, setting useJmx="true” in ActiveMQ.xml config file:
      <broker useJmx="true" brokerName="BROKER1">
      ...
      </broker>
    2. Disable strict-checking by editing file jolokia-access.xml. While building the ActiveMQ docker image, edit file jolokia-access.xml in <Folder ActiveMQ Installed>/webapps/api/WEB-INF/classes/ and comment or remove section below:
    <cors>
    <strict-checking/>
    </cors>
  3. Add the following annotations on your ActiveMQ pods:
 annotations:
    telegraf.influxdata.com/class: sumologic-prometheus
    prometheus.io/scrape: "true"
    prometheus.io/port: "9273"
    telegraf.influxdata.com/inputs: |+
            [[inputs.disk]]
                    mount_points = ["/"]
             [inputs.disk.tags]
                    environment="dev"
                    component="messaging"
                    messaging_system="activemq"
                    messaging_cluster="activemq_on_k8s_CHANGE_ME"
            [[inputs.jolokia2_agent]]
                    urls = ["http://localhost:8161/api/jolokia"]
                    name_prefix = "activemq_"
                    username = "<username_CHANGE_ME>"
                    password = "<password_CHANGE_ME>"
              [inputs.jolokia2_agent.tags]
                    environment="prod_CHANGE_ME"
                    component="messaging"
                    messaging_system="activemq"
                    messaging_cluster="activemq_on_k8s_CHANGE_ME"

              [[inputs.jolokia2_agent.metric]]
                    name  = "OperatingSystem"
                    mbean = "java.lang:type=OperatingSystem"


              [[inputs.jolokia2_agent.metric]]
                    name  = "jvm_runtime"
                    mbean = "java.lang:type=Runtime"
                    paths = ["Uptime"]

              [[inputs.jolokia2_agent.metric]]
                    name  = "jvm_memory"
                    mbean = "java.lang:type=Memory"


              [[inputs.jolokia2_agent.metric]]
                    name = "jvm_garbage_collector"
                    mbean = "java.lang:name=*,type=GarbageCollector"
                    paths = ["CollectionCount"]
                    tag_keys = ["name"]

              [[inputs.jolokia2_agent.metric]]
                    name = "queue"
                    mbean =
                    "org.apache.activemq:brokerName=*,destinationName=*,
                    destinationType=Queue,type=Broker"
                    tag_keys = ["brokerName","destinationName"]

              [[inputs.jolokia2_agent.metric]]
                    name = "topic"
                    mbean =
                    "org.apache.activemq:brokerName=*,destinationName=*,
                    destinationType=Topic,type=Broker"
                    tag_keys = ["brokerName","destinationName"]

              [[inputs.jolokia2_agent.metric]]
                    name = "broker"
                    mbean = "org.apache.activemq:brokerName=*,type=Broker"
                    tag_keys = ["brokerName"]

Enter values for the following parameters (marked CHANGE_ME above):

  • telegraf.influxdata.com/inputs - This contains the required configuration for the Telegraf ActiveMQ Input plugin. Please refer to this doc for more information on configuring the ActiveMQ input plugin for Telegraf. Note: As telegraf will be run as a sidecar the host should always be localhost.

    • In the input plugins section, which is [[inputs.jolokia2_agent]]:
      • url - The URL of the ActiveMQ server for JMX MBeans HTTP Endpoint. Please see this doc for more information on additional parameters for configuring the Jolokia2 input plugin for Telegraf.
      • username: The Username of ActiveMQ’s admin account . The default is “admin”.
      • password: The password of ActiveMQ's admin account. The default is “admin”.
    • In the tags section, [inputs.jolokia2_agent.tags]:
      • environment - This is the deployment environment where the ActiveMQ cluster identified by the value of servers resides. For example: dev, prod or qa. While this value is optional we highly recommend setting it.
      • messaging_cluster - Enter a name to identify this ActiveMQ cluster. This cluster name will be shown in the Sumo Logic dashboards.
    Do not modify these values

    Modifying these values will cause the Sumo Logic apps to not function correctly.

    • telegraf.influxdata.com/class: sumologic-prometheus - Instructs the Telegraf operator what output to use.
    • prometheus.io/scrape: "true" - Ensures our Prometheus will scrape the metrics.
    • prometheus.io/port: "9273" - Tells prometheus what ports to scrape on.
      • telegraf.influxdata.com/inputs
        • In the tags section, [inputs.jolokia2_agent.tags]:
          • component: “messaging” - Used by Sumo Logic apps to identify application components.
          • messaging_system: “activemq” - Identifies the messaging system.
    • For all other parameters, please see this doc for more parameters that can be configured in the Telegraf agent globally.
  1. SumoLogic Kubernetes collection will automatically start collecting metrics from the pods having the labels and annotations defined in the previous step.
  2. Verify metrics in Sumo Logic.

Configure Logs Collection​

This section explains the steps to collect ActiveMQ logs from a Kubernetes environment.

  1. Collect ActiveMQ logs written to standard output. If your ActiveMQ pod is writing logs to standard output, follow the steps below to collect logs:

    1. Make sure that the logs from ActiveMQ are sent to stdout. Follow the instructions below to capture ActiveMQ logs from stdout on Kubernetes. Apply the following labels to the ActiveMQ pods:
      environment: "prod_CHANGE_ME"
        component: "messaging"
        messaging_system: "activemq"
            messaging_cluster: "activemq_on_k8s_CHANGE_ME"
    2. Enter in values for the following parameters (marked in CHANGE_ME above):
    • environment. This is the deployment environment where the ActiveMQ cluster identified by the value of servers resides. For example: dev, prod or qa. While this value is optional we highly recommend setting it.
    • messaging_cluster. Enter a name to identify this ActiveMQ cluster. This cluster name will be shown in the Sumo Logic dashboards.
    Do not modify these values

    Modifying these values will cause the Sumo Logic apps to not function correctly.

    • component: “messaging”. This value is used by Sumo Logic apps to identify application components.
    • messaging_system: “activemq”. This value identifies the messaging system.
    • For all other parameters, see this doc for more parameters that can be configured in the Telegraf agent globally.
    1. The Sumologic-Kubernetes-Collection will automatically capture the logs from stdout and will send the logs to Sumologic. For more information on deploying Sumologic-Kubernetes-Collection, please see this page.

  2. (Optional) Collecting ActiveMQ Logs from a Log File. If your ActiveMQ chart/pod is writing its logs to log files, you can use a sidecar to send log files to standard out. To do this:

    1. Determine the location of the ActiveMQ log file on Kubernetes. This can be determined from the log4j.properties for your ActiveMQ cluster along with the mounts on the ActiveMQ pods.
    2. Install the Sumo Logic tailing sidecar operator.
    3. Add the following annotation in addition to the existing annotations.
      annotations:
            tailing-sidecar: sidecarconfig;<mount>:<path_of_ActiveMQ_log_file>/<ActiveMQ_log_file_name>
      Example:
      annotations:
            tailing-sidecar: sidecarconfig;data:/opt/activemq/data/activemq.log
    4. Make sure that the ActiveMQ pods are running and annotations are applied by using the command:
      kubectl describe pod <ActiveMQ_pod_name>
    5. Sumo Logic Kubernetes collection will automatically start collecting logs from the pods having the annotations defined above.

  3. Add an FER to normalize the fields in Kubernetes environments. Labels created in Kubernetes environments automatically are prefixed with pod_labels. To normalize these for our app to work, we need to create a Field Extraction Rule if not already created for Messaging Application Components. To do so:

    1. Go to Manage Data > Logs > Field Extraction Rules.
    2. Click the + Add button on the top right of the table.
    3. The Add Field Extraction Rule form will appear. Enter the following options:
    • Rule Name. Enter the name as App Observability - Messaging.
    • Applied At. Choose Ingest Time
    • Scope. Select Specific Data
      • Scope: Enter the following keyword search expression:
      pod_labels_environment=* pod_labels_component=messaging
      pod_labels_messaging_system=* pod_labels_messaging_cluster=*
    • Parse Expression. Enter the following parse expression:
      if (!isEmpty(pod_labels_environment), pod_labels_environment, "") as environment
      | pod_labels_component as component
      | pod_labels_messaging_system as messaging_system
      | pod_labels_messaging_cluster as messaging_cluster

Installing ActiveMQ Monitors​

This section and below contain instructions for installing Sumo Logic Monitors for ActiveMQ, the app, and descriptions of each of the app dashboards. These instructions assume you have already set up the collection as described in the Collect Logs and Metrics for the ActiveMQ App page.

  • To install these alerts, you need to have the Manage Monitors role capability.
  • Alerts can be installed by either importing a JSON file or a Terraform script.

Sumo Logic provides out-of-the-box alerts available through Sumo Logic monitors to help you monitor your ActiveMQ clusters. These alerts are built based on metrics and logs datasets and include preset thresholds based on industry best practices and recommendations. For details, see ActiveMQ Alerts.

note

There are limits to how many alerts can be enabled - please see the Alerts FAQ for details.

Method 1: Install the monitors by importing a JSON file:​

  1. Download the JSON file that describes the monitors.
  2. The JSON contains the alerts that are based on Sumo Logic searches that do not have any scope filters and therefore will be applicable to all ActiveMQ clusters, the data for which has been collected via the instructions in the previous sections. However, if you would like to restrict these alerts to specific clusters or environments, update the JSON file by replacing the text messaging_system=activemq with <Your Custom Filter>. Custom filter examples:
    • For alerts applicable only to a specific cluster, your custom filter would be: messaging_cluster=activemq-prod.01
    • For alerts applicable to all clusters that start with activemq-prod: messaging_cluster=activemq-prod*
    • For alerts applicable to a specific cluster within a production environment: messaging_cluster=activemq-1 and environment=prod. This assumes you have set the optional environment tag while configuring collection.
  3. Go to Manage Data > Alerts > Monitors.
  4. Click Add.
  5. Click Import and then copy-paste the above JSON to import monitors.

The monitors are disabled by default. Once you have installed the alerts using this method, navigate to the ActiveMQ folder under Monitors to configure them. See this document to enable monitors to send notifications to teams or connections. Please see the instructions detailed in Step 4 of this document.

Method 2: Install the alerts using a Terraform script​

  1. Generate an access key and access ID for a user that has the Manage Monitors role capability in Sumo Logic using these instructions. Please identify which deployment your Sumo Logic account is in, using this link.
  2. Download and install Terraform 0.13 or later.
  3. Download the Sumo Logic Terraform package for ActiveMQ alerts: The alerts package is available in the Sumo Logic github repository. You can either download it through the “git clone” command or as a zip file.
  4. Alert Configuration: After the package has been extracted, navigate to the package directory terraform-sumologic-sumo-logic-monitor/monitor_packages/ActiveMQ/.
    1. Edit the activemq.auto.tfvars file and add the Sumo Logic Access Key, Access Id, and Deployment from Step 1.
      access_id   = "<SUMOLOGIC ACCESS ID>"
      access_key  = "<SUMOLOGIC ACCESS KEY>"
      environment = "<SUMOLOGIC DEPLOYMENT>"
      The Terraform script installs the alerts without any scope filters, if you would like to restrict the alerts to specific clusters or environments, update the variable 'activemq_data_source'. Custom filter examples:
    • A specific cluster **'messaging_cluster=activemq.prod.01'
    • All clusters in an environment 'environment=prod'
    • For alerts applicable to all clusters that start with activemq-prod, your custom filter would be: 'messaging_cluster=activemq-prod*'
    • For alerts applicable to a specific cluster within a production environment, your custom filter would be:activemq_cluster=activemq-1 and environment=prod (This assumes you have set the optional environment tag while configuring collection)

All monitors are disabled by default on installation, if you would like to enable all the monitors, set the parameter monitors_disabled to false in this file.

By default, the monitors are configured in a monitor folder called “ActiveMQ”, if you would like to change the name of the folder, update the monitor folder name in “folder” key at activemq.auto.tfvars file.

  1. If you would like the alerts to send email or connection notifications, modify the file activemq_notifications.auto.tfvars and populate connection_notifications and email_notifications as per below examples.
Pagerduty Connection Example
connection_notifications = [
    {
      connection_type       = "PagerDuty",
      connection_id         = "<CONNECTION_ID>",
      payload_override      = "{\"service_key\": \"your_pagerduty_api_integration_key\",\"event_type\": \"trigger\",\"description\": \"Alert: Triggered {{TriggerType}} for Monitor {{Name}}\",\"client\": \"Sumo Logic\",\"client_url\": \"{{QueryUrl}}\"}",
      run_for_trigger_types = ["Critical", "ResolvedCritical"]
    },
    {
      connection_type       = "Webhook",
      connection_id         = "<CONNECTION_ID>",
      payload_override      = "",
      run_for_trigger_types = ["Critical", "ResolvedCritical"]
    }
  ]

Replace <CONNECTION_ID> with the connection id of the webhook connection. The webhook connection id can be retrieved by calling the Monitors API.

For overriding payload for different connection types, refer to this document.

Email Notifications Example
email_notifications = [
    {
      connection_type       = "Email",
      recipients            = ["abc@example.com"],
      subject               = "Monitor Alert: {{TriggerType}} on {{Name}}",
      time_zone             = "PST",
      message_body          = "Triggered {{TriggerType}} Alert on {{Name}}: {{QueryURL}}",
      run_for_trigger_types = ["Critical", "ResolvedCritical"]
    }
  ]
  1. Install the Alerts:
    1. Navigate to the package directory terraform-sumologic-sumo-logic-monitor/monitor_packages/ActiveMQ/ and run terraform init. This will initialize Terraform and will download the required components.
    2. Run terraform plan to view the monitors which will be created/modified by Terraform.
    3. Run terraform apply.
  2. Post Installation

If you haven’t enabled alerts and/or configured notifications through the Terraform procedure outlined above, we highly recommend enabling alerts of interest and configuring each enabled alert to send notifications to other users or services. This is detailed in Step 4 of this document.

There are limits to how many alerts can be enabled - please see the Alerts FAQ.

Installing the ActiveMQ App​

Locate and install the app you need from the App Catalog. If you want to see a preview of the dashboards included with the app before installing, click Preview Dashboards.

  1. From the App Catalog, search for and select the app.
  2. Select the version of the service you're using and click Add to Library.

Version selection is applicable only to a few apps currently. For more information, see the Install the Apps from the Library document.

  1. To install the app, complete the following fields.
    1. App Name. You can retain the existing name, or enter a name of your choice for the app.

    2. Data Source. Choose Enter a Custom Data Filter, and enter a custom ActiveMQ cluster filter. Examples:
    • For all ActiveMQ clusters: messaging_cluster=*
    • For a specific cluster: messaging_cluster=activemq.dev.01.

    • Clusters within a specific environment: messaging_cluster=activemq-1 and environment=prod (This assumes you have set the optional environment tag while configuring collection).
  2. Advanced. Select the Location in Library (the default is the Personal folder in the library), or click New Folder to add a new folder.
  3. Click Add to Library.

Once an app is installed, it will appear in your Personal folder, or another folder that you specified. From here, you can share it with your organization.

Panels will start to fill automatically. It's important to note that each panel slowly fills with data matching the time range query and received since the panel was created. Results won't immediately be available, but with a bit of time, you'll see full graphs and maps.

ActiveMQ Alerts​

Sumo Logic has provided out-of-the-box alerts available via Sumo Logic monitors to help you quickly determine if the ActiveMQ database cluster is available and performing as expected.

Alert Type (Metrics/Logs)Alert NameAlert DescriptionTrigger Type (Critical / Warning)Alert ConditionRecover Condition
MetricsActiveMQ - High CPU UsageThis alert fires when CPU usage on a node in a ActiveMQ cluster is high.Critical > = 80 < 80
MetricsActiveMQ - High Host Disk UsageThis alert fires when there is high disk usage on a node in an ActiveMQ cluster.Critical> = 80< 80
MetricsActiveMQ - High Memory UsageThis alert fires when memory usage on a node in an ActiveMQ cluster is high.Critical> = 80< 80
MetricsActiveMQ - High Number of File Descriptors in useThis alert fires when the percentage of file descriptors used by a node in an ActiveMQ cluster is high.Critical> = 80< 80
MetricsActiveMQ - High Storage UsedThis alert fires when there is storage usage on a node that is high in an ActiveMQ cluster.Critical> = 80< 80
MetricsActiveMQ - High Temp UsageThis alert fires when there is high temp usage on a node in an ActiveMQ cluster.Critical> = 80< 80
LogsActiveMQ - Maximum ConnectionThis alert fires when one node in ActiveMQ cluster exceeds the maximum allowed client connection limit.Critical> = 1< 1
MetricsActiveMQ - No Consumers on QueuesThis alert fires when an ActiveMQ queue has no consumers.Critical< 1> = 1
MetricsActiveMQ - No Consumers on TopicsThis alert fires when an ActiveMQ topic has no consumers.Critical< 1> = 1
LogsActiveMQ - Node DownThis alert fires when a node in the ActiveMQ cluster is down.Critical> = 1< 1
MetricsActiveMQ - Too Many ConnectionsThis alert fires when there are too many connections to a node in an ActiveMQ cluster.Critical> = 1000< 1000
MetricsActiveMQ - Too Many Expired Messages on QueuesThis alert fires when there are too many expired messages on a queue in an ActiveMQ cluster.Critical> = 1000< 1000
MetricsActiveMQ - Too Many Expired Messages on TopicsThis alert fires when there are too many expired messages on a topic in an ActiveMQ cluster.Critical> = 1000< 1000
MetricsActiveMQ - Too Many Unacknowledged MessagesThis alert fires when there are too many unacknowledged messages on a node in an ActiveMQ cluster.Critical> = 1000< 1000

Viewing the ActiveMQ Dashboards​

Dashboard Filters with Template Variables​

Template variables provide dynamic dashboards that rescope data on the fly. As you apply variables to troubleshoot through your dashboard, you can view dynamic changes to the data for a fast resolution to the root cause. For more information, see the Filter with template variables help page.

Overview​

The ActiveMQ - Overview dashboard gives you an at-a-glance view of your ActiveMQ deployment across brokers, queues, topics, and messages.

Use this dashboard to:

  • Analyze Memory and CPU utilization.
  • Gain insights into Enqueue messages for your ActiveMQ server.
  • Gain insights into Dequeue messages for your ActiveMQ server.
  • Determine node uptime, the number of nodes, connections, producer, consumer, queues, topics unack messages, across each cluster and ensure they match with expectations.
  • Analysis of near errors.
ActiveMQ dashboards

Brokers​

The ActiveMQ - Brokers dashboard provides an at-a-glance view of the state of your brokers in the ActiveMQ cluster.

Use this dashboard to:

  • Monitor brokers uptime.
  • Analyze unacknowledged messages, serve connections for your ActiveMQ brokers..
  • Gain insights into the limit of resources for your ActiveMQ brokers.
  • Gain insights into enqueue rate, dequeue rate for your ActiveMQ brokers.
  • Determine the amount of current connections, producers, and consumers for your ActiveMQ brokers.
  • find average message size.
ActiveMQ dashboards

Queues​

The ActiveMQ - Queues dashboard provides an at-a-glance view of the state of your queues in ActiveMQ clusters.

Use this dashboard to:

  • Monitor en queue latency, the memory usage of queues.
  • Monitor queues, producers on queues, consumers on queues.
  • Gain insights into en queue rate, dequeue rate for your ActiveMQ queues.
  • Determine the number of current connections, producers, consumers, errors, and expired messages on queues.
  • Find average message size on queues.
ActiveMQ dashboards

Topics​

The ActiveMQ - Topics dashboard provides an at-a-glance view of the state of your topics in ActiveMQ clusters.

Use this dashboard to:

  • Monitor en queue latency, the memory usage of topics.
  • Determine the number of topics, producers on topics, consumers on topics.
  • Gain insights into en queue rate, dequeue rate for your ActiveMQ topics.
  • Analyze current connections, producers, consumers, errors, and expired messages on queues.
  • Find average message size on queues.
ActiveMQ dashboards

Resource Utilization​

The ActiveMQ - Resource Utilization dashboard provides an at-a-glance view of the state of system loads in clusters: CPU usage, memory usage, Swap usage, file descriptor usage, garbage collection rate, heap, and non-heap usage.

Use this dashboard to:

  • Analyze memory, CPU, disk, swap, file descriptor utilization of nodes in clusters.
  • Gain insights into the garbage collection rate for your ActiveMQ nodes.
  • Gain insights into a heap, non-heap usage for your ActiveMQ nodes.
ActiveMQ dashboards

Logs​

This dashboard helps you quickly analyze your ActiveMQ error logs across all clusters.

Use this dashboard to:

  • Identify critical events in your ActiveMQ cluster.
  • Examine trends to detect spikes in Error or Fatal events
  • Monitor Broker added/started and shutdown events in your cluster.
  • Quickly determine patterns across all logs in a given ActiveMQ cluster.
ActiveMQ dashboards
Sumo Logic YouTubeSumo Logic Twitter
Legal
Privacy Statement
Terms of Use

Copyright © 2022 by Sumo Logic, Inc.