Sending PowerStore alerts via SNMP

In this previous article we discussed how to leverage PowerStore's powerful API to collect audit logs and send them to a Syslog server. We used Logstash to do that because of its extensive list of plugins

Sometimes I come across organizations that still have a requirement to send storage array Alerts to SNMP. As in the Syslog case, ultimately most modern monitoring tools today support collecting information via REST API because that's the way the application world is going, i.e. the cloud native way ... it would be fun to see Prometheus monitoring Kubernetes via SNMP :)

However there is still some requirement out there for SNMP traps, so we will explore how to do it. For simplicity we will use the same toolset that we used in the previous article. I encourage you to revisit quickly the article as we did an extensive introduction to Logstash, with practical examples as well as a description of the logs available in PowerStore. I won't cover those aspects in this article

I have tried to write this article in as much details as possible. So if you are already familiar with things like SNMP, feel free to skim ahead

What to include in the alerts?

Whereas the audit log is the typical source for Syslog logging, the Alert and Event API resources are the natural candidates for SNMP. Furthermore the PowerStore MIB is based on the Alert resource.

What's the difference? As you can see in the PowerStore REST API reference guide Event and Alert they have very similar fields. Events are "individual" events (pardon the pun) that get reported whereas Alerts aggregate multiple related events and introduce the idea of "Acknowledged" and "Cleared" to provide a more operational point of view.

The Logstash method we are going to use is very flexible and we could easily collect information from the Events API and map them to SNMP Traps on the Alerts MIB if we wanted to do so

The Event Collection

The first part of this solution is to collect the data from PowerStore's REST API. This is almost identical to our Syslog solution. Firstly, let's ensure Logstash is installed. Don't forget about the Java dependency

Ensure the logstash exec input plugin is installed

Create a shell script to collect the most recent alerts from the PowerStore "Alert" API. I have placed the script in the Logstash's Home directory

These are the contents of the script:

TSTART=`date -u -d 'now -5 minutes' +"%Y-%m-%dT%H:%M:%S.%3N"`
URL="https://10.1.1.1/api/rest/alert?select=*?order=generated_timestamp.desc?generated_timestamp=gt.${TSTART}"
curl -k --location  --header 'Authorization: Basic bTlfasT21AQ3NzY1234CD7' --request GET $URL

Make sure to use the IP address of your array and your array's credentials

Then change ownership and permissions to that the Logstash user can run it

The following is the "input" portion of the Logstash pipeline configuration file. As we discussed in the previous article these files need to be stored in the "/etc/logstash/conf.d" directory and need to have a ".conf" extension. This is the "input" part of the file using the exec plugin to run our shell script

input {
  exec {
    command => "sh -c /usr/share/logstash/getev5mins.sh"
    interval => 300
    codec => json
  }
}

Notice how I am using an interval of 300 seconds to match the 'now -5 minutes' in the shell script. Adjust both to suit your needs

In this example we are using "interval" instead of "schedule" as we did in the Syslog article

Generating the SNMP traps

There is no SNMP output plugins in Logstash. There are input ones (snmp and snmptrap) but they are meant for ingestion so they are no good to us. 

We will use the "exec" plugin as well in conjunction with Net-SNMP to send the traps

In CentOS 7 you can install them like this:

yum install net-snmp
yum install net-snmp-utils

Now you have a number SNMP tools installed in your system. We are going to send some test traps from the command line first to make sure everything is fine. The syntax is as follows: 

snmptrap -v <version> -c <community> <destination> <uptime> <OID_or_MIB> <object> <value_type> <value>

This screenshot shows how I sent a trap to an SNMP Trap Daemon running on the same VM (localhost). As you can see I am running version 2c, the community string is public and I am not specifying any uptime

This is the response I get at the SNMP Trap Deamon

Working with the MIB

In the previous command you might have noticed two long series of numbers separated by dots, ex: 1.3.6.1.4.1.1139. In SNMP speak, this is referred to as OID, or object identifier. Essentially the SNMP structures all the information in a big tree. Each branch has a number and so does every subbranch under it. So the OID in this paragraph is pointing at a location in the SNMP tree that is 7 levels deep. That level is where specific enterprises' subbranches live. Branch 1139 in the example correspondes to "EMC", which is where the PowerStore branch lives

Working with this long OID's a bit awkward and prone to error. A better way to work with SNMP is through MIBs. MIB (Management Information Base) are collections of definitions which define the properties of the managed object within the device to be managed. You can download the MIB for PowerStore from the Dell Support site. By using MIBs we can send traps with friendlier names. For example, we can send the trap like this

snmptrap -v 2c -m ALL -c public localhost '' PowerStore-MIB::powerstoreGenericTrapMajor  powerstoreTrapDescription s 'Capacity Full'

and it produces the same result than this

snmptrap -v 2c -c public localhost '' 1.3.6.1.4.1.1139.205.1.2.2 1.3.6.1.4.1.1139.205.1.1.2 s 'Capacity Full'

As you can guess 205 is the subranch for PowerStore. Also notice how in the first example we are using "-m ALL" to load the MIBs from the default directory, which is "/usr/share/snmp/mibs/". Make sure you place the PowerStore MIB in that directory. 

Likewise, if you want your Trap receiver to decode the trap you will have to import the MIB. Otherwise you will only get OID's instead of meaningful trap names like "Info", "Minor", "Major" ...

In the examples above we only sent a single property "powerstoreTrapDescription" but the MIB defines all the properties you can see below. These properties map directly to the information provided by the REST API

If you want to send multiple properties you can do so by repeating this pattern <object> <value_type> <value>. For example the following command sends both the description and the resource name objects:

snmptrap -v 2c -m ALL -c public localhost '' PowerStore-MIB::powerstoreGenericTrapMinor  \

powerstoreTrapResourceName s 'Volume'\

powerstoreTrapDescription s 'Capacity Full'

The value_type depends on the object. In this example "s" stands for "string". For any TimeStamp objects you will have to use "t" and convert the time stamps to "ticks" ie epoch time

Integrate the SNMP Traps into Logstash

Alright, welcome to the final episode. So how do we integrate the "snmptrap" command into Logstash? We will use the "exec" plugin. This time we will need to use two Logstash skills we didn't cover before: field references and conditionals. 

We need to extract the various fields the REST API returns in the response in order to use them as "objects" of the SNMP trap. In Logstash, this is done with the [field_name] notation. So for example for the following alert:

We can extract the "resource_name" as %{[resource_name]}.

Notice the "severity" field. One thing we will need to do is to map every event to a specific trap depending on its severity. When doing the comparison we use the [] notation as follows:

output {
  if [severity] == "Info" {
    exec {
      command => "snmptrap -v 2c -m ALL -c public localhost '' PowerStore-MIB::powerstoreGenericTrapInfo powerstoreTrapResourceName s '%{[resource_name]}'  powerstoreTrapDescription s '%{[description_l10n]}'"
    }
  }
  if [severity] == "Minor" {
    exec {
      command => "snmptrap -v 2c -m ALL -c public localhost '' PowerStore-MIB::powerstoreGenericTrapMinor  powerstoreTrapResourceName s '%{[resource_name]}'  powerstoreTrapDescription s '%{[description_l10n]}'"
    }
  }
}

The previous example will trigger an "Info" trap for every alert returned by the API that has an "Info" severity and a "Minor" trap for every alert with a "Minor" severity. According to the REST API documentation there are 5 different types of severity: None, Info, Minor, Major and Critical. You will need to create a separate conditional for each severity level you want to report on.

Additionally, in terms of fields, in the previous example we are sending two objects with  each trap: ResourceName and Description. You can add as many objects to the trap as required by following the <object> <value_type> <value> syntax we mentioned earlier where the value follows the  %{[field]} notation.

Well, there you have it, how to send SNMP traps from PowerStore alerts