Debugging

If traps do not arrive at all or do not fall into the correct items, there are a few things to check. If the traps do not appear when sent from a remote host, but work properly when sent from the Zabbix server, check the local firewall on the Zabbix server and make sure incoming UDP packets on port 162 are allowed.

Also make sure that the IP address the Zabbix server sees in the incoming traps matches the address in the SNMP interface for that host.

Sometimes, you might see that traps arrive at the SNMP trap daemon but do not seem to be passed on. It might be useful to debug snmptrapd in this case—luckily, it allows a quite detailed debug output. Exact values to use for various file locations will differ, but the following might work to manually start the daemon while enabling all debug output:

# /usr/sbin/snmptrapd -A -Lf /var/log/net-snmpd.log -p /var/run/snmptrapd.pid -DALL  

Here, -Lf specifies the file to log to and -DALL tells it to enable full debug.

If the received trap is in a numeric format and not very readable, you might have to add specific MIBs to the /etc/snmp/snmp.conf file so that they are found by snmptrapd.

What happens if Zabbix decides that a trap does not belong to any item on any host? This could happen because there are no trap items at all, the fallback item is missing, or the address in the incoming trap is not matched with any of the SNMP interfaces. By default, the Zabbix server logs such traps in the log file. An example record from the log file is as follows:

2271:20150120:124156.818 unmatched trap received from [192.168.168.192]: 12:41:55 2015/01/20 PDU INFO: 
  errorindex                     0 
  transactionid                  1 
  requestid                      1752369294 
  messageid                      0 
  receivedfrom                   UDP: [192.168.168.192]:45375->[192.168.1.13]:162 
  errorstatus                    0 
  version                        1 
  notificationtype               INFORM 
  community                      public 
VARBINDS: 
  DISMAN-EVENT-MIB::sysUpTimeInstance type=67 value=Timeticks: (77578087) 8 days, 23:29:40.87 
  SNMPv2-MIB::snmpTrapOID.0      type=6  value=OID: NET-SNMP-MIB::netSnmpExperimental 
  NET-SNMP-MIB::netSnmpExperimental type=4  value=STRING: "non-matching trap" 

The logging of non-matching traps can be controlled. If we go to Administration | General and choose Other from the drop-down menu in the upper-right corner; the last checkbox there is Log unmatched SNMP traps. Unmarking it will stop logging such traps:

And what if you would like to try out Zabbix's SNMP trap handling without setting up an SNMP trap daemon, perhaps on some development server? That should be very easy as you can simply append trap information to the temporary file. It's a plain-text file, and Zabbix does not know who added content, the trap daemon, user, or somebody else. Just make sure to add all of the data for a single trap in one go.