Riemann and Zabbix: Sending data from riemann to zabbix
Riemann is a stream processing engine (written in Clojure) which can be used to monitor distributed systems. Although it can be used for defining alerts and sending notifications for those alerts, we currently use it like this:
- As a receiving point for metrics / data from a group of systems in an installation
- Applying some filtering and aggregation at the installation level.
- Sending the filtered / aggregated data to a central Zabbix system.
The actual alerting mechanism is handled by Zabbix. Things like trigger definitions, sending notifications, handling acks and escalations, etc.
This might seem like Riemann is redundant (and there is definitely some overlap in functionality), but keeping Riemann in the data pipeline allows us to be more flexible operationally. This is specially in cases when the metrics data we need is coming from application code, and we need to apply some transformations to the data but cannot update the code.
The first problem we faced when trying to do this is: sending data from Riemann to Zabbix is not that straightforward.
Surprisingly, the Zabbix API is not actually meant for sending data points to Zabbix - only for managing it's configuration and accessing historical data.
The recommended way to send data to Zabbix is to use a command line application called zabbix_sender.
Another way would be to write a custom zabbix client in Clojure which follows the Zabbix Agent protocol, which uses JSON over TCP sockets.
The current solution we have taken for this is using
For this, we write filtered values to a predefined text file from Riemann in a format that
zabbix_sender can understand.
The above code writes data into the file
/var/log/riemann/to_zabbix.txt in the following format:
Then, the following script can be run to sending data from this file to Zabbix via
- There should probably be a check on Riemann whether data is correctly being delivered to Zabbix or not. If not, Riemann can send out alerts as well.
- The current solution is a little fragile because it's first writing the data to a file and is dependent on an external service running to ship the data to Zabbix. A better solution would be to integrate directly as a Zabbix agent.