Raspberry Pi, Prometheus and Grafana

Retrospectives on building a scalable monitoring system based on Raspberry Pi, Prometheus, and Grafana.

Introduction

If you’re interested in building a monitoring system for any physical measurements, one simple way is to use a Raspberry Pi. A lot of physical measurements are covered on the web and you’ll have no trouble going from 0 to 1.

Unfortunately, I had trouble finding resources on how to go from 1 to N.

how do you measure several temperature sensors and not only one? How do you manage several Raspberry Pis? How do you mix different signals in a single application?

In this blog post, I’ll list several concepts that can help you go from 1 to N.

I will not go over implementation details that are available on my git: https://github.com/haixuanTao/Thermosensor

I have done this experiment in order for Wakaze saké to grow its brewing capabilities!

1-Wire Protocol

1-Wire protocol on GPIO4 of the Raspberry Pi enables you to connect an infinite amount of sensors to a single GPIO. I would not recommend going above 20. But, with 1-Wire, you will have all your sensors connected to GPIO4 will be listed in /sys/bus/w1/devices/ .

Some sensor don't use 1-Wire protocol though, like pH sensor, but other similar strategies can be put in place.

Prometheus

However, having sensor data is not enough. You’ll need a way to store it, manage it, access it easily.

Prometheus is the go-to monitoring system for any cluster. It is plenty powerful, easy to configure, with tons of integration.

Some killing features of Prom are:

  • Prometheus can manage billions of metrics at the same time from multiple clients.
  • Prometheus makes it really easy to filter and aggregate data.
  • Prometheus is easy to configure across networks, with the possibility to aggregate several Prometheus instances into one federated instance.

There are other people recommending InfluxDB for monitoring Raspberry Pi but InfluxDB seems less universal and resilient. The main difference between InfluxDB and Prometheus is that:

  • For InfluxDB, your application has to push to InfluxDB server, and so your application will depend on InfluxDB configs.
  • For Prometheus, the data is pulled by Prometheus from your applications, and therefore, your applications are agnostic to the monitoring system and its configuration.

Python Prometheus Client

To connect temperature/pH data to your Prometheus Server, you will need a Prometheus Client.

Python Prometheus Client is a great simple client exposing metrics like temperatures and pH for your Prometheus Server.

Technical point, I have found that serving Python with Systemd was easier than with Containers. The reason being that running Python in a container has limited access to the GPIOs of the Raspberry Pi, as well as having many Arm incompatibility. It does make deployments of Python code more tedious for CI/CD and networking but it is still manageable.

Grafana

Prometheus in itself does not provide an extensive UI. And so, you will need Grafana which is a visualization application that can display Prometheus data.

It has tons of customization, plugins, and alerting as well as predefined dashboards that you can download from the web. Grafana really gives a professional look to your monitoring application. It is very mature and easy to recommend.

Kubernetes

To manage your Prometheus and Grafana application you can use Kubernetes. Kubernetes is an operating system for your cluster.

Kubernetes will:

  • manage network across applications. If you hate knowing by heart IP address, application port, and port forwarding techniques. Kubernetes solves those issues.
  • manage reproducibility. Any application that runs on one Kube cluster can run on any Kube cluster.
  • manage installation with Helm.

Using k3s, you can install it on every Raspberry Pi and connect them together pretty simply. Self-hosting a Kube cluster on Raspberry Pis early stage will make migrating to the cloud way easier. As you will have your infrastructure as code, going to a cloud-managed Kubernetes like GKE, AKS, or EKS will be a breeze.

The temporary hardware installation:


Rhett Trickett picture

Awesome project!

So from the picture above it looks like you are running one Pi for every two brewing tanks and each tank gets a temperature and PH sensor. Or are you running many sensors to a single Pi using the wire protocol? Is that the I2C address-based protocol?

Then are you running Prometheus and Grafana on a server Pi or cloud service?

I've been experimenting with building low cost sensors using ESP32 dev boards (like the NodeMCU) to monitor temperature, humidity, light and soil moisture for plants. The sensors either report into a Raspberry Pi on my home network or to a cloud service VM.

Then I wrote a small Node API to store and retrieve sensor data so I'm close to going from one to N. I've got some unfinished blog posts about building the sensor prototypes that I need to finish up and publish but I have a few challenges left to solve.

So Prometheus and Grafana look very interesting and flexible in this regard and I'll definitely be checking them out. Thanks for the tip!

Anson also works with various projects like this. I'm sure he'd be interested in this one. Nice work, love the application to sake brewing!

Haixuan Xavier Tao picture

Currently, we only run a small set of sensors on 2 tanks, but we can very easily monitor 12 tanks at once with one raspberry pi. The objective is to produce sake in Japan, France, and the US, all linked together. This is where being able to scale comes to play.

For Temperature it is 1- Wire protocol, and for pH. I'm using a SPI Protocol through an Analog Digital converter.

ESP32s are definitely a more cost-efficient board, but It may be more cumbersome to debug. With the Raspberry Pi, I can SSH from anywhere to debug it.

Your project looks great :) Yep, if you want a quick and easy front, you should definitely check Grafana out :)

Thanks for reading and taking the time to comment :)