IoT Data Pipeline — From Sensor to Dashboard Without the Complexity

Most IoT tutorials cover either the device side (ESP32 publishing data) or a specific platform's dashboard. The middle — how data actually gets from the sensor to the visualisation in a way you understand and control — is often left as "use this cloud service."

Cloud IoT platforms are convenient for some use cases but they add cost, create dependency, and are often over-engineered for a project with five sensors. This post covers a complete, practical data pipeline architecture that runs on a Raspberry Pi, handles dozens of sensors, stores data indefinitely, and produces dashboards — all locally, all for the cost of a Pi and some time.

The stack

MQTT broker (Mosquitto): receives all sensor data. Sensors publish to topic paths like "home/sensors/temperature/bedroom". The broker routes messages to all subscribers.

Node-RED or Telegraf (processing): subscribes to MQTT topics and forwards data to the time-series database. Node-RED provides visual flow programming and can handle data transformation, alerting, and conditional logic. Telegraf is simpler and more efficient if you just need to route data.

InfluxDB (storage): a time-series database optimised for sensor data. Handles continuous ingestion efficiently, stores tagged time-series data, and supports the queries dashboards need. InfluxDB 2.x has a local-only licence tier.

Grafana (visualisation): connects to InfluxDB and renders dashboards. Extremely flexible — line graphs, gauges, alerts, maps, tables. Pre-built IoT dashboard templates are available and save significant setup time.

All four components run on a Raspberry Pi 4 (4GB recommended). The total software stack is free and open source.

Setting it up

Install all four services on the Pi. Docker Compose makes this straightforward — a single compose file pulls all four images and configures the networking between them.

Configure Mosquitto to accept MQTT connections on port 1883. For local use, anonymous authentication is fine. For anything exposed to the internet, use username/password authentication at minimum.

Configure Telegraf with MQTT input plugin: specify the broker address, subscribe to your sensor topic patterns, and output to InfluxDB with appropriate tag extraction from topic paths.

In InfluxDB, create a bucket for sensor data with a retention policy (keep 90 days, for example — raw data accumulates fast).

In Grafana, add InfluxDB as a data source. Build a dashboard with panels for each sensor. Grafana's alert rules can send notifications via email, Telegram, or other channels when values exceed thresholds.

ESP32 firmware: add a simple MQTT publish using the PubSubClient or AsyncMqttClient library. Take a sensor reading every N seconds and publish to the appropriate topic. That's the entire device-side change.

What this enables

With this stack running: every sensor in your home or lab has its data stored and graphable. You can view temperature history for the past 90 days. You can correlate humidity with AC usage. You can set alerts for unusual values. You can export data for offline analysis.

Adding a new sensor requires only that it publishes to an MQTT topic in the right format — no changes to the backend, no configuration except potentially adding a new Grafana panel.

The architecture scales to dozens of sensors without meaningful performance impact on a Pi 4. For hundreds of sensors or high-frequency data (above 1Hz per sensor), you'd want more careful capacity planning, but for most maker deployments the headroom is significant.