Processing Heterogeneous IoT Logs with Fluent Bit and Elasticsearch

A lightweight two-stage approach for parsing, preprocessing and storing semi-structured home automation data The full configuration example is available on GitHub. Home automation systems tend to generate a surprisingly heterogeneous collection of logs and measurements. In this project, most data originates from Domoticz, but additional information is produced by ESP8266/D1 devices, alarm scripts, monitoring tools and other IoT components. Some records are plain text, some contain timestamps and metadata, while others contain JSON embedded inside a log line. Rather than forcing every producer into a single format, I use a two-stage processing architecture based on Fluent Bit and Elasticsearch. The complete chain is: IoT / home automation sources | +----------------+----------------+ | | | Domoticz ESP/D1 Other scripts | | | +----------------+----------------+ | Fluent Bit | parsing + preprocessing | v Elasticsearch | ingest pipeline | final JSON decoding | v dom_v3 index | +----------+----------+ | | Kibana Grafana The complete pipeline normally has a latency of less than one minute in the author's installation. 1. Why two processing stages? The main design decision is to avoid making Fluent Bit responsible for every transformation. Fluent Bit performs the first level of processing: reading the various log files; recognizing several recurring line formats; extracting timestamps and metadata; decoding JSON when it is directly available; removing intermediate fields; forwarding the resulting records to Elasticsearch. Elasticsearch then performs the final JSON decoding through an ingest pipeline. This separation keeps the Fluent Bit configuration relatively simple while giving Elasticsearch responsibility for the final document transformation. 2. Fluent Bit parsers The incoming data belongs to several recurring families. Instead of trying to recognize all of them with one very large regular expression, the configuration uses several smaller parsers. A typical Domoticz format is: [PARSER] Name domoticz_parser_1 Format regex Time_Key dom_timestamp Time_Format %Y-%m-%d %H:%M:%S.%L Time_Offset +0200 Regex ^(?.*) (?\w+)\: (?.*)$ Decode_Field json msgtxt This extracts the timestamp and source and decodes the JSON contained in msgtxt. Another Domoticz format is handled separately: [PARSER] Name domoticz_parser_2 Format regex Time_Key dom_timestamp Time_Format %Y-%m-%d %H:%M:%S.%L Time_Offset +0200 Regex ^(?.*) \((?\w+)\) (?.*) \((?\w+)\)$ Generic JSON-bearing records are covered by additional parsers, including a final fallback for pure JSON. The complete examples are provided in fluent-bit/parsers.conf. 3. Collecting the different sources Fluent Bit's tail input monitors the individual log files. Each source receives its own tag and persistent database: [INPUT] Name tail Tag domoticz Path /path/to/domoticz.log DB /var/spool/fb_domoticz.db The real installation contains several additional sources for ESP/D1 devices, alarms, metrics, monitoring and Tuya-related data. The tag provides a simple way of retaining the origin of each record. 4. Applying several parsers The Domoticz records are processed with the parser filter: [FILTER] Name parser Match domoticz Key_Name log Parser domoticz_parser_1 Parser domoticz_parser_2 Parser domoticz_parser_3 Parser domoticz_parser_4 Parser domoticz_parser_5 Reserve_Data On The individual expressions act as recognizers for recurring formats. This is easier to maintain than one large regular expression attempting to describe every possible record. Reserve_Data On keeps the original record available while parsed fields are added. Intermediate fields can then be removed: [FILTER] Name record_modifier Match domoticz Remove_key msgtxt 5. Sending the preprocessed data to Elasticsearch The resulting records are sent to a common Elasticsearch index: [OUTPUT] Name es Match * Host 127.0.0.1 Port 9200 HTTP_User elastic HTTP_Passwd tls On tls.verify Off Include_Tag_Key On Tag_Key fb_tag Index dom_v3 Type _doc Suppress_Type_Name On Retry_Limit 5 The example deliberately does not contain a real password. The tls.verify Off setting is specific to the author's local installation and should not be copied blindly. A network-accessible Elasticsearch instance should normally use proper certificate verification. 6. Final JSON processing in Elasticsearch The second stage is implemented as an Elasticsearch ingest pipeline: { "description": "dom_v3 ingestion pipeline v1", "processors": [ { "json": { "ignore_failure": true, "add_to_root": true, "field": "log" } } ] } The purpose is to decode JSON that remains in the log field and add the decoded fields to the document root. For example, a simplified intermediate document can contain: { "log": "{\"temperature\":21.7,\"humidity\":48}", "dom_timestamp": "2026-09-10 10:23:41.123", "dom_source": "Sensor", "fb_tag": "domoticz" } After the ingest pipeline, the document can contain: { "dom_timestamp": "2026-09-10 10:23:41.123", "dom_source": "Sensor", "temperature": 21.7, "humidity": 48, "fb_tag": "domoticz" } This example is deliberately simplified; the actual records may contain substantially more nested information. 7. Why not do everything in Fluent Bit? It would be possible to push more processing into Fluent Bit. In this application, however, the separation proved more convenient. Fluent Bit is used for: collection; line-oriented recognition; regular-expression parsing; timestamp extraction; first-level JSON decoding; metadata and filtering. Elasticsearch is used for: final document transformation; JSON decoding; indexing and mapping; subsequent search and visualization. This keeps each component focused on the type of work it handles well. 8. Mappings matter JSON decoding does not automatically guarantee that every field will have the desired Elasticsearch type. Measurements such as temperature and humidity should be indexed as numeric fields, while timestamps should use an appropriate date mapping. This is especially important when the resulting data is queried or visualized by Grafana. For a production deployment, index mappings should therefore be designed explicitly for the metrics expected from the installation. 9. Why this approach works well for home automation Home automation installations rarely produce perfectly homogeneous data. A single installation may contain: application logs; sensor measurements; alarm events; device status; network information; JSON APIs; diagnostic messages; custom scripts. Trying to force every producer to emit exactly the same format can require unnecessary changes to individual applications. The approach described here instead accepts heterogeneity at the input and progressively transforms it into structured Elasticsearch documents. Adding another source generally requires another Fluent Bit input and, when necessary, another parser. 10. Result The architecture has proved sufficient for the author's home automation environment: Various IoT sources | v Fluent Bit | +-- regex parsing +-- timestamp extraction +-- JSON preprocessing +-- metadata +-- filtering | v Elasticsearch | +-- ingest pipeline | +-- final JSON decoding | v dom_v3 | +----+----+ | | Kibana Grafana The complete chain normally has a latency of less than one minute in this installation. The main advantage is not raw processing complexity, but keeping the architecture understandable: Fluent Bit collects and recognizes. Elasticsearch structures and stores. Kibana and Grafana visualize. Conclusion For heterogeneous IoT and home automation data, combining Fluent Bit with Elasticsearch ingest pipelines provides a practical way of progressively converting unstructured log streams into structured data. The key design principle is to avoid making any single component responsible for the complete transformation. The resulting architecture is small enough for a home automation environment while remaining flexible enough to accommodate additional sensors, scripts and data sources. Acknowledgements This article was prepared by Christophe Chaussat, based on his own home automation installation, configuration and experiments. ChatGPT (OpenAI) assisted with the structuring, editing and technical writing of this documentation. The technical architecture and configuration described here are based on the author's own installation and experiments. Licensing The configuration and software examples available on GitHub (cchaussat/fluent-bit-elasticsearch-iot-logging) are released under the MIT License. This article and the accompanying documentation are released under the Creative Commons Attribution 4.0 International License (CC BY 4.0).