node-red-contrib-simple-bu 1.4.2

A Node-RED node that buffers and transmits data reliably with local file-based ring buffer storage.

npm install node-red-contrib-simple-bu

Node-RED Contrib SimpleBU

SimpleBU is a custom Node-RED node that buffers, formats, and transmits measurement data to a server (e.g. deZem M2M endpoints).
It is designed for robustness in environments with unstable internet connections:

• If the server is reachable: data is sent immediately.
• If the server is not reachable: data is stored locally in a ring buffer on disk.
• As soon as connectivity returns, the node will drain buffered files in chronological order.

This ensures no data loss and predictable CSV-formatted uploads.

Features

• CSV output format:
  Each line is id,value,timestamp
  Example:

  temp1,22.5,1756380672
  power,120,1756380672
  

• Robust ring buffer (configurable):

  • Multiple files on disk.
  • Oldest file overwritten once the buffer is full.
  • Each file has a max size limit.

• Async I/O: non-blocking file operations for high throughput.

• HTTP keep-alive: faster repeated requests.

• Automatic retry: buffered files are resent until confirmed by the server.

• Fast flush: configurable thresholds (bytes, lines) to send data quickly when batch grows.

• Validation: strict checking of IDs, values, and message format.

• Debug mode: more verbose logging for troubleshooting.

Configuration

Open the node’s properties panel in Node-RED and configure:

Input Message Format

The node accepts standard Node-RED msg objects.
The important fields are msg.payload and msg.topic.

Case 1: Single value

• msg.payload = number
• msg.topic = string ID (must match regex [A-Za-z0-9._]+)

Example inject node:

{
  "topic": "temp1",
  "payload": 22.5
}

CSV produced:

temp1,22.5,1756380672

Case 2: Multiple values (array)

• msg.payload = array of [id, value] pairs
• msg.topic is ignored

Example inject node:

{
  "payload": [
    ["temp1", 22.5],
    ["power", 120]
  ]
}

CSV produced:

temp1,22.5,1756380672
power,120,1756380672

Invalid messages

• Missing topic when payload is a single number.
• IDs that don’t match [A-Za-z0-9._]+.
• Values that are not finite numbers.

In these cases, the node sets red error status and (if Debug Enabled) logs an error.

Local Buffering (Ring Buffer)

• Data that cannot be sent immediately is saved into buffer files.
• Files are named buff_0.dat, buff_1.dat, … up to Max Files.
• Once all files are full, the oldest file is overwritten.
• Each file is plain UTF-8 CSV (can be inspected manually with cat).

Server Communication

• Data is sent via HTTP POST to the configured Endpoint-URL.
• Headers include:

  Content-Type: text/csv; charset=utf-8; header=absent
  User-Agent: IF:Simple TYPE:<SimpleType> SN:<SimpleSn>
  

• Server must return:
  • HTTP 2xx status AND
  • Body starting with OK (or empty).

Otherwise, the node assumes failure and re-buffers data.

Status Indicators

• 🔵 Buffered: data is accepted and waiting in RAM.
• 🟢 Data sent: batch was sent successfully.
• 🟢 Drained file: buffered file resent and deleted.
• 🟢 Saved: buff_X.dat: data was written to local buffer.
• 🔴 Send failed / Save error: error writing to disk or sending to server.

Performance Notes

• All file operations are asynchronous (non-blocking).
• Buffered files are streamed during upload (efficient for large files).
• Keep-alive connections reduce latency for repeated requests.
• Flush interval defines retry pacing; batch thresholds allow immediate sending when buffer grows quickly.

Troubleshooting

• Files not appearing: Check Storage Path exists and is writable inside the container/host.
• Always “Invalid number”: Verify Max Files and Max File Size are numeric values in GUI.
• Server receives wrong values (e.g. 100 → 1): Update to latest version (fixed bug in number formatting).
• Send failed: Check network connectivity, firewall, or that Endpoint-URL is correct.
• High latency: Lower Flush Interval or adjust batch_max_bytes / batch_max_lines thresholds.

License

MIT License
(c) 2025 deZem GmbH / Emir Dovletyarov