node-red-contrib-simple-bu 1.5.0

A Node-RED node that buffers and transmits data reliably with local file-based ring buffer storage.

npm install node-red-contrib-simple-bu

Node-RED Contrib SimpleBU

SimpleBU converts measurements to CSV (id,value,timestamp) and POSTs them to a SimpleBU endpoint. Version 1.5.0 adds individual measurement timestamps and repairs large-array delivery, offline recovery and concurrent-input data loss.

Install through Node-RED's Manage Palette, or from your Node-RED user directory:

npm install node-red-contrib-simple-bu

Restart Node-RED after upgrading. See CHANGELOG.md for changes and upgrade notes. Existing [id,value] input and ring.dat files remain supported.

Input

One measurement:

{"topic":"temp1","payload":22.5,"timestamp":1790000000}

One message containing many measurements:

const start = Math.floor(Date.now() / 1000) - 15000;
msg.payload = Array.from({ length: 15000 }, (_, i) => ["sensor." + i, i, start + i]);
return msg;

The entire array is accepted as a single input message. Each item is either [id, value, timestamp] or the legacy [id, value]. SimpleBU performs CSV conversion and HTTP chunking itself. Array items are measurements, not files. A triple's timestamp takes priority; a pair uses msg.timestamp, or input arrival time when omitted. Both forms may be mixed. An unused message timestamp is ignored when all rows supply their own time. Explicit null/undefined row timestamps are rejected.

• IDs: nonempty ASCII letters, digits, dot or underscore; finite numeric IDs also work.
• Values: finite JavaScript numbers, including zero and negatives. Decimal serialization preserves the provided number without rounding tiny values or using exponent notation.
• Time: UNIX seconds, modern UNIX milliseconds (abs(value) >= 1e12), a date string accepted by JavaScript Date.parse, or a valid Date. Seconds are floored. Numeric units below that threshold are seconds; for older millisecond dates, use ISO or Date.
• An omitted/null msg.timestamp uses input arrival time. Invalid timestamps are rejected; a triple must always contain an explicit valid timestamp.
• Empty, malformed or invalid arrays are rejected as a whole, before any rows are stored.
• A single CSV row larger than 102,400 bytes is rejected. The array itself can be much larger.

Sending and buffering

Validated input is appended to ring.dat before sending. Appends, POSTs and file compaction are serialized per node, preserving FIFO input order. Large queues (at least 100 KiB) start sending immediately; small batches wait for the flush timer. A drain sends consecutive chunks until the queue is empty or a request fails; there is no flush-interval delay between chunks. Individual CSV rows are never split.

The node reads at most 100 KiB per HTTP request and compacts the remaining file once per drain attempt. It does not rebuild or rewrite the entire remaining queue per POST. On startup, existing disk data is retried without waiting for a new input message. On graceful close/redeploy, in-flight HTTP is cancelled and already queued input is written to disk; unsent data stays there for the next instance.

The server must return HTTP 2xx and either an empty/whitespace body or OK on its first line. Additional lines may contain diagnostics (the deZem test endpoint returns a hex echo). A blank first line followed by an error is not an acknowledgement. Other bodies/statuses, timeouts and connection errors retain data for retry. Redirects are not followed: configure the final upload URL directly. Headers:

Content-Type: text/csv; charset=utf-8; header=absent
User-Agent: IF:Simple TYPE:<SimpleType> SN:<SimpleSn>

Configuration

Each active node needs its own storage path. Sharing a directory within one process is rejected. Do not share a path across separate Node-RED processes either; the ownership guard is process-local. Existing ring.dat files remain compatible. An incomplete trailing row is retained with an error; back up and repair that file before accepting more input, rather than silently merging new rows into corrupt data.

Input validation/storage errors use Node-RED done(error) / Catch nodes. A successful input completion means local processing finished; server failure can still leave data buffered. Runtime delivery failures are also reported independently of Debug Enabled.

Delivery limits

• When the disk cap is exceeded, the oldest complete rows are dropped with a warning. Choose enough capacity for the largest input plus the expected offline backlog.
• Delivery is at least once under successful persistence and eventual recovery, subject to the cap. A lost HTTP acknowledgement, disk error during compaction or crash before compaction can replay already accepted chunks. Server-side deduplication is required if duplicates are unacceptable.
• Persistence uses ordinary filesystem writes, not an fsync transaction. Hardware/power failures are not covered. Input waiting behind an active send remains in process memory until its disk write; a forced process kill can lose that pending input.
• Graceful shutdown preserves pending input, but a forced kill or Node-RED shutdown deadline can interrupt work. Unbounded producer traffic still needs upstream backpressure.
• Large arrays are formatted once in memory; the 100 KiB HTTP cap is not a maximum JavaScript heap limit. A single 10,000/15,000-row message is covered by tests.

Development and tests

Runtime requires Node.js >=14. Tests use Node.js >=18 and real local HTTP and filesystem operations; only the Node-RED node-registration/lifecycle interface is substituted.

npm install
npm test

Tests cover single 10,000/15,000-row arrays, exact CSV/order, retry and partial recovery, overlapping inputs, startup, shutdown, timestamps/values, cap boundaries, errors and redirects. Version 1.5.0 passed 37 automated tests, Node-RED 3.0.2 integration tests, and 10,000/15,000-record tests against deZem's test endpoint with exact CSV echo checks. Echo verification confirms receipt by the test endpoint, not persistent database storage.

License

MIT License. © 2025 deZem GmbH / Emir Dovletyarov.