node-red-contrib-elt 1.0.28

Node-RED nodes for ELT operations with automatic Singer.io plugin management and schema-driven configuration

npm install node-red-contrib-elt

node-red-contrib-elt

A collection of Node-RED nodes for ELT (Extract, Load, Transform) operations using Singer.io taps and targets with automatic plugin management and schema-driven configuration.

Features

• Extractor Node: Extract data from various sources using Singer.io taps
• Loader Node: Load data to various destinations using Singer.io targets
• Automatic Plugin Management: Virtual environments are set up automatically
• Schema-Driven Configuration: Dynamic UI forms based on plugin schemas
• File-Based Communication: Extractor saves results to files, loader reads from them
• Retry Logic: Configurable retry attempts and timeouts
• Visual Status Indicators: Real-time status with attempt counters

Installation

Via Node-RED Palette Manager (Recommended)

1. Open Node-RED in your browser
2. Go to Menu → Manage palette
3. Click Install tab
4. Search for node-red-contrib-elt
5. Click Install

Via Command Line

cd ~/.node-red
npm install node-red-contrib-elt

Quick Start

1. Add Extractor Node:

   • Drag extractor node from the palette
   • Configure plugin (e.g., tap-postgres)
   • Set database connection details

2. Add Loader Node:

   • Drag loader node from the palette
   • Configure plugin (e.g., target-postgres)
   • Set destination database details

3. Connect Nodes:

   • Wire extractor output to loader input
   • Deploy the flow

Available Plugins

Extractors (Taps)

• tap-postgres: Extract from PostgreSQL databases

Loaders (Targets)

• target-csv: Load to CSV files
• target-postgres: Load to PostgreSQL databases

Configuration

Each node provides a dynamic configuration form based on the selected plugin's schema:

• Database Connection: Host, port, database name, credentials
• Retry Settings: Max attempts, timeout values
• Plugin-Specific Options: Varies by plugin type

Environment Setup

The package automatically sets up Python virtual environments with required Singer.io plugins during installation. If you encounter environment issues:

1. Go to Menu → Manage palette
2. Remove the node-red-contrib-elt package
3. Install it again to trigger environment setup

Usage Example

[inject] → [extractor] → [loader] → [debug]

1. Inject Node: Triggers the extraction
2. Extractor Node: Extracts data and saves to file
3. Loader Node: Reads file and loads to destination
4. Debug Node: Shows completion status

Troubleshooting

Environment Issues

• Error: "Environment not ready - reload module from palette"
• Solution: Remove and reinstall the package via palette manager

Configuration Issues

• Error: "No configuration found"
• Solution: Double-click node and configure plugin settings

Connection Issues

• Error: "Failed to connect"
• Solution: Verify database credentials and network connectivity

Requirements

• Node-RED: v2.0.0 or higher
• Node.js: v14.0.0 or higher
• Python: v3.6 or higher
• virtualenvwrapper: For automatic environment management

Support

• Issues: GitHub Issues
• Documentation: See DEVELOPER.md for technical details
• Examples: Check the /examples directory

License

ISC License

Keywords

node-red, elt, etl, data-pipeline, singer, tap-postgres, target-postgres, meltano