node-red-contrib-ibm-igc 0.2.2
Node-RED nodes for integrating with IBM Information Governance Catalog
node-red-contrib-ibm-igc
This module provides a set of nodes in Node-RED for integrating with IBM Information Governance Catalog ("IGC").
Pre-requisites
In addition to Node-RED, the module relies on two other NPM packages: ibm-iis-commons and ibm-igc-rest.
Install
To install the stable version run the following command in your Node-RED user directory (typically ~/.node-red):
npm i node-red-contrib-ibm-igc
Open your Node-RED instance and you should have IGC nodes available in the palette (under storage) and a new config node available once you drag one into a flow.
IGC in
Input can be drawn from Information Governance Catalog in the following ways. This uses both GET- and POST-based Information Governance Catalog REST APIs to retrieve metadata using the most efficient means possible.
In general, the set of results for the retrieval will have a structure of { items: [], paging: {} }, where only the first 10 items retrieved will be included in the initial payload. The paging property provides sufficient detail to retrieve the subsequent page of results each time, and can be passed-on using the URL Passthrough option described below. When only a single (or very few) assets are retrieved, of course, the paging portion may be absent -- so it is worth checking for it via a Function node if you plan on making use of it.
Complex Query
Using the Complex Query option, you can run an arbitrary search. For example, you can pass a Query like this:
{
"properties": ["name"],
"types": ["term"],
"where":
{
"operator": "and",
"conditions": [
{
"property": "modified_on",
"operator": ">=",
"value": "1483232400"
}
]
}
}
to get a listing of the names of all business terms that were modified on or after January 1, 2017 (value provided as UNIX time).
More detailed options on query constructs can be found in the attachments linked to the following tech-notes:
If Query is empty, will look for a query as a JSON object in msg.query.
URL Passthrough
Using the URL Passthrough mode allows you to invoke any arbitrary GET-based Information Governance Catalog REST API URL. For example, you can pass any URL returned by a previous query's next property (for pagination), or the _url property (to get more details on that particular asset).
If URL is empty, will look for a URL as a string in msg.url.
(Note: if specific host details are supplied in the URL they will be overridden by the Server selected via the config node for IGC -- this is necessary to ensure credentials are passed appropriately).
RID
Using the RID option you can retrieve a single asset's details by its unique Information Governance Catalog ID (RID). You can also (optionally) provide a list of specific Properties to retrieve for the asset: in general this is a good practice, as it is more efficient than retrieving all properties. Note that when doing so, specifying the Type of the asset then also becomes mandatory.
If RID is empty, will look for a RID as a string in msg.rid; if Properties is empty, will look for properties as an array in msg.properties; if Type is empty, will look for the type as a string in msg.type.
IGC out
Output can be stored into Information Governance Catalog in the following ways. This uses the appropriate Information Governance Catalog REST API to make updates to metadata, whether creating new assets, updating existing assets, or deleting existing assets. All operations will push their results out to the output port for optional consumption; if any error has occured, an error will be raised with the details -- use a Catch node to capture these.
Create
The Create operation should only be used when an asset does not already exist in IGC with the details that need to be stored. If the asset already exists, and you simply wish to set some additional properties on the asset (or replace existing ones), use the Update operation instead. If you attempt to Create an asset that already exists, an error will result.
Using the Create operation, you need to provide both the Type of the asset to create and the Details of the property values to set within it. Note that different assets have different required properties as part of their creation; in general this will require at least the name property.
The output link will contain the RID of the created asset, as a string.
For example, use a Type of category and provide Details like this:
{
"name": "Test Category"
}
to create a new top-level category named "Test Category".
If Type is empty, will look for the type as a string in msg.type; if Details is empty, will look for the detailed properties as a JSON object in msg.details.
Update
The Update operation expects a unique IGC identifier (RID) to specify which specific piece of metadata to update. Details are also needed to define what on that piece of metadata should be updated. This second parameter is expected to be a JSON object, and should contain only the properties that need to change on the asset.
The output link will be the changed properties of the asset (typically matching the provided Details), as a JSON object.
For example, you can use the following Details:
{
"short_description": "a new short description",
"labels": {
"items": [ "6662c0f2.22257cc4.p864keo17.vbbgrpe.26j4m1.mgcmn0lh4fb2ggdjb7ujo", "6662c0f2.22257cc4.p864keo18.0o97qi8.f1ar57.dmjlsjtjghbifmi6p2tc4" ],
"mode": "add"
}
}
to update the short description of the piece of metadata, and also add (append) two new labels.
Note:
- related items must be specified by their RID
- using
replaceas the mode will overwrite any existing relationships - (if you use
replaceas the mode and provide an empty array, all of those relationships will be removed from that asset)
If RID is empty, will look for the RID as a string in msg.rid; if Details is empty, will look for the detailed properties as a JSON object in msg.details.
Delete
Like the Update operation, the Delete operation requires a unique IGC identifier (RID) to specify which specific piece of metadata to delete. No Details object is needed as the entire asset (and all of its properties) will be deleted.
On successful deletion, the standard output will be empty (just a payload containing only Node-RED's _msgid).
If RID is empty, will look for the RID as a string in msg.rid.
Example flow
The example flow pasted below retrieves key term details (ID, name, short description, long description, and related assets) for all terms published in IGC, iterating through multiple pages of results:
[
{
"id": "8dd27fec.9c3dc",
"type": "tab",
"label": "Get Terms and Related Assets"
},
{
"id": "8ed2d18a.14748",
"type": "ibm-igc",
"z": "",
"host": "yourHostForIGC",
"port": "9445",
"name": "IGC_HOST"
},
{
"id": "e5403847.44885",
"type": "IGC in",
"z": "8dd27fec.9c3dc",
"name": "Search IGC",
"server": "8ed2d18a.14748",
"search": "_query_",
"query": "",
"url": "",
"rid": "",
"ridtype": "",
"ridproperties": "",
"x": 157,
"y": 156,
"wires": [
[
"1adade39.6d3afa"
]
]
},
{
"id": "f9394507.c723c",
"type": "inject",
"z": "8dd27fec.9c3dc",
"name": "Query",
"topic": "",
"payload": "{ \"properties\": [ \"name\", \"short_description\", \"long_description\", \"assigned_assets\" ], \"types\": [ \"term\" ], }",
"payloadType": "json",
"repeat": "",
"crontab": "",
"once": false,
"x": 91,
"y": 65,
"wires": [
[
"cf38780b.47dd28"
]
]
},
{
"id": "1adade39.6d3afa",
"type": "function",
"z": "8dd27fec.9c3dc",
"name": "Split Results",
"func": "return [ msg.items, msg.paging ];",
"outputs": "2",
"noerr": 0,
"x": 284,
"y": 203,
"wires": [
[
"ec20d121.4fd68"
],
[
"809e88cc.d933f8"
]
]
},
{
"id": "809e88cc.d933f8",
"type": "function",
"z": "8dd27fec.9c3dc",
"name": "Next page?",
"func": "if (msg.hasOwnProperty('next')) {\n const payload = {};\n payload.url = msg.next;\n node.send(payload);\n}\n",
"outputs": 1,
"noerr": 0,
"x": 459,
"y": 223,
"wires": [
[
"235b3e32.ac6d22"
]
]
},
{
"id": "ec20d121.4fd68",
"type": "function",
"z": "8dd27fec.9c3dc",
"name": "Parse details",
"func": "const payload = {};\n\npayload.id = msg._id;\npayload.name = msg.name;\npayload.short_desc = msg.hasOwnProperty('short_description') ? msg.short_description : \"\";\npayload.long_desc = msg.hasOwnProperty('long_description') ? msg.long_description : \"\";\npayload.assigned_assets = [];\n\nif (msg.hasOwnProperty('assigned_assets')) {\n for (let i = 0; i < msg.assigned_assets.items.length; i++) {\n const itm = msg.assigned_assets.items[i];\n payload.assigned_assets.push({ \"type\": itm._type, \"name\": itm._name, \"id\": itm._id });\n }\n}\n\nreturn { payload: payload };",
"outputs": 1,
"noerr": 0,
"x": 460,
"y": 179,
"wires": [
[
"df84cb0.eb09038"
]
]
},
{
"id": "df84cb0.eb09038",
"type": "file",
"z": "8dd27fec.9c3dc",
"name": "",
"filename": "TermsAndRelationships.json",
"appendNewline": true,
"createDir": false,
"overwriteFile": "false",
"x": 682,
"y": 179,
"wires": []
},
{
"id": "235b3e32.ac6d22",
"type": "IGC in",
"z": "8dd27fec.9c3dc",
"name": "Get next page",
"server": "8ed2d18a.14748",
"search": "_url_",
"query": "",
"url": "",
"rid": "",
"ridtype": "",
"ridproperties": "",
"x": 352,
"y": 298,
"wires": [
[
"1adade39.6d3afa"
]
]
},
{
"id": "cf38780b.47dd28",
"type": "change",
"z": "8dd27fec.9c3dc",
"name": "",
"rules": [
{
"t": "set",
"p": "query",
"pt": "msg",
"to": "payload",
"tot": "msg"
}
],
"action": "",
"property": "",
"from": "",
"to": "",
"reg": false,
"x": 157,
"y": 109,
"wires": [
[
"e5403847.44885"
]
]
}
]
Note that you'll need to open the "Configuration nodes" from Node-RED's UI menu and double-click the "IGC_HOST" configuration under "On all flows" to change the details to your own -- hostname, port, username and password. There will need to be some form of network connectivity from where Node-RED is running to the host and port you define there in order for the nodes to work.
Other potentially useful resources
For exploring the REST APIs offered by IBM Information Governance Catalog, you might also want to explore the ibm-igc-postman project containing collections for Postman, to experiment further with the various REST API end-points. And as indicated earlier (linking again for convenience):
