Thermal Image sensor experiment as presence detector
Thermal Image sensor experiment as presence detector
This was a "failed" experiment project to use an AMG8833 sensor, ESP32, MQTT and Node-Red to use as a presence detector in the door. Since PIR sensors can only tell movement, I wanted to experiment if a low definition (AGM8833 is a 8x8 pixel thermal sensor) thermal image sensor can be used to tell if somebody is the room, stays in various parts of the room etc. It is a failed project, because normal body temperature pretty much fades into background noise after about 2 meters. It can certainly tell presence in closer distance, but it is not suitable for an entire room. The sensor is relatively expensive (cca. $50 on Banggood) so it is no viable to map a room with multiple sensors.
What I completed in this project:
- ESP32 sketch reads the image sensor every second and sends the "image" data over MQTT
- ESP32 is also connected to a BME680 environment sensor (temperature, humidity, pressure, voletile gas compound) which is also send over MQTT every 10 seconds
- MQTT data is read by Node-Red and the thermal image is displayed on the Dashboard
- Rules can be defined that are evaluated for each image e.g. max or average temperature on the image, or part of the image.
Useful links
- Youtube video explaining this entire project: https://youtu.be/o97A5IZ9Piw
- Arduino sketch on github: https://github.com/nygma2004/amg8833_thermal_imager
- AMG8833 sensor from Banggood: https://www.banggood.com/CJMCU-833-AMG8833-8x8-Thermal-Camera-IR-Infrared-Array-Thermal-Imaging-Sensor-Board-p-1268056.html
- BME680 sensor: https://www.aliexpress.com/item/32961369966.html
How does it looks like in Node-Red
The below screenshot shows an example of the Node-Red flow.
On the left side the actual thermal image is shown using a blue-red colour palette. That is getting auto scaled according to the measures values. The individual pixels show the exact measured values as well. On the middle the rule engine evaluates the different scenarios and provides a result based on the current image. "Am I at the desk" rule looks at the top right 4x4 pixels and checks if the average temperature is above 27C.
[{"id":"948243a3.6a4c2","type":"tab","label":"Thermal Image","disabled":false,"info":""},{"id":"c0f81119.2fa26","type":"mqtt in","z":"948243a3.6a4c2","name":"","topic":"/thermal/status","qos":"0","datatype":"auto","broker":"cea5258a.b34038","x":800,"y":140,"wires":[["250280cb.2d764"]]},{"id":"6ca67c67.5158f4","type":"debug","z":"948243a3.6a4c2","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":1190,"y":140,"wires":[]},{"id":"305b5b3.715b4a4","type":"mqtt in","z":"948243a3.6a4c2","name":"","topic":"/thermal/bme","qos":"0","datatype":"auto","broker":"cea5258a.b34038","x":110,"y":140,"wires":[["8b04d5df.c30948"]]},{"id":"174906f2.bea8d9","type":"debug","z":"948243a3.6a4c2","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":490,"y":140,"wires":[]},{"id":"7f0330d.502b7d","type":"mqtt in","z":"948243a3.6a4c2","name":"","topic":"/thermal/thermal","qos":"0","datatype":"auto","broker":"cea5258a.b34038","x":120,"y":400,"wires":[["5ff1fcf0.3046e4","cb9daecb.6bbc4","caf0a6e1.b96d88"]]},{"id":"5ff1fcf0.3046e4","type":"debug","z":"948243a3.6a4c2","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":750,"y":400,"wires":[]},{"id":"250280cb.2d764","type":"json","z":"948243a3.6a4c2","name":"","property":"payload","action":"","pretty":false,"x":970,"y":140,"wires":[["6ca67c67.5158f4","eed49414.163348","bc13ee80.e8f6a"]]},{"id":"8b04d5df.c30948","type":"json","z":"948243a3.6a4c2","name":"","property":"payload","action":"","pretty":false,"x":290,"y":140,"wires":[["174906f2.bea8d9","4db792fa.d7f1dc","200d202d.a95fc","92289ab8.1b78b8","f9b556b5.d5f058"]]},{"id":"cb9daecb.6bbc4","type":"function","z":"948243a3.6a4c2","name":"Render image","func":"let camColors = [\"#0a00bb\",\"#0b00ba\",\"#0c00ba\",\"#0d00b9\",\"#0e00b8\",\"#0f00b7\",\"#1000b7\",\"#1100b6\",\"#1200b5\",\"#1300b4\",\"#1400b4\",\"#1500b3\",\"#1600b2\",\"#1700b1\",\"#1800b1\",\"#1800b0\",\"#1900af\",\"#1a00ae\",\"#1b00ae\",\"#1c00ad\",\"#1d00ac\",\"#1e00ac\",\"#1f00ab\",\"#2000aa\",\"#2100a9\",\"#2200a9\",\"#2300a8\",\"#2400a7\",\"#2500a6\",\"#2600a6\",\"#2700a5\",\"#2800a4\",\"#2900a3\",\"#2a00a3\",\"#2b00a2\",\"#2c00a1\",\"#2d00a0\",\"#2e00a0\",\"#2f009f\",\"#30009e\",\"#31009e\",\"#32009d\",\"#33009c\",\"#33009b\",\"#34009b\",\"#35009a\",\"#360099\",\"#370098\",\"#380098\",\"#390097\",\"#3a0096\",\"#3b0095\",\"#3c0095\",\"#3d0094\",\"#3e0093\",\"#3f0093\",\"#400092\",\"#410091\",\"#420090\",\"#430090\",\"#44008f\",\"#45008e\",\"#46008d\",\"#47008d\",\"#48008c\",\"#49008b\",\"#4a008a\",\"#4b008a\",\"#4c0089\",\"#4d0088\",\"#4e0087\",\"#4e0087\",\"#4f0086\",\"#500085\",\"#510085\",\"#520084\",\"#530083\",\"#540082\",\"#550082\",\"#560081\",\"#570080\",\"#58007f\",\"#59007f\",\"#5a007e\",\"#5b007d\",\"#5c007c\",\"#5d007c\",\"#5e007b\",\"#5f007a\",\"#600079\",\"#610079\",\"#620078\",\"#630077\",\"#640077\",\"#650076\",\"#660075\",\"#670074\",\"#680074\",\"#690073\",\"#690072\",\"#6a0071\",\"#6b0071\",\"#6c0070\",\"#6d006f\",\"#6e006e\",\"#6f006e\",\"#70006d\",\"#71006c\",\"#72006b\",\"#73006b\",\"#74006a\",\"#750069\",\"#760069\",\"#770068\",\"#780067\",\"#790066\",\"#7a0066\",\"#7b0065\",\"#7c0064\",\"#7d0063\",\"#7e0063\",\"#7f0062\",\"#800061\",\"#810060\",\"#820060\",\"#83005f\",\"#84005e\",\"#85005e\",\"#85005d\",\"#86005c\",\"#87005b\",\"#88005b\",\"#89005a\",\"#8a0059\",\"#8b0058\",\"#8c0058\",\"#8d0057\",\"#8e0056\",\"#8f0055\",\"#900055\",\"#910054\",\"#920053\",\"#930052\",\"#940052\",\"#950051\",\"#960050\",\"#970050\",\"#98004f\",\"#99004e\",\"#9a004d\",\"#9b004d\",\"#9c004c\",\"#9d004b\",\"#9e004a\",\"#9f004a\",\"#a00049\",\"#a00048\",\"#a10047\",\"#a20047\",\"#a30046\",\"#a40045\",\"#a50044\",\"#a60044\",\"#a70043\",\"#a80042\",\"#a90042\",\"#aa0041\",\"#ab0040\",\"#ac003f\",\"#ad003f\",\"#ae003e\",\"#af003d\",\"#b0003c\",\"#b1003c\",\"#b2003b\",\"#b3003a\",\"#b40039\",\"#b50039\",\"#b60038\",\"#b70037\",\"#b80036\",\"#b90036\",\"#ba0035\",\"#bb0034\",\"#bb0034\",\"#bc0033\",\"#bd0032\",\"#be0031\",\"#bf0031\",\"#c00030\",\"#c1002f\",\"#c2002e\",\"#c3002e\",\"#c4002d\",\"#c5002c\",\"#c6002b\",\"#c7002b\",\"#c8002a\",\"#c90029\",\"#ca0028\",\"#cb0028\",\"#cc0027\",\"#cd0026\",\"#ce0026\",\"#cf0025\",\"#d00024\",\"#d10023\",\"#d20023\",\"#d30022\",\"#d40021\",\"#d50020\",\"#d60020\",\"#d6001f\",\"#d7001e\",\"#d8001d\",\"#d9001d\",\"#da001c\",\"#db001b\",\"#dc001b\",\"#dd001a\",\"#de0019\",\"#df0018\",\"#e00018\",\"#e10017\",\"#e20016\",\"#e30015\",\"#e40015\",\"#e50014\",\"#e60013\",\"#e70012\",\"#e80012\",\"#e90011\",\"#ea0010\",\"#eb000f\",\"#ec000f\",\"#ed000e\",\"#ee000d\",\"#ef000d\",\"#f0000c\",\"#f1000b\",\"#f1000a\",\"#f2000a\",\"#f30009\",\"#f40008\",\"#f50007\",\"#f60007\",\"#f70006\",\"#f80005\",\"#f90004\",\"#fa0004\",\"#fb0003\",\"#fc0002\",\"#fd0001\",\"#fe0001\",\"#ff0000\"];\nlet MINTEMP = 20.0;\nlet MAXTEMP = 35.0;\n\n\nlet html = \"<table border=\\\"0\\\" padding=\\\"0\\\" spacing=\\\"0\\\">\";\n\nlet image = msg.payload.split(\",\");\n\n// auto calculate min and max\nMINTEMP = 200.0;\nMAXTEMP = 0.0;\nfor (let k=0;k<64;k++) {\n let value = parseFloat(image[k]);\n if (value<MINTEMP) { MINTEMP = value; }\n if (value>MAXTEMP) { MAXTEMP = value; }\n}\nif (MAXTEMP-MINTEMP<10) { MAXTEMP = MINTEMP + 10.0; }\n\nfor (let i=0;i<8;i++) {\n html = html + \"<tr>\"\n for (let j=0;j<8;j++) {\n let mycolor = parseFloat(image[i*8+j]);\n let temp = image[i*8+j];\n if (mycolor<MINTEMP) { mycolor = MINTEMP; }\n if (mycolor>MAXTEMP) { mycolor = MAXTEMP; }\n mycolor = Math.floor(255*(mycolor-MINTEMP)/(MAXTEMP-MINTEMP));\n //html = html + mycolor + \",\";\n if (mycolor>253) { mycolor = 253; }\n mycolor = camColors[mycolor];\n html = html + \"<td bgcolor=\\\"\"+mycolor+\"\\\" width=\\\"40px\\\" height=\\\"40px\\\" align=\\\"center\\\" valign=\\\"middle\\\"><font size=\\\"2\\\" color= \\\"#CCCCCC\\\">\"+temp+\"</font></td>\"\n }\n html = html + \"</tr>\"\n}\n\nhtml = html + \"</table>\"\n\nmsg.payload = html;\n\nreturn [msg,{\"payload\":MINTEMP},{\"payload\":MAXTEMP}];","outputs":3,"noerr":0,"initialize":"","finalize":"","x":400,"y":460,"wires":[["a3c4087e.bc2c28","ce18a3aa.bdd22"],["cd5cd1d9.5125e"],["ebfecb70.d74198"]]},{"id":"a3c4087e.bc2c28","type":"debug","z":"948243a3.6a4c2","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":650,"y":460,"wires":[]},{"id":"ce18a3aa.bdd22","type":"ui_template","z":"948243a3.6a4c2","group":"38a41d52.5ca782","name":"","order":2,"width":"10","height":"10","format":"<div ng-bind-html=\"msg.payload\"></div>","storeOutMessages":true,"fwdInMessages":true,"templateScope":"local","x":660,"y":520,"wires":[[]]},{"id":"cd5cd1d9.5125e","type":"ui_text","z":"948243a3.6a4c2","group":"38a41d52.5ca782","order":1,"width":0,"height":0,"name":"","label":"Min Temp","format":"{{msg.payload}}","layout":"row-spread","x":660,"y":560,"wires":[]},{"id":"ebfecb70.d74198","type":"ui_text","z":"948243a3.6a4c2","group":"38a41d52.5ca782","order":1,"width":0,"height":0,"name":"","label":"Max Temp","format":"{{msg.payload}}","layout":"row-spread","x":670,"y":600,"wires":[]},{"id":"dd6ed6a2.6c4e68","type":"inject","z":"948243a3.6a4c2","name":"Max Temp > 32","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"setrule","payload":"{\"name\":\"maxtemp\",\"aggregation\":\"max\",\"value\":32,\"weight\":[[1,1,1,1,1,1,1,1],[1,1,1,1,1,1,1,1],[1,1,1,1,1,1,1,1],[1,1,1,1,1,1,1,1],[1,1,1,1,1,1,1,1],[1,1,1,1,1,1,1,1],[1,1,1,1,1,1,1,1],[1,1,1,1,1,1,1,1]]}","payloadType":"json","x":130,"y":720,"wires":[["caf0a6e1.b96d88"]]},{"id":"caf0a6e1.b96d88","type":"function","z":"948243a3.6a4c2","name":"Rule Engine","func":"if (msg.topic===\"setrule\") {\n context.set(\"rule_\"+msg.payload.name,msg.payload);\n node.status({fill:\"green\",shape:\"dot\",text:\"Rule stored\"});\n return;\n}\n\n\n// Run through the rules\nlet rulearray = context.keys();\nlet output = [];\nlet image = msg.payload.split(\",\");\nlet rulecount = 0;\n\n// Cycle through the context values, all rule are stored as \"rule_\"\nfor (let i=0;i<rulearray.length;i++) {\n // Check if this is a rule variable that need to be evaludated\n if (rulearray[i].substring(0,5)===\"rule_\") {\n // get the actual rule stored in the context\n let thisrule = context.get(rulearray[i]);\n // remove the \"rule_\" from the beginning\n rulearray[i]=rulearray[i].substring(5,rulearray[i].length);\n let response = {\"topic\": rulearray[i], \"payload\": {}};\n response.payload.name=rulearray[i];\n response.payload.result = false;\n rulecount++;\n let calc = 0;\n response.payload.debug=[];\n response.payload.treshold=thisrule.value;\n response.payload.value=0;\n response.payload.count=0;\n for (let row=0;row<8;row++) {\n for (let col=0;col<8;col++) {\n let mycell = parseFloat(image[row*8+col])*thisrule.weight[row][col];\n response.payload.debug.push(mycell);\n // Run calculation if the aggregation rule is max\n if (thisrule.aggregation===\"max\") {\n if (mycell>thisrule.value) {\n response.payload.result = true;\n response.payload.value=mycell;\n }\n }\n // Run calculation if the aggregation rule is average\n if (thisrule.aggregation===\"average\") {\n if (thisrule.weight[row][col]>0) {\n response.payload.value = response.payload.value+mycell;\n response.payload.count++;\n }\n }\n } // column loop\n } // row loop\n // Do final calculation if the aggregation rule is average\n if (thisrule.aggregation===\"average\") {\n response.payload.value = 0.0+response.payload.value/response.payload.count;\n response.payload.result = response.payload.value > thisrule.value;\n } \n output.push(response);\n }\n}\nnode.status({fill:\"green\",shape:\"ring\",text:\" \"+rulecount+\" rule(s) evaluated\"});\nreturn [output];\n","outputs":1,"noerr":0,"initialize":"","finalize":"","x":410,"y":720,"wires":[["22e3a623.d55f2a","cd8f1d51.69d65"]]},{"id":"22e3a623.d55f2a","type":"debug","z":"948243a3.6a4c2","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":750,"y":700,"wires":[]},{"id":"251c1e5a.8b88f2","type":"inject","z":"948243a3.6a4c2","name":"Corner > 30","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"setrule","payload":"{\"name\":\"topleftcorner\",\"aggregation\":\"max\",\"value\":30,\"weight\":[[1,1,1,0,0,0,0,0],[0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0]]}","payloadType":"json","x":130,"y":760,"wires":[["caf0a6e1.b96d88"]]},{"id":"cd8f1d51.69d65","type":"switch","z":"948243a3.6a4c2","name":"","property":"topic","propertyType":"msg","rules":[{"t":"eq","v":"maxtemp","vt":"str"},{"t":"eq","v":"topleftcorner","vt":"str"},{"t":"eq","v":"atdesk","vt":"str"}],"checkall":"true","repair":false,"outputs":3,"x":590,"y":780,"wires":[["d85ea9b7.8d1278"],["e6615ebc.b3d11"],["11bb4b1d.3fe185"]]},{"id":"d85ea9b7.8d1278","type":"ui_template","z":"948243a3.6a4c2","group":"2e00d8e7.308948","name":"MaxTemp","order":8,"width":"","height":"","format":"<div layout=\"row\" layout-align=\"space-between\">\n <p>Max Temp > 32</p>\n <p ng-style=\"{color: msg.payload.result ? 'green' : 'grey'}\">\n <b>{{msg.payload.result ? 'Yes' : 'No'}} ({{msg.payload.value}})</b>\n </p>\n</div>","storeOutMessages":false,"fwdInMessages":false,"resendOnRefresh":false,"templateScope":"local","x":780,"y":760,"wires":[[]]},{"id":"e6615ebc.b3d11","type":"ui_template","z":"948243a3.6a4c2","group":"2e00d8e7.308948","name":"TopLeftCorner","order":8,"width":"","height":"","format":"<div layout=\"row\" layout-align=\"space-between\">\n <p>Top Left Corner > 30</p>\n <p ng-style=\"{color: msg.payload.result ? 'green' : 'grey'}\">\n <b>{{msg.payload.result ? 'Yes' : 'No'}} ({{msg.payload.value}})</b>\n </p>\n</div>","storeOutMessages":false,"fwdInMessages":false,"resendOnRefresh":false,"templateScope":"local","x":800,"y":800,"wires":[[]]},{"id":"f1fc2bcb.c4f418","type":"inject","z":"948243a3.6a4c2","name":"At the desk?","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"setrule","payload":"{\"name\":\"atdesk\",\"aggregation\":\"average\",\"value\":27,\"weight\":[[0,0,0,0,1,1,1,1],[0,0,0,0,1,1,1,1],[0,0,0,0,1,1,1,1],[0,0,0,0,1,1,1,1],[0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0]]}","payloadType":"json","x":130,"y":800,"wires":[["caf0a6e1.b96d88"]]},{"id":"11bb4b1d.3fe185","type":"ui_template","z":"948243a3.6a4c2","group":"2e00d8e7.308948","name":"At Desk","order":8,"width":"","height":"","format":"<div layout=\"row\" layout-align=\"space-between\">\n <p>Am I at the desk?</p>\n <p ng-style=\"{color: msg.payload.result ? 'green' : 'grey'}\">\n <b>{{msg.payload.result ? 'Yes' : 'No'}} ({{msg.payload.value}})</b>\n </p>\n</div>","storeOutMessages":false,"fwdInMessages":false,"resendOnRefresh":false,"templateScope":"local","x":780,"y":840,"wires":[[]]},{"id":"eed49414.163348","type":"ui_text","z":"948243a3.6a4c2","group":"edc77171.40e6d","order":0,"width":0,"height":0,"name":"","label":"RSSI","format":"{{msg.payload.rssi}}","layout":"row-spread","x":1190,"y":240,"wires":[]},{"id":"bc13ee80.e8f6a","type":"ui_text","z":"948243a3.6a4c2","group":"edc77171.40e6d","order":0,"width":0,"height":0,"name":"","label":"Uptime [min]","format":"{{msg.payload.uptime}}","layout":"row-spread","x":1210,"y":280,"wires":[]},{"id":"4db792fa.d7f1dc","type":"ui_text","z":"948243a3.6a4c2","group":"a41754f8.9c73b8","order":0,"width":0,"height":0,"name":"","label":"Temperature","format":"{{msg.payload.temperature}} °C","layout":"row-spread","x":550,"y":200,"wires":[]},{"id":"200d202d.a95fc","type":"ui_text","z":"948243a3.6a4c2","group":"a41754f8.9c73b8","order":0,"width":0,"height":0,"name":"","label":"Humidity","format":"{{msg.payload.humidity}} %","layout":"row-spread","x":540,"y":240,"wires":[]},{"id":"92289ab8.1b78b8","type":"ui_text","z":"948243a3.6a4c2","group":"a41754f8.9c73b8","order":0,"width":0,"height":0,"name":"","label":"Pressure","format":"{{msg.payload.pressure}} hPa","layout":"row-spread","x":540,"y":280,"wires":[]},{"id":"f9b556b5.d5f058","type":"ui_text","z":"948243a3.6a4c2","group":"a41754f8.9c73b8","order":0,"width":0,"height":0,"name":"","label":"VOC","format":"{{msg.payload.gas}} kOhms","layout":"row-spread","x":530,"y":320,"wires":[]},{"id":"cea5258a.b34038","type":"mqtt-broker","z":"","broker":"192.168.1.80","port":"1883","clientid":"node-red","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","willTopic":"","willQos":"0","willPayload":""},{"id":"38a41d52.5ca782","type":"ui_group","z":"","name":"Thermal Image","tab":"dd1d391f.548498","order":1,"disp":true,"width":"10","collapse":false},{"id":"2e00d8e7.308948","type":"ui_group","z":"","name":"Rule Evaluation Results","tab":"dd1d391f.548498","order":2,"disp":true,"width":"6","collapse":false},{"id":"edc77171.40e6d","type":"ui_group","z":"","name":"Device Status","tab":"dd1d391f.548498","order":3,"disp":true,"width":"6","collapse":false},{"id":"a41754f8.9c73b8","type":"ui_group","z":"","name":"Environment","tab":"dd1d391f.548498","order":4,"disp":true,"width":"6","collapse":false},{"id":"dd1d391f.548498","type":"ui_tab","z":"","name":"Thermal Imaging","icon":"image","disabled":false,"hidden":false}]
