Some Node Red pointers would be appreciated

This is all @PeteKnight 's fault :laughing: After reading his posts I am not addicted to Node Red and spending far too much time looking at it!

I’m just after some general ‘good practice’ and hints for a newcomer, as I can see it can become quite complex. Current thoughts:

  1. Flow tabs - is it best to group flows into categories? (e.g. areas of home, lighting, power etc…)

  2. I want to use Blynk, Node Red Dashboard and Siri to control most things (I always have my apple phone or watch on me). I’m using " node-red-contrib-homekit-bridged 0.6.2" which works, although node red needs restarting after deploying due to a known bug.

I want each of the three systems to update the state of the other ones (remain sync’d). Is there a standard way of doing this? I read about a rbe node which I will look at.

Apologies if the question is not well formulated, it’s still all cooking in my brain :smile:

Here’s what I have so far:

Garage heater (TPLink HS100 smart socket):

Garage outside lights: (Sonoff Basic with Tasmota firmware)

Also got a BME280 connecting to a Wemos D1 mini, sending temp/humidity/pressure via MQTT to the Node Red Dashboard:

Thanks!

Hi @877, I love it when we have another convert!

Question 1
I think it’s about personal preference as much as anything. The same applies to coding, and if you use tabs in Arduino then you could adopt a similar approach to the way that you use the tabs in Node-Red.
The same applies to how you choose to use tabs in Blynk to arrange your controls.

I like to have my Node-Red tabs laid-out in a way that I can see all of my flow without having the view zoomed-out so far that I can’t make-out the details.
I tend to have one device per tab, but if I have multiple devices in the same room, or following the same rules (lights come on at dusk and go off at midnight lets say) then I’ll group them one one tab.

You can connect flows together using the Link nodes if you wish, or you can also use global variables that are visible across all flows.

Question 2
First of all, I spent a lot of time messing around with the Blynk dashboard and eventually decided that it wasn’t really worth the effort.
Keeping all of your Node-Red and Blynk controls synchronised, as well as, keeping them synced with commands from physical switches and Alexa-controlled commands makes the flows far too complex for my liking and the dashboard doesn’t give you anything that cant be done better in Blynk.
My initial use for the Node-Red dashboard was in case Blynk was down (I use the cloud server). In this scenario it would give me another way of controlling my stuff. However, a simple Inject node does the same thing, and I’ve never yet had a problem with Blynk being down anyway.

It is quite easy to get into a looping scenario. It looks like this could cause repetitive toggling of the values:
image

One way around this is to use the RBE node, which allows you to disregard values unless they change, or throw-away values unless they are significantly different from the previous one.

It’s also worth looking in to the node.status functionality when you write your own functions. This is what puts the little messages below the node and can be a really useful debugging tool, as well as allowing yo to see at a glance what the status of your flow is.

Hope this helps, at least a little bit…

Pete.

1 Like

Thank you, I have to say it was a lot to absorb, I still find it a little overwhelming at times.

But it’s exciting the amount of control it can give, and the flexibility of design.

I think you meant to say you decided ‘Node red’ dashboard wasn’t worth the effort? I thought it looks quite good at first impressions, I was thinking of using an old iPad mini as an always on display, showing weather etc. Swiping would show the garage and status of heater, lights, alarm etc… I’m interested as to why you abandoned it.

I have seen the node.status on @scargill ‘s blog originally, it does look very good. That’s my next thing is to add the bigtimer :slight_smile:

Thanks

Yes!

As I said earlier, I think Blynk is better in every way - more configurable, more functionality, nicer looking etc - and trying to keep Node-Red and Blynk controls synchronised just isn’t worth the effort.
Why not run Blynk on the iPad mini instead?

Yes, I love BigTimer. I also use Pete’s Timeout Node quite a lot.

The one problem with using the Blynk ws plug-in for Node-Red is that the devices in Blynk only show as being offline in the app if your Node-Red session is down. This is because it’s Node-Red that’s maintaining the heartbeat communication with the Blynk server, so you never know if a slave device on your system, is off-line.

I get around this by adding a ‘Hearbeat’ MQTT message from each device. I get the devices to send their current RSSI signal strength reading to Node-Red via an MQTT message every 5 seconds or so. I then connect this incoming MQTT message to a Timeout node like this:

The Timeout node is configured like this:

This will turn my LED on (255) if we’re getting heartbeat messages and off (0) if not). You can also take another branch off of the incoming RSSI and send it out to a Blynk display widget if you want to see the actual RSSI value.

Pete.

1 Like

Thanks Pete,

I appreciate the advice and help. I was thinking about that last part, the devices being offline. It bothered me that node red is happily sending commands, meanwhile the actual devices could be offline. I will look at integrating the heartbeat code, I’ve never used RSSI but a search provides the info : https://www.arduino.cc/en/Reference/WiFiRSSI

Cheers :+1:

There is also another way of telling when a device is disconnected. Most examples of how to use PubSubClient use a simple connection string like this, followed by a “hello world” notification:

    if (client.connect(clientId.c_str())) {
      Serial.println("connected");
      // Once connected, publish an announcement...
      client.publish("outTopic", "hello world");

If you delve into the docs for PubSubClient you’ll find that you can actually do quite a lot more. This is what the documentation says:

  MQTT Connection syntax:
  boolean connect (client_id, username, password, willTopic, willQoS, willRetain, willMessage)
  Connects the client with a Will message, username and password specified.
  Parameters
    client_id : the client ID to use when connecting to the server.
    username : the username to use. If NULL, no username or password is used (const char[])
    password : the password to use. If NULL, no password is used (const char[])
    willTopic : the topic to be used by the will message (const char[])
    willQoS : the quality of service to be used by the will message (int : 0,1 or 2)
    willRetain : whether the will should be published with the retain flag (int : 0 or 1)
    willMessage : the payload of the will message (const char[])
  Returns
    false - connection failed.
    true - connection succeeded

The “Last Will and Testament” functionality of MQTT is how the server can tell you if a device has disconnected. By using these ‘willTopic’ and ‘willMessage’ parameters in the connection string you can replace the simple “hello world” message that never changes to a message that will say “Alive” when its online and “Dead” when it’s offline (or whatever alternative words you choose to use).

This is my alternative to the standard connection string above…

  if(MQTTclient.connect(mqtt_client_id.c_str(), mqtt_username, mqtt_password, (base_mqtt_topic + "/Status").c_str(),0, 1, "Dead"))
  {
    // We get here if the connection was successful... 
    Serial.println(F("MQTT Connected"));
    mqtt_connect_count++;

    // Once connected, publish some announcements...
    // These all have the Retained flag set to true, so that the value is stored on the server and can be retrieved at any point
    // Check the .../Status topic to see that the device is still online before relying on the data from these retained topics
    MQTTclient.publish((base_mqtt_topic + "/MQTT_Client_ID").c_str(),mqtt_client_id.c_str(),true);   
    MQTTclient.publish((base_mqtt_topic + "/Status").c_str(),"Alive",true);

Pete.

1 Like

Thanks @PeteKnight for the information, I must admit it’s a lot to take in and I have been dipping in and out trying to understand it.

I thought I would start with the ‘simpler’ heartbeat setup. Here is my original code for my lounge BME280 sensor:

//MQTT lounge weather station v001
//
//Hardware: Wemos D1 mini, BME280 sensor wired as I2C, Raspberry Pi 3 node red and moquitto MQTT server.
//Description: BME temp, humidity and pressure readings are published to the MQTT server (also printed to serial).
//Only updated values are printed, OTA support included.
//
// Code based on https://www.home-assistant.io/blog/2015/10/11/measure-temperature-with-esp8266-and-report-to-mqtt/
//
// Add support for OTA***************************************
#include <ArduinoOTA.h>
// **********************************************************
#include <ESP8266WiFi.h>
#include <Wire.h>
#include <PubSubClient.h>
#include <BME280I2C.h>

#define wifi_ssid ""
#define wifi_password ""

#define mqtt_server "192.168.1.xx"
#define mqtt_user "admin"
#define mqtt_password ""

#define temperature_topic "house/lounge/sensor/temperature"
#define humidity_topic "house/lounge/sensor/humidity"
#define pressure_topic "house/lounge/sensor/pressure"

WiFiClient espClient;
PubSubClient client(espClient);

BME280I2C bme;    // Default : forced mode, standby time = 1000 ms
                  // Oversampling = pressure ×1, temperature ×1, humidity ×1, filter off,



void setup() {

    // Add support for OTA***************************************
  ArduinoOTA.onError([](ota_error_t error) { ESP.restart(); });
  ArduinoOTA.setHostname("Wemos no.3 MQTT Lounge weather");  
  ArduinoOTA.begin();  /* setup the OTA server */
  // **********************************************************
  
  Serial.begin(115200);
  setup_wifi();
  client.setServer(mqtt_server, 1883);

  while(!Serial) {} // Wait

  Wire.begin();

  while(!bme.begin())
  {
    Serial.println("Could not find BME280 sensor!");
    delay(1000);
  }

  // bme.chipID(); // Deprecated. See chipModel().
  switch(bme.chipModel())
  {
     case BME280::ChipModel_BME280:
       Serial.println("Found BME280 sensor! Success.");
       break;
     case BME280::ChipModel_BMP280:
       Serial.println("Found BMP280 sensor! No Humidity available.");
       break;
     default:
       Serial.println("Found UNKNOWN sensor! Error!");
  }
}

void setup_wifi() {
  delay(10);
  // We start by connecting to a WiFi network
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(wifi_ssid);

  WiFi.begin(wifi_ssid, wifi_password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}


void reconnect() {
  // Loop until we're reconnected
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Attempt to connect
    // If you do not want to use a username and password, change next line to
    // if (client.connect("ESP8266Client")) {
    if (client.connect("ESP8266Client", mqtt_user, mqtt_password)) {
      Serial.println("connected");
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      // Wait 5 seconds before retrying
      delay(5000);
    }
  }
}



bool checkBound(float newValue, float prevValue, float maxDiff)
{
  return !isnan(newValue) &&
         (newValue < prevValue - maxDiff || newValue > prevValue + maxDiff);
}

long lastMsg = 0;
float oldtemp = 0.0;  //set a float to get decimal places
int oldhum = 0.0;  //set as int to get whole numbers only
int oldpres = 0;  // NEW
float diff = 1.0;



void loop() {

  // Add support for OTA***************************************
  ArduinoOTA.handle();
  // **********************************************************
  
  if (!client.connected()) {
    reconnect();
  }
  client.loop();

  long now = millis();
  if (now - lastMsg > 1000) {
    lastMsg = now;

    float newTemp = bme.temp();
    int newHum = bme.hum();
    int newPres = bme.pres();  // NEW

    if (checkBound(newTemp, oldtemp, diff)) {
      oldtemp = newTemp;
      Serial.print("New temperature:");
      Serial.println(String(oldtemp).c_str());
      client.publish(temperature_topic, String(oldtemp ,1).c_str(), true);
    }

    if (checkBound(newHum, oldhum, diff)) {
      oldhum = newHum;
      Serial.print("New humidity:");
      Serial.println(String(oldhum).c_str());
      client.publish(humidity_topic, String(oldhum).c_str(), true);
    }
    
    if (checkBound(newPres, oldpres, diff)) {
      oldpres = newPres;
      Serial.print("New pressure:");
      Serial.println(String(oldpres/100).c_str());
      client.publish(pressure_topic, String(oldpres/100).c_str(), true);
    }
    
  }
}

To add heartbeat, I have added this code:

#define heartbeat_topic "house/lounge/sensor/heartbeat"  // setup heartbeat topic
long rssi   //setup variable to store RSSI value

void loop()
rssi =  WiFi.RSSI();
Serial.print("RSSI:");
Serial.println(rssi);
client.publish(heartbeat_topic, String(rssi).c_str(), true);

Which is working and sending the RSSI to node red (confirmed with debug node).

Questions:
I think the publishing of heartbeat rssi should not go in the loop, as it will spam the MQTT server many times per second, I was thinking of using a timer function - correct?

Would you be able to advise on the correct layout if I’m not asking too much? :blush:

Bonus question: the above code only sends the updated BME280 values to MQTT when they change from the last value, is their an easier way to manage this in node red?

EDIT - updated post as I fixed the original problem just after posting - typical :blush:

I use SimpleTimer and put the RSSI code in a function (maybe call it Heartbeat) that’s called every 5 seconds.
It’s just the same as you would do in. Blynk - define the timer object, initiate the timer instance in void setup and add timer.run to void loop.

I tend to send things like temperature every 5 seconds regardless of whether there’s been a change. You can push this to a Blynk virtual pin in Node-Red if you want, and these will be averaged by Blynk to give a 1 minute average (the minimum granularity).

If you only want to do something if a value changes, or if it changes by a certain amount, then the RBE node in Node-Red is a very handy tool.

I’m about to do a bit of gardening, but when I’m done I’ll share some code for making some of the topic stuff you’re doing a bit easier.

Pete.

1 Like

Thank you! I really appreciate it, I do enjoy the learning process, but it’s nice to be nudged in the right direction :smile:

I really ought to cut the grass too!

Okay, halfway through cutting the hedge the good old B&D trimmer died on me. Turned out to be a broken wire inside, which was a fairly easy fix. Then it rained, then the wind picked-up and blew my clippings everywhere, but finally got it done and everything tided-away. Now back to the interesting stuff…

I’ve been doing some work on my ‘basic’ MQTT sketch, but it’s now got a bit out of hand. I worked on the principal that it’s easier to add-in all the bells and whistles and get them working, then delete/comment them out if I didn’t need them. Unfortunately that’s left me with a sketch that’s a bit unwieldy to share, so I need to spend some time making a simpler version.
In the meantime, here are a few pointers that you can use if you wish…

I’ve adopted a similar naming convention to you with MQTT topics. after a while I decided that I didn’t want to keep repeating the first part of the topic in multiple places in the sketch (the “house/lounge/sensor/” part in your example). So, I’ve defined a variable to hold that, like this:

String base_mqtt_topic = "house/lounge/sensor";  // Start of the MQTT Topic name used by this device

This is a String variable type for a reason. PubSubClient requires the topic to be a char variable type, but you cant do string concatenations on chars. So, I do my concatenation with String variables, then turn the result into a char.

So, my heartbeat function contains these lines of code:

  MQTTclient.publish((base_mqtt_topic + "/RSSI").c_str(),String(WiFi.RSSI()).c_str(),true);
  MQTTclient.publish((base_mqtt_topic + "/Free_RAM").c_str(),String(ESP.getFreeHeap()).c_str(),true);
  MQTTclient.publish((base_mqtt_topic + "/Uptime").c_str(),uptime_dd_hh_mm_ss,true);

The uptime is created in a similar way to what you were doing in the thread about printing serial output to Blynk terminal widget:

void FormatUptime()
{
uptime = millis();   
int systemUpTimeSec = int((uptime / (1000)) % 60);
  int systemUpTimeMin = int((uptime / (1000 * 60)) % 60);
  int systemUpTimeHr = int((uptime / (1000 * 60 * 60)) % 24);
  int systemUpTimeDay = int((uptime / (1000 * 60 * 60 * 24)) % 365);
  sprintf (uptime_dd_hh_mm_ss, "%02d:%02d:%02d:%02d", systemUpTimeDay, systemUpTimeHr, systemUpTimeMin,systemUpTimeSec);
} //End of void FormatUptime

This also needs a global variable to be declared like this:

char uptime_dd_hh_mm_ss [12]; // Character array to hold the uptime in Day:Hour:Minute:Second format

The other thing you might not have realised is that each device (ESP8266 etc) must have its own MQTT client ID. If you have two devices with the same client ID then you won’t get any errors or warnings, but everything will stop working.
You’ve hard-coded your MQTT Client ID into your connection string, “ESP8266Client” in this case:

  if (client.connect("ESP8266Client", mqtt_user, mqtt_password)) {

I prefer to use a variable to hold the MQTT Client ID, and I use the same ID for the MQTT Hostname:

Global variable:

char *client_id = "Wemos_no_3_MQTT_Lounge_weather";  // Used for OTA hostname and MQTT Client ID

Modified MQTT connection string:

if (client.connect(client_id, mqtt_user, mqtt_password)) {

Modified OTA command:

  ArduinoOTA.setHostname(client_id); 

Obviously none of this stuff is essential, but it can make life much easier and cut out a lot of head-scratching when it all stops working.

Pete.

1 Like

Thanks Pete that’s all good info, I know what you mean about having one ‘master’ code to suit all, I have done similar templates for sensors, TFT displays, etc… and it can get a little out of control. Especially when you then switch to MQTT or something :laughing:

I have read your post a couple of times and implemented some of it into my sketch so far. Particularly the MQTT client ID being unique was a useful tip, I’m sure I would have fell foul of that in the near future :slight_smile:

When you say your turn the result into a char, that is via the .c_str() function I assume. I was wondering what that means, char arrays etc… are some things I really need to get my brain around!

Do you just use debug nodes to read the free memory/uptime I’m curious?

Thanks! :+1:

Yes, should have said that. I really don’t understand chars as well as I probably should, but .c_str() function seems to be a get out of jail free card so I use it whenever I need to.

If you want you can connect the MQTT node direct to a Blynk ws write node which pushes the data to a display widget.

I use the Windows version of MQTT Explorer:

I’m running v0.2.5, which is classed as pre-release, but seems fine. You have to click the ‘assets’ link.
I struggled a bit to find the right download link for windows, but this is the correct one:
https://github.com/thomasnordquist/MQTT-Explorer/releases/download/0.0.0-v0.2.5/MQTT-Explorer-Setup-0.2.5.exe

MQTT Explorer is nice because it allows you to explore the topic structure easily, and shows you history and graphs for things like RSSI and Free Memory…

You’ll see that I really did go over the top with my ‘basic’ configuration. When the device firsts boots and connects to Wi-Fi/MQTT I output lots of information including:
I* P Address

  • MAC Address
  • Filename of the firmware file
  • Compiled date and time of the firmware file
  • Client ID
  • Heartbeat frequency (this is randomized between 4 and 7.5 seconds on my devices)

The Heartbeat routine outputs:

  • RSSI
  • Free Memory
  • Uptime

I also count how many times the millis() value has hit the 49 day mark and rolled-over back to zero. This plus my Uptime value will tell me how long the device has been up for.

I also update counter for Wi-Fi reconnect count and MQTT reconnect count

I also played with also running Blynk directly on a device, with a different Auth code to the one used for the device in Node-Red, so that I could use the serial to Terminal widget code that we were using in the other thread. This is disabled on this device, but if it’s enabled then I also publish MQTT messages with Blynk Auth code, connection status, re-connect count etc.

The other MQTT topics are to do with controlling RGB LEDs.

In reality, this has become far too complex, but it is nice to have the ability to drop these functions in without having to do any real head-scratching and debugging if I need them.

If you want code to be able to capture and publish any of these things via MQTT then just let me know :grinning:

Pete.

Thanks again Pete, I read your message last night on my phone but it’s such a pain to reply on it!

I have downloaded MQTT Explorer which looks great, thanks for the link. It’s really interesting to see the messages pinging around, and to have a list of topics easily available.

Yes it does look like you have been adding lots of features! The uptime rollover /MQTT reconnect/WiFi reconnect look particularly good if you fancy sharing - share as much as you like! sorry if I’m being cheeky :smile:

One thing I have noticed with the heartbeat nodes, is that sometimes after a power interruption of the Wemos, the ‘My Timeout’ node seems to stay at “Timeout!” with a red square. If I press the reset button on the Wemos it starts up again.

Many thanks for your time :+1:

Okay, I started trying to extract more pieces of code from my base setup, but then decided to create a simplified version to share.

I’ve done this and put it on GitHub (because there are quite a few files and that’s the easiest way to share them).
I’ve updated my Home Automation topic with the details:

Pete.

1 Like

Good idea to add to your original post, I’ll check it out now - thanks! :+1:

I’m guessing that this is caused by the device failing to connect to either Wi-Fi or MQTT, but without knowing if the other MQTT messages are being published it’s hard to know.

I did discover a small bug in the current (1.1.1) version of the Timeout node, that I’ve alerted Pete Scargill to. The Node is meant to send the SAFE message on startup, but it isn’t doing this. The workaround is to tick the “Repeat message every second” checkbox.
I don’t think this would cause your problem though.

I put a warning about version 2.5.0 of the Arduino core in the latest post of the Home Automation thread.
If you’re using v2.5.0 (the latest version) then you’ll probably suffer random problems connecting to Wi-Fi. It’s worth downgrading to 2.4.2.

Pete.

The MQTT messages are getting through so it’s not a connection issue.

I noticed the same thing, I currently have mine set to send #23C48E (red) or #D3435C (green) to a setproperty node linked to a LED, as I like seeing a light either way. But doing this I have to manually inject 255 one time only to light up the LED. I will think of a solution later, I just re-wrote my garage weather station for MQTT and my brain hurts… :slight_smile:

PS - I am using 2.4.2 core as I did read your posts a while back about there being an issue with 2.5.0,

Thanks!!

There is actually another way to get Node-Red to do something automatically on Start-up/Deploy.

The Inject node has the option to “Inject once after x seconds”…

image

When this option is selected a little “1” appears next to the node:

image

This can also be done to trigger a Blynk.syncVirtial(Vpin) or Blynk.syncAll() equivalent on startup:

image

The inject node defaults to 0.1 second for the delay, which I’ve found is too fast for the Blynk ws nodes, so I change it to 1 second which works fine for me.

Pete.

1 Like

Nice that will do it, thank you! :smile:

@PeteKnight hi there, I thought I could post here without cluttering the forum with another “node red” thread. I might get put on some watch list :smile:

I was wondering if you had updated to 2.5.2 Arduino core?

I’ve had trouble with one of my Wemos’ not re-connecting to the WiFi after an outage.

Thx