Thank you Pete!
Got it working, and I now better understand the flow of things…
Great,
Not sure if you’ve seen this topic…
It was written in the Legacy days, but very little has changed.
Pete.
I will check it out…
Hi PeteKnight, I follwoed your other tutorials ti transition from some extensive blynk legacy projects to Blank 2.0 + MQTT + Node Red.
May I know how you make sure that after a crash or re-start of the ESP you get the last values saved in the Blynk cloud to your variables on ESP? (communicated via MQTT of course).
On legacy app I was simply using SyncAll function on the ESP platform, now this is not possible anymore as I am not running any blynk code anymore on the ESP. Of course I could trigger this somehow via NodeRed, but the timing and instance of this sync is important and this should be controlled from hardware/ESP if I am not mistaken. Would appreciate your answer here.
I have seen this answer here, but the timing of the sync event is usually important. How does Node Red know when to trigger this sync event? I could of course periodicall call it, but this is not necessarily a clean implementation.
"Yes. The Write Event node will trigger whenever a BLYNK_WRITE() would trigger in the sketch. Writing to a virtual pin from a device won’t do that.
You can trigger a Sync node in Node-Red to cause the server to send the current virtual pin value, which will trigger the Write Event node.
Pete.
"
It depends on the scenario you are talking about.
Do you have an example that you can describe where you need your ESP to get the latest Blynk server value on startup?
As far as I’m concerned, here are really two different possible situations that need different approaches…
-
ESP goes offline then back online again, and you want to sync the latest Blynk server value
-
Node-Red goes offline then back online again and you want to sync the latest Blynk server value
In situation (1) you might have a command like an on/off status and you want to set a relay that tye ESP device is controlling to the correct status. In this case, the on/off message would have been sent from Node-Red to the ESP device via MQTT.
If the message was sent with the “Retain” flag set to true then that MQTT message value will be retained on your MQTT server.
When you reboot your ESP device it wil connect to the MQTT server and when it subscribes to the topic which relates to this on/off message it will automatically be sent the latest value, and your device will respond to that and set your relay to the correct state.
In Scenario (2) you’ll want to know what the current widget state that controls this relay is set to, as it could have changed while your Node-Red server was offline.
You can use the Sync/SyncAll node in the Blynk IoT contrib to fetch this from the Blynk server, and you can trigger that with an Inject node set to trigger at startup after x seconds.
If you use the SyncAll option (which I don’t recommend) then you have to be careful that your datastream is set-up to respond to this request, by checking the “Sync with latest server value every time device connects to the cloud" option in the Datastream Advanced settings for each virtual datastream you wish to sync.
Pete.
Hi PeteKnight.
The retain option is definetly addressing almost all cases I could think of, indeed.
I will start deploying and do some testing. Very accurate and great response. Thanks a lot!
Let’s see how long this implementation takes.
By the way, have you meanwhile come along any better option than using blynk? Such as Node Red with Dashboard on local machine + same Node Red on a cloud based VPS for example? I was thinking of bridging the values via Mosquitto or NodeRed from local Mosquitto MQTT to a cloud based MQTT and this way be able to use NodeRed Dashboard and avoid blynk. Just an idea…
No, that’s why I’m still here on the Blynk forum, advocating using MQTT as the transport method, Node-Red as the logic/rules engine and Blynk as the UI.
I started-off with MQTT and Node-Red, and dabbled with the Node-Red dashboard - and hated it. Searching for a better UI bought me to Blynk, and I’m still here over 6 years later.
I did initially try to mirror my Blynk dashboards into Node-Red dashboard, but it made my flows too complicated and I never used the Node-Red dashboard anyway.
Also, I don’t normally store any data on my Node-Red server. I use Blynk’s cloud server to store my historical data to be displayed in Superchart etc. Although I do have a SQL database that I store some specific data in, but most of the time I prefer to let Blynk handle that for me.
I don’t understand the architecture of a local plus VPS cloud based mirror. Why wouldn’t you just run one local Node-Red instance and make it accessible outside of your network.
Pete.
1 Like
Making Node Red accessible from outside my network—> no problem
Making NodeRed Dashboard accessible, full different story. No luck for me.
I am following your suggestion and actually do like it. If Blynk decides one day to deploy 3.0 and I need again to change all my projects, it will be a nightmare though. haha
Let’s see how I will progress. I have more than 20 relays and 20 sensors that all work together including timers etc. Let’s see.
Also I was heavily depending on webhooks and some thingspeak rest API functions. Let’s hope I will be able to replicate those without any issues, from what I saw so far webhooks are not as widely supported on 2.0 as on 1.0.
Do you have any clue how I cann access the content of my 1.0 apps? I cannot see anymore what is saved in the webhooks as all servers are shut down and the apps only load in offline mode…
Cheers
It’s probably 5 years since I dropped that way of working, but it does work.
You’d be surprised. If your logic and rules-engine is in Node-Red then changing to a different UI is actually very easy.
If you’re using webhooks in Blynk then you’re taking the wrong approach. You should be calling the webhook in Node-Red and processing the data there, then passing it to either Blynk or your device in the format that it’s expecting. The same with Thingspeak. There’s at least one Thingspeak contrib for Node-Red and you should be using that or a Node-Red webhook.
No, the servers are decommissioned, so the data is gone I’m afraid.
Pete.
Again, your answers are spot-on.
Fully agree, once my logic in implemented and all my virtual write and read functions from legacy 1.0 are converted to mqtt protocol, hence topic subscribe and post, I guess I will be way more flexible than I was before and get all other benefits such as Alexa integrations etc. (although I do not need them for now). This keeps me very excited though as this additional flexibility will pay off in the end I think.
Your answer in relation to webhooks, I fully agree. I actually already implemented part of the webhook nodered thingspeak nodes, you are fully right here.
Another question:
I also had used the bridge function with blynk legacy. What do you suggest here? As I was sending arrays of integers, I would like to replicate this as much as possible. Would you suggest to use:
A:
ESP NOW to communicate between ESP clients (ESP-B send data to ESP-A via ESP NOW) and local MQTT to communicate between Node Red on a RSP-PI as broker and ESP-A as client?
B:
Or alternatively ESP-A to ESP-B via MQTT broker via MQTT protocol? Can you actually send array of data? Double/floats etc?
Many questions but very relevant in my eyes…
Approach B, or a variation on it, is the way to go.
You can build a collection of data, either delimited or as JSON, but I wouldn’t bother.
You’ll probably find it easier to send each piece of data as a separate MQTT message to its own topic.
Once again, I’d really need to see an example to fully understand, but separate MQTT messages would be my preference.
Pete.
I see.
Here is an example.
I would like to avoid sending data to 10 different topics.
ESP-B, send data:
bridge2.virtualWrite(V100, tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7, tmp8, tmp9); //this will send the data to the master collector ESP32
ESP-A, receive data:
BLYNK_WRITE(V100) { // needed to receive data from the other ESP32s or NodeMCUs that are sending data to virtual ports //int pinData = param.asInt(); // for (int kk = 0; kk <= 9; kk++) { collectorVariable[kk]=param[kk].asInt() * 1.00 / 100; delay(10); // Serial.println(collectorVariable[kk]); } }
I assume that tmp0-9 are temperature readings from sensors attached to ESP-B ?
What does ESP-A do with these values when it receives them?
Does it apply some logical tests and comparisons and control some relays for example, based on these values?
If so then my approach would be to send the values from ESP-B to Node-Red, and do the logical tests and comparisons there. If relays attached to ESP-A need to be turned on or off as a result then I’d send the appropriate MQTT commands to that device.
This turns ESP-A into more of a dumb device, with no complex logic comparisons being performed in its code, it simply listens for instructions and follows them.
The beauty of this approach is that it makes your logic much more visible and far easier to change, because its all done in the Node-Red flow and you can makes the incoming and outgoing values, and the intermediate calculations and statuses visible in the flow editor rather than having to hook-up two serial monitors and try to debug the code on your ESPs.
Pete.
Thanks a lot for the answer. Actually I do have a display on ESP-A to display some values. I will see what I will finally do, whether the logic to remain on the board on this to happen in Node Red.
Just finished the mqtt functions:
Sure somebody may want to use this. Seems to be working fine for now…
#include <WiFi.h>
#include <PubSubClient.h>
#include <Wire.h>
// Replace the next variables with your SSID/Password combination
const char* ssid = "Casa";
const char* password = "s7y62u5gdztsd";
//MQTT Broker Details
// MQTT Broker, it should be noted that the port from HassIO is not needed here (http://192.168.1.127:8123/), therefore only 192.168.1.127 is needed.
//For MQTT Explorer both are needed, port 8123 and port 1883
const char *mqtt_broker = "192.168.1.127";;
const char *mqtt_username = "mqtt_andreas";
const char *mqtt_password = "Ab.6186.,tt";
const int mqtt_port = 1883;
//const char *ID = "esp8266_tempbase"; // Name of our device, must be unique
//const char *topic = "esp32/test";
WiFiClient espClient1;
PubSubClient client(espClient1);
long lastMsg = 0;
char msg[50];
int value = 0;
float temperature = 0;
float humidity = 0;
void setup() {
Serial.begin(115200);
setup_wifi();
client.setCallback(callback);
}
void loop() {
if (!client.connected()) {
reconnect();
}
client.loop();
long now = millis();
if (now - lastMsg > 5000) {
lastMsg = now;
temperature = 100.22;
// Convert the value to a char array
char tempString[8];
dtostrf(temperature, 1, 2, tempString);
Serial.print("Temperature: ");
Serial.println(tempString);
client.publish("esp8266-tempbase/sht35_t1", tempString);
humidity = 38.38;
// Convert the value to a char array
char humString[8];
dtostrf(humidity, 1, 2, humString);
Serial.print("Humidity: ");
Serial.println(humString);
client.publish("esp8266-tempbase/sht35_h1", humString);
}
}
void callback(char* topic, byte* message, unsigned int length) {
Serial.print("Message arrived on topic: ");
Serial.print(topic);
Serial.print("... Message: ");
String messageTemp;
// If a message is received on the topic esp32/relay1, you check if the message is either "on" or "off".
if (strcmp(topic,"esp32/relay1")==0){
for (int i = 0; i < length; i++) {
Serial.print((char)message[i]);
messageTemp += (char)message[i];
}
Serial.println();
// Changes the output state according to the message
if (String(topic) == "esp32/relay1") {
Serial.print("Changing output to: ");
if(messageTemp == "on"){
Serial.println("on");
}
else if(messageTemp == "off"){
Serial.println("off");
}
}
}
if (strcmp(topic,"esp32/input/minPH")==0) {
for (int i = 0; i < length; i++) {
Serial.print((char)message[i]);
messageTemp += (char)message[i];
}
Serial.println();
float minPH = messageTemp.toFloat();
minPH=1000.365*minPH;
Serial.println("testestest");
Serial.print(minPH);
}
if (strcmp(topic,"blue")==0) {
// this one is blue...
}
if (strcmp(topic,"green")==0) {
// i forgot, is this orange?
}
}
void setup_wifi() {
delay(10);
// We start by connecting to a WiFi network
Serial.println();
Serial.print("Connecting to ");
Serial.println(ssid);
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("");
Serial.println("WiFi connected");
Serial.println("IP address: ");
Serial.println(WiFi.localIP());
}
void reconnect() {
// Loop until we're reconnected
while (!client.connected()) {
//connecting to a mqtt broker
client.setServer(mqtt_broker, mqtt_port);
client.setCallback(callback);
while (!client.connected()) {
String client_id = "esp32-client-";
client_id += String(WiFi.macAddress());
Serial.printf("The client %s connects to the local mqtt broker\n", client_id.c_str());
if (client.connect(client_id.c_str(), mqtt_username, mqtt_password)) {
Serial.println("Local mqtt broker connected");
// subscribe to topic
client.subscribe("esp32/relay1");
client.subscribe("esp32/relay2");
client.subscribe("esp32/relay3");
client.subscribe("esp32/relay4");
client.subscribe("esp32/relay5");
client.subscribe("esp32/relay6");
client.subscribe("esp32/relay7");
client.subscribe("esp32/relay8");
client.subscribe("esp32/input/minPH");
} else {
Serial.print("failed with state ");
Serial.print(client.state());
delay(2000);
}
}
}
}
float bytesToFloat( unsigned char b0, unsigned char b1, unsigned char b2, unsigned char b3)
{
float output;
//may need to reverse these
*((unsigned char*)(&output) + 3) = b3;
*((unsigned char*)(&output) + 2) = b2;
*((unsigned char*)(&output) + 1) = b1;
*((unsigned char*)(&output) + 0) = b0;
return output;
}
//and we call it passing in the bytes from the payload@12racebike when you post code to the forum it needs to have triple backticks a5 the beginning and, not blockquotes as you’ve used.
Please edit your post and correct the formatting.
Triple backticks look like this:
```
Pete.
hope this is fine… thanks.
1 Like
This is a bit cumbersome, and there’s a better way to do it, by re-structuring your MQTT topics
I tend to organise my topics by location (room), function and device - but it doesnt really matte how you do this so long as it makes sense to you.
At device level I tend to organise my topics in a way that allows me to use wildcards to subscribe to them. Take this example of a device that performs two functions - it has an IR transmitter and also a temperature and humidity sensor attached. This allows me top control an aircon unit and ceiling fans via Infra Red, and also send current temperature and humidity readings to MQTT to allow me to know if I need to change the settings of these devices relative to the target temperature I’ve defined.
My device is located in the upstairs lounge, so I start with this as my MQTT topic…
Upstairs/Lounge/IR_Tx_and_Temp_Sensor/
at this ‘root’ level for my device I get the device to publish lots of high-level info. Some of thsi is published once at start-up with a Retain flag, other info is updated at 5 second intervals…
- MQTT_Client_ID (has to be unique for each device, so uses the ESP chip ID)
- Status (Dead/Alive from the Last Will and Testament)
- Filename (Helps me find the .ino file again)
- Compiled_Date
- Compiled_Time
- MAC_Address
- IP_Address
- WiFi_Connect_Count
- MQTT_Connect_Count
- RSSI
- Free_RAM
- Uptime
Having these at this root level for the device makes it easy to see the info at a glance.
I then have a CMD_IN topic, where all of my incoming commands will be directed, and this will have sub-topics that relate to the things that are controlled by the device - in your case it might look like this…
CMD_IN/Relay1/Power
CMD_IN/Relay2/Power
CMD_IN/Relay3/Power
CMD_IN/Relay4/Power
CMD_IN/Relay5/Power
CMD_IN/Relay6/Power
CMD_IN/Relay7/Power
CMD_IN/Relay8/Power
This allows you to use a wildcard to subscribe to the CMD_IN topic and all of its sub topics, like this…
client.subscribe("Upstairs/Lounge/IR_Tx_and_Temp_Sensor/CMD_IN/#");
where the # wildcard means subscribe to all child topics.
I also have a DATA_OUT topic that I use to send feedback data about the current state of the thing I’m controlling (in your case a relay) that comes from actually reading the state of the pin I’m controlling. This tells me if the “on” command actually resulted in the device’s pin being pulled LOW.
I also use this part of the topic structure to add a memo note that gives me more info about the relay, like this
DATA_OUT/Relay_1/Power = 1
DATA_OUT/Relay_1/Memo = Relay used to control the lights near the TV
I adopt this structure because if you subscribe to a topic and then publish to that same topic the you’ll get an infinite feedback loop.
The CMD_IN topic and its children is only for incoming messages
The DATA_OUT topic is never subscribed to by the device, and this (or the ‘root’ topic) is where the device writes data to so that it is passed to Node-Red.
All of this might seem like a lot of effort just to avoid subscribing to eight topics instead of one, but at some point in future you’ll add a new topic to Node-Red and write the callback handler for it, then spend an hour or two trying to work-out why it isn’t working. Then you’ll eventually realise that you forgot to subscribe to the new topic!
I also take a different approach with my callback handler for incoming MQTT messages. I hate working with char variables, so turn them into strings as quickly as possible. It’s then very easy to turn tehse into integers etc if you wat to, but it’s not always necessary.
I also split the message into topic and payload and use these variables in my if statements later…
void MQTTcallback(char* topic, byte* payload, unsigned int length)
{
Serial.print(F("Message arrived ["));
Serial.print(topic);
Serial.print(F("] ["));
for (unsigned int i=0;i<length;i++)
{
Serial.print((char)payload[i]);
}
Serial.println(F("]"));
String msg_topic = String((char *)topic);
String msg_payload = String((char *)payload);
msg_payload.remove(length); // Trim any unwanted characters off the end of the string
// We now have two string variables, 'msg_topic' and 'msg_payload' that we can use in 'if' statements below...
if (msg_topic==base_mqtt_topic + "/CMD_IN/Fan/OnOff")
{
if (msg_payload == "Off")
{
Fan_Sender.sendRaw(rawFan_Off, 143, khz);
}
If you want to use the Last Will and Testament feature then you need to have a more sophisticated client connect command…
/*
MQTT Connection syntax:
boolean connect (client_id, username, password, willTopic, willQoS, willRetain, willMessage)
Connects the client with a Will message, username and password specified.
Parameters
client_id : the client ID to use when connecting to the server.
username : the username to use. If NULL, no username or password is used (const char[])
password : the password to use. If NULL, no password is used (const char[])
willTopic : the topic to be used by the will message (const char[])
willQoS : the quality of service to be used by the will message (int : 0,1 or 2)
willRetain : whether the will should be published with the retain flag (int : 0 or 1)
willMessage : the payload of the will message (const char[])
Returns
false - connection failed.
true - connection succeeded
*/
if(MQTTclient.connect(mqtt_client_id.c_str(), mqtt_username, mqtt_password, (base_mqtt_topic + "/Status").c_str(),0, 1, "Dead"))
{
// We get here if the connection was successful...
The “Dead” part at the end of the connect command, which really translates to “/Status/Dead”, tells the MQTT server server to write the word “Dead” to the “Status” topic if the device disappears off the radar.
I then publish “Alive” to the “/Status” topic when I’ve connected to the MQTT server with a Retained flag so that I can see at a glance whether the device is connected or not…
MQTTclient.publish((base_mqtt_topic + "/Status").c_str(),"Alive",true);
As you can see, I have a string variable called “base_mqtt_topic” that set once in the sketch and it’s used as the prefix for all of my subsequent MQTT Publish commands…
String base_mqtt_topic = "Upstairs/Lounge/IR_Tx_and_Temp_Sensor/";
Hope this helps.
Pete.
Thanks a lot for your message, I will need to do some coding and testing and will replay once I got my full head around this new concept. Very helpful!. Thanks a lot. Some of the code which was there was indeed to be refined, such as usage of wildcards etc. I will then come back with a proper reply:-)
Still di not fully understand the dead/alive part. I do see the idea behind it and it is needed indeed, knowing whetehr the device is online or not, but the implementation I did not fully understand yet…
Mainly I am making reference to this:
“If you want to use the Last Will and Testament feature then you need to have a more sophisticated client connect command…”
I think there was a minor character too much in your example above, but the concept is clear. Thanks a lot…
Talk to you soon and thanks again!
this:
MQTTclient.publish((base_mqtt_topic + "/Status").c_str(),"Alive",true);
MQTTclient.publish((base_mqtt_topic + "Status").c_str(),"Alive",true);Based on your proposals I came to a conclusion of how I send my mqtt messages via esp. Thank you very much for your support. This is very clean and readable, so far working like a charm. Still need to implement the receive / mqtt listener and see what the best way is for me. Here comes my code:
Basic function for sending mqtt messages, when calling the function it is finally as easy as it used to be with blynk, sending virtual variables…
void mqttSend(float value, String baseTopic ,String topic) {
float tempValue = value;
char tempString[8];
dtostrf(value, 1, 2,tempString);
Serial.print((topic + ": "));
Serial.println(tempString);
client.publish((baseTopic + topic).c_str(),tempString,true); //alternative syntax: client.publish((base_mqtt_topic + "dsdb1820_t3").c_str(),tempString3,true);
}
mqttSend(tempDS18B20[0],base_mqtt_topic,"DS18B20_1"); //this calls the declared mqtt function and sends a float value as charachter array via mqtt
mqttSend(tempDS18B20[1],base_mqtt_topic,"DS18B20_2");
mqttSend(tempDS18B20[2],base_mqtt_topic,"DS18B20_3");Nice approach, I hadn’t previously thought of passing the values I wanted to publish via MQTT to a parametrised function.
Did you figure-out the Last Will and Testament stuff?
The LWT value is set by the server when the device stops responding to ‘pings’, but it takes about 20 seconds on my setup, which I guess is the default timeout time for Mosquitto.
Pete.