Force Blynk Write Event / Server Inner Workings

[segallis]

I could use some help trying to understand the inner workings of the Blynk server.

My application and requirements are as follows: I have a local Blynk server, which also runs several python background jobs to manage various IOT devices. I run micropython on various ESP8266 endpoints. I have multiple Blynk apps that can control the same HW endpoints. I have multiple endpoints that are controlled via the same Blynk app. I only utilize virtual pins on each part. These are fixed requirements and I am not interested in alternate solutions (at this point).

I realize Blynk is not really designed for my particular use case. On the other hand, I pretty much have most of it working and think it will work for my atypical application. There are a few quirks with the blynk server and it’s interaction with the app and endpoints that I am trying to understand fully, so please correct my misinformation and provide clarification.

I’ll use the example of an ESP8266 as well as a server background job - both running python, both using the same AUTH token:

Any button press on the Blynk app is reliably sent by the server to BOTH endpoints. However, if either endpoint does a virtual_write, that vpin only makes it to the server and the app, not the other other endpoint. So the app and endpoints look different to the server.

If I do a sync_virtual from an endpoint, the server only sends the vpin state to the endpoint that sent the sync. Plus the sync only captures the current state so in the case of a on-off momentary toggle, the sync request misses it completely. (a kludgy workaround is doing a sync every second and having the momentary be longer than 1 second, but that is not ideal)

Is there anyway to have an endpoint command the server to send the updated vpin to all endpoints, the same way the app appears to? Ideally, I would like to be able to generate a write event from the python background job running on my server (and avoid periodic syncs by the endpoints).

Thanks,
-Greg

[PeteKnight]

As you’ve realised, Blynk isn’t designed to work this way.
The expected scenario is that each endpoint (device in Blynk speak) has its own unique Auth token.
When a device sends data to the server in the form of a virtualWrite, there is no echoing-back of this status, otherwise it would trigger the corresponding BLYNK_WRITE (in C++ terms, not sure what the Python equivalent is), which could result in an endless and unbreakable loop.

When one device needs to communicate with another - via the server - then Bridge code is used to send this data to the other device’s Auth token.

Anything you get working now will be something that works by accident rather than design, and may well get broken by future updates to the server software or the library that you are using.

Pete.

[segallis]

In python, the virtual_write sends the vpin value from endpoint to the server. The server then sends this to the app. If it did send it back to the endpoint, it would be captured by the blynk.on handler (formerly VIRTUAL_WRITE). Thought named similarly, there are separate and would not result in a loop unless the code deliberately made the loop. e.g. endpoint one sends a virtual_write of V4 to endpoint two which has a V4 VIRTUAL_WRITE handler (listener). Endpoint one would not have this listener since it a control on endpoint two. Think of both endpoints being a single virtual device - only one would use V4 as the output control.

Just curios, doesn’t node-red connect to the server with the same AUTH token as well? Does node-red look different to the blynk server than another endpoint?

BTW, I just discovered that this “forced write event” than I am looking for can be emulated by an HTTP GET. If I type http://192.168.1.20/{AUTH TOKEN}/update/V4?value=1 the new vpin value is sent by the blynk server to both endpoints. Worst case, I can use this as a workaround, although it would make more sense to use the calls that already exist in the blynk library if possible.

And I have no intention of making something work by accident. I have been making systems do what I want for for over 4 decades, and I don’t settle for less - nor do I leave things to chance. I find out how things tick and then I engineer a solution.

-G

[segallis]

[When one device needs to communicate with another - via the server - then Bridge code is used to send this data to the other device’s Auth token.]

Am curious what you mean. How does one device “send this data to the other device’s Auth token”? This sounds like exactly what I am trying to do. And doesn;t that bridge code look like another endpoint with the same auth token to the server?

Or do you mean bridge code within the blynk server itself? I haven’t found good documentation on how to use that server capability if it exists. I see the MSG_BRIDGE (15) command, but the server reported
“Bridge not initialized” when I tried using it.

-G

[PeteKnight]

@segallis

In python, the virtual_write sends the vpin value from endpoint to the server. The server then sends this to the app. If it did send it back to the endpoint, it would be captured by the blynk.on handler (formerly VIRTUAL_WRITE). Thought named similarly, there are separate and would not result in a loop unless the code deliberately made the loop. e.g. endpoint one sends a virtual_write of V4 to endpoint two which has a V4 VIRTUAL_WRITE handler (listener). Endpoint one would not have this listener since it a control on endpoint two. Think of both endpoints being a single virtual device - only one would use V4 as the output control.

But what you are describing here is two endpoints sharing the same Auth token, which is not how Blynk is designed to work.

In C++ a Blynk.virtualWrite(vPin, value) sends the value to the specified virtual pin on the server.
BLYNK_WRITE(vPIN) is in effect a callback that is triggered whenever the value of a virtual pin changes on the server. This is normally triggered by a change in the widget attached to the virtual pin in the app, or via an API call. It can also be triggered by a Blynk.syncVirtual(vPin) command which forces the server to push out the current value of the virtual pin.

@segallis

doesn’t node-red connect to the server with the same AUTH token as well?

I’m not sure what you mean by “same Auth token”. Each Node-Red connection to the server uses an Auth token, and represents a device created in the app. When using Node-Red it’s bad practice to also run Blynk code on the devices being used, and if you do choose to run Blynk and MQTT code side by side on a device then you shouldn’t use the same Auth token on the device as is used in a Node-Red connection.
Node-Red does allow you to use fewer Auth tokens/server connections (depending on how many virtual pins you want to use, and how you prefer to organise your data structure), because the devices themselves are invisible to Blynk as they communicate with Node-Red just by MQTT messaging.
This gives you 256 virtual pins to use as data channels to-and-from the Blynk server with a single connection. Personally, I find it a bit clunky to do it this way, and it also make it impossible to use Device Tiles or the Device Selector in the way that they are intended, so I tend to have a hybrid of Node-Red connections that represent individual devices and other that amalgamate data from multiple devices - depending on the use case.

The bottom line is that each device/Node-Red connection/endpoint should have a unique Auth token.

@segallis

BTW, I just discovered that this “forced write event” than I am looking for can be emulated by an HTTP PUT. If I type http://192.168.1.20/{AUTH TOKEN}/update/V4?value=1 the new vpin value is sent by the blynk server to both endpoints. Worst case, I can use this as a workaround

I find that it’s usually better to use the GET method for both reading and writing data, but your workaround is still fudging the issue and here are no guarantees that the behaviour you have observed will remain the same in future.

As I said before, if you want a bulletproof/futureproof solution then it’s one unique Auth token per endpoint.

Pete.

[segallis]

I had typed PUT, but meant GET

Maybe I remember incorrectly, but I thought the node red auth token had to match the ESP8266 token in the example “family-control-of-same-device” you helped me with?

If node red has a different auth token from the esp, then how does the blynk server send the vpin update from node red to the esp?

Also, why would an http GET ever not work? In fact, I’m leaning towards using that in my server python code as a permanent and super flexible solution. My initial concept is that every endpoint WILL have a unique auth token, and my python background “router” will send VPIN updates to the server using the appropriate endpoint’s unique auth token.

But why can’t I have this “router” also listening to VPIN updates from the server with the same auth token as the endpoint? I suspect any solution, such as node red, has a version of what I am describing.

Thanks,
-Greg

[PeteKnight]

@segallis

If node red has a different auth token from the esp, then how does the blynk server send the vpin update from node red to the esp?

The way that I use Node-Red, and the way that I’d recommend anyone to use it with Blynk, is that the ESP doesn’t have any Auth token, doesn’t run any Blynk code and and doesn’t connect to the Blynk server itself. Communication between the ESP and Node-Red is via MQTT messaging and it’s Node-Red alone that talks to the Blynk server…
The data flow is ESP <–MQTT Messages–> Node-Red ↔ Blynk Server ↔ Blynk App

@segallis

why would an http GET ever not work?

It will always update the server, but how that then translates to that update being seen by multiple devices sharing the same Auth token and how those devices process that update is the area where you don’t know what the future will hold. You’re using the system in a way it’s not meant to be used (multiple end points sharing the same Ayth token) and that’s where the uncertainty lies as far as I’m concerned.
In theory, a widget button tap, a Blynk.syncVirtual and an API update should all produce the same result, but your testing shows that this isn’t the case when you have multiple devices using the same Auth token. As a result, you are exploiting an observed but undocumented behaviour, and asking what that behaviour should ever change in future.

@segallis

But why can’t I have this “router” also listening to VPIN updates from the server with the same auth token as the endpoint?

Because that’s not how the system is designed to work!

Pete.

[segallis]

Ahh. I was going by your example from before:

That example had the Dummy use the same Auth Token as the ESP, i.e. two end points with the same token. Which is why I was asking.

I suppose I could consider using MQTT messages between all the ESP endpoints and my background python server app and let it be the only “endpoint” the blynk server sees. Although, for many of my applications, I won’t need the python bridge, so I like the consistency of using the blynk lib on the ESP. Then for the cases where I want to link devices or apps, I can use the python bridge with the appropriate auth token.

I suspect if it works today, it will work next year as the lib and server won’t change until I decide to change them since it is all my HW.

I’ll prob dig into the server code and do some reverse engineering to see exactly how it processes and responds to app and lib write events so see if there is any chance of that functionality ever changing. I doubt that potential exists given that the server really only knows sockets, not endpoints. So the only question is probably how does the server handle multiple sockets.

Since the server is designed to always support multiple apps - i.e. different phones with the same login - and therefore multiple sockets, I suspect what I am doing has no risk of breaking later. I suspect what I will find is simply that the only way the server ever sends out a write event is if the apps originates it, or if the HTTP GET originates it. And probably that the server will echo that properly originating write event to all open sockets, regardless of how many). At least that is my hope and as I said, I will get to the bottom of how it works.

-G

[PeteKnight]

@segallis

That example had the Dummy use the same Auth Token as the ESP, i.e. two end points with the same token. Which is why I was asking.

No, these are the same endpoint (an ESP), as the nodes both have the same Node-Red connection. This was demonstrating how to control the same endpoint from two separate App projects.

@segallis

Since the server is designed to always support multiple apps - i.e. different phones with the same login - and therefore multiple sockets, I suspect what I am doing has no risk of breaking later

When you do dig in to this you’ll see that the App to server communication is done on one port (9443 on the local server) and the non SSL device to server communication is done on another (port 8080). Also, the communication protocols are different (as you’d expect) so I think it’s quite a big leap to make this assumption.

If you’re interested in the protocols, which you’d probably need to be to reverse engineer the communication handlers in the server and library code, then there is some useful info here…
https://github.com/blynkkk/blynk-server#blynk-protocol
Obviously the app source code is not public domain, but that’s not really relevant for your project.

Pete.

[segallis]

Thanks for your time helping me out on this!

Maybe I’m confused about the prior example. Your picture and text had:

I’ve set-up 3 devices (with non politically correct names )…

Master (Admin app)
Slave (Family app)
Dummy (to give an auth code that is used by the ESP).

So you showed two nodes (write_out and sync , I believe) on the right with the same auth token as the ESP. That’s why I was saying the blynk server sees two endpoints (sockets) with the same token (or is that 3? - but def not 1).

Fundamentally, that example had a node on the left that listened to a family member’s app - meaning the left node had the same token as the app. In this case that node is the endpoint wrt to the server, right? And there is only one endpoint.

On the right - whether it be one or two nodes - there is a node that has the same token as the ESP. So the server sees two endpoints (sockets) that share the same token. IF not, then maybe I am missing how the node on the right is able to the tell the blynk server to send anything to the ESP. They have have to have the same token?

What am I missing? Does node red use something other than the same socket connection to the blynk server as the ESP?

Beyond that, I tested with multiple python “endpoints”. The server reports each endpoint with the same device id and dash id, but each connection gets a different login id.

18:04:20.708 DEBUG- completeLogin. [id: 0x68d0f190, L:/…
18:04:20.708 TRACE- Connected device id 0, dash id 2146604307

18:04:25.603 DEBUG- completeLogin. [id: 0x8cae91e6, L:/…
18:04:25.604 TRACE- Connected device id 0, dash id 2146604307

Not sure if this a good thing or a bad thing. But all write events make to all endpoint and everything works as expected.

Worst case, if I need to get super conservative, I can prob get by with my background python bridge only needing to use HTTP GETs, which is officially supported and documented in the blynk server docs. And not have any sockets “connected” to the blynk server with the python lib. It is not as elegant a solutions as having the python bridge directly a part of the blynk connected network, but it will work reliably and would be consistent with the intended (partially crippled) functionality of the blynk server.

-G

[PeteKnight]

@segallis

So you showed two nodes (write_out and sync , I believe) on the right with the same auth token as the ESP. That’s why I was saying the blynk server sees two endpoints (sockets) with the same token (or is that 3? - but def not 1).

No, The server see one endpoint.

Let me demonstrate with the Node-Red configuration for my Spanish holiday home…

The Configuration Nodes page in Node-Red shows four Blynk connections (devices/endpoints with unique Auth tokens). The one called Spain_Outside_Lights is used 72 times on different Node-Red nodes, Spain_Lounge_Lights is used 53 times, Development is used 5 times and Spain_Weather_Station is used 11 times.

This is what the Device Statuses look like in the Blynk app…

Development is not visible, because as it’s name suggests it is used for development work and although the connection is configured in Node-Red and it is used 5 times, the connection and the flows that use it are disabled, so Blynk doesn’t see it.

Blynk has no concept of what these connections are used for in Node-Red, it doesn’t know that they exist at all. All that the Blynk server sees are three connections, each with their individual Auth tokens.

I have about 25 different devices around the building all talking to Node-Red via MQTT broker. Some of these are controllable via or push data out to Blynk, others don’t, but the in total there are 136 nodes that either accept incoming data from Blynk or push data out to Blynk, on these three devices/endpoints that Blynk uses.

Does that make it clearer?

@segallis

HTTP GETs, which is officially supported and documented in the Blynk server docs.

Yes it is, but the point you’re choosing to ignore is that nowhere in the Blynk documentation is sharing or Auth tokens supported. In fact, if you search the forum for guidance on this from the developers they will tell you that it is not supported and should be avoided.
The API communicates only with the Blynk server. The current version of the server and client software work together when an API call is received by the server, and the current end result meets your requirements in terms of the behaviour you are observing. That is very different to having an officially supported and documented end-to-end process, and I have problems reconcil;ing this apoproach with your assertion that…

@segallis

And I have no intention of making something work by accident. I have been making systems do what I want for for over 4 decades, and I don’t settle for less - nor do I leave things to chance.

However, that’s not my problem, I’ve pointed out the potential issues several times and where you go from here is up to you.

Pete.

[segallis]

Hmmm, looks like your house project and the “family_shared_app” example are totally different. The original example (that was actually working), had blinklib on the ESP, so node-red had the same auth token, which was consistent with your instruction for that example (“Dummy, to give an auth code that is used by the ESP”)

Obviously if all your endpoints are MQTT and not blynk, then sure, your node red nodes are the “single” endpoint that the blynk server sees. But once you start with the idea that an endpoint WILL run blynklib, then any connectivity to node-red will certainly use that same token, else node-red can’t tell the blynk server to do anything with respect to that ESP endpoint.

So sure, if you run MQTT at each endpoint, then the blynk server only has to see one endpoint per token. No way this can be done with any endpoint running blynklib. By definition, any other control point MUST use that same token.

Can’t say I agree with your assessment of the HTTP GET API. Using the HTTP GET with the same auth token as the device you are trying to control is explicitely described in the Blynk documentation. That is exactly what it is intended to do, and , in fact, there wouldn’t be much use for that API if it isn;t supposed to use an existing auth token. Think about it… what auth token should that GET use if not a token that is used by the app and endpoint.

I appreciate your concerns, but I just don’t see the risks. Certainly, no risk if I have one app and one endpoint , as that is exactly the use case for Blynk in the first place. The fact they have an HTTP GET API does not alter that.

Besides that, the actual testing thus far appears reliable - even with multiple endpoint sharing the same toke, as long as the only control comes from wither the app or an HTTP GET.

As you said, it’s on me. I already have a high-pass filter & pulse-integrator on the ESP8266 GPIO to GUARANTEE I NEVER experience a false activation on my garage door opener on reboot/power failure/hardware failure/etc, so rest assured I won’t settle for any solution that isn’t 100% reliable.

Thanks again for the insight! I’ll let you know what I find when I dissect the server code.

-Greg