Server "Success" Response To Field Device - uPython

SmsAlert · December 8, 2021, 9:32am

Hi Fellow Blynk’ers

I am busy working on a temperature management device and appear to have hit a snag, so firstly let me add what has already been actioned:

Overview
Hardware: ESP32-WROOM-32 (Eval Kit for now)
uPython Version: esp32-20210902-v1.17
Blynk Library: 1.2 (see notes for link to Volodymyr Shymanskyy Library which is Ver 1.0)
Platform: Blynk Server

Started the project on the old Blynk version and was close to commercialising the product when the new version was released.
I started with the basics offered by Volodymyr Shymanskyy ( https://github.com/vshymanskyy/blynk-library-python/blob/master/BlynkLib.py )
I worked through the docs and examples and have amended and adjusted this base code to a more robust solution and added a few features to assist with commercialisation. One of the features I added was the “ACK” check option from the server.
The ACK function - Push data to the platform (virtual write for now) and then check the response from the platform (usually “200”) so that confirmation is received to ensure data is transferred.
I have reviewed the uPython version (and upgraded) on the ESP32 as well as have tested on 5. different devices (hardware) and am receiving the same issue.
This issue appears to be more recent (last 2-3 weeks) as I keep getting disconnects.
I did review the “Post Rule” of no more than 20 messages per second (or along those lines), so have eliminated this option.
The main application has completed +/- 80 000 cycles with 5-9 disconnects (this is bad as the old version was 1 in 800 000 cycles), so the platform and mobile apps appear to be ok regarding the “virtual pin” set up.
Also tried the various server locations and this did not help either.
Looked around over the last 2-3 weeks to see any online docs ( and various forums) covering the server side responses, but with no luck

So here I am

Blynk User Interfacing:

Blynk Platform for management usage.
Mobile phones: Samsung (android) and Apple
All 3 these are working well.

Routine basics (for understanding):

data is compiled to the binary string and correct format.
Data pack is pushed to the Blynk server
The “process” routine can have data passed to it or if no data passed it will read from the receive buffer. This read is managed in a loop with maximum 2 second delay before it exits.
if data is found during the loop it will jump out an then process accordingly through a series of “if” statements

The Issue (Debug Log below):

When the device connects and authenticates I get the “200” response.
When I update (Input 1 in this case) 2 parameters are updated
a. The colour property
b. The input state (ON/OFF)
When I update the property (colour) I get the “200” reply
When I update the Input status I don’t get the “200” reply

DEBUG Trace:
Login
(Note: “xxxxx” is where the actual token is)
Push to platform
—> Blynk Log: Connect: Last Received: 16609 :: Last Send: 0 :: Last Ping: 0
—> Blynk Log: Login: Token: xxxxxxxxxxxxxxxxxxxx
—> Blynk Log: > Sending CMD: 29
—> Blynk Log: > Sending ARGS: xxxxxxxxxxxxxxxxxxxx
—> Blynk Log: > Sending KWARGS:
—> Blynk Log: > Sending ID: 1
—> Blynk Log: > Outbound Data Packet CMD: 29, ID: 1, Data: xxxxxxxxxxxxxxxxxxxx
—> Blynk Log: > Outbound Data Packet: b’\x1d\x00\x01\x00 xxxxxxxxxxxxxxxxxxxx’
—> Blynk Log: > Outbound Data b’\x1d\x00\x01\x00 xxxxxxxxxxxxxxxxxxxx’

*Data packet sent and now enter the “process” routine, looking for ID and SUCCESS (200) *
—> Blynk Log: < Check For Data From Server
—> Blynk Log: < Server Data: b’\x00\x00\x01\x00\xc8’

This trace looks for the ID (matches send and receive ID’s) and the SUCCESS (200), if both are valid then data is transferred successfully.
—> Blynk Log: < Inbound Data Packet: CMD: 0, ID: 1,STA: 200
—> Blynk Log: < BIN Data: b’\x00\x00\x01\x00\xc8’
—> Blynk Log: < Server Acknowledge: CMD: 0 ID: 1
—> Blynk Log: Version Info: [‘ver’, ‘1.2.0’, ‘h-beat’, 50, ‘buff-in’, 1024, ‘dev’, ‘esp32 uPython’]
—> Blynk Log: > Sending CMD: 17
—> Blynk Log: > Sending ARGS: ver 1.2.0 h-beat 50 buff-in 1024 dev esp32 uPython
—> Blynk Log: > Sending KWARGS:
—> Blynk Log: > Sending ID: 2
—> Blynk Log: > Outbound Data Packet CMD: 17, ID: 2, Data: ver 1.2.0 h-beat 50 buff-in 1024 dev esp32 uPython
—> Blynk Log: > Outbound Data Packet: b’\x11\x00\x02\x002ver\x001.2.0\x00h-beat\x0050\x00buff-in\x001024\x00dev\x00esp32 uPython’
—> Blynk Log: > Outbound Data b’\x11\x00\x02\x002ver\x001.2.0\x00h-beat\x0050\x00buff-in\x001024\x00dev\x00esp32 uPython’
—> Blynk Log: < Check For Data From Server
—> Blynk Log: < Server Data: b’\x00\x00\x02\x00\xc8’
—> Blynk Log: < Inbound Data Packet: CMD: 0, ID: 2,STA: 200
—> Blynk Log: < BIN Data: b’\x00\x00\x02\x00\xc8’
—> Blynk Log: < Server Acknowledge: CMD: 0 ID: 2

This trace looks for the ID (matches send and receive ID’s) and the “200”, if both are valid then data is transferred successfully.
—> Blynk Log: < Server Upload Successful

Send Ping ( this works as required)
—> Blynk Log: > Sending CMD: 6
—> Blynk Log: > Sending ARGS:
—> Blynk Log: > Sending KWARGS:
—> Blynk Log: > Sending ID: 4
—> Blynk Log: > Outbound Data Packet CMD: 6, ID: 4, Data:
—> Blynk Log: > Outbound Data Packet: b’\x06\x00\x04\x00\x00’
—> Blynk Log: > Outbound Data b’\x06\x00\x04\x00\x00’
—> Blynk Log: < Check For Data From Server
—> Blynk Log: < Server Data: b’\x00\x00\x04\x00\xc8’
—> Blynk Log: < Inbound Data Packet: CMD: 0, ID: 4,STA: 200
—> Blynk Log: < BIN Data: b’\x00\x00\x04\x00\xc8’
—> Blynk Log: < Server Acknowledge: CMD: 0 ID: 4
—> Blynk Log: < Server Upload Successful

Issue starts from here
Update Terminal (App and Platform)
1 Second elapsed, sending data to the server
—> Blynk Log: > Sending CMD: 20
—> Blynk Log: > Sending ARGS: vw 0 —> Loop: 0 RST: 0
—> Blynk Log: > Sending KWARGS:
—> Blynk Log: > Sending ID: 6
—> Blynk Log: > Outbound Data Packet CMD: 20, ID: 6, Data: vw 0 —> Loop: 0 RST: 0
—> Blynk Log: > Outbound Data Packet: b’\x14\x00\x06\x00\x19vw\x000\x00—> Loop: 0 RST: 0\n’
—> Blynk Log: > Outbound Data b’\x14\x00\x06\x00\x19vw\x000\x00—> Loop: 0 RST: 0\n’

This waits a full 2 seconds and no “ACK” is received
—> Blynk Log: < Check For Data From Server

Switch IN 1 ON
—> Blynk Log: > Sending CMD: 20
—> Blynk Log: > Sending ARGS: vw 1 1
—> Blynk Log: > Sending KWARGS:
—> Blynk Log: > Sending ID: 11
—> Blynk Log: > Outbound Data Packet CMD: 20, ID: 11, Data: vw 1 1
—> Blynk Log: > Outbound Data Packet: b’\x14\x00\x0b\x00\x06vw\x001\x001’
—> Blynk Log: > Outbound Data b’\x14\x00\x0b\x00\x06vw\x001\x001’
—> Blynk Log: < Check For Data From Server
No data is received here when the “virtual write” is done

Update the colour property
—> Blynk Log: > Sending CMD: 19
—> Blynk Log: > Sending ARGS: 1 color #00FF00
—> Blynk Log: > Sending KWARGS:
—> Blynk Log: > Sending ID: 12
—> Blynk Log: > Outbound Data Packet CMD: 19, ID: 12, Data: 1 color #00FF00
—> Blynk Log: > Outbound Data Packet: b’\x13\x00\x0c\x00\x0f1\x00color\x00#00FF00’
—> Blynk Log: > Outbound Data b’\x13\x00\x0c\x00\x0f1\x00color\x00#00FF00’
—> Blynk Log: < Check For Data From Server
—> Blynk Log: < Server Data: b’\x00\x00\x0c\x00\xc8’
—> Blynk Log: < Inbound Data Packet: CMD: 0, ID: 12,STA: 200
—> Blynk Log: < BIN Data: b’\x00\x00\x0c\x00\xc8’
—> Blynk Log: < Server Acknowledge: CMD: 0 ID: 12

But here we have a successful payload transaction (this is executed immediately after the previous command
—> Blynk Log: < Server Upload Successful

Switch IN 1 OFF
Here I chanced the “loop” timer to 10 seconds and still nothing.
—> Blynk Log: > Sending CMD: 20
—> Blynk Log: > Sending ARGS: vw 1 0
—> Blynk Log: > Sending KWARGS:
—> Blynk Log: > Sending ID: 20
—> Blynk Log: > Outbound Data Packet CMD: 20, ID: 20, Data: vw 1 0
—> Blynk Log: > Outbound Data Packet: b’\x14\x00\x14\x00\x06vw\x001\x000’
—> Blynk Log: > Outbound Data b’\x14\x00\x14\x00\x06vw\x001\x000’
—> Blynk Log: < Check For Data From Server

So, I must be doing something wrong, if anyone can assist it would be great.

PeteKnight · December 8, 2021, 9:59pm

Hi @SmsAlert I know nothing at all about uPython, but I have looked at the Blynk communication protocol in some detail.
I think you’re making the assumption that the Bklynk server sends a 200 (C8) back to the hardware after each virtual write, and that the uPython library is losing this somehow, but as far as I can tell this isn’t the case.

As you’ve discovered, some commands do return a 200 acknowledgement, but not virtual writes.

If you use C++ and add the #define BLYNK_DEBUG command you get quite a comprehensive output of the communication between the device and the server.
The meaning of these commands and responses are discussed to a certain extent in this topic…

That topic was based on Blynk Legacy, the Blynk IoT protocol definitions have changed slightly since, and are documented here:

github.com

blynkkk/blynk-library/blob/master/src/Blynk/BlynkProtocolDefs.h

/**
 * @file       BlynkProtocolDefs.h
 * @author     Volodymyr Shymanskyy
 * @license    This project is released under the MIT License (MIT)
 * @copyright  Copyright (c) 2015 Volodymyr Shymanskyy
 * @date       Jan 2015
 * @brief      Blynk protocol definitions
 *
 */

#ifndef BlynkProtocolDefs_h
#define BlynkProtocolDefs_h

enum BlynkCmd
{
    BLYNK_CMD_RESPONSE       = 0,

    BLYNK_CMD_PING           = 6,

    BLYNK_CMD_BRIDGE         = 15,

This file has been truncated. show original

I think the only way you’re going to get the acknowledgement you need is to call a sync_virtual immediately after your virtual_write and compare the two values.

Pete.

SmsAlert · December 9, 2021, 8:20am

Hi Pete

Thanks for the feedback, yip uPython is different but have to admit that working on this project did bring the design time down. And have to admit it works like a charm, except for this little hiccup.

When I started with Volodymyr’s code in the original version, it just worked and was a real winner, there were no concerns on the server responses so did not even look at this at that stage. Just before we were moving to production, the new servers came online

When implementing the new library, it was a back to basics type scenario so looking around the web and GitHub I had quite a bit of hacking to do from the original code (C++, Arduino and etc), and I did find the server “instructions” and “responses” and then implemented them in code, so the added defines (responses and instructions) were added and am in the process of implementing them.

‘’’
#------------------------------------------------------------------------------

Global Defines

#------------------------------------------------------------------------------
MSG_RSP = const(0) # Server Response to Received Message
MSG_LOGIN = const(2) # Login Command
MSG_PING = const(6) # Ping Command
MSG_TWEET = const(12) # Not Tested
MSG_EMAIL = const(13) # Email Command
MSG_NOTIFY = const(14) # Notification Command (APP Only)
MSG_BRIDGE = const(15)
MSG_HW_SYNC = const(16) # Syncronise Hardware Command
MSG_INTERNAL = const(17) # Internal Message Command
MSG_SEND_SMS = const(18) # Not Tested
MSG_PROPERTY = const(19) # Property Command
MSG_HW = const(20) # Hardware Command
MSG_HW_LOGIN = const(29) # Login Command
MSG_REDIRECT = const(41) # TODO: not implemented
MSG_DBG_PRINT = const(55) # TODO: not implemented
MSG_EVENT_LOG = const(64) # Event logging Command (Must be defined on the platform and then Linked)
MSG_EVENT_CLEAR = const(65) # Not Tested

Server Responses

STA_QUOTA_LIMIT_EXCEPTION = const(1) # TODO: not implemented
STA_ILLEGAL_COMMAND = const(2) # TODO: not implemented
STA_NOT_REGISTERED = const(3) # TODO: not implemented
STA_ALREADY_REGISTERED = const(4) # TODO: not implemented
STA_NOT_AUTHENTICATED = const(5) # TODO: not implemented
STA_NOT_ALLOWED = const(6) # TODO: not implemented
STA_DEVICE_NOT_IN_NETWORK = const(7) # TODO: not implemented
STA_NO_ACTIVE_DASHBOARD = const(8) # TODO: not implemented
STA_INVALID_TOKEN = const(9) # TODO: not implemented
STA_ILLEGAL_COMMAND_BODY = const(11) # TODO: not implemented
STA_GET_GRAPH_DATA_EXCEPTION = const(12) # TODO: not implemented
STA_NTF_INVALID_BODY = const(13) # TODO: not implemented
STA_NTF_NOT_AUTHORIZED = const(14) # TODO: not implemented
STA_NTF_ECXEPTION= const(15) # TODO: not implemented
STA_TIMEOUT = const(16) # TODO: not implemented
STA_NO_DATA_EXCEPTION = const(17) # TODO: not implemented
STA_DEVICE_WENT_OFFLINE = const(18) # TODO: not implemented
STA_SERVER_EXCEPTION = const(19) # TODO: not implemented
STA_NOT_SUPPORTED_VERSION = const(20) # TODO: not implemented
STA_ENERGY_LIMIT = const(21) # TODO: not implemented
STA_OPERATION_ACCEPT = const(23) # TODO: not implemented
STA_OPERATION_DECLINE = const(24) # TODO: not implemented
STA_SUCCESS= const(200) # Data Uploaded Successfully

Connection Status

DISCONNECTED = const(0)
CONNECTING = const(1)
CONNECTED = const(2)
AUTHENTICATED = const(3)
‘’’

Just seems strange that the Success(200) response from the server would be limited to certain instructions and not others, I do have a virtual sync in use, but have attached this to an output (app switch), temp max (app numeric input) and temp min (app numeric input) so the devices sends a notification if above or below these settings, ane keeps the app / web portal and edge device in sync.

From a design perspective to push data to the server and not get some form of ACK leaves too many flaws in the design and deeming the reliability of the data transfer to a poor quality. So implementing the virtual sync after each write just makes the code slower, and does not make logical sense so not sure if you agree with me here…

Ah, on the C++, yeah, am a little out dated in coding here, mostly spend time in Python and C, will see if there is a way to get the debug comms between the server and edge device in python. If you have time to check on your side, we can maybe compare notes.

Awesome find (thanks for that), did not even realise that it is on the virtual write only that this happens, so will focus there to see if there is something in the way I am pushing the data (int, str etc)

P.S. Saw your thread on the home automation project, happy to ship you one of our devices once we ironed out all the niggly’s to add to your project

Monitor 2 temps (-55 to +125)
Enviro sensor (Ambient, Baro, Humodity)
2 x Inputs (doors or what ever)
1 Output (1A NC/NC)
Flood Sensing

PeteKnight · December 9, 2021, 9:16am

I don’t really understand this statement.
In C++ Blynk.syncVirtual(vPin) is used to force the server to send the latest value for vPin to the device. This is normally used at start-up or re-connection, to get the app (or web dashboard) to send the latest widget values.

When these widget values are sent, the BLYNK_WRITE(vPin) callback is triggered. This callback contains the code that you want to be processed when the widget value changes.

So, changes in widget values such as your “output (app switch), temp max (app numeric input) and temp min (app numeric input)” would be handled by the BLYNK_WRITE(vPin) callback. The Blynk.syncVirtual(vPin) is just a way to get these widget values returned without ‘dirtying’ the widget settings.

Yes, I agree, but I think that’s the very reason why an acknowledgement wasn’t implemented for virtualWrites. People like to keep blasting the server with dozens of virtual writes per second, and if an acknowledgement was sent for each of them then it would slow things down.

I hadn’t done any debug testing on the new Blynk IoT, so I tested things with C++ yesterday.
I used a simple sketch which wrote a data value to pin V0 every two seconds, and implemented the value each time.
When the value reaches 10 a Blynk.syncVirtual(V0) is called. This is the serial output…

08:47:35.609 -> [4309] Connected to WiFi
08:47:35.609 -> [4309] IP: 192.168.1.51
08:47:35.609 -> [4309] 
08:47:35.609 ->     ___  __          __
08:47:35.609 ->    / _ )/ /_ _____  / /__
08:47:35.609 ->   / _  / / // / _ \/  '_/
08:47:35.609 ->  /____/_/\_, /_//_/_/\_\
08:47:35.609 ->         /___/ v1.0.1 on ESP8266
08:47:35.609 -> 
08:47:35.609 -> [4319] Connecting to blynk.cloud:80
08:47:35.609 -> [4353] <[1D|00|01|00] AUTH TOKEN REDACTED
08:47:35.660 -> [4367] >[00|00|01|00|C8]
08:47:35.660 -> [4367] Ready (ping: 13ms).
08:47:35.660 -> [4367] Free RAM: 48472
08:47:35.700 -> [4434] <[11|00|02|00]over[00]1.0.1[00]h-beat[00]45[00]buff-in[00]1024[00]dev[00]ESP8266[00]fw-type[00]TEMPLATE ID REDACTED[00]build[00]Dec  8 2021 21:37:46[00]tmpl[00]TEMPLATE ID REDACTED[00]
08:47:35.700 -> [4452] >[00|00|02|00|C8]
08:47:37.696 -> [6441] <[14|00|03|00|06]vw[00]0[00]1
08:47:39.710 -> [8441] <[14|00|04|00|06]vw[00]0[00]2
08:47:41.709 -> [10441] <[14|00|05|00|06]vw[00]0[00]3
08:47:43.728 -> [12441] <[14|00|06|00|06]vw[00]0[00]4
08:47:45.714 -> [14441] <[14|00|07|00|06]vw[00]0[00]5
08:47:47.709 -> [16441] <[14|00|08|00|06]vw[00]0[00]6
08:47:49.731 -> [18441] <[14|00|09|00|06]vw[00]0[00]7
08:47:51.691 -> [20441] <[14|00|0A|00|06]vw[00]0[00]8
08:47:53.714 -> [22441] <[14|00|0B|00|06]vw[00]0[00]9
08:47:55.707 -> [24441] <[14|00|0C|00|07]vw[00]0[00]10
08:47:55.707 -> Sync called here.....................
08:47:55.758 -> [24509] <[10|00|0D|00|04]vr[00]0
08:47:55.799 -> [24518] >[14|00|0D|00|09]
08:47:55.799 -> [24519] >vw[00]0[00]10.0
08:47:55.799 -> Sync was just processed..............
08:47:57.734 -> [26441] <[14|00|0E|00|07]vw[00]0[00]11
08:47:59.729 -> [28441] <[14|00|0F|00|07]vw[00]0[00]12
08:48:01.703 -> [30441] <[14|00|10|00|07]vw[00]0[00]13
08:48:03.684 -> [32441] <[14|00|11|00|07]vw[00]0[00]14
08:48:05.711 -> [34441] <[14|00|12|00|07]vw[00]0[00]15

As you’ll see, all the traffic is one way, from the device to the server, once this incrementing loop starts…

<[14|00|03|00|06]vw[00]0[00]1
<[14|00|04|00|06]vw[00]0[00]2
<[14|00|05|00|06]vw[00]0[00]3

until the loop value == 10, then the syncVirtual is called…

<[14|00|0C|00|07]vw[00]0[00]10
Sync called here.....................
<[10|00|0D|00|04]vr[00]0    // [virtual read for V0 sent from hardware to server]
>[14|00|0D|00|09]           // [server responds, but I don't understand this] 
>vw[00]0[00]10.0            // [server sends latest V0 value of 10.0 (float)]
Sync has been processed..............
<[14|00|0E|00|07]vw[00]0[00]11

As you can see, there are no acknowledgements in C++ until a syncVirtual is called.

Is this a purely Blynk device, or does it do MQTT too?
As you’ve probably seen, I don’t actually run any Blynk code on my devices everything goes via MQTT and Node-Red.

Pete.

SmsAlert · December 9, 2021, 11:41am

ah. this makes sense

Just for Clarity

Only difference in our testing

Using a secure connection
Connect: Server: “blynk.cloud” : Port: “443”

0x14 = 20 > I have this as a Hardware Message in my “defines”
0x10 = 16 > I have this as Hardware Sync in my defines
0x0D = 13 > Usually Message ID position
0x09 = 9 > Usually response “code” (0xC8 is Success) I have this as Invalid Token in my defines

so to put this all together: Hardware Invalid Token (not properly defined, str, int, double) is how I would read this.

Yes that is exactly how it is used, so on each connect I sync the necessary parameters (as per on the App/Platform):
Temp 1 Max setting
Temp 1 Min setting
Temp 2 Max setting
Temp 2 Min setting
Output State

Forgive the minimal info, forget that others can’t read my mind…

This is where the solution lies (I think), based on the “blasting” methods used, this would make perfect sense why this is stated in the documentation
“Make sure you don’t send a huge number of virtualWrite commands per seconds. Blynk Cloud doesn’t allow to send more than 20 requests per seconds. Connections that exceeds the limit are closed”

Yes unfortunately

Pete, If we agree on the below resolve, then we can close this thread.

So to conclude (for sake of those that follow later):

The “virtual write” issue, there will be no ACK (200) after the action is executed, as this allows for fast data push from the edge and the ACK (200) will just delay the data transfer process (improve real time monitoring, I guess)
Other commands, that are critical i.e. Login, properties, and the likes will get an ACK (200) as confirmation of data received by the server. but is the responsibility of the developer to “use or lose” as needed.

PeteKnight · December 9, 2021, 12:02pm

Yes, that’s how I read it, but it makes no sense as the Auth token has already been validated, and the token wasn’t sent as part of the previous command. Hence my “I don’t understand this” comment. Either way, it’s not really relevant to your problem.

That’s my take on it, and that if you want to have that confirmation then you need to have a process that consists of…

Virtual write value x to a virtual pin
Force a sync on that virtual pin and retrieve the current value
Check that current value == x

As far as your new hardware is concerned, if you want to extend the potential market then adding-in some MQTT functionality might expand the appeal of your device.
Lots of people use MQTT as the basis for their HA and even industrial automation systems.
Home Assistant is also a growing market and you could either add native HA integration, or allow HA users to build their own based on MQTT.
Of course you could add this at a later date, via a firmware update if you wished.

Pete.

SmsAlert · December 9, 2021, 12:22pm

Thanks Pete, appreciate the help, now off to some other bugs… and we can take this topic as solved and closed

Yeah, did think around this, hence the Blynk, was a faster time to market, busy with some NB-IoT stuff in the MQTT / LwM2M options, will probably move away from Blynk on these projects, busy eval on a local company offer for the MQTT hosting, might give the Blynk a try in these…

PeteKnight · December 9, 2021, 1:02pm

I’ve marked it as Solved.
We usually leave topics open so that others can post comments later if they wish. Sometimes that’s a disadvantage (4 years later someone says “I have exactly the same problem, how did you solve it?”) but generally it works well.

Pete.

SmsAlert · December 9, 2021, 2:23pm

perfect, Thanks Pete