I’m losing the first few characters, and only the first few when my ESP8266 sends to the terminal on my Blynk app on my Android phone. The data is an ASCII log file from a controller consisting of around 10 lines of 16 characters, each suffixed with CR and LF. As nobody else seems to have this issue it points to a problem with the data itself which is a simple 38,400 bauds serial stream sent in response to a request from the ESP8266. The data flow is:
Button on Blynk app sends to ESP8266
ESP8266 picks it up and sends a serial command to the controller (actually 0x02,0x4C,0x4C,0x0D,0x0A)
Controller responds with the log data text
ESP8266 sends the text to the virtual terminal on the Blynk app.
If I connect directly to the controller with a PC serial application it works perfectly every time.
The data is serial and meets all the voltage requirements (3.3V uni-polar) and a logic analyzer on the serial lines confirm the data is as expected. The only unusual aspect is the controller can intentionally start a reply with either 0x04 or 0x05 as part of the data routing. These are intercepted as the first character in the response and used to control indicators (LEDs) or to say the following text is a push message. Any response that does not start with those bytes is passed verbatim to the terminal window.
Question: How does the terminal window in the Blynk app respond to non-printable ASCII characters? Does it for example recognize or do anything when it sees ASCII values lower than 0x20 (space character)?
I seem to remember that Blynk uses UTF-8 strings, so anything below 20 isn’t likely to display correctly. If you’re not to hung-up about seeing EXACTLY what is coming out of your ASCII file then you could always replace them with placeholders that indicate their function (you can even use emojis if you wish)
I hadn’t considered it using UTF-8 but the odd thing is it does work reliably if one of those control codes is the first character. Basically, the controller prefixes 0x04 to the string and the ESP8266 converts the next characters to enable or disable virtual LEDs in the app. Similarly 0x05 tells it to take the rest of the text (it is a status alert) and send it as a push notification. All that works fine, it is strings that start with plain text that don’t work and even then it is only the first few characters. I’m leaning towards the idea that there is some timing constraint between the controller getting its command and the response being sent back. The connection to the Android phone is via WiFi to a router then by landline to the internet and back via 3G. The internet connection here is jokingly called ‘superfast’ which in reality means it occasionally peaks at 1Mb but is usually around 128Kb/s!
I wrote the controller code but it is nearly filling the 64K program memory in a PIC so I’m hesitant to make it any bigger. I might try adding another prefix to terminal messages to see if it ‘works around’ the problem.
Still banging my head with this problem. Tests show it has nothing at all to do with control characters but the issue persists. I investigated by connecting my browser to the cloud and seeing what had been sent from the ESP8266 and the beginning of the messages are also missing there so it points to a problem on the transmission side. I have tried delays and even splitting the block of information into two halves and sending each with a ‘terminal.flush()’ after them but it didn’t fix things.
My theory is a clash of timing between the ESP8266 receiving and storing bytes and the wireless side sending them to the cloud. I tried pacing the bytes send to the serial port but that didn’t help. What is consistent is the failure pattern, sometimes it works fine, most of the time it misses the same length (about 60 characters) from the beginning. This is the offending chunk of code:
void CheckSerial() // triggered by timer, update app terminal with serial RX data
{
int PacketSize;
String Packet;
if (Serial.available())
{
digitalWrite(LED_BUILTIN, LOW); // internal blue LED on
HostString = Serial.readStringUntil(0x1A);
PacketSize = HostString.length();
switch(HostString[0])
{
case 0x03: // header for LED update
if(HostString.indexOf('Y') != -1) MAIL_LED.on(); else MAIL_LED.off();
if(HostString.indexOf('G') != -1) GATE_LED.on(); else GATE_LED.off();
if(HostString.indexOf('B') != -1) BELL_LED.on(); else BELL_LED.off();
if(HostString.indexOf('W') != -1) OSL_LED.on(); else OSL_LED.off();
break;
case 0x04: // text alert header
Blynk.notify("Alert: "+ HostString);
break;
default: // terminal window has no prefix
{
terminal.clear();
delay(200);
//Packet = HostString.substring(0,PacketSize / 2);
//terminal.print(Packet);
//terminal.flush();
//Packet = HostString.substring(PacketSize / 2, PacketSize);
//terminal.print(Packet);
terminal.print(HostString);
}
}
terminal.flush();
HostString = ""; // Clear holding String
Packet = "";
digitalWrite(LED_BUILTIN, HIGH); // internal blue LED off
}
}
void SyncToNTP(void) // read NTP and format clock setting command
{
timeClient.begin(); // Get NTP time and format as a clock setting command
timeClient.update();
time_t utcCalc = timeClient.getEpochTime();
if((year(utcCalc) / 100) == 20) // crude check the date is valid (=20xx)
{
sprintf(HostSetTime,"C%02d%02d%02d%02d%02d%02d",hour(utcCalc),minute(utcCalc),second(utcCalc),day(utcCalc),month(utcCalc),year(utcCalc)%100);
Serial.write(0x02); // STX
Serial.println(HostSetTime);
}
}
///////////////////////////////////////////////////////////
// Basic OS routines
void setup()
{
pinMode(LED_BUILTIN, OUTPUT);
digitalWrite(LED_BUILTIN, HIGH); // internal blue LED off
Serial.begin(38400);
Blynk.begin(auth, ssid, pass);
NET_LED.on();
timer.setInterval(2000L, CheckSerial);
}
//In the loop function include Blynk.run() command.
void loop()
{
Blynk.run();
timer.run();
}
The remaining code is to do with handling requests from the Blynk app on the Android device and converting them to serial commands to my controller and all that works fine.
Any ideas? (shooting the programmer is out of the question I’m afraid)
design your own protocol avoiding the control character, for example instead of sending control char 0x04, you send something else (for example ASCII 0x04, or *04, etc.)
Decode at your ESP8266, then send correct command, by replacing your something else with control chars to the controller.
you can design and add ACK and RESEND protocol to prevent package loss on the Internet, etc.
It’s really difficult to find out and solve the vague issue without the real experiment and equipment. I’m afraid only you can solve this.
If you have more time, read the BlynkProtocol.h to find out more why it’s rejecting your control chars as it misunderstands your control chars as its command, then it finds out it’s not in correct Blynk protocol format, then rejects the whole package.
BlynkProtocol.h is still has some kind of bug, generating Packet Too Big, etc… errors if you’re using some kind of Serial Interface, such as ESP-01 AT command, BlueTooth.
I’m was in middle of finding out and fixing it some time before when I was trying to write, not successfully yet, the Blynk WiFiManager for Arduino UNO Mega / ESP-01 using Serial / Stream interface . But I’ve deferred this job for later as I’ve finished that of ESP8266 / ESP32.
Firstly, many thanks to you Pete for assisting. Increasing the RXBufferSize did fix the receive problem. Looks like I approached the problem from the wrong angle, I assumed that because it occasionally worked correctly the fault was when it didn’t receive the rest of the time but in fact it was pure luck the timing was adequate when it did work!
The control characters seems to be a red herring, changing them makes no difference and I can’t see any clashes in the protocol that would cause issues.
However, I still can’t get some of the program working and it seems to be for a similar reason. This code doesn’t work reliably:
‘’’
sprintf(ToTerm,“0=%02X 1=%02X 2=%02X 3=%02X 4=%02X”,LogChars[0], LogChars[1], LogChars[2], LogChars[3], LogChars[4]);
terminal.println(ToTerm);
terminal.flush();
delay(200);
‘’’
There seemed to be all kinds of problems with String functions when the length is long (my data is ~700 bytes of text) so I captured everything into a char array instead and scan it for delimiters. The capture is fine but I can’t sync the LED widgets reliably. The original code sent a string containing the letters ‘BGWY’ (Blue Green White Yellow) in any combination if the corresponding Blynk widget was to illuminate on the Android device, it wasn’t reliable. I tried a different method of encoding the LED states into a bit map in a single byte in the controller and decoding them to Blynk commands in the ESP8266 but with the same result. The debugging code in the fragment posted above shows the bytes are all as expected. My guess is a flood problem by sending five sequential commands. Before I try using a timer to pace the commands, is there a better way to replicate multiple LED widgets ‘in one go’ or is the only option to set/reset them sequentially?
Flooding usually occurs if you send more than 10 commands per second to the Blynk server (or maybe fewer commands less than 100ms apart).
You can use a small delay between sending the commands, and maybe put a Blynk.run(); command after sending the first command and before initiating the small delay, to prevent a disconnection.
Thanks Pete.
From results here it looks like the flood is based on 100mS rather than the absolute number per second. I’m not sure if delay() is a blocking routine or not and this application handles serial data packets at unpredictable times so I did it using a timer instead. If it helps anyone else, this is the relevant code:
void SetLeds()
{
if(LedsNeedSetting == 0) return; // exit ASAP if nothing to do
char LedSwitch = LedState & BitMask;
switch(BitMask)
{
case 0x01: if(LedSwitch) BELL_LED.on(); else BELL_LED.off(); // Blue
break;
case 0x02: if(LedSwitch) GATE_LED.on(); else GATE_LED.off(); // Green
break;
case 0x04: if(LedSwitch) ALARM_LED.on(); else ALARM_LED.off(); // Red
break;
case 0x08: if(LedSwitch) OSL_LED.on(); else OSL_LED.off(); // White
break;
case 0x10: if(LedSwitch) MAIL_LED.on(); else MAIL_LED.off(); // Yellow
break;
default: BitMask = 0x01;
LedsNeedSetting = 0;
}
BitMask = BitMask << 1;
}
///////////////////////////////////////////////////////////
// Basic OS routines
void setup()
{
pinMode(LED_BUILTIN, OUTPUT);
digitalWrite(LED_BUILTIN, HIGH); // internal blue LED off
Serial.begin(38400);
Serial.setRxBufferSize(1024);
Blynk.begin(auth, ssid, pass);
timer.setInterval(2000L, CheckSerial);
timer.setInterval(300L, SetLeds);
}
//In the loop function include Blynk.run() command.
void loop()
{
Blynk.run();
timer.run();
}
It is called with the bits set in ‘LedState’ and ‘LedsNeedSetting’ = 1 to enable the event and it turns itself off after reading bit 4. Note an extra LED has been added since the last post. It seems to be very reliable now.
There is more to add yet but now the communications problems are sorted it should be fairly easy. The RxBufferSize was causing the receive loss and flooding was the transmission problem. Note that the ‘SetLeds’ routine in my last post will only follow one ‘case’ per tick of the timer so it nicely paces the commands to the app. I considered making the ‘BitMask’ a static variable and passing the ‘LedState’ as a function parameter but I feared problems with the routine being entered periodically by the timer without the parameters being set properly each time so I made them global instead.
Excellent, glad we got to the bottom of it.
The RxBuffer thingy was something I came across when I was trying to mirror serial output to a terminal widget. I kept randomly losing bits of data and something you said in one of your posts sounded similar, hence the suggestion.
Anyway, good luck with the rest of the project. Shout if you get stuck and we’ll try to help.