I need to read a TXT file from TextUploader.com.
This host reply with a RAW text data after a GET to a personal account page like this one http://textuploader.com/dscre.
So, I need to capture the RAW text data replied from TextUploader.com in a Vpin.
I think is needed a BLYNK_WRITE for this widget? It’s possibile?
Good! I will wait! 
You could make it yourself. I have an example of how to get stuff from the Web, if you want.
Yes! How I can make it?
I made a little function to get some values from an API. It returns a 0 or 1 based on success of failure (like a pro!). It’s based on the ESP board (Wemos) board, so I’m not really sure how to make it for Ethernet, but the principle will be the same I think. The function below also does a little parsing, but you can probably figure that out ;). It fills a variable (array) with each line of text retrieved from the URL/textfile/callback/however you want to call it.
char server[] = "api.sunrise-sunset.org";
WiFiClient client;
bool getrawdata()
{
if(debug) { Serial.println("Disconnecting Blynk"); }
Blynk.disconnect(); // Disconnect Blynk, cause API call has to be made
// Put current date in for retrieving correct date
char currentDateBuffer[12];
sprintf(currentDateBuffer,"%04u-%02u-%02u ", year(currentTime), month(currentTime), day(currentTime) );
delay(750);
// Make request http://api.sunrise-sunset.org/json?lat=36.7201600&lng=-4.4203400&date=2016-08-25 (with variable data of course)
if (client.connect(server, 80))
{
if(debug) { Serial.println("Connecting to server ..."); }
client.print("GET /json?lat=53.097001&lng=6.054036&date=");
client.print(currentDateBuffer);
client.println(" HTTP/1.1");
client.println("Host: api.sunrise-sunset.org");
client.println("Accept: text/html");
client.println("Connection: close");
client.println();
if(debug) { Serial.println("Connected to server"); }
delay(750);
// Read all characters into one big, noisy array full of stuff (contents[])
if(debug) { Serial.println("Now reading data into temp var"); }
int i=0;
String contents[15];
String inputString = "";
inputString.reserve(200);
while (client.available())
{
char c = client.read();
inputString += c;
if (c == '\n')
{
inputString.trim(); // Remove dumbass whitespaces and too much CR, LF etc.
if(debug) { Serial.print(i); }
if(debug) { Serial.print(": Received non-useless data, processing: "); }
if(debug) { Serial.println(inputString); }
contents[i] = inputString;
inputString = "";
i++;
}
}
if(contents[0] == "HTTP/1.1 200 OK")
{
if(debug) { Serial.println(contents[0]); }
if(debug) { Serial.println("Stopping client"); }
client.stop();
// Result set to this var
useFullData = contents[11];
if(debug) { Serial.println("Connecting Blynk"); }
Blynk.connect();
return 1;
}
else
{
if(debug) { Serial.print("Error received: "); }
if(debug) { Serial.println(contents[0]); }
return 0;
}
}
}
2 Likes
@Lichtsignaal I put your extract into a full Blynk sketch yesterday and got up at 3.30am this morning to see the sunrise.
It was pitch black for 3 hours!!!
http://sunrise-sunset.org/api confirms the times given by the API are UTC, not local time.
Website for Nicoisa at http://sunrise-sunset.org/search?location=Nicosia%2C+Cyprus does show local times of around 6.30am.
Can you make any sense of the yearly graph at the bottom of the Nicosia page? I understand the 2 jumps during the year for DST but I can’t understand why it shows around 9am for sunrise times during the summer when it is 3 hours earlier than this. Can’t be a UTC issue as that would be 3 hours the opposite way i.e. 3am.
We use the TimeLord library for Sunrise and Sunset data. I notice the example they provide on Github is written by none other than “Mr Arduino” Nick Gammon https://github.com/probonopd/TimeLord/blob/master/examples/SunriseSunset/SunriseSunset.ino
Okay, but now we are a little bit off topic… the post is about read data from web with Webhook…
If we would talk about Sunset and Sunrise with arduino… same as @Costas I use a library, in details I use a fork of TimeLord that is from André Gonçalves .
@naamah75 not off topic at all. Your OP asked about something that is not currently available from Blynk.
@Lichtsignaal said try DIY. You asked how and he provided the details.
I was merely pointing out that there are existing libraries available for many Webhooks rather than re-inventing the wheel.
Well, that it is, but it’s also an example on how to retrieve data from a web url.
@Costas the graph represents the sunset/sunrise times, but not the twilight times. Twilight happens in advance of sunrise and after sunset. And those are the ones which takes longer if you are further from the equator. And yes, the times are UTC, I convert them myself in another piece of code later on 
Furthermore, the API gives back different types of sunrise/sunset times whereas the TimeLord library only uses one algorithm, so each has it’s advantages/disadvantages
Only thing I can think of is that they messed up their timezones somewhere or maybe the webpage converts for local time?
I have no idea tbh. xD
@Lichtsignaal I liked your extract as it provides easy parsing of API calls without the normal additional json libraries. I was having trouble with cookies from wunderground.com which meant the index wasn’t constant.
Eventually got it running and the full sketch to parse sunrise and sunset times, without disconnecting from the Blynk server, is provided below.
/* Webhooksv2a.ino based on sketch extract by @Lichtsignaal
* http://community.blynk.cc/t/read-data-from-the-web-with-webhook/8334/6
* Terminal on V0 and button in PUSH mode on V2
*/
//#define BLYNK_DEBUG // enable for debugging Blynk problems
#define BLYNK_PRINT Serial // Comment this out to disable prints and save space
#include <ArduinoOTA.h>
#include <ESP8266WiFi.h>
#include <BlynkSimpleEsp8266.h>
#include <SimpleTimer.h> // Essential for almost all sketches
SimpleTimer timer;
WidgetTerminal terminalW(V0);
char server[] = "api.wunderground.com"; // same as: const char* server = "api.wunderground.com";
WiFiClient client;
#define debug 0 // reduced Serial Monitor output
bool validData = false; // assume bad data until validated
unsigned int sunriseseconds;
unsigned int sunsetseconds;
String sunrise;
String sunset;
int foundsunrise;
int foundsunset;
unsigned int numberofattempts = 3; // set maximum number of attempts to try to get valid data from webservice
// will try 4 times from V2 but 1 attempt is ok now
char OTAhost[] = "Webhook"; // Optional, but very useful
char ssidstr[] = "OfficeGargoyle"; // enter your Router SSID
char passstr[] = "1234567890"; // enter your Router AP password
char authstr[] = "AB012345678901234567890123456789"; // enter your Blynk token
char serverstr[] = "blynk-cloud.com"; // change to IP for local server connection
char WUNDERGROUND_REQ[] =
"GET /api/API_KEY/astronomy/q/COUNTRY/CITY.json HTTP/1.1\r\n" // enter your WUNDERGROUD API_KEY, COUNTRY and CITY
"User-Agent: ESP8266/0.1\r\n"
"Accept: */*\r\n"
"Host:api.wunderground.com\r\n"
"Connection: close\r\n"
"\r\n";
BLYNK_WRITE(V2){
int getWUsunrise = param.asInt();
terminalW.println();
if(getWUsunrise == 1){
int datachecks = 0;
while(validData == false){
getrawdata(); // keep checking for valid data
if(validData == false){ // rechecked after calling getrawdata();
terminalW.println("Invalid data, will try again");
terminalW.flush();
}
datachecks++;
if(datachecks > numberofattempts){
Serial.println("Check API");
terminalW.println("Check API");
terminalW.flush();
break; // tried enough times so bail out of while loop
}
}
gotValidData();
}
}
void gotValidData(){
if(validData == true){
terminalW.println("Data check passed");
terminalW.print("Sunrise seconds ");
terminalW.println(sunriseseconds);
terminalW.print("Sunset seconds ");
terminalW.println(sunsetseconds);
terminalW.flush();
validData = false; // resetting flag for further use
}
}
void reconnectBlynk() { // reconnect to server if we get disconnected
if (!Blynk.connected()) {
if(Blynk.connect()) {
BLYNK_LOG("Reconnected");
terminalW.print("Reconnected with local IP of ");
terminalW.println(WiFi.localIP());
terminalW.flush();
} else {
BLYNK_LOG("Not reconnected");
}
}
}
void getrawdata()
{
if(debug) { Serial.println("Disconnecting Blynk"); }
//Blynk.disconnect(); // Disconnect Blynk, cause API call has to be made
//delay(750);
int i=0;
String contents[65]; // maximum number of lines, not characters per line
String inputString = "";
foundsunrise = 0;
foundsunset = 0;
inputString.reserve(1400); // actual data approx 662 plus headers say 438 (measured size is around 1071 characters BEFORE trim
int countcharacters = 0;
if(debug) {
Serial.print("connecting to ");
Serial.println(server);
}
WiFiClient client; // Use WiFiClient class to create TCP connections
const int httpPort = 80;
if (!client.connect(server, httpPort)) {
if(debug) { Serial.println("connection failed"); }
return;
}
String url = "/api/527c5df849ee9364/astronomy/q/Cyprus/Paralimni.json HTTP/1.1\r\n"; // We now create a URI for the request
if(debug) {
Serial.print("Requesting URL: ");
Serial.println(url);
}
client.print(String("GET ") + url + " HTTP/1.1\r\n" + // This will send the request to the server
"Host: " + server + "\r\n" +
"Connection: close\r\n\r\n");
unsigned long timeout = millis();
while (client.available() == 0) {
if (millis() - timeout > 5000) {
if(debug) { Serial.println(">>> Client Timeout !"); }
client.stop();
return;
}
}
while(client.available()){ // Read all the lines of the reply from server and print them to Serial
char c = client.read();
inputString += c;
countcharacters++;
if (c == '\n')
{
inputString.trim(); // Remove dumbass whitespaces and too much CR, LF etc.
//if(inputString.startsWith("Set-Cookie:") == false){ // skip any lines that refer to setting cookies as they are once / session
if(debug) {
Serial.print(i);
Serial.print(": Received: ");
Serial.println(inputString);
}
contents[i] = inputString;
i++;
yield();
if(foundsunrise == 0){ // ensures we only search for 1st instance
if(inputString.substring(1, 8) == "sunrise"){ // sunrise
foundsunrise = i;
if(debug) {
Serial.print("Found sunrise at index ");
Serial.println(foundsunrise);
}
}
}
if(foundsunset == 0){ // ensures we only search for 1st instance
if(inputString.substring(1, 7) == "sunset"){ // sunset
foundsunset = i;
if(debug) {
Serial.print("Found sunset at index ");
Serial.println(foundsunset);
}
}
}
inputString = "";
}
}
if(debug) {
Serial.println();
Serial.println("closing connection");
}
if((foundsunrise != 0) && (foundsunset != 0)){
validData = true;
if(debug) {
Serial.println("Data is valid");
Serial.println(contents[foundsunrise]);
Serial.println(contents[foundsunrise + 1]);
Serial.println(contents[foundsunset ]);
Serial.println(contents[foundsunset + 1]);
}
int risehourlength = (contents[foundsunrise].length()); // length 11 means before 10am i.e. single digit number
int riseminutelength = (contents[foundsunrise + 1].length()); // length 12 means less than 10 minutes passed the hour i.e. single digit number
int sethourlength = (contents[foundsunset].length()); // length 11 means before 10am i.e. single digit number
int setminutelength = (contents[foundsunset + 1].length()); // length 12 means less than 10 minutes passed the hour i.e. single digit number
if(debug) { Serial.print("Sunrise at "); }
if(risehourlength == 11){
if(debug) {
Serial.print("0");
Serial.print(contents[foundsunrise].charAt(8));
}
sunriseseconds = ((contents[foundsunrise].charAt(8) - '0') * 3600);
}
else{
if(debug) {
Serial.print(contents[foundsunrise].charAt(8));
Serial.print(contents[foundsunrise].charAt(9));
}
sunriseseconds = ((contents[foundsunrise].charAt(8) - '0') * 10 * 3600) + ((contents[33].charAt(9) - '0') * 3600);
}
if(debug) { Serial.print(":"); }
if(riseminutelength == 12){
if(debug) {
Serial.print("0");
Serial.println(contents[foundsunrise + 1].charAt(10));
}
sunriseseconds = sunriseseconds + ((contents[foundsunrise + 1].charAt(10) - '0') * 60);
}
else{
if(debug) {
Serial.print(contents[foundsunrise + 1].charAt(10));
Serial.println(contents[foundsunrise + 1].charAt(11));
}
sunriseseconds = sunriseseconds + ((contents[foundsunrise + 1].charAt(10) - '0') * 10 * 60) + ((contents[foundsunrise + 1].charAt(11) - '0') * 60);
}
if(debug) {
Serial.print("Seconds from midnight to SUNRISE: ");
Serial.println(sunriseseconds);
}
if(debug) { Serial.print("Sunset at "); }
if(sethourlength == 11){
if(debug) {
Serial.print("0");
Serial.print(contents[foundsunset].charAt(8));
}
sunsetseconds = ((contents[foundsunset].charAt(8) - '0') * 3600);
}
else{
if(debug) {
Serial.print(contents[foundsunset].charAt(8));
Serial.print(contents[foundsunset].charAt(9));
}
sunsetseconds = ((contents[foundsunset].charAt(8) - '0') * 10 * 3600) + ((contents[foundsunset].charAt(9) - '0') * 3600);
}
if(debug) { Serial.print(":"); }
if(setminutelength == 12){
if(debug) {
Serial.print("0");
Serial.println(contents[foundsunset + 1].charAt(10));
}
sunsetseconds = sunsetseconds + ((contents[foundsunset + 1].charAt(10) - '0') * 60);
}
else{
if(debug) {
Serial.print(contents[foundsunset + 1].charAt(10));
Serial.println(contents[foundsunset + 1].charAt(11));
}
sunsetseconds = sunsetseconds + ((contents[foundsunset + 1].charAt(10) - '0') * 10 * 60) + ((contents[foundsunset + 1].charAt(11) - '0') * 60);
}
if(debug) {
Serial.print("Seconds from midnight to SUNSET: ");
Serial.println(sunsetseconds);
}
//validData = false; // reset flag not needed here as it is done in V2
}
else{
validData = false;
if(debug) { Serial.println("Data is INVALID, check API"); }
}
if(debug) { Serial.println("Connecting Blynk"); }
//Blynk.connect();
}
void setup() {
Serial.begin(115200);
Serial.println("\nStarting");
Blynk.begin(authstr, ssidstr, passstr, serverstr);
int mytimeout = millis() / 1000;
while (Blynk.connect() == false) { // wait here for upto 10s until connected to the server
if((millis() / 1000) > mytimeout + 8){ // try to connect to the server for less than 9 seconds
break; // continue with the sketch regardless of connection to the server
}
}
Serial.print("IP address: ");
Serial.println(WiFi.localIP());
ArduinoOTA.setHostname(OTAhost); // for local OTA updates
ArduinoOTA.begin();
timer.setInterval(5000L, reconnectBlynk); // check every 5s if still connected to server
terminalW.println();
terminalW.print("Local IP: ");
terminalW.println(WiFi.localIP());
terminalW.flush();
getrawdata(); // now also done with V2 momentary switch
gotValidData();
}
void loop() {
if (Blynk.connected()) { // to ensure that Blynk.run() function is only called if we are still connected to the server
Blynk.run();
}
timer.run();
ArduinoOTA.handle(); // for local OTA updates
yield();
}
2 Likes
Great work! Currently I’m busy building a lighting system for Lego houses and this might come in handy there too …
1 Like
The great thing about your code is that it can be easily adapted for all the many, great API’s that are currently available.
I did think about making the sketch more generic whereby you would enter a keyword in the sketch or say Blynk Terminal such as “Sunrise”, an offset for the index where the data for the keyword is located and a few integer variables to pull out the data i.e. to replace 1, 8 for example in the following line:
if(inputString.substring(1, 8) == "sunrise"){
1 Like
Hmm, you could be on to something there. I think maybe even more generic would be something to parse XML files. Since they are probably the standard of API’s. I just don’t have a lot of experience using XML files. The trick would probably to determine how many levels of parameters you have and how to store them dynamically in arrays and such. I think C or C++ would be better for this since they handle arrays a lot better than a simple MCU.
Btw, I see you use chars and strings and such to get the time’s with leading zero’s and whatnot. I highly recommend using UNIX time. It’s so much easier to deal with once you get the hang of it. And it saves a lot of memory in the end.
For reference, this is how I process the captured data:
/*
* Sunrise/sunset timer via API for use with RTC Widget in Blynk
*
* (C) 2016 B. Visee, info@lichtsignaal.nl
* This code is licensed under MIT license
*/
bool procesdata()
{
for(int i=0;i<NUM_LEDS;i++)
{
leds[i] = CRGB::Pink;
FastLED.show();
}
// Temp variable for storing dates and times
String parameter[10][2];
// Remove first {"results":
int firstOpening = useFullData.indexOf('{');
int secondOpening = useFullData.indexOf('{', firstOpening + 1);
useFullData.remove(0, secondOpening + 1);
// Remove ending ,"status":"OK"}
int firstClosing = useFullData.indexOf('}');
useFullData.remove(firstClosing);
if(debug) { Serial.println(useFullData); }
// Put parameters and values into the temporary array parameters[][]
for(int i=0;i<10;i++)
{
firstOpening = useFullData.indexOf('\"');
secondOpening = useFullData.indexOf('\"', firstOpening + 1);
parameter[i][0] = useFullData.substring(firstOpening+1, secondOpening);
useFullData.remove(0, secondOpening+1);
firstOpening = useFullData.indexOf('\"');
secondOpening = useFullData.indexOf('\"', firstOpening + 1);
parameter[i][1] = useFullData.substring(firstOpening+1, secondOpening);
useFullData.remove(0, secondOpening+1);
if(debug)
{
Serial.print("Parameter: ");
Serial.print(parameter[i][0]);
Serial.print(" contains: ");
Serial.println(parameter[i][1]);
}
}
// Convert times into usable UNIX times
firstOpening = parameter[0][1].indexOf(':');
secondOpening = parameter[0][1].indexOf(':', firstOpening+1);
int thirdOpening = parameter[0][1].indexOf(' ', secondOpening+1);
// Store all the times in the time variables
for(int i=0;i<10;i++)
{
if(i != 2)
{
int tempHour = parameter[i][1].substring(0,firstOpening).toInt();
int tempMinute = parameter[i][1].substring(firstOpening+1,secondOpening).toInt();
int tempSecond = parameter[i][1].substring(secondOpening+1,thirdOpening).toInt();
// Correct for 24h clock
if(parameter[i][1].endsWith("PM") )
{
tempHour = tempHour + 12;
}
tmElements_t tm;
tm.Hour = tempHour;
tm.Minute = tempMinute;
tm.Second = tempSecond;
tm.Day = day(currentTime);
tm.Month = month(currentTime);
tm.Year = year(currentTime) - 1970;
switch(i)
{
case 0:
sunrise = myTZ.toLocal(makeTime(tm), &tcr);
sunriseOriginal = myTZ.toLocal(makeTime(tm), &tcr); // Keep backup of original times for custom times
if(debug) { Serial.println(sunrise); }
break;
case 1:
sunset = myTZ.toLocal(makeTime(tm), &tcr);
sunsetOriginal = myTZ.toLocal(makeTime(tm), &tcr); // Keep backup of original times for custom times
if(debug) { Serial.println(sunset); }
break;
case 3:
solar_noon = myTZ.toLocal(makeTime(tm), &tcr);
if(debug) { Serial.println(solar_noon); }
break;
case 4:
civil_twilight_begin = myTZ.toLocal(makeTime(tm), &tcr);
if(debug) { Serial.println(civil_twilight_begin); }
break;
case 5:
civil_twilight_end = myTZ.toLocal(makeTime(tm), &tcr);
if(debug) { Serial.println(civil_twilight_end); }
break;
case 6:
nautical_twilight_begin = myTZ.toLocal(makeTime(tm), &tcr);
if(debug) { Serial.println(nautical_twilight_begin); }
break;
case 7:
nautical_twilight_end = myTZ.toLocal(makeTime(tm), &tcr);
if(debug) { Serial.println(nautical_twilight_end); }
break;
case 8:
astronomical_twilight_begin = myTZ.toLocal(makeTime(tm), &tcr);
if(debug) { Serial.println(astronomical_twilight_begin); }
break;
case 9:
astronomical_twilight_end = myTZ.toLocal(makeTime(tm), &tcr);
if(debug) { Serial.println(astronomical_twilight_end); }
break;
}
}
}
if(debug) { Serial.println("Converted all times to correct timezone"); }
// Check if both hours of sunrise and sunset are filled, we assume it's ok then.
if( (hour(sunrise) != 0) && (hour(sunset) != 0) )
{
for(int i=0;i<NUM_LEDS;i++)
{
leds[i] = CRGB::Black;
FastLED.show();
}
return 1;
}
else
{
// Set LED's in error state
for(int i=0;i<NUM_LEDS;i++) { leds[i] = CRGB::Red; FastLED.show(); }
return 0;
}
}
1 Like
@Lichtsignaal for my requirement it is only really the seconds from midnight for sunrise and sunset that I require, rather than the time per se. As you have probably seen with debug set as 1 it does show the actual times in Serial Monitor but only the sunriseseconds andsunsetseconds are used in the Blynk project and passed to the Terminal widget.
I think my use of Strings may be because many moons ago I started with GW Basic and have always found them easier to work with.
It was not ideal for Arduino’s and very much frowned upon but now we have so much memory to work with on the ESP’s String use is more acceptable.
When I have a minute I will study your code as I’m sure there are things I will find useful, thanks.
Yr welcome. It all depends on what you want to achieve of course. I’ve noticed the calculations with time are easier with UNIX time, but if your input is limited, you could do without
I figured Unix time was a pain, but it is really easy with the Time library if you take a little time (pun intended) to study it 
@Costas @Lichtsignaal this request -
http://api.sunrise-sunset.org/json?lat=36.7201600&lng=-4.4203400&date=2016-08-25
returns
{"results":{"sunrise":"5:43:55 AM","sunset":"6:55:05 PM","solar_noon":"12:19:30 PM","day_length":"13:11:10","civil_twilight_begin":"5:17:12 AM","civil_twilight_end":"7:21:47 PM","nautical_twilight_begin":"4:45:18 AM","nautical_twilight_end":"7:53:42 PM","astronomical_twilight_begin":"4:12:05 AM","astronomical_twilight_end":"8:26:55 PM"},"status":"OK"}
Do you use all this fields or just 1 specific? Like sunrise?
@Dmitriy for me it would just be sunrise and sunset at the moment as any “twilight” adjustments could be done with a slider. Ideally though all fields from all API’s should be available. Not an easy task I know.