Didn't find Let's Encrypt certificates but Certbot says they are there

Ok, I’ve been struggling with this for a few days now but I cannot get the local Blynk server to find my Let’s Encrypt certificates. I tried to have it automatically generate them which did not work, so I have done it manually and it still can’t find them.

I am running Blynk on a Raspberry Pi 3 Model B using this command
java -jar server-0.29.3-java8.jar

I manually updated the Let’s Encrypt certificates with Certbot-Auto. I can verify they exist with this:

pi@hassio:~/Blynk $ sudo ~/certbot/certbot-auto certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Found the following certs:
  Certificate Name: home.binarybeach.com
    Domains: home.binarybeach.com blynk.binarybeach.com mqtt.binarybeach.com service.binarybeach.com www.binarybeach.com
    Expiry Date: 2018-04-02 21:49:29+00:00 (VALID: 87 days)
    Certificate Path: /etc/letsencrypt/live/home.binarybeach.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/home.binarybeach.com/privkey.pem
-------------------------------------------------------------------------------

I also ran this so that I didn’t have to run the Blynk server with sudo

pi@hassio:~/Blynk $ sudo chmod 755 /etc/letsencrypt/live/home.binarybeach.com/fullchain.pem
pi@hassio:~/Blynk $ sudo chmod 755 /etc/letsencrypt/live/home.binarybeach.com/privkey.pem

I have added mail.properties and it contains this

mail.smtp.auth=true
mail.smtp.starttls.enable=true
mail.smtp.host=smtp.gmail.com
mail.smtp.port=587
mail.smtp.username=conklin.jason@gmail.com
mail.smtp.password=redacted

and my server.properties

#hardware mqtt port
hardware.mqtt.port=8440

#hardware ssl port
hardware.ssl.port=8441

#hardware plain tcp/ip port
hardware.default.port=8442

#http and web sockets port
http.port=8080

#https and web sockets port
https.port=9443

#application ssl port
app.ssl.port=8443

#address to bind to. by default bounded to all interfaces
listen.address=

#by default server uses embedded in jar cert to simplify local server installation.
#WARNNING DO NOT USE THIS CERTIFICATES ON PRODUCTION OR IN WHERE ENVIRNOMENTS REAL SECURITY REQUIRED.
#provide either full path to files either use '.' for specifying current directory. For instance "./myfile.crt"
server.ssl.cert=/etc/letsencrypt/live/home.binarybeach.com/fullchain.pem
server.ssl.key=/etc/letsencrypt/live/home.binarybeach.com/privkey.pem
server.ssl.key.pass=

#by default System.getProperty("java.io.tmpdir")/blynk used
data.folder=/home/pi/Blynk

#folder for logs.
logs.folder=./logs

#log debug level. trace|debug|info|error. Defines how precise logging will be.
log.level=trace

#maximum number of devices allowed per account
user.devices.limit=25

#maximum number of tags allowed per account
user.tags.limit=100

#defines maximum allowed number of user dashboards. Needed to limit possible number of tokens.
user.dashboard.max.limit=100

#defines maximum allowed widget size in KBs as json string.
user.widget.max.size.limit=20

#user is limited with 100 messages per second.
user.message.quota.limit=100

#maximum allowed number of notification queue. Queue responsible for processing email, pushes, twits sending.
#Because of performance issue - those queue is processed in separate thread, this is required due
#to blocking nature of all above operations. Usually limit shouldn't be reached.
notifications.queue.limit=2000

#Number of threads for performing blocking operations - push, twits, emails, db queries.
#Recommended to hold this value low unless you have to perform a lot of blocking operations.
blocking.processor.thread.pool.limit=6

#this setting defines how often we can send mail/tweet/push or any other notification. Specified in seconds
notifications.frequency.user.quota.limit=15

#this setting defines how often we can send webhooks. Specified in miliseconds
webhooks.frequency.user.quota.limit=1000

#this setting defines how big could be response for webhook GET request. Specified in kbs
webhooks.response.size.limit=72

#maximum size of user profile in kb's
user.profile.max.size=128

#number of strings to store in terminal widget
terminal.strings.pool.size=25

#number of strings to store in map widget
map.strings.pool.size=25

#number of strings to store in lcd widget
lcd.strings.pool.size=6

#maximum number of rows allowed
table.rows.pool.size=100

#period in millis for saving all user DB to disk.
profile.save.worker.period=60000

#period in millis for saving stats to disk.
stats.print.worker.period=60000

#max size of web request in bytes, 256 kb (256x1024) is default
web.request.max.size=524288

#maximum number of points that are fetched during CSV export
#43200 == 60 * 24 * 30 - minutes points for 1 month
csv.export.data.points.max=43200

#specifies maximum period of time when hardware socket could be idle. After which
#socket will be closed due to non activity. In seconds. Default value 15 if not provided.
#leave it empty for infinity timeout
hard.socket.idle.timeout=15

#enable DB
enable.db=false

#enable raw data storage to DB
enable.raw.db.data.store=false

#size of async logger ring buffer. should be increased for loads >2-3k req/sec
async.logger.ring.buffer.size=2048

#initial amount of energy
initial.energy=100000

#ADMINISTRATION SECTION

admin.rootPath=/admin

#used for reset password page and certificate generation.
#by default current server IP is taken. could be replaced with more friendly hostname.
#it is recommended to override this property with your server IP to avoid possible problems of host resolving
server.host=blynk.binarybeach.com

#email used for certificate registration, could be omitted in case you already specified it in mail.properties
contact.email=conklin.jason@gmail.com

#network interface to determine server's current IP.
#only the first characters of the interface's name are needed.
#the default setting eth will use the first ethX interface found (i.e. eth0)
net.interface=eth

#comma separated list of administrator IPs. allow access to admin UI only for those IPs.
#you may set it for 0.0.0.0/0 to allow access for all.
#you may use CIDR notation. For instance, 192.168.0.53/24
allowed.administrator.ips=0.0.0.0/0,::/0

# default admin name and password. that will be created on initial server start
admin.email=conklin.jason@gmail.com
admin.pass=also_redacted_to_protect_the_innocent

#comma separated list of users allowed to create accounts. leave it empty if no restriction required.
allowed.users.list=

Currently I have ports 80 and 443 forwarded to a Synology Diskstation and on there I have a reverse proxy that sends everything for blynk.binarybeach.com on port 80 to port 8080 and port 433 to 9443. This works, I can start the server and I can access it from anywhere. (I’ve tried port forwarding on the Raspi to forward 80 to 8080 and 443 to 9443 and changing my router to forward directly to the Raspi and not the Synology, but same results.)

However, every time I start the server, it cannot find the Let’s Encrypt Certificates. This is my blynk.log

17:07:28.524 DEBUG- Using SLF4J as the default logging framework
17:07:28.540 INFO - Using data dir '/home/pi/Blynk'
17:07:28.625 DEBUG- -Dio.netty.noUnsafe: false
17:07:28.625 DEBUG- Java version: 8
17:07:28.628 DEBUG- sun.misc.Unsafe.theUnsafe: available
17:07:28.631 DEBUG- sun.misc.Unsafe.copyMemory: available
17:07:28.632 DEBUG- java.nio.Buffer.address: available
17:07:28.634 DEBUG- direct buffer constructor: available
17:07:28.637 DEBUG- java.nio.Bits.unaligned: available, false
17:07:28.637 DEBUG- jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable prior to Java9
17:07:28.637 DEBUG- java.nio.DirectByteBuffer.<init>(long, int): available
17:07:28.638 DEBUG- sun.misc.Unsafe: available
17:07:28.639 DEBUG- -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
17:07:28.639 DEBUG- -Dio.netty.bitMode: 32 (sun.arch.data.model)
17:07:28.643 DEBUG- -Dio.netty.noPreferDirect: false
17:07:28.643 DEBUG- -Dio.netty.maxDirectMemory: 235274240 bytes
17:07:28.643 DEBUG- -Dio.netty.uninitializedArrayAllocationThreshold: -1
17:07:28.646 DEBUG- java.nio.ByteBuffer.cleaner(): available
17:07:28.657 DEBUG- Starting reading user DB.
17:07:28.879 DEBUG- Reading user DB finished.
17:07:28.879 INFO - Region : local. Host : blynk.binarybeach.com.
17:07:28.954 DEBUG- -Dio.netty.eventLoopThreads: 8
17:07:29.037 DEBUG- -Dio.netty.noKeySetOptimization: false
17:07:29.037 DEBUG- -Dio.netty.selectorAutoRebuildThreshold: 512
17:07:29.065 DEBUG- org.jctools-core.MpscChunkedArrayQueue: available
17:07:29.080 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@1e5f737
17:07:29.101 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@f9c5b7
17:07:29.102 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@1271612
17:07:29.102 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@d7f8b4
17:07:29.102 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@309d4d
17:07:29.103 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@102d01
17:07:29.103 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@1883b97
17:07:29.104 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@ab778a
17:07:29.104 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@3958e7
17:07:29.171 DEBUG- -Dio.netty.leakDetection.level: disabled
17:07:29.171 DEBUG- -Dio.netty.leakDetection.targetRecords: 4
17:07:29.177 DEBUG- Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@668b6e
17:07:29.924 DEBUG- Default protocols (JDK): [TLSv1.2, TLSv1.1, TLSv1] 
17:07:29.925 DEBUG- Default cipher suites (JDK): [TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA]
17:07:30.100 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@13d4a8c
17:07:30.100 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@1de24cc
17:07:30.101 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@1c7df28
17:07:30.105 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@f842ca
17:07:30.108 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@1c81773
17:07:30.109 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@114b6c2
17:07:30.110 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@990ed7
17:07:30.111 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@5fddc
17:07:30.153 DEBUG- -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
17:07:30.154 DEBUG- -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
17:07:30.198 DEBUG- -Dio.netty.allocator.numHeapArenas: 2
17:07:30.198 DEBUG- -Dio.netty.allocator.numDirectArenas: 2
17:07:30.199 DEBUG- -Dio.netty.allocator.pageSize: 8192
17:07:30.199 DEBUG- -Dio.netty.allocator.maxOrder: 11
17:07:30.200 DEBUG- -Dio.netty.allocator.chunkSize: 16777216
17:07:30.200 DEBUG- -Dio.netty.allocator.tinyCacheSize: 512
17:07:30.200 DEBUG- -Dio.netty.allocator.smallCacheSize: 256
17:07:30.201 DEBUG- -Dio.netty.allocator.normalCacheSize: 64
17:07:30.201 DEBUG- -Dio.netty.allocator.maxCachedBufferCapacity: 32768
17:07:30.201 DEBUG- -Dio.netty.allocator.cacheTrimInterval: 8192
17:07:30.202 DEBUG- -Dio.netty.allocator.useCacheForAllThreads: true
17:07:30.230 DEBUG- -Dio.netty.allocator.type: pooled
17:07:30.230 DEBUG- -Dio.netty.threadLocalDirectBufferSize: 65536
17:07:30.231 DEBUG- -Dio.netty.maxThreadLocalCharBufferSize: 16384
17:07:30.502 INFO - Initializing gmail smtp mail transport. Username : conklin.jason@gmail.com. SMTP host : smtp.gmail.com:587
17:07:31.268 INFO - Didn't find Let's Encrypt certificates.
17:07:31.268 INFO - Automatic certificate generation is turned ON.
17:07:31.316 DEBUG- -Dio.netty.buffer.bytebuf.checkAccessible: true
17:07:31.316 DEBUG- Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@13e12b5
17:07:31.415 DEBUG- hard.socket.idle.timeout = 15
17:07:31.435 DEBUG- app.socket.idle.timeout = 600
17:07:31.745 DEBUG- hard.socket.idle.timeout = 15
17:07:31.784 DEBUG- -Dio.netty.processId: 8869 (auto-detected)
17:07:31.793 DEBUG- -Djava.net.preferIPv4Stack: false
17:07:31.793 DEBUG- -Djava.net.preferIPv6Addresses: false
17:07:31.798 DEBUG- Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
17:07:31.801 DEBUG- /proc/sys/net/core/somaxconn: 128
17:07:31.804 DEBUG- -Dio.netty.machineId: b8:27:eb:ff:fe:6d:6b:73 (auto-detected)
17:07:31.911 INFO - Hardware plain tcp/ip server listening at 8442 port.
17:07:31.915 INFO - Hardware SSL server listening at 8441 port.
17:07:31.917 INFO - Application server listening at 8443 port.
17:07:31.919 INFO - HTTP API and WebSockets server listening at 8080 port.
17:07:31.921 INFO - HTTPS API, WebSockets and Admin page server listening at 9443 port.
17:07:31.922 INFO - Mqtt hardware server listening at 8440 port.
17:07:31.944 DEBUG- Adding new user conklin.jason@gmail.com. App : Blynk

I just get no indication of what when wrong getting the certificates. I’ve also tried changing server.host to home.binarybeach.com for the same results.

I’m just out of ideas to try here.

Shouldn’t it be fullchain.crt?

Running certbot-auto didn’t give me a fullchain.crt file, only fullchain.pem, but from the README.md here https://github.com/blynkkk/blynk-server in the example it says to use fullchain.pem

Then add to your server.properties file (in folder with server.jar)

  server.ssl.cert=/etc/letsencrypt/live/YOUR_HOST/fullchain.pem
  server.ssl.key=/etc/letsencrypt/live/YOUR_HOST/privkey.pem
  server.ssl.key.pass=

And what this case? What error did you get?

So, if I delete the server.ssl values and start up the server I get this from the command line:

Blynk Server 0.29.4-SNAPSHOT successfully started.
All server output is stored in folder '/home/pi/Blynk/logs' file.
Generating own initial certificates...
Error during certificate generation.
Network error

But what sort of network error? After the server starts up I can get to the admin page at both http://blynk.binarybeach.com/admin and https://blynk.binarybeach.com/admin, and I can access the Internet from the Raspi. I only see in the log file where it does a “GET https://acme-v01.api.letsencrypt.org/directory” and nothing about the error response. I’m baffled, and I really appreciate you looking at this!

My blynk.log file

17:30:23.228 DEBUG- Using SLF4J as the default logging framework
17:30:23.244 INFO - Using data dir '/home/pi/Blynk'
17:30:23.329 DEBUG- -Dio.netty.noUnsafe: false
17:30:23.330 DEBUG- Java version: 8
17:30:23.333 DEBUG- sun.misc.Unsafe.theUnsafe: available
17:30:23.335 DEBUG- sun.misc.Unsafe.copyMemory: available
17:30:23.337 DEBUG- java.nio.Buffer.address: available
17:30:23.339 DEBUG- direct buffer constructor: available
17:30:23.342 DEBUG- java.nio.Bits.unaligned: available, false
17:30:23.342 DEBUG- jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable prior to Java9
17:30:23.342 DEBUG- java.nio.DirectByteBuffer.<init>(long, int): available
17:30:23.342 DEBUG- sun.misc.Unsafe: available
17:30:23.344 DEBUG- -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
17:30:23.344 DEBUG- -Dio.netty.bitMode: 32 (sun.arch.data.model)
17:30:23.347 DEBUG- -Dio.netty.noPreferDirect: false
17:30:23.348 DEBUG- -Dio.netty.maxDirectMemory: 235274240 bytes
17:30:23.348 DEBUG- -Dio.netty.uninitializedArrayAllocationThreshold: -1
17:30:23.351 DEBUG- java.nio.ByteBuffer.cleaner(): available
17:30:23.362 DEBUG- Starting reading user DB.
17:30:25.051 DEBUG- Reading user DB finished.
17:30:25.052 INFO - Region : local. Host : blynk.binarybeach.com.
17:30:25.116 DEBUG- -Dio.netty.eventLoopThreads: 8
17:30:25.191 DEBUG- -Dio.netty.noKeySetOptimization: false
17:30:25.191 DEBUG- -Dio.netty.selectorAutoRebuildThreshold: 512
17:30:25.213 DEBUG- org.jctools-core.MpscChunkedArrayQueue: available
17:30:25.227 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@b3fd41
17:30:25.231 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@17277dc
17:30:25.231 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@7e99d8
17:30:25.231 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@4487af
17:30:25.232 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@aceffd
17:30:25.232 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@86b9e3
17:30:25.233 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@aa41f2
17:30:25.233 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@f9c5b7
17:30:25.233 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@1271612
17:30:25.294 DEBUG- -Dio.netty.leakDetection.level: disabled
17:30:25.295 DEBUG- -Dio.netty.leakDetection.targetRecords: 4
17:30:25.300 DEBUG- Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@947bc5
17:30:26.024 DEBUG- Default protocols (JDK): [TLSv1.2, TLSv1.1, TLSv1] 
17:30:26.024 DEBUG- Default cipher suites (JDK): [TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA]
17:30:26.068 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@18d604b
17:30:26.069 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@7cd6cc
17:30:26.069 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@c9d179
17:30:26.069 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@1554c5f
17:30:26.070 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@1faea86
17:30:26.070 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@1888343
17:30:26.071 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@1ecda92
17:30:26.071 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@fc5f04
17:30:26.111 DEBUG- -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
17:30:26.111 DEBUG- -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
17:30:26.151 DEBUG- -Dio.netty.allocator.numHeapArenas: 2
17:30:26.152 DEBUG- -Dio.netty.allocator.numDirectArenas: 2
17:30:26.152 DEBUG- -Dio.netty.allocator.pageSize: 8192
17:30:26.152 DEBUG- -Dio.netty.allocator.maxOrder: 11
17:30:26.152 DEBUG- -Dio.netty.allocator.chunkSize: 16777216
17:30:26.152 DEBUG- -Dio.netty.allocator.tinyCacheSize: 512
17:30:26.152 DEBUG- -Dio.netty.allocator.smallCacheSize: 256
17:30:26.152 DEBUG- -Dio.netty.allocator.normalCacheSize: 64
17:30:26.152 DEBUG- -Dio.netty.allocator.maxCachedBufferCapacity: 32768
17:30:26.152 DEBUG- -Dio.netty.allocator.cacheTrimInterval: 8192
17:30:26.152 DEBUG- -Dio.netty.allocator.useCacheForAllThreads: true
17:30:26.180 DEBUG- -Dio.netty.allocator.type: pooled
17:30:26.180 DEBUG- -Dio.netty.threadLocalDirectBufferSize: 65536
17:30:26.180 DEBUG- -Dio.netty.maxThreadLocalCharBufferSize: 16384
17:30:26.531 INFO - Initializing gmail smtp mail transport. Username : conklin.jason@gmail.com. SMTP host : smtp.gmail.com:587
17:30:26.605 INFO - Didn't find Let's Encrypt certificates.
17:30:26.605 INFO - Automatic certificate generation is turned ON.
17:30:26.652 DEBUG- -Dio.netty.buffer.bytebuf.checkAccessible: true
17:30:26.652 DEBUG- Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@72a540
17:30:26.733 DEBUG- hard.socket.idle.timeout = 15
17:30:26.752 DEBUG- app.socket.idle.timeout = 600
17:30:27.070 DEBUG- hard.socket.idle.timeout = 15
17:30:27.111 DEBUG- -Dio.netty.processId: 8967 (auto-detected)
17:30:27.119 DEBUG- -Djava.net.preferIPv4Stack: false
17:30:27.119 DEBUG- -Djava.net.preferIPv6Addresses: false
17:30:27.124 DEBUG- Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
17:30:27.126 DEBUG- /proc/sys/net/core/somaxconn: 128
17:30:27.129 DEBUG- -Dio.netty.machineId: b8:27:eb:ff:fe:6d:6b:73 (auto-detected)
17:30:27.215 INFO - Hardware plain tcp/ip server listening at 8442 port.
17:30:27.219 INFO - Hardware SSL server listening at 8441 port.
17:30:27.222 INFO - Application server listening at 8443 port.
17:30:27.223 INFO - HTTP API and WebSockets server listening at 8080 port.
17:30:27.225 INFO - HTTPS API, WebSockets and Admin page server listening at 9443 port.
17:30:27.226 INFO - Mqtt hardware server listening at 8440 port.
17:48:06.651 DEBUG- -Dio.netty.recycler.maxCapacityPerThread: 32768
17:48:06.651 DEBUG- -Dio.netty.recycler.maxSharedCapacityFactor: 2
17:48:06.651 DEBUG- -Dio.netty.recycler.linkCapacity: 16
17:48:06.651 DEBUG- -Dio.netty.recycler.ratio: 8
17:48:06.705 DEBUG- In http and websocket unificator handler.
17:48:06.914 DEBUG- In http and websocket unificator handler.
20:29:02.373 DEBUG- Using SLF4J as the default logging framework
20:29:02.388 INFO - Using data dir '/home/pi/Blynk'
20:29:02.473 DEBUG- -Dio.netty.noUnsafe: false
20:29:02.473 DEBUG- Java version: 8
20:29:02.476 DEBUG- sun.misc.Unsafe.theUnsafe: available
20:29:02.479 DEBUG- sun.misc.Unsafe.copyMemory: available
20:29:02.481 DEBUG- java.nio.Buffer.address: available
20:29:02.482 DEBUG- direct buffer constructor: available
20:29:02.485 DEBUG- java.nio.Bits.unaligned: available, false
20:29:02.485 DEBUG- jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable prior to Java9
20:29:02.486 DEBUG- java.nio.DirectByteBuffer.<init>(long, int): available
20:29:02.486 DEBUG- sun.misc.Unsafe: available
20:29:02.487 DEBUG- -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
20:29:02.487 DEBUG- -Dio.netty.bitMode: 32 (sun.arch.data.model)
20:29:02.491 DEBUG- -Dio.netty.noPreferDirect: false
20:29:02.491 DEBUG- -Dio.netty.maxDirectMemory: 235274240 bytes
20:29:02.491 DEBUG- -Dio.netty.uninitializedArrayAllocationThreshold: -1
20:29:02.494 DEBUG- java.nio.ByteBuffer.cleaner(): available
20:29:02.505 DEBUG- Starting reading user DB.
20:29:02.728 DEBUG- Reading user DB finished.
20:29:02.728 INFO - Region : local. Host : blynk.binarybeach.com.
20:29:02.804 DEBUG- -Dio.netty.eventLoopThreads: 8
20:29:02.887 DEBUG- -Dio.netty.noKeySetOptimization: false
20:29:02.888 DEBUG- -Dio.netty.selectorAutoRebuildThreshold: 512
20:29:02.916 DEBUG- org.jctools-core.MpscChunkedArrayQueue: available
20:29:02.931 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@1e5f737
20:29:02.952 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@f9c5b7
20:29:02.953 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@1271612
20:29:02.953 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@d7f8b4
20:29:02.953 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@309d4d
20:29:02.954 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@102d01
20:29:02.954 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@1883b97
20:29:02.955 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@ab778a
20:29:02.955 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@3958e7
20:29:03.022 DEBUG- -Dio.netty.leakDetection.level: disabled
20:29:03.023 DEBUG- -Dio.netty.leakDetection.targetRecords: 4
20:29:03.029 DEBUG- Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@668b6e
20:29:03.780 DEBUG- Default protocols (JDK): [TLSv1.2, TLSv1.1, TLSv1] 
20:29:03.781 DEBUG- Default cipher suites (JDK): [TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA]
20:29:03.957 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@13d4a8c
20:29:03.958 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@1de24cc
20:29:03.958 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@1c7df28
20:29:03.959 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@f842ca
20:29:03.959 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@1c81773
20:29:03.960 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@114b6c2
20:29:03.960 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@990ed7
20:29:03.960 TRACE- instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@5fddc
20:29:04.002 DEBUG- -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
20:29:04.002 DEBUG- -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
20:29:04.045 DEBUG- -Dio.netty.allocator.numHeapArenas: 2
20:29:04.045 DEBUG- -Dio.netty.allocator.numDirectArenas: 2
20:29:04.046 DEBUG- -Dio.netty.allocator.pageSize: 8192
20:29:04.046 DEBUG- -Dio.netty.allocator.maxOrder: 11
20:29:04.046 DEBUG- -Dio.netty.allocator.chunkSize: 16777216
20:29:04.046 DEBUG- -Dio.netty.allocator.tinyCacheSize: 512
20:29:04.046 DEBUG- -Dio.netty.allocator.smallCacheSize: 256
20:29:04.046 DEBUG- -Dio.netty.allocator.normalCacheSize: 64
20:29:04.046 DEBUG- -Dio.netty.allocator.maxCachedBufferCapacity: 32768
20:29:04.046 DEBUG- -Dio.netty.allocator.cacheTrimInterval: 8192
20:29:04.046 DEBUG- -Dio.netty.allocator.useCacheForAllThreads: true
20:29:04.074 DEBUG- -Dio.netty.allocator.type: pooled
20:29:04.075 DEBUG- -Dio.netty.threadLocalDirectBufferSize: 65536
20:29:04.075 DEBUG- -Dio.netty.maxThreadLocalCharBufferSize: 16384
20:29:04.342 INFO - Initializing gmail smtp mail transport. Username : conklin.jason@gmail.com. SMTP host : smtp.gmail.com:587
20:29:05.127 INFO - Didn't find custom user certificates.
20:29:05.132 INFO - Didn't find Let's Encrypt certificates.
20:29:05.132 INFO - Automatic certificate generation is turned ON.
20:29:05.133 WARN - ATTENTION. Server certificate paths (cert : '/home/pi/Blynk', key : '/home/pi/Blynk') not valid. Using embedded server certs and one way ssl. This is not secure. Please replace it with your own certs.
20:29:06.305 DEBUG- -Dio.netty.buffer.bytebuf.checkAccessible: true
20:29:06.306 DEBUG- Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@45ee6e
20:29:06.401 DEBUG- hard.socket.idle.timeout = 15
20:29:06.421 DEBUG- app.socket.idle.timeout = 600
20:29:06.750 DEBUG- hard.socket.idle.timeout = 15
20:29:06.789 DEBUG- -Dio.netty.processId: 9445 (auto-detected)
20:29:06.798 DEBUG- -Djava.net.preferIPv4Stack: false
20:29:06.798 DEBUG- -Djava.net.preferIPv6Addresses: false
20:29:06.803 DEBUG- Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
20:29:06.806 DEBUG- /proc/sys/net/core/somaxconn: 128
20:29:06.809 DEBUG- -Dio.netty.machineId: b8:27:eb:ff:fe:6d:6b:73 (auto-detected)
20:29:06.896 INFO - Hardware plain tcp/ip server listening at 8442 port.
20:29:06.900 INFO - Hardware SSL server listening at 8441 port.
20:29:06.903 INFO - Application server listening at 8443 port.
20:29:06.905 INFO - HTTP API and WebSockets server listening at 8080 port.
20:29:06.907 INFO - HTTPS API, WebSockets and Admin page server listening at 9443 port.
20:29:06.910 INFO - Mqtt hardware server listening at 8440 port.
20:29:06.932 INFO - Starting up certificate retrieval process for host blynk.binarybeach.com and email conklin.jason@gmail.com.
20:29:13.346 DEBUG- create
20:29:13.365 DEBUG- GET https://acme-v01.api.letsencrypt.org/directory
20:29:13.936 DEBUG- Adding new user conklin.jason@gmail.com. App : Blynk
20:30:14.044 DEBUG- -Dio.netty.recycler.maxCapacityPerThread: 32768
20:30:14.044 DEBUG- -Dio.netty.recycler.maxSharedCapacityFactor: 2
20:30:14.044 DEBUG- -Dio.netty.recycler.linkCapacity: 16
20:30:14.045 DEBUG- -Dio.netty.recycler.ratio: 8
20:30:14.237 DEBUG- [id: 0x67380c2b, L:/192.168.1.172:9443 - R:/192.168.1.12:58445] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
20:30:14.337 DEBUG- In http and websocket unificator handler.
20:30:14.391 DEBUG- [id: 0x9079d292, L:/192.168.1.172:9443 - R:/192.168.1.12:58446] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
20:30:14.399 DEBUG- In http and websocket unificator handler.
20:30:14.404 DEBUG- Is unpacked.
20:30:14.404 DEBUG- Getting file from path /home/pi/Blynk/static/login.html
20:30:14.780 DEBUG- [id: 0xb6e708fd, L:/192.168.1.172:9443 - R:/192.168.1.12:58453] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
20:30:14.783 DEBUG- [id: 0x878c3e34, L:/192.168.1.172:9443 - R:/192.168.1.12:58449] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
20:30:14.785 DEBUG- [id: 0x960d64f9, L:/192.168.1.172:9443 - R:/192.168.1.12:58448] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
20:30:14.787 DEBUG- Is unpacked.
20:30:14.787 DEBUG- [id: 0xc7883201, L:/192.168.1.172:9443 - R:/192.168.1.12:58450] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
20:30:14.787 DEBUG- Getting file from path /home/pi/Blynk/static/js/login.js
20:30:14.799 DEBUG- Is unpacked.
20:30:14.800 DEBUG- [id: 0x35baca5b, L:/192.168.1.172:9443 - R:/192.168.1.12:58451] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
20:30:14.801 DEBUG- Is unpacked.
20:30:14.801 DEBUG- Getting file from path /home/pi/Blynk/static/js/core-min.js
20:30:14.801 DEBUG- Getting file from path /home/pi/Blynk/static/js/jquery-2.2.2.min.js
20:30:15.029 DEBUG- Is unpacked.
20:30:15.029 DEBUG- [id: 0xea5ffb90, L:/192.168.1.172:9443 - R:/192.168.1.12:58452] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
20:30:15.029 DEBUG- Getting file from path /home/pi/Blynk/static/js/sha256-min.js
20:30:15.040 DEBUG- [id: 0x0a58c0e0, L:/192.168.1.172:9443 - R:/192.168.1.12:58447] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
20:30:15.042 DEBUG- Is unpacked.
20:30:15.043 DEBUG- Getting file from path /home/pi/Blynk/static/js/bootstrap.min.js
20:30:15.045 DEBUG- Is unpacked.
20:30:15.045 DEBUG- Getting file from path /home/pi/Blynk/static/css/blynk.css
20:30:15.050 DEBUG- Is unpacked.
20:30:15.050 DEBUG- Getting file from path /home/pi/Blynk/static/js/enc-base64-min.js
20:30:15.385 DEBUG- [id: 0x73fc9dfd, L:/192.168.1.172:9443 - R:/192.168.1.12:58454] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
20:30:15.391 DEBUG- Is unpacked.
20:30:15.391 DEBUG- Getting file from path /home/pi/Blynk/static/fonts/ufonts.com_pfdindisplaypro-thin.woff
20:30:27.032 DEBUG- [id: 0x05f38660, L:/192.168.1.172:9443 - R:/192.168.1.12:58455] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
20:30:27.038 DEBUG- In http and websocket unificator handler.
20:30:27.040 DEBUG- Is unpacked.
20:30:27.040 DEBUG- Getting file from path /home/pi/Blynk/static/login.html
20:30:27.098 DEBUG- [id: 0xaab07078, L:/192.168.1.172:9443 - R:/192.168.1.12:58456] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
20:30:27.104 DEBUG- Is unpacked.
20:30:27.105 DEBUG- Getting file from path /home/pi/Blynk/static/css/blynk.css
20:30:32.567 DEBUG- In http and websocket unificator handler.
20:30:32.568 DEBUG- Is unpacked.
20:30:32.569 DEBUG- Getting file from path /home/pi/Blynk/static/login.html
20:30:32.793 DEBUG- Is unpacked.
20:30:32.794 DEBUG- Getting file from path /home/pi/Blynk/static/css/blynk.css
20:30:32.799 DEBUG- Is unpacked.
20:30:32.799 DEBUG- Getting file from path /home/pi/Blynk/static/js/bootstrap.min.js
20:30:32.807 DEBUG- Is unpacked.
20:30:32.807 DEBUG- Is unpacked.
20:30:32.807 DEBUG- Getting file from path /home/pi/Blynk/static/js/core-min.js
20:30:32.807 DEBUG- Getting file from path /home/pi/Blynk/static/js/jquery-2.2.2.min.js
20:30:32.811 DEBUG- Is unpacked.
20:30:32.811 DEBUG- Getting file from path /home/pi/Blynk/static/js/sha256-min.js
20:30:32.822 DEBUG- Is unpacked.
20:30:32.822 DEBUG- Getting file from path /home/pi/Blynk/static/js/enc-base64-min.js
20:30:32.848 DEBUG- Is unpacked.
20:30:32.849 DEBUG- Getting file from path /home/pi/Blynk/static/js/login.js
20:30:33.048 DEBUG- Is unpacked.
20:30:33.049 DEBUG- Getting file from path /home/pi/Blynk/static/fonts/ufonts.com_pfdindisplaypro-thin.woff

I even just changed the router to forward external ports 80 & 443 to Raspi ports 8080 & 9443 to skip the Synology reverse proxy and certificates still won’t work but I can access the admin page from either.

Your host should be available on port 80. So log says that request for certificate retrieval was sent. But look like like Let’s Encrypt wasn’t able to reach your server. By default Blynk starts HTTP on 8080, so you have to add port forwarding before server start 80 -> 8080.

Yes, I agree and have tried it and just tried it again. I can setup port forwarding from external ports 80 & 443 in the router to internal network ports 80 & 443 on the Raspi. Then run:

pi@hassio:~/Blynk $ sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080
pi@hassio:~/Blynk $ sudo iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-port 9443
pi@hassio:~/Blynk $ java -jar server-0.29.3-java8.jar

Blynk Server 0.29.4-SNAPSHOT successfully started.
All server output is stored in folder '/home/pi/Blynk/logs' file.
Generating own initial certificates...
Error during certificate generation.
Network error

Still a network error and some other certificate error, yet from another PC not on the local network I can pull up the admin pages from Blynk which means everything is forwarded as it should be.

…and I can also manually run certbot-auto successfully when setup this way. Is there something with Blynk not liking that I have certificates for multiple subdomains?

I don’t think so. Seems like something specific to your network setup. I don’t know where “Network error” comes from. It could be Let’s Encrypt library message (https://github.com/shred/acme4j/blob/d07e9256e7629cfe6db843348a476ebe2b144c75/acme4j-client/src/main/java/org/shredzone/acme4j/exception/AcmeNetworkException.java), but not Blynk.

I’ll try to check, when I have a time, however 3 weeks ago it was working perfectly.

Ok, thanks. It helps to know that it’s not from Blynk and is from Let’s Encrypt. I’ll try some more things out with the library to see if I can pinpoint the issue. Also interesting that there should be a cause from the Network error and I’m not getting anything there…could be a clue.

I had to copy my certs to the Blynk directory for it to work. Don’t ask my why… :face_with_raised_eyebrow:

server.ssl.cert=/home/blynk/fullchain.pem
server.ssl.key=/home/blynk/privkey.pem

Running Ubuntu Server 16 LTS.

Yeah, thanks for the suggestion. I’ve tried that also though…and I just tried it again based on your suggestion. No go. I did find that acme4j does say “Java 8u101 or higher is required for connecting to the Let’s Encrypt servers.” The Raspberry Pi image I installed was only running:

java version “1.8.0_65”
Java™ SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot™ Client VM (build 25.65-b01, mixed mode)

So, I updated that using this method https://www.raspinews.com/installing-oracle-java-jdk-8-on-raspberry-pi/

Now “java -version” gives me:
java version “1.8.0_152”
Java™ SE Runtime Environment (build 1.8.0_152-b16)
Java HotSpot™ Client VM (build 25.152-b16, mixed mode)

But still doesn’t work. Maybe I’ll have time this weekend to work on it more.

1 Like

I don’t have a RPi myself so can’t replicate the problem = I’m guessing :yum:

Had some earlier problems with both Java and Python (not Blynk related) when updating. Some settings, PATH’s etc still pointed to the previous versions. So I had to completely removed the packages and make a fresh install.

I have NO IDEA if removing your old Java causes problems for other programs on your RPi, but I don’t see why it should.

Check what’s installed (I assume apt works on RPi?!)

 apt list --installed *java*
 apt list --installed *jdk*

Remove them:

 apt remove *java*
 apt remove *jdk*

Using wildcards (*) when removing packages is perhaps a bad idea, use full names instead. :slight_smile:

I’d add the “webupd8team” repo and install it like in this example:

https://www.raspberrypi.org/forums/viewtopic.php?t=101543#p1220629

Another approach is to generate them on a different machine, or try a different client:

Never tried this website, but sounds promising:

https://gethttpsforfree.com/

This is a website that will take you through the manual steps to get your free https certificate so you can make your own website use https!

Good luck!

This is actually good catch. Sad, it didn’t help.

Might not be related but Let’s Encrypt found a possible security breach so they have disabled some features until a fix is found.

@J_Gat did you ever get this resolved? I’m having the exact issue myself.

No, I never did get the certificates to work on the RPi. I have Blynk running in a docker container on a Synology DiskStation now and had no problem getting it to use certificates on there. I still would like to figure this out sometime though.

A bit late to the party here
For anyone using absolute path : /etc/letsencrypts/live/domain/… , just add a dot first.
./etc/letsencrypt/…

And run with sudo.