Just for reference,
Everything is working now, I missed the step were I need to add a BLE widget to my project.
If I use thw BLE widget on my Android app to pair with the ESP32 everything works as expected without connection issues.
Source: Problems with Bluetooth