And it will remain there as this is an ESP feature. Your credentials once stored in ESP will stay there until you erase it. I found this method to be fastest and most effective way for 4MB boards, using 1MB blank bin:
Erasing just first 1MB bank will do the job as well, as credentials are stored there.
You can also use NodeMCU Flasher for the same puspose.