“Special” pins already know their primary function and should just work as needed by default.
Yes, pinMode() should only be needed for “forcing” basic digital I/O to a fixed INPUT, OUTPUT, or INPUT_PULLUP (Not sure if that last one is ESP compatible)
Although with some boards like ESP32 then you can use commands to make most any pin do different stuff (OK, not relevant to this topic, but just tossing it out there )