In principle the datasheet says the ESP32 will operate at 2.3V but there are onboard 1.8V LDOs used to power PSRAM to take into account too. The datasheet recommends a 3.3V 500mA supply to the module unless you also provide separate 1.8V supply.
As it stands, the module is just not getting that from a 4.0V LiPo.
I had a look for a good solution but my component browsing failed until I was recommended the ST LDL1117S33R. It's pin-compatible with a minimum dropout of 350mV. They're not expensive parts so I took a gamble and ordered 100 as the AMS1117 seems to be the default LDO on many cheap development boards.
After a little bit of hot air rework on an ESP32-CAM to replace the LDO, I measured Vout in the same conditions and found a solid 3.25V.
While I don't know if this will help, it feels like it should improve stability under LiPo power and I know it will make a big difference to usable battery life. It has definitely cured the issue where the 'flash LED' on the ESP32-CAM would flicker while streaming and powered by a LiPo.
I may yet make a 'shield' for the ESP32-CAM that includes other components for the project and a buck convertor would be more efficient still but this is a good incremental improvement.