Coincidentally somebody from the LARP community asked me to see if I can make a design that takes four AA batteries and I put this new version together. The 6V from the battery pack works much better than the single 18650 and means they can be swapped easily but sadly makes the camera really quite bulky. Nonetheless I've sent them the files so they can have a go at building some units.
Going forward I've seen that the LDL1117 LDO only has a 0.3-0.6v dropout and is pin compatible. I think carefully desoldering the AMS117 and replacing it with one of these would solve my brownout problems.
An alternative would be a small 'daughterboard' that connects to the headers and has a buck convertor. This would be the more elegant option, especially if I managed to include the power switch and lamp button onboard. I've still not found time to see if I can make an I2S microphone work with the board, but that could also go on a 'daughterboard' if it did.