So after (gulp) nearly a year of using the broom handle, I finally got a fix.
Using an endoscope, I identified that the square nut on the inside was too close to the corner for a socket or a spanner to fit.
What did work was getting a large screwdriver wedged against the side of the nut on the inside and carefully tightening from the outside.
The endoscope was key to this, as it allowed me to see as soon as the screwdriver moved out of position and relocate - they're about $20 on eBay