Thanks for the replies everyone, I may have narrowed it down a bit. The problem has started to become worse, the last couple of days there have been a couple of occasions the car won't start at all, from cold - will just crank until the battery starts to fade.
The fuel pump relay is a genuine Volvo one and not very old (date stamped 2019).
Also, I tested the fuel pumps by jumping between different fuses (as per instructions online) - the main pump near the LH rear wheel can be heard, so the relay definitely works.
However I was not able to get any result from the in tank pump, whether listening through the boot or through the petrol cap.
This leads me to think the in tank pump is bad. This adds up in my mind, because firstly, I could imagine how the car would be difficult to start, but OK once running, if the in tank pump is bad but the main pump is OK - i.e. the failing tank pump means the car can't get enough fuel to start, but once running, enough fuel can get to the main pump for the car to keep running...?
And secondly, the seal on the tank pump access panel has clearly never been broken and after 477,000km I'd not be surprised for the pump to fail.
For extra context - I ended up replacing the fuel pressure regulator and ECU coolant temp sensor with new parts anyway as they are easy jobs.
@jared244 to answer some of your questions - it is a white label ECU. And as mentioned above, I think the relay is fine.
Keen to hear if others agree and if so I'll buy a new in tank pump - only reason I can think of not to, is if it's typically meant to operate quietly causing it to be hard for me to hear.