>>102325072
>>102325115
Check whether you still get stability issues after running
nvidia-smi --lock-gpu-clocks 0,1000 --mode 1
That boost clock range is way too conservative but if it fixes the problem the crashes are most likely caused by power spikes from multiple GPUs randomly aligning.
As far as I can tell software power limits are only enforced on a long timescale (from a hardware perspective) so individual GPUs can momentarily exceed it.