Apologies for reviving this post, but I tried the same yesterday with my Mark V, Helix and Fryette PS2 and have the same issue with squealing when cranking the poweramp. Setup is
Guitar > Gtr In (Helix) > Send 1 (Helix) > Gtr In (Mark V) > Spkr Out (Mark V) > Amp In (PS2) > Line Out (PS2) > Return 1 (Helix) > through IR to Helix outputs and PC sound card.
The idea is to get the Mark V as a switchable FX loop 1 in the Helix to capture the driven poweramp section as well, similar to the built-in amp models. But as soon as I crank the master level to beyond 4-5-ish, the squealing starts. I have managed to resolve it by taking the PS2 balanced line-out to a sound card and patching it through to an output back to the Helix Return 1. This seems to resolve the squeal, but picks up too much additional noise.
Did you manage to resolve the matter and use an attenuated Mesa amp in a Helix FX loop yet?