I am having the same issues where I get a loud hum through the amp when I kick on the distortion channels, even with no effects selected. I prefer the distortion through my amp (Egnater Tourmaster 4100) and picked up the HX effect so I can program which button do what (my Boss ME80 assigns buttons to effects). I opened a ticket with Line 6 and this was the initial response:
Typically, you want to use the cleanest amp setting if you are running any distortion, overdrive, or high gain sounds into the front of the amp. If you wish to use your amp's distortion channel, you will need to be careful if using preamps or other high gain effects on the HX and running them directly into your amp input.
When using the HX Effects, you might also consider running into your amp's effects loop or power amp in, depending on what it has and also your playing situation.
When I go through the effects loop on my amp, I get a much louder squeal unless I kill all of the levels, so that idea is out. I am not running anything else through the amp, so that can't be the culprit. I also unplugged from the unit and plugged straight through to the amp and the hum was gone. No issues going through the Boss pedal either. It just does not seem to like going through a high-gain distortion channel. I trued using the Line 6 updater and plugging the unit into my computer and the updater does not recognize the unit. The HX Edit does, though. I am strongly considering returning the unit and going back to the Boss pedal unless they have some advice that will fix this. for $600, it should not hum.