So.....
Just confirmed, Phantom power does indeed affect the HX Stomp drastically. My phantom power has been on this whole time (two different mixers) and that was causing my issues. I'm pretty happy i figured out what the issue was, as i really wanted to incorporate this in my setup, and now i can. (and the fact i was convincing myself that i was crazy and making it up).
I'd record a video, but not sure the tonality will come through that great, but my impression was switching back and fourth was it's very similar to a set of headphones... with phantom power on, it's like what you hear when the headphones are hanging around your neck, and with phantom power off, its like they're on your ears correctly :).