Took me a while to figure this out also. I ended up hooking output to fx return on my amp. Output is in amp mode. Anything else sounded like a cat with it's tail caught in a mouse trap inside a dumpster with the lid closed. To use headphones, I switch to line mode and turn amp off. There is still some difference between amp and headphones which I will likely have to solve with banks EQ'd for amp and another bank EQ'd for headphones.
I have an Ibanez 150W Solid state head and Marshall 4x12 cab. The variac'd plexi (volume turned to 11 ;-)) through this sounds absolutely killer. Played til my finger tips were black.
I've probably spent 20 hours trying to get "my" sound out of this thing. You tend to forget how long you spent piddling with knobs until you get something new. My advise hang in there. It will take a week to get something out of it and if it just sounds wrong, hook it up differently. There are many options.
