Too many guitarists even with traditional rigs, meaning Tube Amps and foot pedals, have the same problem you just described.  Tweaking in tone, no matter what you're using, takes time, effort, and some skills.

Just like PremiumJones said "you really NEED to tweak to the tones you will use live, at live volume, AND with the full band playing". 

What sounds great by itself usually doesn't work well with everyone else.  Couple things I usually notice when I hear about these situations is the use of too much reverb, too much delay, and too much of the muddy midrange frequencies - usually something in the 300 to 800hz range is out of whack.  Also bass guitar frequencies typically step on the guitar.  Meaning the bass player needs to work with you so he EQs his stuff correctly too.

An average band that has great tone sounds great.  A great band with crappy tone always sounds like crap.

"I have another question, am I better using XLRs into the mixer or jack leads? Or doesn't it matter? I assumed XLRs would be less noisy as they are balanced."

If your PA is run in stereo, (most aren't) the two XLR's will give you stereo output.

If your PA is mono, the XLR's will be summed to mono by the PA, which would be a waste of a mixer channel, since you can use the L-mono from your HD500 and get the same thing from it, and use only one mixer channel.

XLR outputs from the HD500 never sum to mono.

I use the L-mono out to a directbox to the PA. Works well.