If you're plugging an XLR cable into a PA, you're most likely going into a mic preamp.  Technically, the XLR outs on the X3L are mic level, but I've found they are still loud enough that they don't need much gain applied at the channel strip you're plugging into.  It sounds like you either need to turn the gain down at the channel or turn the master volume on the X3L down.

Also, if you're happy with the levels you're getting with the switch set to"amp", there's nothing wrong with simply leaving it that way.

I like to set the Master Volume to control only the ¼ outs so I can use it to set my stage volume without affecting the signal going to the board. But like phil_m said, the XLR outs are not really mic level in this case, I’d say they are somewhere in between mic and line level. I always have problems with sound guys saying the signal it too hot, you would think they could simply turn in input trim on the channel down but some say they can’t. To make my life easier I made an 18dB balanced attenuator and always put it between the POD and the board to keep the sound guys happy.