I do it all the time for rehearsal and gigging with my multi-genre cover band.  I personally see no downside, but some people have hard time getting used to the lack of "physical" interaction with a cranked amp.

I tour using nothing but a direct connection. The only real downside is that you're not flooding the stage with the same level of sound as the drummer and depending on the quality of the house PA and the house wedges (assuming you don't have your own) the sound can be a bit variable. What I've found is that for the most part, the audience gets a great sound (except for anyone standing forward of the PA system). You, on the other hand, might not. To that end we'll be adding powered PA speakers to act as cabs behind us; HH Tessens probably. That should bring the stage level of the guitars up sufficiently to balance everything out and provide us with the sensation of pushing some air around!

I think the Pod's preamp and power amp simulations are great.  The cab/mic simulations work very well for some styles of music, but IMO fail to match the clarity and full frequency response of a real 4x12 for a high gain tone.  I've resorted to using "dual cabs" as a work around: http://line6.com/support/thread/88099

Another option if you wanted better-sounding cab/mics while still going direct would be to simulate them with another piece of gear, such as a DI box with cab/mic simulation (very cheap solution), get a dedicated product for this (such as the TwoNotes Torpedo products), or load 3rd party IR's onto something capable of running them, such as a laptop or MuseBox.  The TwoNotes Torpedo is probably the best option - you can set it up to switch with your Pod patches, so you have a ton of versatility without tap-dancing.  The DI box and laptop/MuseBox solutions work well, but you probably aren't going to be able to set them up to switch between multiple IR's.  you could probably set it up to do so, but I'm not sure how much latency there would be between switching and how much work it would require to set up.

Personally, I think that "dual cabs" using the Pod's onboard cab sims are the best solution for my needs.  I don't need to run lots of effects, so in general DSP limitations are not a huge burden (although they occasionally force me to compromise).  The upside is that I don't have to lug around a power amp and cab, I have more versatility in the kind of sounds I can achieve.  Everything is inside my Pod HD patch, so I don't need to worry about setting multiple pieces of gear to work together.  And I don't have to worry about something like a laptop suddenly freezing, restarting, or getting broken in a live environment.  And I don't have to purchase a TwoNotes product that costs more than my Pod.