My brother and I each have modded V5's and I've modded two for a friend. As far as I'm concerned a simple tube swap or speaker swap will help, but it won't fix the problem. The V5 does not have a volume control, it has a op-amp gain control, and as you have experienced it's OK at low volumes clean, but that's it.
If you read the 15 or so page V5 mod thread here:
http://www.thefret.net/showthread.php?t=9379 there's a bunch of info and ways to mod a V5, what do you want for tone and overdrive characteristics? Deafelectromark was rebuilding the pcb for $40 (I think) and turning it into a real 5W amp. If you want to do it your self there's many ideas and ways to convert the V5 into a usable amp.