My wife has a 6x8 which i use on occasion. Once I actually get more into CG and learn to procrastinate less I'll probably be using it more or getting my own little 4x5.
The only major differences really is the drawing space. It's kinda important I guess since it kinda effects the way you draw. Basically think of the whole drawing area as the size of your monitor as a whole.
So, with a 4x5 it's kinda like you're drawing a teeny tiny caricature on a teeny tiny pad, but instead it's actually turning out as a larger drawing. With a bigger pad, you're drawing a bigger caricature on a bigger pad, but it's coming out the same size on the monitor. For me, it doesn't seem like it makes much of a difference, especially since I will most likely keep doing my actual penciling traditionally, and just ink/color digitally... so I will be buying the relatively cheaper 4x5. It's a Wacom anyway, it will always be good. '-'
You can also adjust the drawing dimensions as lb969 stated earlier.
There's also the differences in pressure sensitivity, but that usually varies by model, and not size dimensions.