I had the same problems you did, really. I think the Witcher is duller at its best than Oblivion was at its worst. It's just an out-and-out failure.
You get the Silver sword when you leave the outskirts. It works better against monsters.
Yeah, the secondary weapons (dagger, axe) are useless at the beginning, but as you get further into the game you get other things, like the dagger can be replaced by a useful torch, and you also get powerful alternative weapons, like a gladius-type sword that works well against human opponents and has extra properties (stun, etc.)
The combat never really gets any better. And as you go on you'll find that inventory management is going to be a major problem.
You're not being too harsh. Honestly if you don't like it at this point in the game then you're not going to like it later. The biggest problem for me, which you've touched upon, is that the quests are not woven into the storyline very well at all, and they're fucking dull. Escort, fedex and collection missions, all of them, throughout the entire game. You might have been able to overlook them, if only there weren't so goddamn much of it. There's 80 HOURS of this shit. It's so disgustingly bloated, padded to the extreme. If the combat and the quests don't bore you to tears the constant sprinting across maps will. I gave up on the game after about 25 hours, and I had just gotten into Chapter 2. I think there are at least 7 chapters (the outskirts are the "prologue").
Now, there may, may be some hope. You've probably noticed the awful translation and VA. Well, sometime in the next few months they'll be releasing a huge-ass patch that replaces all the dialogue and voicework in the game with newer, presumably better content. However, Geralt will still pretty much be tantamount to a walking, talking styrofoam penis, and the female characters would still make fuckin' Frank Miller squeamish (apparently the only thing keeping women off of any and all dicks in this universe is a fear of pregnancy).