Less than you think, and it's not too late. After this season, they're still only 3 of 5 books in. I read all five books between seasons 2 and 3 (after the first two books had aired), and seeing the show didn't make the books less great since there's so much more detail in the books. The only thing the show did when I read the books? I heard the characters' voices as the TV ones (especially Tyrion).
Linds, the fourth one isn't awful, although it is the weakest of the series and it follows the best. The Cersei chapters are pretty bad, though. The books only color the show for me in the sense that I know more than the show reveals, and I know what's going to happen (although not always, and it rarely happens when I think it will).