I believe the reason for not tearing down pre-existing dwellings (if they existed) is the same for not tearing down a dwelling that hasn't had objection from the council for four years: it's a pretty stone-cold thing to say 'hey, you, tear down your home and get moving!', while saying 'don't you go developing this land' is a somewhat reasonable thing for a planning authority to say.
I don't have the time, or really the interest to look into their zoning laws, but I would figure that the intent here is to prevent the land from becoming another suburb. Neighbours said they 'though he was building a barn', which implies that construction of farming buildings probably isn't a problem, and could explain why people didn't think anything of any signs of construction until they actual saw the structure.