Well, the sun is always yellow or orange, and fire did exist before humans learned to control it. Fire isn't exactly a human invention... Plus, the night is always dark and cold, and the colours you can see when it's dark are mostly blue and grey-ish. Plus, ice is often light blue due to the microorganisms that can live there.
However, we don't know when the associations were developed or if they changed, but I'd argue the associations various colours had before the stone age has little impact on how we perceive things today, that their colour associations were the same as ours or not doesn't really matter.