Dirt getting into moving parts will grind them away, especially chains, sprockets, lever pivots, swingarm/suspension linkage bearings. It will also retain moisture and promote corrosion.
At best a dirty bike will loose you money in excess repairs and depreciation, at worst you are putting your life at risk. And yes, I have seen bike that have serious faults or even bits missing that have been hidden by a bit of 'harmless' dirt (in actual fact, usually a massive wad of oily gunge).
No need to go over board and strip it down daily, just a going over it with some mild car shampoo and some decent wax once a month for an average mileage rider will be better than nothing. I definitely recommend using ACF-50 if you ride all year though.