Yep - wear what you like, and feel comfortable in.
But if you do take a tumble, you'll be picking gravel out of your arse / arms / back / legs / everywhere (delete as appropriate) for weeks.

I quite like a bit of pain, but gravel rash isn't good.