I remember how strange it is at first with no mirrors, especially when moving right across the track into the apex on a bend, all your road riding safety kicks in, especially if you can hear a bike close behind you and you yearn to check behind you.
When you think about it though it makes perfect sense. At track speeds you have zero time to think about what's behind you and even if you knew, it's dangerous to move your line to get out of somebodies way.
If the rider behind has 100% responsibility on an overtake it's keeps things simple, what's more, in theory, the rider in fronts line shouldn't change.

I think most TD riders cover their speedo as Simon said, the urge to just have a little peek is very strong at first and you obviously only want to do this at your fastest bit on track - approaching the end of the straight, just the wrong place when you need to get your braking point on target. I know this through personal experience and another Oh**** moment ;-)
What still amazes me is how such a minute change of concentration can have such big consequences, a glance at a crow near the track, a glance at a Marshall you see on his walky-talky, somebody coming off track at the next bend, can all mean you are approaching "your" bend far to fast when you are on the limit anyway !

Reminds me of how much I love it all, although always at the start of a new season when I haven't been on track for a while I get hot just thinking about it lol I always seem to have another mini nervous switch when I see the sign for the race track, but it's funny, as soon as you get on track it's just mega exciting and all you can think about is going quicker and the guy in front of you.
I know it's not a "race" but it's still hard not to try to get round the guy (of neither sex) in front, and is always frustrating when somebody flies by you ! It makes me feel better knowing though that it's not just me, it's 95% of the people on track with you :-)

If there is the slightest bit of you that gets a thrill when you open up the throttle - try it, it's awesome !!!