impact driver wont work, you are only likely to catch the side of your hand and end up with a 'black mans pinch'.
just make sure bike is safe when working on it as you dont want it falling over.
just checking, will the whole lot move front to back? if the spindle runs through the rear caliper bracket use plenty of lube .
work on it tommorrow, it will only go wrong if you keep trying tonight.
dont forget bar that runs from caliper mount to swing arm, may be holding it in place.