You might also tap the starter with hammer. That sounds almost like the bendix is not engaging at all, which could jiggle loose with a few taps. I had an old F-250 with a 7.3 IDI that would do that....each direction! Sometimes it would stick shut and not start, sometimes it would stick open and not disengage. I'd have to watch the voltmeter when I started it, and if it never recovered to charging voltage, I'd have to run out and tap on the starter to get it to disengage before burning up the lead wire.
If it's not stuck, and not engaging, the bendix spring could be broken. It's not a terrible fix, but starters are pretty cheap as it is.