Rik.
Most of the noise comes from the drive system.
The old analogue bandpass filter suffers a large phase shift for a
small change in frequency. The phase gives you the position which once the Q is high enough to get rid of the noise, the angular error is significant.
A small angular position, once the balance becomes finer becomes an error at close 90 degrees.
Don't worry about speed regulation.
Just spin it fast, release the drive belt and process on the
spin down.
You can use a Kalmen filter
Kalman filter - Wikipedia, the free encyclopedia to resolve signals at 0 and 90 degrees, say, call them X and Y.
I reality you simply multiply the signal by plus/minus 1 at the correct times for 0 and 90 degrees, and save them, along with the current speed.
sqrt(X^2 + Y^2) gives you the vector for each measuring plane.
Multiply X and Y by 1/(speed squared), to get rid of the speed problem.
Stuff all the vectors in and array for discrete speed steps, and process as percentiles, and throw out the wild values caused by silly resonances.
You need 0 and 90 degrees signals optically from the shaft, and the signals for each measuring plane.
For this to work it needs to be a high tune system where you measure the unbalance FORCES. Not deflection.
Piezoelectric transducers against a large mass, work well.
You can resolve the vector to the balancing plane(s) from the measuring planes by simple moments about the measuring points treating the shaft as a beam.