The stability pot (potentiometer) on a generator AVR controls the voltage under transient conditions. Stability set too low will result in the generators voltage fluctuating up and down, whereas a too high  stability setting will result in a slower voltage response.

How do you set the STAB potentiometer on a generator?

For setting setting the stability on an AVR, always refer to the manufacturers instructions. Some common AVR types are listed here. Some digital voltage regulators may have the potentiometer turned off in the software, so it may have no effect.

The AVR terminals are LIVE, so ensure you are qualified and take precautions, such as using an insulated screwdriver.

Typically, the correct setting of the stability adjustment can be found by starting the generator,  and running with no load.  Turn the stability fully clockwise, then slowly turn the stability control anti-clockwise until the generator voltage starts to become unstable.
The critically damped or optimum position is slightly clockwise from this point - this is where the machine volts are stable but close to the unstable region.