What limits the adaptability of populations.

A population that lacks genetic variation cannot adapt evolutionarily, period. The more genetic variation that a population possesses, in turn, then the greater its potential to adapt.