Fewer individuals results in greater sampling error.

The smaller the size of a population, the greater the impact of genetic drift and, as a consequence, the lower the impact of natural selection on the population.

You can think of natural selection as being equivalent to "signal" and genetic drift as "noise, with the signal-to-noise ratio smaller – which is a "bad" thing! – the smaller the population size. Smaller populations, as a consequence, evolve less deterministically and instead more stochastically.

In terms of genetic drift, see: impact of population size, impact on allele frequency, impact on genetic variation, and impact on allele fixation.