Multiple problems with buddleia?

Mine are flowering but the plants themselves, even those seemingly unaffected by bugs or disease, haven't grown anything like as much as in past years.

 

Seems random to me - I've seen two plants of the same cultivar close together where one is totally infested with aphids, like a broad bean plant, and a few metres away there's almost no damage on the other. Soft, pot-grown plants - cuttings and seedlings in particular - are more susceptible, but that's true for all pests.

 

Thank you @Thevictorian, that's interesting. I had a look at the smaller plants that had self-seeded, some have the aphid damage but have buds, others no damage, they are right next to each other I have learned a lot about buddleia's, thank you everyone for sharing your knowledge and experience

 

I have the same issue as the OP, wouldn't spraying the buddleias with horticultural soap deter the aphids? Predators don't seem to be sufficient prevention, I've not sprayed any chemicals in my garden for 6 years!