Too much water. Too little water. Too much sun. Too little sun. Root bound in a pot. Some plants in pots require that their roots be cramped for good growth. Root girdling in the ground. Wrong type of fertilizer. Not being fertilized. Weather too hot. Weather too cool. Contaminated compost. Insect eating it. No natural pollinators around. People walking on them. Pets walking, laying down on them or repeatedly digging them up. Planted too close to larger plants. Invasive non-natives smothering. Plants that are not Juglone Tolerant being planted near a Black Walnut tree.
Humidity too high or low. Too much wind. Wildly 'off' PH. Planted in the ground in 'bowl' of soil that is wildly different from the native soil, which can cause drowning or dessication. Altitude too high or low. In addition to pests that actually ingest plant parts, there are parasites (probably not the correct technical term, but that's what these critters *do*) such as scales, thrips, aphids that can cause damage, lack of growth, distorted growth, sometimes fatality. Salt spray from ocean. Toxic run-off from weed'n'feed on lawns, from agricultural fields, highways. Car exhaust along a busy road. Moles, voles, chipmunks, rats, toads, tunneling around roots. Bad vibes.
The type of soil you have could be a factor as well. If you have a lot of clay in your soil, it could make it difficult for the roots of some plants to grow.
Excellent point, Ava! Continually adding organic matter to the surface when available (mulch, leaves, compost,) can help change that very easily over time. The first time I piled fall leaves over a bed, I was instantly hooked the next spring when I dug in and saw the improvement. That was in housing development clay in OH - where they remove the top soil to sell before building the houses. Also works wonders here in AL, with sandy, won't-hold-water soil.