A bunch of factors, including some of those mentioned.
Estrogen-imitators in the food supply (soy being one of them -- it's in a LOT of foods, soybean oil, soy 'protein', etc. -- just read the label). Lack of exercise compared to earlier generations. Foods causing excess weight gain, which can effect T-levels in some individuals. Less smoking of tobacco is also possibly a factor (smoking actually increased T levels -- no joke. Not saying one should smoke, though. Not at all).
And then you have the psychological aspect. There have been studies done showing a connection between T-levels and attitudes, psychological makeups. If you raise a generation or two of young men by constantly telling them that their masculinity is the problem, and they need to be smarmy, passive nice guys, enough of them will be that way, and their T-levels will drop correspondingly.
And there have apparently been studies connecting attitude, outlook, and T-levels. I once read about a study in the 1970's or 80's that showed that 'nice guy' church ministers had lower T-levels than cops, athletes, soldiers, EMS people, factory workers, etc.
So the problem has a lot of diverse causes. I think most of it is diet, exercise, and attitude. But that's just my own take on it.