It is true, If you are primary and lack the testes, then you lack everything produced on testes (except the amounts produced in adrenal glands), but if you have the testes and are primary, many other things may be also affected to some extension at the testes, starting by sperm production...
For Aromatase: If you have a deficit of Aromatase, you have chances to behave as primary but more likely with normal to high T levels. If you are an over-aromatiser, then you may become primary with high E levels, low T Levels and and very sensitive to have High E when getting even small amounts of exogenous Testosterone. None of these two situations seems ti fit to you.
All I say is only a guess... I am far from being and expert on this matter. Just learning because this problem of Low T has reached me at my late fourties..