Yes, as a physician specializing in prostate cancer I would be concerned-- especially with your family history and the fact that you have performed what I call a biological stress test by using exogenous testosterone. The rate of change of PSA over time is a critical component v. the level of concern. A normal digital rectal examination (DRE) is a low level or almost non-significant biologic input in the context of your situation. Always test the PSA using the same lab and no ejaculation within 48 hours preceding the testing. No DRE prior to testing.
Do not go on and off finasteride or dutasteride since this will definitely muddy the picture. Create a simple Word table of your results along with the meds you are taking. This would be a good start. Remember that the TREND is crucial in early diagnosis of anything, from PC to climate change.