Researchers haven't unraveled the mystery of just how testosterone increases libido. It's normal for a man's sex drive to slowly decline from its peak in his teens and 20s, but libido varies widely between men. What one man might consider a low sex drive, another might not. Also, sex drive changes within each man over time and is affected by stress, sleep , and opportunities for sex. For these reasons, defining a "normal" sex drive is next to impossible. Usually, the man himself identifies a lack of sex drive as a problem. Other times, his partner may consider it to be an issue.