Global rise in male infertility as a result of dysfunction of male germ cells has been pointed out by many investigations. Therefore, it is essential to elucidate the molecular mechanisms involving in the development of sperm, which rely on phase-specific gene expression that is regulated by myriads of small non-coding RNAs at transcriptional, post-transcriptional and epigenetic level. Recent advancement in small non-coding RNAs mainly including siRNAs, miRNAs and piRNAs has determined several pathways, which synergically regulate the process of male germ cell development. However, the aberrant expression of them is associated with dysfunction of male germlines, such as sperm arrest or apotosis, which further leads to male infertility. In this review, we focus mainly on the biogenesis and functions of these transcripts in the regulation of mammalian male germ cell development, elucidating the mechanisms between their dysregulation and related dysfunctions of sperm and providing some basic informations for diagnosis and treatment of male infertility.