Fifth-generation (5G) mobile networks have already marked their presence globally, revolutionizing entertainment, business, healthcare, and other domains. While this leap forward brings numerous advantages in speed and connectivity, it also poses new challenges for security protocols. Machine learning (ML) and deep learning (DL) have been employed to augment traditional security measures, promising to mitigate risks and vulnerabilities. This paper conducts an exhaustive study to assess ML and DL algorithms' role and effectiveness within the 5G security landscape. Also, it offers a profound dissection of the 5G network's security paradigm, particularly emphasizing the transformative role of ML and DL as enabling security tools. This study starts by examining the unique architecture of 5G and its inherent vulnerabilities, contrasting them with emerging threat vectors. Next, we conduct a detailed analysis of the network's underlying segments, such as network slicing, Massive Machine-Type Communications (mMTC), and edge computing, revealing their associated security challenges. By scrutinizing current security protocols and international regulatory impositions, this paper delineates the existing 5G security landscape. Finally, we outline the capabilities of ML and DL in redefining 5G security. We detail their application in enhancing anomaly detection, fortifying predictive security measures, and strengthening intrusion prevention strategies. This research sheds light on the present-day 5G security challenges and offers a visionary perspective, highlighting the intersection of advanced computational methods and future 5G security. © 2023 by the authors.