With the emergence of multimedia applications and the upcoming age of Voice over IP (VoIP), Voice setup and resources control protocols such as SIP and H.323 over the Internet are becoming increasingly attractive applications. In the last few years as a real competitor in traditional telephony services (PSTN), SIP has gained much attention when compared with H.323. SIP works at presentation and application layer thus it mainly faces security issue at these layers. The objective of this thesis is to describe the most relevant SIP related security issues and then present security mechanisms that can be deployed to overcome the SIP security related issues.

This project work demonstrates the tasks necessary to enhance the SIP security both inside and outside of the network. It is divided into three main parts, where the first part describes the SIP architecture, for example, the SIP rivals, SIP components and how a SIP system works. The second part is about some vulnerability issues of concern to SIP, study of the proposed security mechanism and also analysis on how possible threats to the SIP system such as call hijacking, message tempering and DoS attack, affect the SIP based VoIP system. The third and final part describes different steps that have been taken to avoid SIP attacks, by implementing some of the proposed security mechanisms.

In order to test the SIP security, a SIP model is designed, which based on security mechanisms such as firewall, IPSec, DMZ and SIP-TLS. The results are conducted into two different scenarios. In the 1^st scenario, the SIP system is tested before implementing the security measurements. In this case, the insecure system was vulnerable to several SIP attacks such as call hijacking, DOS and message tampering. In the 2^nd scenario, the system is tested after the implementation of the proposed security mechanisms, where by the system now is only accessible to the authorized users and services. The tested results are also compared and discussed at the end.