This research studies the security challenges in IoT devices. At first, security challenges have been described and then specifically the security of communication protocols in the IoT has been addressed. Finally, among different communication protocols, ZigBee and Z-Wave protocols have been chosen for this study. The criterion for choosing these two protocols is the level of security they provide for IoT devices to protect them against unauthorized access and hacking. Security, frequency, power consumption and data rate are the characteristics that have been discussed in the review of these two protocols. In the end, a comparison of the various features of these two protocols clarified that the security of IoT devices in each of these protocols depends on the type of the IoT device, the required range and other requirements, however, in most cases the ZigBee protocol showed more security than Z-Wave.