This paper presents the results of a study performed on Parallel Convolutional Neural Network (PCNN) toward detecting heart abnormalities from the heart sound signals. The PCNN preserves dynamic contents of the signal in a parallel combination of the recurrent neural network and a Convolutional Neural Network (CNN). The performance of the PCNN is evaluated and compared to the one obtained from a Serial form of the Convolutional Neural Network (SCNN) as well as two other baseline studies: a Long- and Short-Term Memory (LSTM) neural network and a Conventional CNN (CCNN). We employed a well-known public dataset of heart sound signals: the Physionet heart sound. The accuracy of the PCNN, was estimated to be 87.2% which outperforms the rest of the three methods: the SCNN, the LSTM, and the CCNN by 12%, 7%, and 0.5%, respectively. The resulting method can be easily implemented in an Internet of Things platform to be employed as a decision support system for the screening heart abnormalities.