The rapid growth of new emerging computing technologies has encouraged many organizations to outsource their data and computational requirements. Such services are expected to always provide security principles such as confidentiality, availability and integrity; therefore, a highly secure platform is one of the most important aspects of cloud-based computing environments. A considerable improvement over traditional security strategies is achieved by understanding how malware behaves over the entire behavioural space. In this paper, we propose a new approach to improve the capability of cloud service providers to model users’ behaviours. We applied a particle swarm optimization-based probabilistic neural network (PSO-PNN) for the detection and recognition process, in the first module of the recognition process, we meaningfully converted the users’ behaviours to an understandable format and then classified and recognized the malicious behaviours by using a multi-layer neural network. We took advantage of the UNSW-NB15 dataset to validate the proposed solution by characterizing different types of malicious behaviours exhibited by users. Evaluation of the experimental results shows that the proposed method is promising for use in security monitoring and recognition of malicious behaviours. © 2019 Elsevier Ltd