The Big Video Data generated in today’s smart cities has raised concerns from its purposeful usage perspective, where surveillance cameras, among many others are the most prominent resources to contribute to the huge volumes of data, making its automated analysis adifcult task in terms of computation and preciseness. Violence detection (VD), broadly plunging under action and activity recognition domain, is used to analyze Big Video data for anomalous actions incurred due to humans. The VD literature is traditionally basedon manually engineered features, though advancements to deep learning based standalone models are developed for real-time VD analysis. This paper focuses on overview of deepsequence learning approaches along with localization strategies of the detected violence.This overview also dives into the initial image processing and machine learning-based VD literature and their possible advantages such as efciency against the current complex models. Furthermore,the datasets are discussed, to provide an analysis of the current models, explaining their pros and cons with future directions in VD domain derived from anin-depth analysis of the previous methods. © The Author(s), under exclusive licence to Springer Nature B.V. 2022.