The decisions derived from AI-based clinical decision support systems should be explainable and transparent so that the healthcare professionals can understand the rationale behind the predictions. To improve the explanations, knowledge graphs are a well-suited choice to be integrated into eXplainable AI. In this paper, we introduce a knowledge graph-based explainable framework for AI-based clinical decision support systems to increase their level of explainability.