Black-box Machine Learning (ML) models are widely used in Affective Computing applications, including in high-stake tasks, such as autonomous driving or mental health applications. Despite their state-of-the-art performance, blackbox models are usually criticized as non-interpretable as they only produce a final prediction without providing insight into the decision-making process. Previous research on explainable affective computing applications mainly focuses on post-hoc techniques or reverting to traditional ML approaches, which may compromise either the explainability or the performance. Recently, concept-based models have demonstrated great success in general object classification tasks by training the ML model to learn both the classification label as well as the underlying concepts that contribute to the prediction.