In this paper, the author examines the concept of explorable explanations. It has emerged as a genre of educational software within the last 7 years, yet descriptions of it are vague at best. The author works with the genre through a generic design approach that consists of an analysis of existing explorables and the design of three iterations of the author's own explorable explanation on the topic of neural networks. 22 examples, of which 9 are presented in-depth, are analyzed with educational theory and games research theory as tools. It is found that explorable explanations tend to be digital experiences with a high degree of interactivity that attempt to teach facts, concepts and procedures to the user. Furthermore, the author embarks on a design process of creating explorable explanations of their own to understand what can be relevant when designing and evaluating an explorable explanation. The paper is concluded with reflections on the employed method in the project. Future work is also briefly outlined about what impact the analysis and design work can have on the practice of other designers seeking to work with the genre, as well as to other researchers.