The “graying” of the population is proceeding rapidly, and the prevalence of people living with dementia is rising. Dementia is a syndrome due to the disease of the brain, which is characterized by a progressive, global deterioration in the intellect including memory, learning, orientation, language, comprehension, and judgment. This entails an increased need for assistance with daily tasks. In most cases, family members perform the caregiving role. However, providing care and support for someone with dementia is associated with an enormous emotional, physical, and social stress. Therefore, it is necessary for caregivers to take periods of rest regularly and to take care of their own health. In Germany, assisted vacations have been established as an important type of potential support for family caregivers. They go on a vacation together with their care recipient. During their stay, professionals and volunteers assist with the care and, as a result, relieve the family caregivers. Unfortunately, few vacation accommodations meet the specific needs of the target group. Especially, a proper design of the physical environment can positively affect functionality, behavior, and well-being among people with dementia and, thus, also benefit the caregivers. Former research on designing dementia-friendly environments has been heterogenic, and little is known about how to design the built environment for people with dementia outside nursing homes. The purpose of this paper is to explore ways of designing holiday accommodations for people with dementia and their family caregivers. The methods used were, first, a literature review on design and family caregiving for people with dementia. This aimed at identifying dementia-friendly physical features and at determining dementia caregivers’ characteristics. Second, a case study of existing holiday homes was carried out to detect strengths and weaknesses of their built environments. In combination, caregiver’s demands on the design of the built environment were assessed through qualitative interviews with caregivers. Consequently, design criteria were developed through a synthesis of the result of the previous steps. Finally, the resulting design criteria were transferred into a concept for a prototypical holiday accommodation. Results suggest that a holiday accommodation, which meets the demands of the target group on the built environment, has to be barrier-free and dementia-friendly designed; has to offer situational orientation cues; and provides space for activities, communication, and retreat. This paper represents a first approach of exploring ways to design dementia-friendly holiday homes. There is a need to further examine the resulting ideas.