The Earth’s climate is projected to warm significantly in the 21st century, and this will affect human societies in many ways. Since sleep is a basic human need and part of everyone’s life, the question of how temperature affects human sleep naturally arises. This paper examines the effect of daily mean temperature on sleep duration using nationally representative Hungarian time use surveys between 1976 and 2010. Compared to a day with an average temperature of 5–10 °C, colder temperatures do not influence sleep duration. However, as daily mean temperatures rise, sleep duration starts to strongly decline. The effect of a hot (>25 °C) day is −13.3 minutes, but if preceded by a few other hot days, the effect is even stronger, −24.7 minutes. The estimated sleep loss is especially large on weekends and public holidays, for older individuals, and men. Combining the estimated effects with temperature projections of twenty-four climate models shows that the warming climate will substantially decrease sleep duration. The projected impacts are especially large when taking into account the effects of heatwave days. This study also shows that different groups in society are likely to be affected in significantly different ways by a warming climate.