Many aspects of human and animal interaction, such as the frequency of contacts of an individual, the number of interaction partners, and the time between the contacts of two individuals, are characterized by heavy-tailed distributions. These distributions affect the spreading of, e.g., infectious diseases or rumors, often because of impacts of the right tail of the distributions (i.e., the large values). In this paper we show that when it comes to inter-event time distributions, it is not the tail but the small values that control spreading dynamics. We investigate this effect both analytically and numerically for different versions of the susceptible-infected-recovered model on different types of networks.