This paper proposes a fast and sufficiently accurate algorithm for calculating fractional time-derivatives of stress and strain in the viscoelastic damper. The algorithm also has an advantage of requiring substantially smaller memory space. At every step, numerical integration using past stress and strain data is required, but the proposed algorithm needs only a small portion of such data: it selects and stores the data at much longer interval than the original algorithm. The algorithm is applied to two types of viscoelastic damper models, which are "short duration model" and "long duration model". Its numerical efficiency and accuracy are demonstrated through comparison with the original and precise algorithm.