Microblogging services such as Twitter and Plurk allow users to easily access and share different types of social multimedia (e.g. images and videos) in the cyber world. However, the massive amount of information available causes information overload, which prevents users from quickly accessing popular and important digital content. This paper studies the problem of predicting the popularity of social multimedia content embedded in short microblog messages. A property of social multimedia is that it can be continuously re-shared, thus its popularity may revive or evolve over time. We exploit the idea of concept drift to capture this property. We formulate the problem using a classification-based approach and propose to tackle two tasks, re-share classification and popularity score classification. Two categories of features are devised and extracted, including information diffusion and explicit multimedia meta information. We develop a concept drift-based popularity predictor by ensembling multiple trained classifiers from social multimedia instances in different time intervals. The key idea lies in dynamically determining the ensemble weights of classifiers. Experiments conducted on Plurk and Twitter datasets show the high accuracy of the popularity classification and the results on detecting popular social multimedia are promising.