Approximate factor models and their extensions are widely used in forecasting and economic analysis due to their ability to extracting useful information from a large number of relevant variables. In these models, candidate predictors are typically subject to some common components. In this paper, we consider to efficiently estimate an approximate factor model in which the candidate predictors are additionally subject to idiosyncratic large uncommon components such as jumps or outliers. By assuming that occurrences of the uncommon components are rare, we propose an estimation procedure to simultaneously disentangle and estimate the common and uncommon components. We formulate the estimation problem as a penalized least squares problem in which a norm penalty function is imposed on the uncommon components. To solve the estimation problem, we propose an algorithm, which iteratively solves a principal component analysis (PCA) problem and a one dimensional shrinkage estimation problem. The algorithm is flexible in incorporating methods for selecting the number of common components. We then compare finite-sample efficiency of the proposed method and traditional PCA method with simulations. We also demonstrate performances of the proposed method with empirical applications on predicting yearly growths of important macroeconomic indicators.