While the role of attention in determining the neural fate of unattended emotional items has been investigated in the past, it remains unclear whether bottom-up and top-down factors have differential effects in shaping responses evoked by such stimuli. To study the effects of bottom-up and top-down factors on the processing of neutral and fearful faces, we employed functional magnetic resonance imaging (fMRI) while participants performed attentional tasks that manipulated these factors. To probe the impact of top-down mechanisms on the processing of face distractors, target letters either had to be found among several distinct nontarget letters (attentional load condition) or among identical nontarget letters (baseline condition). To probe the impact of bottom-up factors, we decreased the salience of the targets by reducing their size and contrast relative to baseline (salience condition). Our findings revealed that bottom-up and top-down manipulations produced dissociable effects on amygdala and fusiform gyrus responses to fearful-face distractors when task difficulty was equated. When the attentional load of the main task was high, weaker responses were evoked by fearful-face distractors relative to baseline during the early trials. By contrast, decreasing target salience resulted in increased responses relative to baseline. The present findings suggest that responses evoked by unattended fearful faces are modulated by several factors, including attention and stimulus salience.