Inhibition of Return (IOR) refers to a slowing of response times (RTs) for visual stimuli repeated at the same spatial location, as compared to stimuli occurring at novel locations. The functional mechanisms and the neural bases of this phenomenon remain debated. Here we present FORTIOR, a model of the cortical control of visual IOR in the human brain. The model is based on known facts about the anatomical and functional organization of fronto-parietal attention networks, and accounts for a broad range of behavioral findings in healthy participants and brain-damaged patients. FORTIOR does that by combining four principles of asymmetry: a) Asymmetry in the networks topography, whereby the temporoparietal junction (TPJ) and ventrolateral prefrontal cortex (vlPFC) nodes are lateralized to the right hemisphere, causing higher activation levels in the right intraparietal sulcus (IPS) and frontal eye field (FEF) nodes. b) Asymmetry in inter-hemispheric connectivity, in which inter-hemispheric connections from left hemisphere IPS to right hemisphere IPS and from left hemisphere FEF to right hemisphere FEF are weaker than in the opposite direction. c) Asymmetry of visual inputs, stipulating that the FEF receives direct visual input coming from the ipsilateral visual cortex, while the right TPJ and vlPFC and IPS nodes receive input from both the contralateral and the ipsilateral visual fields. d) Asymmetry in the response modality, with a higher response threshold for the manual response system than that required to trigger a saccadic response. This asymmetry results in saccadic IOR being more robust to interference than manual IOR. FORTIOR accounts for spatial asymmetries in the occurrence of IOR after brain damage and after non-invasive transcranial magnetic stimulation on parietal and frontal regions. It also provides a framework to understand dissociations between manual and saccadic IOR, and makes testable predictions for future experiments to assess its validity.