Fix dropping grants without unmapping if they have no file descriptor (such as those from anonymous maps)