Accidentally made file references not count as references and therefore also close the fd. My bad.