Fix panic when the start matches

~/.local/<tab> crashed on my computer, because the script tried to remove the prefix directory two times.