Prevent the build cache for the installer getting clobbered

Previously it was getting built by two different toolchains (nightly from rustup and the relibc prefix) Also some env vars differed between both invocations causing blake3 and sodiumoxide to get rebuilt.