The problem here was that we always added the base address, and we assumed that all addresses we access are relative but this is not the case in case of non pie binaries. The issue is that all addresses were base+offset. so if we added the base again it will of course generate wrong address.
Note that this patch doesn't fix the issue that the issue that ld.so gets loaded at the default load address for non pie binaries.