Avoid panics (whenever possible)

There are quite a few places in the code where unwrap() and expect() are used and could lead to panics. Remove as many of these as is possible. If possible to remove all or most then add a #disallow order at the root of the crate and fail in CI if any are re-introduced.