I believe you are correct; if the unsafe code can cause undefined behavior if input data is not following a specific contract, then the entire function should be labeled unsafe so the caller knows that.
The other option is to check to make sure the contract is valid, and return an error or panic if it is not. That function would be sound, as no inputs cause undefined behavior.
For anyone who is confused: This is exploiting an old soundness bug in the Rust compiler that is still present. The GitHub issue page has this comment from maintainers: