At no point should you ever be so concerned about protecting a licensed professional’s feelings that you don’t ask for what you want, in this case to be left alone. If they get their feelings hurt, that’s totally on them, because they’re (supposed to be) the professional in this situation.
In the US, medical records within a practice or facility are able to be accessed on a “need to know” basis by those working with the patient. If your doc refers you to another specialty, whoever comes to see you will have access to your medical record. Strict disclosure laws apply to releasing info to entities outside the facility