Dear Hello123, hopefully this reply is still of interest given the 2 months elapsed since your post.
1 - In my opinion: yes, as sponsor you have the right to terminate the contract with this site. How to phrase this best and which contract § to cite should be aligned with your legal department. At least you could ask how the instituation intends to tackle the situation (perhaps they have an alternative PI and other staff to take over).
2 - I assume there was an official initiation visit. The site may have investigational product at their disposal. So, yes, a close out should be done. Even if the site did not include subjects, they were "open for recruitment". Depending on what has been done beforehand (e.g. return of investigational product) it may be an option to do the close out remotely. You need to check the specific case. You also want to ensure that any documentation provided (like an Investigators Brochure) will be kept confidential. In my opinion you can also perform a short close out visit with another nomimee of the site who has access to e.g. study material or documents.