In addition to deleting a Contact, you're also able to mark them as 'Left Company'. This retains all of the information about the Contact, as well as retaining any records which have been created under it.
Step 1: Locate the Contact
Once you've located the required Contact, select the three dots on the right-hand side and select 'Left Company'.
Step 2: Rejoin the Contact
If you need to restore the Contact (i.e. you've accidentally marked them as having left the company, or they have rejoined the organisation), simply select the 'Rejoined' button on the right. When you then view a list of Contacts at the Company level, you'll see that the Contact has a new status applied to it.