As above, pressing the delete button should instead do a "Remove tag", if it is removable; not remove the file (if linked).

I think if you press delete with the tag name highlighted, it should try and remove from old and new metadata. If you have just one of the columns highlighted it should just remove it from one or the other.