Normally you can't link an entity to itself. We generally prevent it by changing the link when using "Use in relationship" and not returning the same entity in the search results for the "Relate to ..." inline search. If you try to force it, e.g. http://beta.musicbrainz.org/edit/relationship/create?type0=recording&type1=recording&entity0=9c531d53-78c6-4502-882e-8eb39136b1a1&entity1=9c531d53-78c6-4502-882e-8eb39136b1a1&returnto=http%3A%2F%2Fbeta.musicbrainz.org%2Frecording%2F9c531d53-78c6-4502-882e-8eb39136b1a1 it gives an error.
The relationship editor does not prevent a user from selecting the same entity nor from submitting the edit, e.g. http://beta.musicbrainz.org/edit/19187830