Sometimes I get almost duplicate URLs with both http and https versions (I have no idea why this happens as the MB artist record should only have at most one of these types in the Official Artist Website relationship), and sometimes the sort order is different (despite there being a sort instruction at line 107).

I will try to find the time to look at this at some point, but in essence to avoid duplicated URLs:

1. if we get an http URL we need to check and delete the equivalent https record; and
2. if we get an https URL we need to check for the equivalent http record and if it exists discard the https version.

or vice versa. This will not however fix the issue where sometime it gets an http and sometimes an https - and I don't think we can safely convert all to either http or https.