Unicode Private Use Area characters are stripped during edit process

ExportXMLWordPrintable

      When editing the following release, Unicode "private use" characters were silently stripped from the tracklist when navigating to the submission page: https://musicbrainz.org/edit/71525782

      I then tried to create an artist with the name "" at https://test.musicbrainz.org/artist/create, but it failed with the following ISE:

      Caught exception in MusicBrainz::Server::Controller::Artist->create "The edit could not be created. Exception (MusicBrainz::Server::Exceptions::DatabaseError): 23514 DBD::Pg::st execute failed: ERROR: new row for relation "artist" violates check constraint "only_non_empty"
DETAIL: Failing row contains (1990049, 7b1f3365-f03d-4535-bbf7-c547f1943d90, , , null, null, null, null, null, null, null, null, null, , 0, 2020-07-21 05:23:19.510745+00, f, null, null). [for Statement "INSERT INTO artist (gender,begin_date_day,sort_name,begin_date_year,comment,type,begin_date_month,end_area,begin_area,end_date_month,gid,name,ended,area,end_date_year,end_date_day) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) RETURNING id" with ParamValues: 1=undef, 2=undef, 3='', 4=undef, 5='', 6=undef, 7=undef, 8=undef, 9=undef, 10=undef, 11='7b1f3365-f03d-4535-bbf7-c547f1943d90', 12='', 13='0', 14=undef, 15=undef, 16=undef] at lib/MusicBrainz/Server/Controller.pm line 82.
 at lib/Sql.pm line 335."

      As it turns out, there was a recent change made to `remove_invalid_characters` that included the following construct: `\p{Co}` - I believe this is causing the issue.

      My particular use case ("") would require unblocking of the following private use area: U+E000..U+F8FF.

            Assignee:
            Michael Wiencek
            Reporter:
            jacobbrett
            Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved:

                Version Package
                2021-03-15