Uploaded image for project: 'Picard'
  1. Picard
  2. PICARD-1359

Crash with tagger integration when using DuckDuckGo Privacy Essentials

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Normal Normal
    • 2.1
    • 2.0.4
    • None
    • None
    • Win 10, 64. Firefox latest (62.0.2 64bit). Greasemonkey with one script: Import Discogs (disabled).

      Since this weekend, as reported in the forum by two users, the tagger will make Picard close down. There are no crash reports to be found in debug mode.

      Retest with fresh install, no plugins, user scripts in Firefox also disabled.

      Forum post: https://community.metabrainz.org/t/crash-error-language-preferences/399901/3

          [PICARD-1359] Crash with tagger integration when using DuckDuckGo Privacy Essentials

          The mitigation of the crash from our side is getting into the next release. Also the developers of the Duckduckgo extension are aware of the issue (which is not only a problem for MB.org) and have already reverted some related changes. So hopefully a new release of the extension will also solve the issue from this side.

          In the meantime please whitelist musicbrainz.org in the extension.

          Philipp Wolfer added a comment - The mitigation of the crash from our side is getting into the next release. Also the developers of the Duckduckgo extension are aware of the issue (which is not only a problem for MB.org) and have already reverted some related changes. So hopefully a new release of the extension will also solve the issue from this side. In the meantime please whitelist musicbrainz.org in the extension.

          Bob Swift added a comment -
          We also should merge zas changes to avoid the crash

          I agree, but we might consider moving the debug dump line into the try wrapper so that it only appears if there is a problem, That will save a few log entries.

          Bob Swift added a comment - We also should merge zas changes to avoid the crash I agree, but we might consider moving the debug dump line into the try wrapper so that it only appears if there is a problem, That will save a few log entries.

          Philipp Wolfer added a comment - - edited

          For users the current fix would be to add "https://musicbrainz.org" to the whitelist of the DuckDuckGo Privacy Essentials extension.

          We also should merge zas changes to avoid the crash, but at the end we can not really handle this case so "it just works". There have been talks about using a different approach, e.g. by implementing a custom protocol, at PICARD-715 and PICARD-529, which would also circumvent issues like this. But basically in this case this should best be fixed by the extension, I have sent a "broken site" report, maybe they respond:

          UPDATE: Found out they have a Github project, reported issue at https://github.com/duckduckgo/duckduckgo-privacy-extension/issues/323

          Philipp Wolfer added a comment - - edited For users the current fix would be to add "https://musicbrainz.org" to the whitelist of the DuckDuckGo Privacy Essentials extension. We also should merge zas changes to avoid the crash, but at the end we can not really handle this case so "it just works". There have been talks about using a different approach, e.g. by implementing a custom protocol, at PICARD-715 and PICARD-529 , which would also circumvent issues like this. But basically in this case this should best be fixed by the extension, I have sent a "broken site" report, maybe they respond: UPDATE: Found out they have a Github project, reported issue at https://github.com/duckduckgo/duckduckgo-privacy-extension/issues/323

          Just a confirmation that I can reproduce the warning in developer mode and that turning off the Site Privacy Protection removes the closing of Picard.

          Thanks for all your work on this!

          Michel Vorenhout added a comment - Just a confirmation that I can reproduce the warning in developer mode and that turning off the Site Privacy Protection removes the closing of Picard. Thanks for all your work on this!

          Bob Swift added a comment -

          Okay, I found that if I turned off the Site Privacy Protection for musicbrainz.org in the Firefox "DuckDuckGo Privacy Essentials" extension everything seems to be working just fine.  It would seem that this is not a problem with the Picard code.

          I suspect that this issue might be alleviated by including a Content Security Policy setting in the web page header. I'm still trying to digest the information found at https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP.

          Bob Swift added a comment - Okay, I found that if I turned off the Site Privacy Protection for musicbrainz.org in the Firefox " DuckDuckGo Privacy Essentials " extension everything seems to be working just fine.  It would seem that this is not a problem with the Picard code. I suspect that this issue might be alleviated by including a Content Security Policy setting in the web page header. I'm still trying to digest the information found at https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP .

          Bob Swift added a comment -

          The developer console in Firefox is showing:

          Content Security Policy: Upgrading insecure request ‘http://127.0.0.1:8000/openalbum?id=0624ba55-1e47-43a8-87fb-640e251b58f1&t=1538511382’ to use ‘https’
          

          I'm still trying to find where I can view / edit the Content Security Policy.

           

          Bob Swift added a comment - The developer console in Firefox is showing: Content Security Policy: Upgrading insecure request ‘http://127.0.0.1:8000/openalbum?id=0624ba55-1e47-43a8-87fb-640e251b58f1&t=1538511382’ to use ‘https’ I'm still trying to find where I can view / edit the Content Security Policy.  

          Zas added a comment - - edited

          Not really, because that's still not what we expect as request,so either this is what the browser sends, or we have some obscure bug in our code.

          Can you just check the request on the browser side ?

          Done.

          So Firefox is expecting an https connection to Picard ... ok. Why ?

           

          Note: it is http only, we don't manage any certificate for ssl connection or whatever, so the browser has to use http, not https.

          Zas added a comment - - edited Not really, because that's still not what we expect as request,so either this is what the browser sends, or we have some obscure bug in our code. Can you just check the request on the browser side ? Done. So Firefox is expecting an https connection to Picard ... ok. Why ?   Note: it is http only, we don't manage any certificate for ssl connection or whatever, so the browser has to use http, not https.

          Bob Swift added a comment -

          This is coming from Firefox 62.0.2 (64-bit), and it appears that the link is:

           

          http://127.0.0.1:8000/openalbum?id=0624ba55-1e47-43a8-87fb-640e251b58f1&t=1538444537
          

          Copying the link address and pasting it into a new window works perfectly in both Firefox and Chrome.  Just something strange when clicking the link on the page.  If I right-click the link and open it in a new window, again everything is fine.

          It appears that when the link is clicked directly, Firefox is forcing it to use HTTPS as:

          https://127.0.0.1:8000/openalbum?id=0624ba55-1e47-43a8-87fb-640e251b58f1&t=1538444537
          

          which I'm guessing is the problem.

          Bob Swift added a comment - This is coming from Firefox 62.0.2 (64-bit), and it appears that the link is:   http://127.0.0.1:8000/openalbum?id=0624ba55-1e47-43a8-87fb-640e251b58f1&t=1538444537 Copying the link address and pasting it into a new window works perfectly in both Firefox and Chrome.  Just something strange when clicking the link on the page.  If I right-click the link and open it in a new window, again everything is fine. It appears that when the link is clicked directly, Firefox is forcing it to use HTTPS as: https://127.0.0.1:8000/openalbum?id=0624ba55-1e47-43a8-87fb-640e251b58f1&t=1538444537 which I'm guessing is the problem.

          Bob Swift added a comment -

          If I modify the _process_request() method in browser.py to:

              def _process_request(self):
                  conn = self.sender()
                  conn.write(b"HTTP/1.1 200 OK\r\nCache-Control: max-age=0\r\n\r\nNothing to see here.")
                  rawline = conn.readLine().data()
                  log.debug("Browser integration request: %r", rawline)
                  rawline = rawline.decode('iso-8859-1')
                  rawline = rawline.encode('utf-8')
                  log.debug("Browser integration request (recoded): %r", rawline)
                  try:
                      line = rawline.decode()
                  except UnicodeDecodeError as e:
                      log.error(e)
                      return
                  finally:
                      conn.disconnectFromHost()
          

          there is no error generated, and the relevant part of the log is:

          D: 13:09:03,501 C:\Junk\Download\picard-4526ebb95a4a81f37af888703321cb23ae6a7799\picard\webservice\ratecontrol._out_of_backoff:225: ('musicbrainz.org', 80): oobackoff; delay: 1000ms -> 1000ms; slow start; window size 1.000 -> 2.000
          D: 13:09:09,509 C:\Junk\Download\picard-4526ebb95a4a81f37af888703321cb23ae6a7799\picard\browser\browser._process_request:70: Browser integration request: b'\x16\x03\x01\x02\x00\x01\x00\x01\xfc\x03\x03\xbc\xdeK\xf6\xf7\x1a`\x0cdxw\x1a\x85;\x81\x0c\xfc \xbb,pd\xf3\xda\xd0\x88I,\x9dFo\xac \x16\xf4?5,F\xa6\xb2\x16\x8e\x05\x1f\xa4\xdf`lh\x88Co\xdfC6\x10\xe0h\xb1\xddz\xc0/\xdb\x00$\x13\x01\x13\x03\x13\x02\xc0+\xc0/\xcc\xa9\xcc\xa8\xc0,\xc00\xc0\n'
          D: 13:09:09,509 C:\Junk\Download\picard-4526ebb95a4a81f37af888703321cb23ae6a7799\picard\browser\browser._process_request:73: Browser integration request (recoded): b'\x16\x03\x01\x02\x00\x01\x00\x01\xc3\xbc\x03\x03\xc2\xbc\xc3\x9eK\xc3\xb6\xc3\xb7\x1a`\x0cdxw\x1a\xc2\x85;\xc2\x81\x0c\xc3\xbc \xc2\xbb,pd\xc3\xb3\xc3\x9a\xc3\x90\xc2\x88I,\xc2\x9dFo\xc2\xac \x16\xc3\xb4?5,F\xc2\xa6\xc2\xb2\x16\xc2\x8e\x05\x1f\xc2\xa4\xc3\x9f`lh\xc2\x88Co\xc3\x9fC6\x10\xc3\xa0h\xc2\xb1\xc3\x9dz\xc3\x80/\xc3\x9b\x00$\x13\x01\x13\x03\x13\x02\xc3\x80+\xc3\x80/\xc3\x8c\xc2\xa9\xc3\x8c\xc2\xa8\xc3\x80,\xc3\x800\xc3\x80\n'
          

          Hope that helps narrow things down a bit.

          Bob Swift added a comment - If I modify the _process_request() method in browser.py to: def _process_request(self): conn = self.sender() conn.write(b "HTTP/1.1 200 OK\r\nCache-Control: max-age=0\r\n\r\nNothing to see here." ) rawline = conn.readLine().data() log.debug( "Browser integration request: %r" , rawline) rawline = rawline.decode( 'iso-8859-1' ) rawline = rawline.encode( 'utf-8' ) log.debug( "Browser integration request (recoded): %r" , rawline) try : line = rawline.decode() except UnicodeDecodeError as e: log.error(e) return finally : conn.disconnectFromHost() there is no error generated, and the relevant part of the log is: D: 13:09:03,501 C:\Junk\Download\picard-4526ebb95a4a81f37af888703321cb23ae6a7799\picard\webservice\ratecontrol._out_of_backoff:225: ('musicbrainz.org', 80): oobackoff; delay: 1000ms -> 1000ms; slow start; window size 1.000 -> 2.000 D: 13:09:09,509 C:\Junk\Download\picard-4526ebb95a4a81f37af888703321cb23ae6a7799\picard\browser\browser._process_request:70: Browser integration request: b'\x16\x03\x01\x02\x00\x01\x00\x01\xfc\x03\x03\xbc\xdeK\xf6\xf7\x1a`\x0cdxw\x1a\x85;\x81\x0c\xfc \xbb,pd\xf3\xda\xd0\x88I,\x9dFo\xac \x16\xf4?5,F\xa6\xb2\x16\x8e\x05\x1f\xa4\xdf`lh\x88Co\xdfC6\x10\xe0h\xb1\xddz\xc0/\xdb\x00$\x13\x01\x13\x03\x13\x02\xc0+\xc0/\xcc\xa9\xcc\xa8\xc0,\xc00\xc0\n' D: 13:09:09,509 C:\Junk\Download\picard-4526ebb95a4a81f37af888703321cb23ae6a7799\picard\browser\browser._process_request:73: Browser integration request (recoded): b'\x16\x03\x01\x02\x00\x01\x00\x01\xc3\xbc\x03\x03\xc2\xbc\xc3\x9eK\xc3\xb6\xc3\xb7\x1a`\x0cdxw\x1a\xc2\x85;\xc2\x81\x0c\xc3\xbc \xc2\xbb,pd\xc3\xb3\xc3\x9a\xc3\x90\xc2\x88I,\xc2\x9dFo\xc2\xac \x16\xc3\xb4?5,F\xc2\xa6\xc2\xb2\x16\xc2\x8e\x05\x1f\xc2\xa4\xc3\x9f`lh\xc2\x88Co\xc3\x9fC6\x10\xc3\xa0h\xc2\xb1\xc3\x9dz\xc3\x80/\xc3\x9b\x00$\x13\x01\x13\x03\x13\x02\xc3\x80+\xc3\x80/\xc3\x8c\xc2\xa9\xc3\x8c\xc2\xa8\xc3\x80,\xc3\x800\xc3\x80\n' Hope that helps narrow things down a bit.

          Zas added a comment - - edited

          Wtf, is that b'\x16\x03\x01\x02\x00\x01\x00\x01\xfc\x03\x03\xa7\x7f\x96\xb7P\x97 \x89\xcd\xb3\x1cK\xf3>\x91\x9b\n' ???

          Which browser is sending this ?

          See what i have on Linux/Firefox:

          Browser integration request: b'GET /openalbum?id=ed6a3bb0-d787-4b64-880e-ee542691d3b0&t=1538497477 HTTP/1.1\r\n'

           

          Can you check what is sent by your browser (using internal debugger) ? If this is what is sent by the browser the problem is elsewhere. If it isn't ...

          Zas added a comment - - edited Wtf, is that b'\x16\x03\x01\x02\x00\x01\x00\x01\xfc\x03\x03\xa7\x7f\x96\xb7P\x97 \x89\xcd\xb3\x1cK\xf3>\x91\x9b\n' ??? Which browser is sending this ? See what i have on Linux/Firefox: Browser integration request: b'GET /openalbum?id=ed6a3bb0-d787-4b64-880e-ee542691d3b0&t=1538497477 HTTP/1.1\r\n'   Can you check what is sent by your browser (using internal debugger) ? If this is what is sent by the browser the problem is elsewhere. If it isn't ...

            zas Zas
            michelv Michel Vorenhout
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved:

                Version Package
                2.1