Uploaded image for project: 'Picard'
  1. Picard
  2. PICARD-657

Gateway Timeout with large releases

    • Icon: Improvement Improvement
    • Resolution: Done
    • Icon: Normal Normal
    • None
    • 1.3
    • Other
    • None

      on big releases the server gives a timeout (see: http://tickets.musicbrainz.org/browse/MBS-3841 )
      it seems the current approach to this issue for the web-interface will be lazy loading with collapsed discs, so i guess that won't help here within picard.
      maybe it is possible to fetch the data in chunks for every disc of a release so picard won't run into this timeout?

          [PICARD-657] Gateway Timeout with large releases

          Closing this in favor of PICARD-2398. The timeout issues themselves have been long ago dealt with by the server side changes in MBS-10404.

          Philipp Wolfer added a comment - Closing this in favor of PICARD-2398 . The timeout issues themselves have been long ago dealt with by the server side changes in MBS-10404 .

          PICARD-2137 suggest to show the user a warning when this happens.

          One idea to handle this would also be to allow reloading relationships for a single track, e.g. via context menu. Or maybe in case of this warning it could be offered to the user to load the individual relationships per recording, with the notice that this might take a long time.

          Philipp Wolfer added a comment - PICARD-2137 suggest to show the user a warning when this happens. One idea to handle this would also be to allow reloading relationships for a single track, e.g. via context menu. Or maybe in case of this warning it could be offered to the user to load the individual relationships per recording, with the notice that this might take a long time.

          Sophist added a comment -

          My PR was not merged - perhaps because it resulted it sub-optimal results by retrying the request without track relation metadata.

          However, I guess it would be possible to do as the OP suggested and combine the above with a set of track-by-track requests for the missing data.

          Sophist added a comment - My PR was not merged - perhaps because it resulted it sub-optimal results by retrying the request without track relation metadata. However, I guess it would be possible to do as the OP suggested and combine the above with a set of track-by-track requests for the missing data.

          Sophist added a comment -

          PR-437 submitted as a partial solution to this issue.

          Sophist added a comment - PR-437 submitted as a partial solution to this issue.

          Sophist added a comment - - edited

          As Philipp says, this is primarily a server issue. However, a simple workaround we can code into Picard relatively easily is to handle 504 errors as follows:

          If we get a 504 AND we have track_relationships enabled, then reissue the web service call in the priority queue as if track relationships is not set.

          Ok - we will only get a subset of the data we want, but it is better than a "cannot load album" message and you will at least get the album name and track list loaded into Picard.

          If I get the chance in the next couple of weeks I will try to submit a Picard PR to try this.

          ------------------------------

          A longer term Picard-based solution will be difficult in ws/2 - the only complete solution is to do something similar to the web solution and break it into multiple simpler queries, but unlike the web interface where a user is typically only wanting details of a single release, Picard often loads many albums - so breaking it up into multiple smaller web-service calls to e.g. get each track relationship individually will result in an order of magnitude more (rate-limited) web service calls.

          However, if we are desperate, then I guess we can combine these two methods and only issue track-by-track web service calls to get track relationships IF we have received a 504 error. This is a more complicated, and for the limited number of releases with this problem perhaps not essential anyway - so once I have done the simpler workaround, I will leave this to others if there is a demand.

          P.S. It occurs to me that the releases where we get 504 Gateway Timeout errors are those with a lot (in the hundreds) of tracks - so switching to a track-by-track retrieval of track relationships will be slow - but if the user is prepared to wait, it will work.

          P.P.S. I can now see how to achieve this technically if the consensus is we should code it.

          Sophist added a comment - - edited As Philipp says, this is primarily a server issue. However, a simple workaround we can code into Picard relatively easily is to handle 504 errors as follows: If we get a 504 AND we have track_relationships enabled, then reissue the web service call in the priority queue as if track relationships is not set. Ok - we will only get a subset of the data we want, but it is better than a "cannot load album" message and you will at least get the album name and track list loaded into Picard. If I get the chance in the next couple of weeks I will try to submit a Picard PR to try this. ------------------------------ A longer term Picard-based solution will be difficult in ws/2 - the only complete solution is to do something similar to the web solution and break it into multiple simpler queries, but unlike the web interface where a user is typically only wanting details of a single release, Picard often loads many albums - so breaking it up into multiple smaller web-service calls to e.g. get each track relationship individually will result in an order of magnitude more (rate-limited) web service calls. However, if we are desperate, then I guess we can combine these two methods and only issue track-by-track web service calls to get track relationships IF we have received a 504 error. This is a more complicated, and for the limited number of releases with this problem perhaps not essential anyway - so once I have done the simpler workaround, I will leave this to others if there is a demand. P.S. It occurs to me that the releases where we get 504 Gateway Timeout errors are those with a lot (in the hundreds) of tracks - so switching to a track-by-track retrieval of track relationships will be slow - but if the user is prepared to wait, it will work. P.P.S. I can now see how to achieve this technically if the consensus is we should code it.

          I had made some tests some time ago but never commented here. In my experience the most trouble in loading times is caused by the "Use track relationships" option, which is heavy on the server. When this is disabled even large releases like this can be loaded. So disabling this option can currently be a workaround for affected releases, but of course has the downside that track level relationships will not be available.

          The proposed workaround of loading disc by disc is not doable with the current web service (you can only load releases or tracks, but not single discs). What could maybe work is in case of a timeout and "Use track relationships" enabled, redo the request without track relationships (thus automating the workaround above). In theory Picard could then load the data track by track, but this would a) be horrible slow and b) put additional load on the server.

          In the end this is mostly a server side issue and not solvable by Picard alone.

          See also my post on http://forums.musicbrainz.org/viewtopic.php?pid=30253#p30253

          Philipp Wolfer added a comment - I had made some tests some time ago but never commented here. In my experience the most trouble in loading times is caused by the "Use track relationships" option, which is heavy on the server. When this is disabled even large releases like this can be loaded. So disabling this option can currently be a workaround for affected releases, but of course has the downside that track level relationships will not be available. The proposed workaround of loading disc by disc is not doable with the current web service (you can only load releases or tracks, but not single discs). What could maybe work is in case of a timeout and "Use track relationships" enabled, redo the request without track relationships (thus automating the workaround above). In theory Picard could then load the data track by track, but this would a) be horrible slow and b) put additional load on the server. In the end this is mostly a server side issue and not solvable by Picard alone. See also my post on http://forums.musicbrainz.org/viewtopic.php?pid=30253#p30253

          Ryan Harper added a comment -

          Ryan Harper added a comment - This release also gives this same error: https://musicbrainz.org/release/cf74e971-b791-4c23-ab2d-2189e6f8195a

          ebz777 added a comment - An example of the timeout error messages that can be found in the picard log file for large releases: E: 22:40:32 Network request error for http://musicbrainz.org/ws/2/release/0e7cd180-3323-4ca2-a101-60f967d7d292?inc=release-groups+media+recordings+artist-credits+artists+aliases+labels+isrcs+collections+artist-rels+release-rels+url-rels+recording-rels+work-rels+recording-level-rels+work-level-rels+tags: Error downloading http://musicbrainz.org/ws/2/release/0e7cd180-3323-4ca2-a101-60f967d7d292?inc=release-groups+media+recordings+artist-credits+artists+aliases+labels+isrcs+collections+artist-rels+release-rels+url-rels+recording-rels+work-rels+recording-level-rels+work-level-rels+tags - server replied: Gateway Time-out (QT code 301, HTTP code 504) E: 22:40:32 Error downloading http://musicbrainz.org/ws/2/release/0e7cd180-3323-4ca2-a101-60f967d7d292?inc=release-groups+media+recordings+artist-credits+artists+aliases+labels+isrcs+collections+artist-rels+release-rels+url-rels+recording-rels+work-rels+recording-level-rels+work-level-rels+tags - server replied: Gateway Time-out

            sophist Sophist
            ehrgeiz lorenz pressler
            Votes:
            4 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved:

                Version Package