Uploaded image for project: 'Picard'
  1. Picard
  2. PICARD-1963

Possible stack overflow when loading files

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: High High
    • 2.5.0b1
    • 2.4.4
    • Other
    • None

      Under certain circumstances Picard can crash with a stack overflow when loading a larger amount of files.

      I can only reliable reproduce this on one system (KDE Neon, Python 3.6.9, Qt 5.14.2, PyQt 5.14.1) with a certain set of files, but in theory it can happen everywhere.

      The basic recursion happening is this:

      1. After a file's load method has finished the callback tagger._file_loaded is set as a callback. This callback is not called immediately, but is put on the event queue via a ProxyToMainEvent by the picard.util.thread module.
      2. Under certain circumstances many of those ProxyToMainEvent(tagger._file_loaded) events can pile up
      3. When invoked tagger._file_loaded calls tagger.move_files, which in return calls QCoreApplication.processEvents
      4. QCoreApplication.processEvents processes the next event in the queue by calling tagger.even. If this again is a ProxyToMainEvent(tagger._file_loaded) this again calls tagger.move_files and QCoreApplication.processEvents. We get a recursion

      With proper timing and enough files to be loaded this can get into a situation where the maximum stack depth gets used up.

      File "./picard/tagger.py", line 402 in event
      File "./picard/util/thread.py", line 47 in run
      File "./picard/file.py", line 218 in _loading_finished
      File "./picard/tagger.py", line 456 in _file_loaded
      File "./picard/tagger.py", line 473 in move_files
      File "./picard/tagger.py", line 402 in event
      File "./picard/util/thread.py", line 47 in run
      File "./picard/file.py", line 218 in _loading_finished
      File "./picard/tagger.py", line 456 in _file_loaded
      File "./picard/tagger.py", line 473 in move_files
      File "./picard/tagger.py", line 402 in event
      File "./picard/util/thread.py", line 47 in run
      File "./picard/file.py", line 218 in _loading_finished
      ...

      See crash.log for full output.

          [PICARD-1963] Possible stack overflow when loading files

          Graham Gibby added a comment - - edited

          I think I'm experiencing this as well, since adding >51 files
          causes Picard to immediately close.

          Having issues with MediaMonkey Batch Discogs tagger with the same set of files... hrm.

          Version 2.4.4 on Windows 10

          Graham Gibby added a comment - - edited I think I'm experiencing this as well, since adding >51 files causes Picard to immediately close. Having issues with MediaMonkey Batch Discogs tagger with the same set of files... hrm. Version 2.4.4 on Windows 10

          Sophist added a comment -

          That sounds like a good improvement - and which might pave the way for multi-processing.

          Sophist added a comment - That sounds like a good improvement - and which might pave the way for multi-processing.

          I know, my brain hurts today . Actually my first instinct was that we should replace the way we communicate back to the main thread, but in the end went with the simpler solution.

          As a side effect I rewrote picard.util.thread to use concurrent.futures, which is IMHO much cleaner than QThreads. Not sure if this would have some unexpected side effects, but I'll probably submit this for Picard 2.6

          Philipp Wolfer added a comment - I know, my brain hurts today  . Actually my first instinct was that we should replace the way we communicate back to the main thread, but in the end went with the simpler solution. As a side effect I rewrote picard.util.thread to use concurrent.futures, which is IMHO much cleaner than QThreads. Not sure if this would have some unexpected side effects, but I'll probably submit this for Picard 2.6

          Sophist added a comment -

          I will have to think about this - trying to work out what happens with multiple threads in all circumstances with multiple types of call-back code and use of processEvents() is a non-trivial activity which makes my brain hurt.

          Sophist added a comment - I will have to think about this - trying to work out what happens with multiple threads in all circumstances with multiple types of call-back code and use of processEvents() is a non-trivial activity which makes my brain hurt.

          Philipp Wolfer added a comment - - edited

          AFAIK a signal wouldn't be much different from sending a QEvent. It's basically the same mechanism. You'd need some loop to check for incoming signals. For the current implementation with QEvent this is done by Qt's main loop.

          The main difference if we would use separate signals for the thread callbacks would be that it would not mix up with other Qt events, so we can e.g. run processEvents() in a long running code block to allow handling some UI events without also triggering the thread callbacks. But then eventually we find that it would be great to run the callbacks during some long processing as well, because usually those callbacks are about UI updates (otherwise the code should be part of the thread itself) and having the UI update once in a while feels good. So we add a tagger.process_signals() method and we are back were we started

          Or what did I miss?

          Philipp Wolfer added a comment - - edited AFAIK a signal wouldn't be much different from sending a QEvent. It's basically the same mechanism. You'd need some loop to check for incoming signals. For the current implementation with QEvent this is done by Qt's main loop. The main difference if we would use separate signals for the thread callbacks would be that it would not mix up with other Qt events, so we can e.g. run processEvents() in a long running code block to allow handling some UI events without also triggering the thread callbacks. But then eventually we find that it would be great to run the callbacks during some long processing as well, because usually those callbacks are about UI updates (otherwise the code should be part of the thread itself) and having the UI update once in a while feels good. So we add a tagger.process_signals() method and we are back were we started Or what did I miss?

          Sophist added a comment -

          This is not directly related to this issue, but IMO the handoff between worker threads and main thread for callbacks should be handled using signals or queues as this is cleaner and less likely to result in excessive recursion and stack overflows.

          Sophist added a comment - This is not directly related to this issue, but IMO the handoff between worker threads and main thread for callbacks should be handled using signals or queues as this is cleaner and less likely to result in excessive recursion and stack overflows.

          GitHub Bot added a comment -

          See code changes in pull request #1653 submitted by phw.

          GitHub Bot added a comment - See code changes in pull request #1653 submitted by phw .

            outsidecontext Philipp Wolfer
            outsidecontext Philipp Wolfer
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:

                Version Package
                2.5.0b1