Uploaded image for project: 'Zapped: AcousticBrainz'
  1. Zapped: AcousticBrainz
  2. AB-439

Improve multithreaded code and speed up HL extractor

    • Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Normal Normal
    • None
    • None
    • None
    • None

      We're finding that the HL extractor is struggling to keep up with the number of submissions that we're receiving

      The HL extractor works well, but the amount of threading code needed to perform its job means that it's quite difficult to modify it if needed.

      We can make the code easier to maintain by replacing the custom threading code with concurrent.futures

      An additional improvement to speed up the extractor: The extractor binary can take multiple arguments, in order to process many files at a time. As the expensive part of the hl process is actually loading the models into place, we can speed up this process by just loading them once and processing a large number of files in a single process.

          Loading...
          Uploaded image for project: 'Zapped: AcousticBrainz'
          1. Zapped: AcousticBrainz
          2. AB-439

          Improve multithreaded code and speed up HL extractor

            • Icon: Improvement Improvement
            • Resolution: Fixed
            • Icon: Normal Normal
            • None
            • None
            • None
            • None

              We're finding that the HL extractor is struggling to keep up with the number of submissions that we're receiving

              The HL extractor works well, but the amount of threading code needed to perform its job means that it's quite difficult to modify it if needed.

              We can make the code easier to maintain by replacing the custom threading code with concurrent.futures

              An additional improvement to speed up the extractor: The extractor binary can take multiple arguments, in order to process many files at a time. As the expensive part of the hl process is actually loading the models into place, we can speed up this process by just loading them once and processing a large number of files in a single process.

                    Unassigned Unassigned
                    alastairp Alastair Porter
                    Votes:
                    0 Vote for this issue
                    Watchers:
                    1 Start watching this issue

                      Created:
                      Updated:

                        Version Package

                          Unassigned Unassigned
                          alastairp Alastair Porter
                          Votes:
                          0 Vote for this issue
                          Watchers:
                          1 Start watching this issue

                            Created:
                            Updated:

                              Version Package