Uploaded image for project: 'Zapped: AcousticBrainz'
  1. Zapped: AcousticBrainz
  2. AB-466

similarity: build annoy index files in parallel

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • None
    • None
    • None

      With 24 million items, building a similarity index takes about 16 minutes to complete, using 100% CPU. If we performed this process in threads for each index type we could reduce the total time needed to rebuild all indices.

          Loading...
          Uploaded image for project: 'Zapped: AcousticBrainz'
          1. Zapped: AcousticBrainz
          2. AB-466

          similarity: build annoy index files in parallel

            • Icon: Improvement Improvement
            • Resolution: Unresolved
            • Icon: Normal Normal
            • None
            • None
            • None
            • None

              With 24 million items, building a similarity index takes about 16 minutes to complete, using 100% CPU. If we performed this process in threads for each index type we could reduce the total time needed to rebuild all indices.

                    Unassigned Unassigned
                    alastairp Alastair Porter
                    Votes:
                    0 Vote for this issue
                    Watchers:
                    1 Start watching this issue

                      Created:
                      Updated:

                        Version Package

                          Unassigned Unassigned
                          alastairp Alastair Porter
                          Votes:
                          0 Vote for this issue
                          Watchers:
                          1 Start watching this issue

                            Created:
                            Updated:

                              Version Package