Some of the changes meant for PICARD-1843 ended up postponed due to some rework.
As a result, clustering of my 19k file library took ~10m with clusters collapsed, compared to the original ~5h (a 30x speedup).

By offloading clustering to a worker thread and moving files in batches per cluster, clustering time of the same library is further reduced to ~3m with clusters collapsed and ~5m with clusters expanded.

That translates into a 60x speedup over the original with expanded clusters, and makes it practically irrelevant to keep clusters expanded while clustering.

On the loading side, freezing the window during album loading finalization speed things up. In the same library, loading times fell from 10m down to 7m (avg. 32 songs/sec->45 songs/sec).

Update:

Additional performance gains by setting a size hint that wasted 35% of CPU cycles during load trying to guess the row height to fit the icons. More details on: https://github.com/metabrainz/picard/pull/1555#issuecomment-644475419

Loading times fell from 7m to 5m30s (50m on 2.3.2, resulting in a 9x speedup) while clustering times fell from 5min to 2min (5h17m on 2.3.2, resulting in a 144x speedup).