In the high level extractor we process one lowlevel recording at a time with the essentia extractor binary. This binary takes a long time to load all of the model files, but a comparatively short time to actually calculate the highlevel file.

We added a functionality to the highlevel extractor to be able to process more than one file at a time, and started work on adding this support to the AB extractor tool (https://github.com/metabrainz/acousticbrainz-server/pull/88), but it had some bugs, and is designed for the old database table structure. We should try again. This could help us to process new dataset models faster (AB-399, AB-400)

I would like to switch this tool from using a manually created thread pool/threading to something with a bit more testing behind it, like concurrent.futures: https://docs.python.org/3/library/concurrent.futures.html (there is a python 2 backport at https://pypi.org/project/futures/)