To reproduce:

• Have an album where only a single track has exactly one matched files, but there are also unmatched files
• Use "Search for similar tracks..." on the track with the file
• In the search result select an album different from the existing one

Result: The new album gets loaded, the file gets moved there and the old album gets removed
Expected result: The old album is kept, as it contains unmatched files