• Type: Bug
• Resolution: Fixed
• Priority: High
• Fix Version/s: 2.1
• Affects Version/s: 2.0.4
• Component/s: User Interface
• Labels:

  None

[PICARD-1339] Removing unclustered files can be very slow

Collapse comment: Philipp Wolfer added a comment - 2018-09-10 11:39

Philipp Wolfer added a comment - 2018-09-10 11:39

The committed fix has side effects. The issue is that remove_file() is not only used for actually removing a file completely, but also when moving files between clusters and/or tracks. In these cases the parent cluster/track currently is not updated anymore.

But actually the same performance issues apply there: Moving a huge amount of files between one cluster/track and another causes the same unnecesary update_metadata_images calls. So we should fix these cases as well properly.

Expand comment: Philipp Wolfer added a comment - 2018-09-10 11:39

Philipp Wolfer added a comment - 2018-09-10 11:39 The committed fix has side effects. The issue is that remove_file() is not only used for actually removing a file completely, but also when moving files between clusters and/or tracks. In these cases the parent cluster/track currently is not updated anymore. But actually the same performance issues apply there: Moving a huge amount of files between one cluster/track and another causes the same unnecesary update_metadata_images calls. So we should fix these cases as well properly.

Collapse comment: Zas added a comment - 2018-09-07 13:18

Zas added a comment - 2018-09-07 13:18

After investigation it appears the issue is even worse, as it concerns any removal or move action (cluster to cluster), removing 11 files leads to 88 calls to update_metadata_images ...

Expand comment: Zas added a comment - 2018-09-07 13:18

Zas added a comment - 2018-09-07 13:18 After investigation it appears the issue is even worse, as it concerns any removal or move action (cluster to cluster), removing 11 files leads to 88 calls to update_metadata_images ...