• Type: Improvement
• Resolution: Unresolved
• Priority: Normal
• Fix Version/s: None
• Affects Version/s: None
• Component/s: None
• Labels:

  None

[AB-314] Don't allow more than X recordings in a data dump file

Collapse comment: Param Singh added a comment - 2017-11-14 17:42

Param Singh added a comment - 2017-11-14 17:42

First cut for lowlevel json dumps: https://github.com/metabrainz/acousticbrainz-server/pull/241

Expand comment: Param Singh added a comment - 2017-11-14 17:42

Param Singh added a comment - 2017-11-14 17:42 First cut for lowlevel json dumps: https://github.com/metabrainz/acousticbrainz-server/pull/241

Collapse comment: Alastair Porter added a comment - 2017-10-24 18:52

Alastair Porter added a comment - 2017-10-24 18:52

I'd like to run some tests to see how many files we should put in each archive. We currently have 6 million submissions, so a first dump will have 6m / x parts - does it matter if we put 100k files per archive and end up with 120 dump files (60 ll, 60 hl) or should we try and minimise this?

Also test the file size of a 200k, 500k, 1m archive. Our original dump was 30GB. For archival purposes it probably doesn't matter if we're around this size, but smaller files are less unwieldy for people to download.

Check how many submissions we have on average per month. If for example we have ~110k, I'd prefer to put the limit higher so that we don't always create 100k + 10k files each month.

Some variation of this code could probably also be used for AB-97, if we have the ability to dump X files and not write a record to the incremental dumps table then we can use this dump function to make a sample archive.

Expand comment: Alastair Porter added a comment - 2017-10-24 18:52

Alastair Porter added a comment - 2017-10-24 18:52 I'd like to run some tests to see how many files we should put in each archive. We currently have 6 million submissions, so a first dump will have 6m / x parts - does it matter if we put 100k files per archive and end up with 120 dump files (60 ll, 60 hl) or should we try and minimise this? Also test the file size of a 200k, 500k, 1m archive. Our original dump was 30GB. For archival purposes it probably doesn't matter if we're around this size, but smaller files are less unwieldy for people to download. Check how many submissions we have on average per month. If for example we have ~110k, I'd prefer to put the limit higher so that we don't always create 100k + 10k files each month. Some variation of this code could probably also be used for AB-97 , if we have the ability to dump X files and not write a record to the incremental dumps table then we can use this dump function to make a sample archive.