Uploaded image for project: 'ListenBrainz'
  1. ListenBrainz
  2. LB-682

Improve the process for importing listens into HDFS

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Fixed
    • Icon: Normal Normal
    • None
    • listenbrainz-labs
    • None

      We should update the way that we import listens into spark. Some places for improvement are -

      • Use a context manager for creating temporary directories locally, this way we don't have to worry about deleting the directories if an error occurs or after the import is done
      • We don't create the '/temp' directory in HDFS explicitly, which might be confusing for readers who are new to the code.
      • We should also use different variable names for directories inside HDFS and locally in order to reduce confusion.

          [LB-682] Improve the process for importing listens into HDFS

          amCap1712 added a comment -

          This was fixed at the time of reworking spark dumps to use parquet format.

          amCap1712 added a comment - This was fixed at the time of reworking spark dumps to use parquet format.

            kartik1712 amCap1712
            ishaanshah Ishaan Shah
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved:

                Version Package