Uploaded image for project: 'ListenBrainz'
  1. ListenBrainz
  2. LB-247

bigquery-writer listen batching

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Normal Normal
    • gsoc release
    • data store
    • None

      The bigquery writer is currently reading one "block" of listens from RabbitMQ and then sending it to BigQuery. This is currently 50 listens at a time, so updating a backlog takes a really long time. 

      Ideally, the writer would read up to a max number of listens from RabbitMQ, submit a much larger batch, making things much faster. We'll need to take care that we can't possibly lose the listens until they are successfully submitted to BQ. I think this needs careful management of the RabbitMQ consumption acknowledgement.

            iliekcomputers Param Singh
            rob Robert Kaye
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved:

                Version Package
                gsoc release