Uploaded image for project: 'ListenBrainz'
  1. ListenBrainz
  2. LB-141

Influx and bigquery writers may discard good data

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Fixed
    • Icon: Normal Normal
    • None
    • data store
    • None

      As we saw in LB-140 we may find errors in blocks of 1000 listens that we currently discard wholesale. That means that if one listen is rejected, we lose 999 good listens, which is unacceptable.

      What we need to do next is be more intelligent about error handling. If a block fails to submit to either BQ or influx, then we need to bisect the block and submit the good half and continue bisecting the bad block until we can submit all the good listens. 

      I wonder if we could write some sort of common class that can be used between the two writers. I think recursion would be ok to use, since even a block size of 100,000 would max give 17 levels of recursion....

            iliekcomputers Param Singh
            rob Robert Kaye
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved:

                Version Package