Uploaded image for project: 'Picard'
  1. Picard
  2. PICARD-2836

fpcalc results ignored even if fingerprint is generated

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Normal Normal
    • None
    • 2.8.5
    • None
    • Debian bookworm amd64

      I've noticed that fingerprint calculation seems to be failing (or at least not used) fairly often due to an fpcalc problem that doesn't seem to be a real problem.

      I'm often getting the following error (with rc = 3) from fpcalc (so far mainly noticed on mp3 files):

      ERROR: Error decoding audio frame (End of file)

      There seems to be no problem at all with the mp3 encoding of these files, as mp3val doesn't see any problems with them.

      I think this might be a bug in the underlying ffmpeg libraries that fpcalc seems to be using?

      In any case, fingerprinting is often failing even though there is nothing wrong with the source files, and a fingerprint is actually being generated.  Picard is just ignoring the fingerprint (presumably) because fpcalc is returning an error when there isn't actually anything wrong with the source file.

          [PICARD-2836] fpcalc results ignored even if fingerprint is generated

          Decoding error due to end of file could very well also indicate a truncated file which would produce a different (partial) fingerprint. We don't know, and I don't see a way to tell. Of course truncated file can happen otherwise just as well without an error indicating it. But here we know something was not fully right, without a way to know if it had an impact.

          Anyway, this is pretty much an issue to deal with in fpcalc. Thanks for creating the ticket there.

          Philipp Wolfer added a comment - Decoding error due to end of file could very well also indicate a truncated file which would produce a different (partial) fingerprint. We don't know, and I don't see a way to tell. Of course truncated file can happen otherwise just as well without an error indicating it. But here we know something was not fully right, without a way to know if it had an impact. Anyway, this is pretty much an issue to deal with in fpcalc. Thanks for creating the ticket there.

          Robo Tardis added a comment -

          My point is that the other ticket is about a whole class of decoding errors, and this ticket is about a specific error (end of file/stream) that should not be considered an error, so not exactly a duplicate.

          The end of an audio stream should not be considered an error.  While there is a chance that a partial frame may be left, any decoder should process as much as possible, and will skip any remainder.

          In my particular cases, there's not even a partial frame that is causing the problem.  In either case, discarding or restricting the use of the fingerprint seems like the wrong decision.  I agree with you totally if there are decoding errors within the stream (that is, where additional audio data follows the decoding error).

          One correction, fpcalc (at least within Debian) has been patched to support ffmpeg 5.  But I agree that support for ffmpeg 6 would be good.

          As far as reporting elsewhere, I have done so:

          AcoustID/Chromaprint: https://github.com/acoustid/chromaprint/issues/137

          I've also reported it in the Debian bug tracker against the libchromaprint1, libchromaprint-tools, and ffmpeg packages.

           

          Robo Tardis added a comment - My point is that the other ticket is about a whole class of decoding errors, and this ticket is about a specific error (end of file/stream) that should not be considered an error, so not exactly a duplicate. The end of an audio stream should not be considered an error.  While there is a chance that a partial frame may be left, any decoder should process as much as possible, and will skip any remainder. In my particular cases, there's not even a partial frame that is causing the problem.  In either case, discarding or restricting the use of the fingerprint seems like the wrong decision.  I agree with you totally if there are decoding errors within the stream (that is, where additional audio data follows the decoding error). One correction, fpcalc (at least within Debian) has been patched to support ffmpeg 5.  But I agree that support for ffmpeg 6 would be good. As far as reporting elsewhere, I have done so: AcoustID/Chromaprint: https://github.com/acoustid/chromaprint/issues/137 I've also reported it in the Debian bug tracker against the libchromaprint1, libchromaprint-tools, and ffmpeg packages.  

          It is a duplicate as the other ticket is exactly about this class of problems where fpcalc continues processing despite of ffmpeg errors and returns with status code 3. The submission of fingerprint was disabled as a precaution, as it is unclear whether such errors have an impact on the fingerprint or not and if so to what degree. Given the popularity of Picard and the fact that it did not even use the fingerprint of such files before I'd rather  not risk turning Picard into a source of wrong submissions.

          I don't see how we can decide whether a specific case of errors decoding an audio frame have an impact or not. And given the error actually originates from ffmpeg, which is a quickly changing library, there is potentially and endless amount of variations here.

          If you think this is actually an issue of how fpcalc is using ffmpeg this then should be better reported there. You could also check whether such decoding issues show up when using ffmpeg directly. If this happens with a current ffmpeg version it would be good to report it there. But it's not unlikely this particular issue has already been fixed. Current fpcalc is built against ffmpeg 4. It would be desirable that fpcalc gains support for ffmpeg 6.

          Philipp Wolfer added a comment - It is a duplicate as the other ticket is exactly about this class of problems where fpcalc continues processing despite of ffmpeg errors and returns with status code 3. The submission of fingerprint was disabled as a precaution, as it is unclear whether such errors have an impact on the fingerprint or not and if so to what degree. Given the popularity of Picard and the fact that it did not even use the fingerprint of such files before I'd rather  not risk turning Picard into a source of wrong submissions. I don't see how we can decide whether a specific case of errors decoding an audio frame have an impact or not. And given the error actually originates from ffmpeg, which is a quickly changing library, there is potentially and endless amount of variations here. If you think this is actually an issue of how fpcalc is using ffmpeg this then should be better reported there. You could also check whether such decoding issues show up when using ffmpeg directly. If this happens with a current ffmpeg version it would be good to report it there. But it's not unlikely this particular issue has already been fixed. Current fpcalc is built against ffmpeg 4. It would be desirable that fpcalc gains support for ffmpeg 6.

          Robo Tardis added a comment -

          I've created a workaround for this problem by creating and configuring the attached python wrapper for fpcalc within picard:

          #!/usr/bin/python3

          from subprocess import run
          import sys

          sys.argv[0] = '/usr/bin/fpcalc'
          p = run(sys.argv, capture_output=True)

          force_success = False
          if (p.returncode == 3 and p.stderr.decode().strip() == 'ERROR: Error decoding audio frame (End of file)'):
              force_success = True
          else:
              sys.stderr.write(p.stderr.decode())

          sys.stdout.write(p.stdout.decode())

          if force_success:
              exit(0)
          else:
              exit(p.returncode)

          Robo Tardis added a comment - I've created a workaround for this problem by creating and configuring the attached python wrapper for fpcalc within picard: #!/usr/bin/python3 from subprocess import run import sys sys.argv [0] = '/usr/bin/fpcalc' p = run(sys.argv, capture_output=True) force_success = False if (p.returncode == 3 and p.stderr.decode().strip() == 'ERROR: Error decoding audio frame (End of file)'):     force_success = True else:     sys.stderr.write(p.stderr.decode()) sys.stdout.write(p.stdout.decode()) if force_success:     exit(0) else:     exit(p.returncode)

          Robo Tardis added a comment - - edited

          This seems to stem from some sort of problem in the use of

          fpcalc -> libchromaprint1 -> FFmpegAudioReader::Read() -> avcodec_receive_frame()

          Robo Tardis added a comment - - edited This seems to stem from some sort of problem in the use of fpcalc -> libchromaprint1 -> FFmpegAudioReader::Read() -> avcodec_receive_frame()

          Robo Tardis added a comment -

          Also, has this problem been traced through and reported upstream to ffmpeg?

          Robo Tardis added a comment - Also, has this problem been traced through and reported upstream to ffmpeg?

          Robo Tardis added a comment -

          This isn't exactly a duplicate of that ticket, since in this particular case (the specific error seen) the problem isn't a real problem and submission should still be allowed.

          The fix in the other ticket prevents fingerprints from being submitted, which in the case of this specific error (message, not all cases of rc = 3), should still be allowed.

          Robo Tardis added a comment - This isn't exactly a duplicate of that ticket, since in this particular case (the specific error seen) the problem isn't a real problem and submission should still be allowed. The fix in the other ticket prevents fingerprints from being submitted, which in the case of this specific error (message, not all cases of rc = 3), should still be allowed.

          Thanks for your report. The observation was correct. There are certain non critical decoding problems in ffmpeg that get reported but don't prevent fpcalc from calculating a fingerprint.

          But this has just been fixed in the latest Picard 2.11 release. See PICARD-2813

          Philipp Wolfer added a comment - Thanks for your report. The observation was correct. There are certain non critical decoding problems in ffmpeg that get reported but don't prevent fpcalc from calculating a fingerprint. But this has just been fixed in the latest Picard 2.11 release. See PICARD-2813

            Unassigned Unassigned
            RoboTardis Robo Tardis
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved:

                Version Package