Uploaded image for project: 'Picard'
  1. Picard
  2. PICARD-2427

Files not found with NFS share on Windows 10

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Normal Normal
    • None
    • 2.8.0rc1
    • File Move & Rename
    • None
    • Windows 10, NFS mounted linux filesystem Picard 2.8.0dev1

      While trying to load a new set of files into the unclustered files pane while using the filebrowser, the source file window shows errors for many file names and the error is file not found.

      I was willing to write this off to a round trip from windows to linux back to windows, but every other windows app that I tried eg. foobar2000, groove, windows notepad have no problem finding and opening the files.

      The file name looks like some attempt to escape certain characters in the file path, but it could just be a double substitution of the \ character.  I am not a windows expert when it comes to file naming.

      The files were ripped on a windows7 system running dbPoweramp.  Then the network shares on a windows10 system were used to copy the file to the normal working directory on the NFS mounted file system.  Then the attempt was made to import the files into Picard for tagging and clustering.

      No go.  I've attached the log and a screenshot showing the error condition.

      I don't remember this happening on 2.7.3.  I may also have the wrong component.

       

          [PICARD-2427] Files not found with NFS share on Windows 10

          Mark Huth added a comment -

          Thanks for the update.  I've worked around the problem for now, and will play with looking at the Windows long file name support.  Looking forward to an an update of 2.8

          Mark Huth added a comment - Thanks for the update.  I've worked around the problem for now, and will play with looking at the Windows long file name support.  Looking forward to an an update of 2.8

          Some good news: With the recent changes for PICARD-2076 and PICARD-1570 the long paths are properly working in Picard, including the renaming. Support for renaming / moving to long paths must be explicitly enabled in Picard's options, though. I confirmed that this also works with an NFS share. The changes will be part of Picard 2.8

          For your problem right now you might also consider enabling long path support in Windows by registry change, see https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=cmd#enable-long-paths-in-windows-10-version-1607-and-later for details. This might already improve things and might make you saving problem go away.

          Just be aware that even with this setting Picard is only partially supporting long paths without the other fixes (loading and saving should work, renaming not). Also Windows itself, especially Explorer, still has some quirks and does e.g. not allow you to rename or create such files / folders.

          Philipp Wolfer added a comment - Some good news: With the recent changes for PICARD-2076 and PICARD-1570 the long paths are properly working in Picard, including the renaming. Support for renaming / moving to long paths must be explicitly enabled in Picard's options, though. I confirmed that this also works with an NFS share. The changes will be part of Picard 2.8 For your problem right now you might also consider enabling long path support in Windows by registry change, see https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=cmd#enable-long-paths-in-windows-10-version-1607-and-later for details. This might already improve things and might make you saving problem go away. Just be aware that even with this setting Picard is only partially supporting long paths without the other fixes (loading and saving should work, renaming not). Also Windows itself, especially Explorer, still has some quirks and does e.g. not allow you to rename or create such files / folders.

          I started looking into this again and will provide support for Picard to be able to load such long paths in PICARD-1570. Full support for long paths without shortening when renaming will be done as part of PICARD-2076, but this might come later.

          Philipp Wolfer added a comment - I started looking into this again and will provide support for Picard to be able to load such long paths in PICARD-1570 . Full support for long paths without shortening when renaming will be done as part of PICARD-2076 , but this might come later.

          Didn't know Windows does support NFS out of the box, but now I tried this on my test VM.

          As far as I could test the issues are not related to non-ASCII characters, some of the paths affected by this in your logs do not have any special character, and I had no trouble on the Windows side of loading sand saving files with umlauts. Only issue seems to be that Windows' NFS implementation does not support Unicode encoding, and NFS passes the file names through as given so they end up with Windows encoding on the Linux side.

          Anyway, the issue at hand seems to be the long path issue as discussed in PICARD-1570 and PICARD-2076. I actually think this is a direct duplicate of PICARD-2076. Paths below the path length limit work fine for me.

          Philipp Wolfer added a comment - Didn't know Windows does support NFS out of the box, but now I tried this on my test VM. As far as I could test the issues are not related to non-ASCII characters, some of the paths affected by this in your logs do not have any special character, and I had no trouble on the Windows side of loading sand saving files with umlauts. Only issue seems to be that Windows' NFS implementation does not support Unicode encoding, and NFS passes the file names through as given so they end up with Windows encoding on the Linux side. Anyway, the issue at hand seems to be the long path issue as discussed in PICARD-1570 and PICARD-2076 . I actually think this is a direct duplicate of PICARD-2076 . Paths below the path length limit work fine for me.

          Mark Huth added a comment -

          It really is an NFS mount on Windows.  The servers are all linux and are not running Samba.

          The files names in question all appear correctly in the Windows file browser, and double-click sends them to the media player (Foobar2000 in this case).  Additionally, I can open the file in Notepad to take a look at the raw file with the embedded tags.  Running into the path length problem has previously just resulted in an attempt to access a truncated file path, which shows in the log and on the bottom status bar.  Because if it, I have generally managed to shorten the elements in the path, so the actual path length is less than the Windows10 limit.  The filenames show up correctly in the browser pane in Picard and I can drag them into the unclustered pane.  The problem shows up after clustering or after scanning when trying to save updates, with or without rename active.

          As I've played around with this problem a bit, I have come to suspect that the paths that fail contain certain foreign language characters.  Checking the "Replace non-ASCII characters" option in the file naming option panel seems to create filenames that no longer have the problem once I have used the DragnDrop to fix up the original error.

          Linux just stores the foreign characters with an escape code (\$xyz) in the file path/filename.  I would prefer to leave the foreign characters (diacritical marks, German umlaut and essette) in the mix, as many Linux apps correctly display the characters, as do most Windows apps, but I can live with removing them.  If I see anything more that forms a pattern I'll let you know.

          I have seen some problems with Windows graphic characters in the filenames.  For example, some classical works have a tempo indication.  Someone managed to get cute with the data on Brainz, finding a character sequence that looks like a quarter note.  Saving that file on Linux resulted in ????= ???? embedded in the linux path.  Linux is happy with that, since only Null and / are prohibited, but Windows can't deal with it.  I can't even manage to rename it on Windows using the file browser, since there are two separate ???? strings, and changing the first and attempting to move to the second causes an illegal name error.  I've had to fix it on linux before I could work with the file again.

          It's possible that the Python filesystem APIs and the normal Windows APIs don't work quite the same way.

          Mark Huth added a comment - It really is an NFS mount on Windows.  The servers are all linux and are not running Samba. The files names in question all appear correctly in the Windows file browser, and double-click sends them to the media player (Foobar2000 in this case).  Additionally, I can open the file in Notepad to take a look at the raw file with the embedded tags.  Running into the path length problem has previously just resulted in an attempt to access a truncated file path, which shows in the log and on the bottom status bar.  Because if it, I have generally managed to shorten the elements in the path, so the actual path length is less than the Windows10 limit.  The filenames show up correctly in the browser pane in Picard and I can drag them into the unclustered pane.  The problem shows up after clustering or after scanning when trying to save updates, with or without rename active. As I've played around with this problem a bit, I have come to suspect that the paths that fail contain certain foreign language characters.  Checking the "Replace non-ASCII characters" option in the file naming option panel seems to create filenames that no longer have the problem once I have used the DragnDrop to fix up the original error. Linux just stores the foreign characters with an escape code (\$xyz) in the file path/filename.  I would prefer to leave the foreign characters (diacritical marks, German umlaut and essette) in the mix, as many Linux apps correctly display the characters, as do most Windows apps, but I can live with removing them.  If I see anything more that forms a pattern I'll let you know. I have seen some problems with Windows graphic characters in the filenames.  For example, some classical works have a tempo indication.  Someone managed to get cute with the data on Brainz, finding a character sequence that looks like a quarter note.  Saving that file on Linux resulted in ????= ???? embedded in the linux path.  Linux is happy with that, since only Null and / are prohibited, but Windows can't deal with it.  I can't even manage to rename it on Windows using the file browser, since there are two separate ???? strings, and changing the first and attempting to move to the second causes an illegal name error.  I've had to fix it on linux before I could work with the file again. It's possible that the Python filesystem APIs and the normal Windows APIs don't work quite the same way.

          Don't get confused by the double backslashes in the log output, that's just Python's string escaping for the backslash, because the backslash itself is an escape character. Every two backslashes stands for a single backslash.

          Can you give some more details how the files are stored and accessed. You write something about NFS, but I assume you don't actually mean the NFS (https://de.wikipedia.org/wiki/Network_File_System), but probably a normal Windows share (SMB / CIFS)? Where is this share located, on another Windows system (which version) or on a Linux system using SAMBA? How do you open the files in Picard (using the drag and drop from Explorer, using the Add files dialog, or the file browser)?

          If I understand you correctly this happens only with some files, not with all. So if you e.g. open the following file path in Explorer, does Windows open the file with the default application?

          \\192.168.1.63\\mnt\\raid4\\music\\downloads
          Leonard Bernstein Edition - Concertos and Orchestral Works
          Also Sprach Zarathustra-Don Juan-Till Eulenspiegel-Leonard Bernstein
          01_Also sprach Zarathustra (Thus Spoke Zoroaster), tone poem for orchestra, Op. 30 (TrV 176)- Einleitung.flac

          It might indeed be related to the long file path, which exceeds the standard Windows file path limit, in which case this would be a duplicate of PICARD-1570 . In that case opening the file via the file dialog might behave different from drag and drop.

          Philipp Wolfer added a comment - Don't get confused by the double backslashes in the log output, that's just Python's string escaping for the backslash, because the backslash itself is an escape character. Every two backslashes stands for a single backslash. Can you give some more details how the files are stored and accessed. You write something about NFS, but I assume you don't actually mean the NFS ( https://de.wikipedia.org/wiki/Network_File_System ), but probably a normal Windows share (SMB / CIFS)? Where is this share located, on another Windows system (which version) or on a Linux system using SAMBA? How do you open the files in Picard (using the drag and drop from Explorer, using the Add files dialog, or the file browser)? If I understand you correctly this happens only with some files, not with all. So if you e.g. open the following file path in Explorer, does Windows open the file with the default application? \\192.168.1.63\\mnt\\raid4\\music\\downloads Leonard Bernstein Edition - Concertos and Orchestral Works Also Sprach Zarathustra-Don Juan-Till Eulenspiegel-Leonard Bernstein 01_Also sprach Zarathustra (Thus Spoke Zoroaster), tone poem for orchestra, Op. 30 (TrV 176)- Einleitung.flac It might indeed be related to the long file path, which exceeds the standard Windows file path limit, in which case this would be a duplicate of PICARD-1570 . In that case opening the file via the file dialog might behave different from drag and drop.

          Mark Huth added a comment - - edited

          In some cases, I have been able to manually pull the file into the cluster.  It looks like the extra
          backslashes in the program generated file name may exceed the file name length limit.  I'm not sure about the however.

          Mark Huth added a comment - - edited In some cases, I have been able to manually pull the file into the cluster.  It looks like the extra backslashes in the program generated file name may exceed the file name length limit.  I'm not sure about the however.

            Unassigned Unassigned
            mhuth1776-1 Mark Huth
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved:

                Version Package