Uploaded image for project: 'Picard'
  1. Picard
  2. PICARD-294

Picard can write files on NTFS partition to "alternate data streams"

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 1.0
    • File Move & Rename
    • None

      Imported from http://bugs.musicbrainz.org/ticket/5790

      Original ticket said:

      I'm very upset with this software. I think it has just deleted my hole song library with no warning. I'm using ubuntu 10.04 and installed picard from the repository. What I've done:

      1. Opened picard 2. Pressed "add folder" and selected my musics folder. They were in a NTFS partition. 3. Pressed "Lookup" it scanned for a long time and found tags for most songs, filling the right column. 4. Checked "Options->Rename files" and "Options->Move files" 5. Pressed "Save"

      All my files have been moved to nowhere. The original folder was missing. Then I took a look at "Options->Options->Move files" and there was just a dot "." in the directory box.

      BrianFreud replied:

      Given that it's NTFS, and depending on the naming string used, it's also possible that the files were written to an alternate stream attached to the files' directory. (See http://manpages.ubuntu.com/manpages/jaunty/man8/ntfs-3g.8.html ; the NTFS-3G driver does support reading and writing to streams). I know I did accidentally somehow get Picard to write to an alternate stream once, back when I was still using Windows, though I don't recall exactly what I'd done to the naming string that managed to do it.

      ...

      voiceinsideyou did a lot more investigating and found:

      There is the sysinternals "streams" utility, for Windows. [0]

      I believe in order to have got it to write to an ADS, you would have had to somehow make Picard write files using a file naming string that had a colon in it. I believe opening and writing to a file handle with "filename:adsname" is enough to make Picard do this.[1] I've tested this myself on an NTFS volume and it works through black magic.

      So my guess is that when doing this on *Nix, if you're writing to an NTFS volume, perhaps it's very easy to break things if you have "Replace Windows-incompatible characters" disabled. Possibly Python and Picard just blindly tries to write files with : in them which could make things get hidden in ADSes.

      Still not quite sure how you could get it to move an entire collection hidden in an ADS though; without a somewhat unlucky file naming string. Perhaps some time later I'll get VirtualBox/VMWare going and see if I can replicate something like this in Picard itself.

      [0] http://technet.microsoft.com/en-us/sysinternals/bb897440 [1] http://robindavid.comli.com/?p=446

      ...

      Indeed I was able to replicate this on Ubuntu using the ntfs-3g driver with streams_interface=windows (which is not the default). Witha file renaming string of ":" and move to ".", replace windows incompatible chars turned off.

      Before:

      voiceinsideyou@ubuntu:/media/ntfs-3g/4 Corners/The Foundations$ ls -al
      total 17332
      drwxrwxrwx 1 root root    4096 2011-08-14 18:51 .
      drwxrwxrwx 1 root root       0 2010-02-13 13:24 ..
      -rwxrwxrwx 1 root root 4941952 2011-08-09 21:16 06-4 Corners-On the Downlow.mp3
      -rwxrwxrwx 1 root root 4916277 2011-08-09 21:16 07-4 Corners-Push It Up.mp3
      -rwxrwxrwx 1 root root 3582080 2011-08-09 21:16 08-4 Corners-Walk Right.mp3
      -rwxrwxrwx 1 root root 4292736 2011-08-09 21:16 09-4 Corners-Strife.mp3
      

      Picard renames

      D: 3077994176 18:51:57 Updating file <File #111 u'06-4 Corners-On the Downlow.mp3'>
      D: 3077994176 18:51:57 Saving file u'/media/ntfs-3g/4 Corners/The Foundations/06-4 Corners-On the Downlow.mp3'
      D: 3077994176 18:51:57 Updating file <File #112 u'07-4 Corners-Push It Up.mp3'>
      D: 3077994176 18:51:57 Updating file <File #113 u'08-4 Corners-Walk Right.mp3'>
      D: 3077994176 18:51:57 Updating file <File #114 u'09-4 Corners-Strife.mp3'>
      D: 3077994176 18:51:57 Moving file u'/media/ntfs-3g/4 Corners/The Foundations/06-4 Corners-On the Downlow.mp3' => u'/media/ntfs-3g/4 Corners/The Foundations/:/: (4).mp3'
      D: 3077994176 18:51:57 Updating file <File #111 u': (4).mp3'>
      D: 3077994176 18:51:57 Saving file u'/media/ntfs-3g/4 Corners/The Foundations/07-4 Corners-Push It Up.mp3'
      D: 3077994176 18:51:57 Moving file u'/media/ntfs-3g/4 Corners/The Foundations/07-4 Corners-Push It Up.mp3' => u'/media/ntfs-3g/4 Corners/The Foundations/:/: (5).mp3'
      D: 3077994176 18:51:57 Updating file <File #112 u': (5).mp3'>
      D: 3077994176 18:51:57 Saving file u'/media/ntfs-3g/4 Corners/The Foundations/08-4 Corners-Walk Right.mp3'
      D: 3077994176 18:51:57 Moving file u'/media/ntfs-3g/4 Corners/The Foundations/08-4 Corners-Walk Right.mp3' => u'/media/ntfs-3g/4 Corners/The Foundations/:/: (6).mp3'
      D: 3077994176 18:51:57 Updating file <File #113 u': (6).mp3'>
      D: 3077994176 18:51:57 Saving file u'/media/ntfs-3g/4 Corners/The Foundations/09-4 Corners-Strife.mp3'
      D: 3077994176 18:51:57 Moving file u'/media/ntfs-3g/4 Corners/The Foundations/09-4 Corners-Strife.mp3' => u'/media/ntfs-3g/4 Corners/The Foundations/:/: (7).mp3'
      D: 3077994176 18:51:57 Updating file <File #114 u': (7).mp3'>
      

      After

      voiceinsideyou@ubuntu:/media/ntfs-3g/4 Corners/The Foundations$ ls -al
      total 4
      drwxrwxrwx 1 root root 4096 2011-08-14 18:51 .
      drwxrwxrwx 1 root root    0 2010-02-13 13:24 ..
      

      The streams

      G:\4 Corners\The Foundations\.:
            :/: (1).mp3:$DATA 6425332
            :/: (2).mp3:$DATA 5574784
            :/: (3).mp3:$DATA 1038472
            :/: (4).mp3:$DATA 4941952
            :/: (5).mp3:$DATA 4916277
            :/: (6).mp3:$DATA 3582504
            :/: (7).mp3:$DATA 4292736
                :/:.mp3:$DATA 6012126
      

      So this is certainly possible.... likely... hmmm, not sure about that.

          Loading...
          Uploaded image for project: 'Picard'
          1. Picard
          2. PICARD-294

          Picard can write files on NTFS partition to "alternate data streams"

            • Icon: Bug Bug
            • Resolution: Unresolved
            • Icon: Normal Normal
            • None
            • 1.0
            • File Move & Rename
            • None

              Imported from http://bugs.musicbrainz.org/ticket/5790

              Original ticket said:

              I'm very upset with this software. I think it has just deleted my hole song library with no warning. I'm using ubuntu 10.04 and installed picard from the repository. What I've done:

              1. Opened picard 2. Pressed "add folder" and selected my musics folder. They were in a NTFS partition. 3. Pressed "Lookup" it scanned for a long time and found tags for most songs, filling the right column. 4. Checked "Options->Rename files" and "Options->Move files" 5. Pressed "Save"

              All my files have been moved to nowhere. The original folder was missing. Then I took a look at "Options->Options->Move files" and there was just a dot "." in the directory box.

              BrianFreud replied:

              Given that it's NTFS, and depending on the naming string used, it's also possible that the files were written to an alternate stream attached to the files' directory. (See http://manpages.ubuntu.com/manpages/jaunty/man8/ntfs-3g.8.html ; the NTFS-3G driver does support reading and writing to streams). I know I did accidentally somehow get Picard to write to an alternate stream once, back when I was still using Windows, though I don't recall exactly what I'd done to the naming string that managed to do it.

              ...

              voiceinsideyou did a lot more investigating and found:

              There is the sysinternals "streams" utility, for Windows. [0]

              I believe in order to have got it to write to an ADS, you would have had to somehow make Picard write files using a file naming string that had a colon in it. I believe opening and writing to a file handle with "filename:adsname" is enough to make Picard do this.[1] I've tested this myself on an NTFS volume and it works through black magic.

              So my guess is that when doing this on *Nix, if you're writing to an NTFS volume, perhaps it's very easy to break things if you have "Replace Windows-incompatible characters" disabled. Possibly Python and Picard just blindly tries to write files with : in them which could make things get hidden in ADSes.

              Still not quite sure how you could get it to move an entire collection hidden in an ADS though; without a somewhat unlucky file naming string. Perhaps some time later I'll get VirtualBox/VMWare going and see if I can replicate something like this in Picard itself.

              [0] http://technet.microsoft.com/en-us/sysinternals/bb897440 [1] http://robindavid.comli.com/?p=446

              ...

              Indeed I was able to replicate this on Ubuntu using the ntfs-3g driver with streams_interface=windows (which is not the default). Witha file renaming string of ":" and move to ".", replace windows incompatible chars turned off.

              Before:

              voiceinsideyou@ubuntu:/media/ntfs-3g/4 Corners/The Foundations$ ls -al
              total 17332
              drwxrwxrwx 1 root root    4096 2011-08-14 18:51 .
              drwxrwxrwx 1 root root       0 2010-02-13 13:24 ..
              -rwxrwxrwx 1 root root 4941952 2011-08-09 21:16 06-4 Corners-On the Downlow.mp3
              -rwxrwxrwx 1 root root 4916277 2011-08-09 21:16 07-4 Corners-Push It Up.mp3
              -rwxrwxrwx 1 root root 3582080 2011-08-09 21:16 08-4 Corners-Walk Right.mp3
              -rwxrwxrwx 1 root root 4292736 2011-08-09 21:16 09-4 Corners-Strife.mp3
              

              Picard renames

              D: 3077994176 18:51:57 Updating file <File #111 u'06-4 Corners-On the Downlow.mp3'>
              D: 3077994176 18:51:57 Saving file u'/media/ntfs-3g/4 Corners/The Foundations/06-4 Corners-On the Downlow.mp3'
              D: 3077994176 18:51:57 Updating file <File #112 u'07-4 Corners-Push It Up.mp3'>
              D: 3077994176 18:51:57 Updating file <File #113 u'08-4 Corners-Walk Right.mp3'>
              D: 3077994176 18:51:57 Updating file <File #114 u'09-4 Corners-Strife.mp3'>
              D: 3077994176 18:51:57 Moving file u'/media/ntfs-3g/4 Corners/The Foundations/06-4 Corners-On the Downlow.mp3' => u'/media/ntfs-3g/4 Corners/The Foundations/:/: (4).mp3'
              D: 3077994176 18:51:57 Updating file <File #111 u': (4).mp3'>
              D: 3077994176 18:51:57 Saving file u'/media/ntfs-3g/4 Corners/The Foundations/07-4 Corners-Push It Up.mp3'
              D: 3077994176 18:51:57 Moving file u'/media/ntfs-3g/4 Corners/The Foundations/07-4 Corners-Push It Up.mp3' => u'/media/ntfs-3g/4 Corners/The Foundations/:/: (5).mp3'
              D: 3077994176 18:51:57 Updating file <File #112 u': (5).mp3'>
              D: 3077994176 18:51:57 Saving file u'/media/ntfs-3g/4 Corners/The Foundations/08-4 Corners-Walk Right.mp3'
              D: 3077994176 18:51:57 Moving file u'/media/ntfs-3g/4 Corners/The Foundations/08-4 Corners-Walk Right.mp3' => u'/media/ntfs-3g/4 Corners/The Foundations/:/: (6).mp3'
              D: 3077994176 18:51:57 Updating file <File #113 u': (6).mp3'>
              D: 3077994176 18:51:57 Saving file u'/media/ntfs-3g/4 Corners/The Foundations/09-4 Corners-Strife.mp3'
              D: 3077994176 18:51:57 Moving file u'/media/ntfs-3g/4 Corners/The Foundations/09-4 Corners-Strife.mp3' => u'/media/ntfs-3g/4 Corners/The Foundations/:/: (7).mp3'
              D: 3077994176 18:51:57 Updating file <File #114 u': (7).mp3'>
              

              After

              voiceinsideyou@ubuntu:/media/ntfs-3g/4 Corners/The Foundations$ ls -al
              total 4
              drwxrwxrwx 1 root root 4096 2011-08-14 18:51 .
              drwxrwxrwx 1 root root    0 2010-02-13 13:24 ..
              

              The streams

              G:\4 Corners\The Foundations\.:
                    :/: (1).mp3:$DATA 6425332
                    :/: (2).mp3:$DATA 5574784
                    :/: (3).mp3:$DATA 1038472
                    :/: (4).mp3:$DATA 4941952
                    :/: (5).mp3:$DATA 4916277
                    :/: (6).mp3:$DATA 3582504
                    :/: (7).mp3:$DATA 4292736
                        :/:.mp3:$DATA 6012126
              

              So this is certainly possible.... likely... hmmm, not sure about that.

                    Unassigned Unassigned
                    bitmap Michael Wiencek
                    Votes:
                    0 Vote for this issue
                    Watchers:
                    2 Start watching this issue

                      Created:
                      Updated:

                        Version Package

                          Unassigned Unassigned
                          bitmap Michael Wiencek
                          Votes:
                          0 Vote for this issue
                          Watchers:
                          2 Start watching this issue

                            Created:
                            Updated:

                              Version Package