Uploaded image for project: 'Other'
  1. Other
  2. OTHER-77

Unify all markup syntax to simple markdown syntax

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Normal Normal
    • Discourse, Forums, Jira, Wiki
    • None

      Current markup syntaxes:

      Wiki, annotations*, edit notes*, forums, Jira, etc. all possible syntax we have, those are internal and they even differ from each other, for example: sometimes the URL comes before, sometimes it comes after.

      Markdown:

      Markdown is an easy-to-read, easy-to-write plain text format, made for web writers: http://daringfireball.net/projects/markdown/

      A widely accepted and more formal specification of Markdown is CommonMark: http://commonmark.org/

      Many flavours of Markdown are listed at https://github.com/commonmark/CommonMark/wiki/Markdown-Flavors

      Status:

      Three reports have been made and attached to this issue by GCI students (see comments) to study existing Markdown variants, implementations, and their relevance to MetaBrainz needs.

      Since the end of Feb 2018, a draft proposition (written in Markdown) for a flavor of Markdown dedicated to *Brainz projects, with specific features, equivalents, breaking changes, and migration paths is available at: https://gist.github.com/yvanzo/c3ffa4dd8410c2dbb4ff73bb41af8935
      Reviews and comments are very welcome, see comments below.

          [OTHER-77] Unify all markup syntax to simple markdown syntax

          jesus2099 added a comment -

          I'm linking MBS-8120 that adds setlists to the (evergrowing) list, as it appeared after 2011.

          jesus2099 added a comment - I'm linking MBS-8120 that adds setlists to the (evergrowing) list, as it appeared after 2011.

          Hi, during the 2019 MusicBrainz Summit, we discussed the documentation toolchain roadmap. See our notes in detail here. There is also a Discourse thread with more info and proof-of-concept of Markdown docs built with Sphinx and ReadTheDocs.

          I marked this ticket as dependent on MBS-10126 since the documentation toolchain ticket will more than likely close this ticket as well. It doesn't account for things like the editing system, but I think getting our docs into Sphinx and porting old docs into the new system would be a big win and solves most of the need for this ticket.

          Curious to hear more thoughts in MBS-10126, please follow-up in that ticket!

          Justin W. Flory added a comment - Hi, during the 2019 MusicBrainz Summit , we discussed the documentation toolchain roadmap. See our notes in detail here . There is also a Discourse thread with more info and proof-of-concept of Markdown docs built with Sphinx and ReadTheDocs. I marked this ticket as dependent on MBS-10126 since the documentation toolchain ticket will more than likely close this ticket as well. It doesn't account for things like the editing system, but I think getting our docs into Sphinx and porting old docs into the new system would be a big win and solves most of the need for this ticket. Curious to hear more thoughts in MBS-10126 , please follow-up in that ticket!

          MonkeyPython added a comment - - edited
          whatever flavour of mark-sideways we chose it should do the following:
          
          /this should be italic/
          _this should be underlined_
          -this should be strikeout-
          **double asterisk is great idea for bold!**
          
          links are fine like [text](http://lolurl.com) or [url](text) or [url | text] or whatever. any of those are fine 
          as long as it's _consistent_ over all the places we do mark-whatever
          
           * this as a bullet list
           * is fine 
           * and easy to remember
            * how we do indentions however, (prefer this one)
           ** must be decided between these two
          
           1 this as a numbered list
           2 works fine
           4 but are unordered going to happen?
           b and what about alphanumeric?
           ii or this way?
          
          
          some way to make sure that :) or the unicode ☺ isn't automatically rendered as 

           (image)

          a logical way is to do :) ☺ as text and ::): or :smile: as image
          option to turn on/off either or is also very good
          (naturally all other emoticons and emoji the same way.)
          in the same vein having an easy way to render text with no formatting a-la "no-code" is a must
          
          some logical way to short link JIRA tickets [MBS-123] and edits [edit 51546475] (and in the future BB revisions too)
          
          
          NOT automatically turning something as typed into something else, eg pasting https://beta.musicbrainz.org/edit/51546475 will not:
          create a shortlink automatically
          create a "previewbox"
          insert any images, newlines or font-changes
          write any text other than the actual link (however, see below)
          
          
          in relation to the above: automatically linking a pasted mbid [8ef2dd1-125e-4f99-8432-fe38e0f32b95] or [https:///instrument/7cbd040f-6217-48d0-ac43-3e8ecb4b4b0b] 
          can also show the title of the thing, not just the link (but again, pasting a bare url contra applying some sort of syntax to it)
          
          That is: having /syntax/ that does any of these is a good idea, "auto-magcially" doing it, is /not/ a good idea.
          
          
          for added usability, the possibility to cross-link links like this
          eg writing the following **_ anywhere_**
           [wiki:Name of Guideline] or [wiki:user:CatFuu] will link to the wikipage or user
           [jira:MBS-4321] will link to the Jira ticket
           [mb:edit543543] will link to a MusicBrainz edit, [bb:r3549] will link to a BookBrainz revision, etc.
           [mb:33cf029c-63b0-41a0-9855-be2a3665fb3b] will link to a MusicBrainz entity
           [cb:f6d79ac0-adea-43f2-87d1-559a1e19d014] will link to a CritiqueBrainz review
          
          and so on
           
          
          show this syntax **on the page**, in all applicable places.
          that is:
           * edit notes
           * annotations (instrument descriptions)
           * jira ticket descriptions and comments
           * community posts and topics
           * mbwiki?
           * CritiqueBrainz reviews (both regular and the short ones) 
            * CritiqueBrainz reviews' comments
           * BookBrainz revision notes, descriptions, annotations etc.
           * any userpage bio or description field (right now only on MusicBrainz, but will be MetaBrainz in the future) including the current community.meb.org ones.
           * The Blog writing software/comments (if possible) 
           * events (setlists)
          
          
          any place where syntax can be used, provide /at least/ the syntax _on the page_ 
          but preferably also buttons that will logically format things,
          eg pasting https://tickets.metabrainz.org/browse/MBS-9691 highlighting and clicking (linkyfythingy button)
           will turn it into [jira:MBS-9691] all nicely formatted and stuff (just pasting does nothing but just turning it into a regular link with the text "https://tickets.metabrainz.org/browse/MBS-9691")
          

          These are just my, best-case-wishlist for mark-inversemöbius
          What makes the most sense for me, I'm sure others have many other ideas and inputs here, but I hope atleast the basic idea is agreeable to most. Thank you :​)

          MonkeyPython added a comment - - edited whatever flavour of mark-sideways we chose it should do the following: /this should be italic/ _this should be underlined_ -this should be strikeout- **double asterisk is great idea for bold!** links are fine like [text](http://lolurl.com) or [url](text) or [url | text] or whatever. any of those are fine as long as it's _consistent_ over all the places we do mark-whatever  * this as a bullet list  * is fine  * and easy to remember   * how we do indentions however, (prefer this one)  ** must be decided between these two  1 this as a numbered list  2 works fine  4 but are unordered going to happen?  b and what about alphanumeric?  ii or this way? some way to make sure that :) or the unicode ☺ isn't automatically rendered as  (image) a logical way is to do :) ☺ as text and ::): or :smile: as image option to turn on/off either or is also very good (naturally all other emoticons and emoji the same way.) in the same vein having an easy way to render text with no formatting a-la "no-code" is a must some logical way to short link JIRA tickets [MBS-123] and edits [edit 51546475] (and in the future BB revisions too) NOT automatically turning something as typed into something else, eg pasting https://beta.musicbrainz.org/edit/51546475 will not: create a shortlink automatically create a "previewbox" insert any images, newlines or font-changes write any text other than the actual link (however, see below) in relation to the above: automatically linking a pasted mbid [8ef2dd1-125e-4f99-8432-fe38e0f32b95] or [https:///instrument/7cbd040f-6217-48d0-ac43-3e8ecb4b4b0b] can also show the title of the thing, not just the link (but again, pasting a bare url contra applying some sort of syntax to it) That is: having /syntax/ that does any of these is a good idea, "auto-magcially" doing it, is /not/ a good idea. for added usability, the possibility to cross-link links like this eg writing the following **_ anywhere_**  [wiki:Name of Guideline] or [wiki:user:CatFuu] will link to the wikipage or user  [jira:MBS-4321] will link to the Jira ticket  [mb:edit543543] will link to a MusicBrainz edit, [bb:r3549] will link to a BookBrainz revision, etc.  [mb:33cf029c-63b0-41a0-9855-be2a3665fb3b] will link to a MusicBrainz entity  [cb:f6d79ac0-adea-43f2-87d1-559a1e19d014] will link to a CritiqueBrainz review and so on   show this syntax **on the page**, in all applicable places. that is:  * edit notes  * annotations (instrument descriptions)  * jira ticket descriptions and comments  * community posts and topics  * mbwiki?  * CritiqueBrainz reviews (both regular and the short ones)   * CritiqueBrainz reviews' comments  * BookBrainz revision notes, descriptions, annotations etc.  * any userpage bio or description field (right now only on MusicBrainz, but will be MetaBrainz in the future) including the current community.meb.org ones.  * The Blog writing software/comments (if possible) * events (setlists) any place where syntax can be used, provide /at least/ the syntax _on the page_ but preferably also buttons that will logically format things, eg pasting https://tickets.metabrainz.org/browse/MBS-9691 highlighting and clicking (linkyfythingy button) will turn it into [jira:MBS-9691] all nicely formatted and stuff (just pasting does nothing but just turning it into a regular link with the text "https://tickets.metabrainz.org/browse/MBS-9691") These are just my, best-case-wishlist for mark-inversemöbius What makes the most sense for me, I'm sure others have many other ideas and inputs here, but I hope atleast the basic idea is agreeable to most. Thank you :​)

          yvanzo added a comment -

          It turns out to be currently unsuitable for our use, due to the following cons:

          Thus, I disabled this add-on for MBS and OTHER projects. Unless these issues are handled, it is likely we have to stick with JIRA’s wiki syntax here. I updated the draft accordingly.

          yvanzo added a comment - It turns out to be currently unsuitable for our use, due to the following cons: It comes with a text editor quite unfriendly compared to the stock JIRA’s wiki text editor. This issue has been reported at https://github.com/fulstech/markin/issues/48 It doesn’t support rendering in activity stream This issue has been reported at https://github.com/fulstech/markin/issues/49 Conflicting syntaxes are badly handled, especially JIRA’s ordered list items # , ## , … which are rendered as Markdown ATX headers. Thus, I disabled this add-on for MBS and OTHER projects. Unless these issues are handled, it is likely we have to stick with JIRA’s wiki syntax here. I updated the draft accordingly.

          yvanzo added a comment -

          Markdown syntax is now rendered for test purpose in OTHER and MBS tickets’ description and comments through Markin add-on. It partially supports JIRA wiki syntax.

          yvanzo added a comment - Markdown syntax is now rendered for test purpose in OTHER and MBS tickets’ description and comments through Markin add-on. It partially supports JIRA wiki syntax.

          yvanzo added a comment -

          Updated the summary with recent changes.

          yvanzo added a comment - Updated the summary with recent changes.

          yvanzo added a comment -

          Reply to kepstin (during meeting) about edit notes storage/conversion:

          That’s a good (technical) point.

          New edit notes will be stored as-is, in the new Markdown-based syntax.

          Old edit notes will need to be converted. Since HTML markup can be included as-is into Markdown text, the easiest conversion seems to be to store pre-rendered old edit notes.

          Since the conversion process will take months, we need a way to decide how to render each edit note appropriately during the migration. This can be done by comparing edit note id to a max value stored into the table `statistic`. To sum up, it can be done gradually and it doesn’t require any schema change.

          yvanzo added a comment - Reply to kepstin (during meeting ) about edit notes storage/conversion : That’s a good (technical) point. New edit notes will be stored as-is, in the new Markdown-based syntax. Old edit notes will need to be converted. Since HTML markup can be included as-is into Markdown text, the easiest conversion seems to be to store pre-rendered old edit notes. Since the conversion process will take months, we need a way to decide how to render each edit note appropriately during the migration. This can be done by comparing edit note id to a max value stored into the table `statistic`. To sum up, it can be done gradually and it doesn’t require any schema change.

          yvanzo added a comment -

          Reply to freso:

          1. Name: OK, I changed it to Brainzed Markdown for now.
          2. doc: Syntax: It is both a legacy of current edit note syntax and a convenient trigger for auto-completion of documentation references. What makes you un_happy_ with it?
          3. MusicBrainz Edit URL Rendering: BookBrainz revision URL can be rendered in the same way, I just focused on replacing MusicBrainz markup features for now.
            (Nobody enjoys skimming full URLs to read a post. Embellished/shortened URLs are intended to improve readability by only displaying the most useful part. Users will probably be pleased with it. Although regulars might be surprised at first, I don’t think anyone will be shocked. The full URL can still be read from target link, and it is possible to specify URL as link title instead.)
          4. MetaBrainz Tickets Renderer: Actually, there can be only one renderer per field. Fortunately, Markin renderer supports a sufficient subset of JIRA markup.
          5. BookBrainz: Yes, I have the same memory. I just made sure there is no legacy markup to handle here. It is likely that MusicBrainz specific extensions will be transposed to BookBrainz.
          6. MusicBrainz Event Setlist: Even though the schema gets changed to handle setlist akin tracklist, we still need a plain text format akin tracklist one. Markdown is very appropriate for that purpose as it is easy to read/edit even without renderer.

          yvanzo added a comment - Reply to freso : Name: OK, I changed it to Brainzed Markdown for now. doc: Syntax: It is both a legacy of current edit note syntax and a convenient trigger for auto-completion of documentation references. What makes you un_happy_ with it? MusicBrainz Edit URL Rendering: BookBrainz revision URL can be rendered in the same way, I just focused on replacing MusicBrainz markup features for now. (Nobody enjoys skimming full URLs to read a post. Embellished/shortened URLs are intended to improve readability by only displaying the most useful part. Users will probably be pleased with it. Although regulars might be surprised at first, I don’t think anyone will be shocked. The full URL can still be read from target link, and it is possible to specify URL as link title instead.) MetaBrainz Tickets Renderer: Actually, there can be only one renderer per field. Fortunately,  Markin renderer supports a sufficient subset of JIRA markup. BookBrainz: Yes, I have the same memory. I just made sure there is no legacy markup to handle here. It is likely that MusicBrainz specific extensions will be transposed to BookBrainz. MusicBrainz Event Setlist: Even though the schema gets changed to handle setlist akin tracklist, we still need a plain text format akin tracklist one. Markdown is very appropriate for that purpose as it is easy to read/edit even without renderer.

          yvanzo added a comment - - edited

          Reply to jesus2099:

          It seems to me that Edit #123457 would still be more familiar and explicit than just #123456.

          You can still type “edit” before the URL to obtain the same result.
          The current automatic rendering of the word “edit” has localization issue.

          In annotations we can make protocol-less, host-less links to entities with [artist:mbid] or [label:mbid|description] like in that exemple, etc.
          It would be nice to keep some kind of simple entity links.

          Protocol-less, host-less entity link requires more effort to type than to copy/paste full URL. In your example, link syntax will be replaced with: [COLUMBIA](https://musicbrainz.org/label/011d1192-6f65-45bd-85c4-0400dd45693e)

          1. Brackets can be automatically added using the toolbar, see Community forums UI.
          2. Protocol and host won’t be taken into account when rendered on a MusicBrainz server.
          3. Although it is a bit longer to read as plain text, the longest part is the MBID in either case.

          In edit notes we can type very fast links to MB wiki like [AC], [ETI], [RG] or [ReleaseGroup].

          Having both scheme-less Wikilink syntax [ReleaseGroup] and Markdown link syntax [url] would be confusing. That’s why I suggested to keep the syntax doc:ReleaseGroup only, with the additional benefit that the scheme doc: could be used to trigger auto-completion for documentation page name.

          yvanzo added a comment - - edited Reply to jesus2099 : It seems to me that Edit #123457 would still be more familiar and explicit than just #123456 . You can still type “edit” before the URL to obtain the same result. The current automatic rendering of the word “edit” has localization issue. In annotations we can make protocol-less, host-less links to entities with [artist:mbid] or [label:mbid|description] like in that exemple , etc. It would be nice to keep some kind of simple entity links. Protocol-less, host-less entity link requires more effort to type than to copy/paste full URL. In your example, link syntax will be replaced with: [COLUMBIA] ( https://musicbrainz.org/label/011d1192-6f65-45bd-85c4-0400dd45693e ) Brackets can be automatically added using the toolbar, see Community forums UI. Protocol and host won’t be taken into account when rendered on a MusicBrainz server. Although it is a bit longer to read as plain text, the longest part is the MBID in either case. In edit notes we can type very fast links to MB wiki like [AC] , [ETI] , [RG] or [ReleaseGroup] . Having both scheme-less Wikilink syntax  [ReleaseGroup] and Markdown link syntax [url] would be confusing. That’s why I suggested to keep the syntax doc:ReleaseGroup only, with the additional benefit that the scheme doc: could be used to trigger auto-completion for documentation page name.

          jesus2099 added a comment - - edited

          It seems to me that Edit #123457 would still be more familiar and explicit than just #123456.

          Two things I use but did not see covered:

          In annotations we can make protocol-less, host-less links to entities with [artist:mbid] or [label:mbid|description] like in that exemple, etc.
          It would be nice to keep some kind of simple entity links.

          In edit notes we can type very fast links to MB wiki like [AC], [ETI], [RG] or [ReleaseGroup].

          jesus2099 added a comment - - edited It seems to me that Edit #123457 would still be more familiar and explicit than just #123456 . — Two things I use but did not see covered: In annotations we can make protocol-less, host-less links to entities with [artist:mbid] or [label:mbid|description] like in that exemple , etc. It would be nice to keep some kind of simple entity links. In edit notes we can type very fast links to MB wiki like [AC] , [ETI] , [RG] or [ReleaseGroup] .

          I thought the proposal was that raw URLs like https://musicbrainz.org/edit/123456 would be detected as edit links, not #123456 (which I agree would be a bad idea).

          Alec Edgington added a comment - I thought the proposal was that raw URLs like https://musicbrainz.org/edit/123456 would be detected as edit links, not #123456 (which I agree would be a bad idea).

          jesus2099 added a comment -

          I want to read the draft thoroughly and make a good comprehensive comment like Freso.
          But I just have a small remark I don’t want to delay:

          I was wondering if we shouldn’t keep our current parsing for detecting Edit links, that seems to be something like /\b[eE]dit\s#\d+\b/ (Edit #123456 or edit #123456) rather than switching to very short /\b#\d+\b/ (#123456) that seems little risky to me…

          jesus2099 added a comment - I want to read the draft thoroughly and make a good comprehensive comment like Freso. But I just have a small remark I don’t want to delay: I was wondering if we shouldn’t keep our current parsing for detecting Edit links, that seems to be something like /\b [eE] dit\s#\d+\b/ (Edit #123456 or edit #123456) rather than switching to very short /\b#\d+\b/ (#123456) that seems little risky to me…

          Freso added a comment -

          Left some comments on the gist, repeating here for perpetuity:

          I think we should keep *Brainz names for data projects, so I'd prefer something like BrainzMark (or BrainzMarkup?) instead, but this is bikeshedding.

          https://gist.github.com/yvanzo/c3ffa4dd8410c2dbb4ff73bb41af8935#musicbrainz-documentation-reference-shortcut-syntax-autocompletion-and-postprocessing
          I'm not sure if I'm very happy about the `doc:` syntax.

          https://gist.github.com/yvanzo/c3ffa4dd8410c2dbb4ff73bb41af8935#musicbrainz-edit-url-postprocessing
          Another con: MusicBrainz specific, doesn't take e.g., BookBrainz edits or similar into account. (Also changes display for users who may expect the URL to show and get #… instead. Somewhat minor con, but still something.)

          https://gist.github.com/yvanzo/c3ffa4dd8410c2dbb4ff73bb41af8935#tickets-issue-tracker
          I think it is possible to have multiple parser plugins at the same time too, so we could have both Jira markup and BrainzMarkup living side-by-side and people would be free to use either. (Need to verify this though.)

          https://gist.github.com/yvanzo/c3ffa4dd8410c2dbb4ff73bb41af8935#bookbrainz
          AFAIK, BB is/was waiting for something like BrainzMarkup before implementing formatting? Maybe I misremember things.

          https://gist.github.com/yvanzo/c3ffa4dd8410c2dbb4ff73bb41af8935#musicbrainz
          Re: Event set lists, I feel these should rather be moved to some format more akin to release track lists, rather than revamping the syntax for them. This of course needs some schema design and a schema change release (unless it can be done entirely with relationships), but I feel this would be a better path forward for these.

          https://gist.github.com/yvanzo/c3ffa4dd8410c2dbb4ff73bb41af8935#tagging-username
          About normalisation, there is at least https://tickets.metabrainz.org/browse/MBS-9557 and a linked ticket about this.

          Freso added a comment - Left some comments on the gist, repeating here for perpetuity: I think we should keep *Brainz names for data projects, so I'd prefer something like BrainzMark (or BrainzMarkup?) instead, but this is bikeshedding. https://gist.github.com/yvanzo/c3ffa4dd8410c2dbb4ff73bb41af8935#musicbrainz-documentation-reference-shortcut-syntax-autocompletion-and-postprocessing I'm not sure if I'm very happy about the `doc:` syntax. https://gist.github.com/yvanzo/c3ffa4dd8410c2dbb4ff73bb41af8935#musicbrainz-edit-url-postprocessing Another con: MusicBrainz specific, doesn't take e.g., BookBrainz edits or similar into account. (Also changes display for users who may expect the URL to show and get #… instead. Somewhat minor con, but still something.) https://gist.github.com/yvanzo/c3ffa4dd8410c2dbb4ff73bb41af8935#tickets-issue-tracker I think it is possible to have multiple parser plugins at the same time too, so we could have both Jira markup and BrainzMarkup living side-by-side and people would be free to use either. (Need to verify this though.) https://gist.github.com/yvanzo/c3ffa4dd8410c2dbb4ff73bb41af8935#bookbrainz AFAIK, BB is/was waiting for something like BrainzMarkup before implementing formatting? Maybe I misremember things. https://gist.github.com/yvanzo/c3ffa4dd8410c2dbb4ff73bb41af8935#musicbrainz Re: Event set lists, I feel these should rather be moved to some format more akin to release track lists, rather than revamping the syntax for them. This of course needs some schema design and a schema change release (unless it can be done entirely with relationships), but I feel this would be a better path forward for these. https://gist.github.com/yvanzo/c3ffa4dd8410c2dbb4ff73bb41af8935#tagging-username About normalisation, there is at least https://tickets.metabrainz.org/browse/MBS-9557 and a linked ticket about this.

          Reviews and comments are very welcome.

          Looks great to me.

          Alec Edgington added a comment - Reviews and comments are very welcome. Looks great to me.

          yvanzo added a comment -

          Here is a draft proposition for a flavor of Markdown dedicated to *Brainz projects, with specific features, equivalents, breaking changes, and migration paths: MarkBrainz Draft #84880.
          Reviews and comments are very welcome.

          yvanzo added a comment - Here is a draft proposition for a flavor of Markdown dedicated to *Brainz projects, with specific features, equivalents, breaking changes, and migration paths: MarkBrainz Draft #84880 . Reviews and comments are very welcome.

          jesus2099 added a comment -

          And setlists, and wiki, and editor biographies.

          jesus2099 added a comment - And setlists, and wiki, and editor biographies.

          jesus2099 added a comment -

          When this is studied, I think the migration of existing annotations and edit notes must be defined as well.

          jesus2099 added a comment - When this is studied, I think the migration of existing annotations and edit notes must be defined as well.

          Here is my report focusing mainly on the GitHub flavoured markdowns.haroo_stack_gitlab_markdown_comp.pdf

          Daniel Vlasits added a comment - Here is my report focusing mainly on the GitHub flavoured markdowns. haroo_stack_gitlab_markdown_comp.pdf

          Enjeck Cleopatra added a comment - Report: metabrainz (1).pdf

          Eshan Singh added a comment - - edited

          I made a report on this - see markdown_comparison.pdf.

          Eshan Singh added a comment - - edited I made a report on this - see markdown_comparison.pdf .

          A great consequence of this would be having a clean way to format quotes in annotations, like when quoting another website (I often do that when adding information from the Internet Archive).

          Stéphane Guillou added a comment - A great consequence of this would be having a clean way to format quotes in annotations, like when quoting another website (I often do that when adding information from the Internet Archive).

            Unassigned Unassigned
            jesus2099 jesus2099
            Votes:
            14 Vote for this issue
            Watchers:
            20 Start watching this issue

              Created:
              Updated:

                Version Package