Skip to content

Commit 63d48c6

Browse files
vschvsch
committed
compatibility for list processing rewritten. CommonMark passes.
1 parent abf9ca3 commit 63d48c6

174 files changed

Lines changed: 10907 additions & 4253 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.idea/dictionaries/vlad.xml‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.idea/typing_corrector_settings.xml‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.idea/workspace.xml‎

Lines changed: 3623 additions & 3034 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎MarkdownProcessorsEmulation.md‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
Markdown Processors Emulation
2+
=============================
3+
4+
Parsers are classified by "Family" for their list processing characteristics, by far the
5+
greatest point of deviation between all markdown parsers. They differ the greatest on how they
6+
determining whether a text line is:
7+
8+
- indented code of the current item or parent list item's
9+
- lazy continuation of current item
10+
- next list item
11+
- current item's sub-item
12+
13+
Some processors, like Kramdown and Markdown, use the list's first item indent to determine when
14+
text is an item, lazy continuation or indented code. CommonMark (spec.txt 0.27),
15+
uses the last item's content indent for this determination. MultiMarkdown, uses a
16+
fixed indent of 4 spaces from left edge or last block quote marker to make that determination.
17+
18+
After trying to work this out by trial and error and only having moderate success, I decided to
19+
make more rigorous to reduce interaction between options and to be able tweak individual family
20+
emulation accuracy.
21+
22+
The following definitions are used:
23+
24+
- `index`: offset from start of line being processed
25+
- `line indent`: index of first non-blank from index set by parent element for the line
26+
- `line column`: would be index of first non-blank if tabs were expanded to 4 space boundaries
27+
- `item indent`: `line indent` of first line of item (one with list item marker)
28+
- `item column`: `line column` of first line of item (one with list item marker)
29+
- `item content offset`: item prefix length + # of trailing white spaces following the item prefix
30+
- `item content indent`: `item indent` + `item content offset`
31+
- `item content column`: `item column` + `item content offset`
32+
- `list indent`: first item's `item indent`
33+
- `list column`: first item's `item column`
34+
- `list content indent`: first item's `item content indent`
35+
- `list content column`: first item's `item content column`
36+
- `list last indent`: last item's `item indent`
37+
- `list last column`: last item's `item column`
38+
- `list last content indent`: last item's `item content indent`
39+
- `list last content column`: last item's `item content column`
40+
- `list nesting`: number of direct list ancestors of an item (counting stops on non-list or
41+
non-list-item parent block), always >= 1 since every list item has a list parent
42+
- `first parent list`: last list block after non-list or non-list item block
43+
44+
Family types:
45+
46+
- CommonMark: version 0.27 of the spec, all common mark parsers
47+
- Definitions/Defaults:
48+
- `ITEM_INDENT` = 4 <!-- not used -->
49+
- `CODE_INDENT` = 4
50+
- `current indent` = `line indent`
51+
- Start List Conditions:
52+
- `item indent` < `CODE_INDENT`: new list with new item
53+
- `item content indent` >= `CODE_INDENT`: empty item, indented code
54+
- Continuation Conditions:
55+
- `current indent` >= `list last content indent` + `CODE_INDENT`: indented code
56+
- `current indent` >= `list last content indent`: sub-item
57+
- `current indent` >= `list indent`: list item
58+
59+
- MultiMarkdown: Pandoc, Pegdown, all fixed indent type processors
60+
- Definitions/Defaults:
61+
- `ITEM_INDENT` = 4
62+
- `CODE_INDENT` = 8
63+
- `current indent` = `line column` - `first parent list column` + `first parent list
64+
indent` - (`list nesting` - 1) * `ITEM_INDENT`
65+
- Start List Conditions:
66+
- `current indent` < `ITEM_INDENT`: new list with new item
67+
- Continuation Conditions:
68+
- `current indent` >= `CODE_INDENT`: indented code
69+
- `current indent` >= `ITEM_INDENT`: sub-item
70+
- `current indent` < `ITEM_INDENT`: list item
71+
72+
- Kramdown:
73+
- Definitions/Defaults:
74+
- `ITEM_INDENT` = 4
75+
- `CODE_INDENT` = 8
76+
- `current indent` = `line indent`
77+
- Start List Conditions:
78+
- `current indent` < `ITEM_INDENT`: new list with new item
79+
- Continuation Conditions:
80+
- `current indent` >= `list content indent` + `CODE_INDENT`: indented code
81+
- `current indent` >= `list content indent` + `ITEM_INDENT`:
82+
- if had blank line in item && have previous list item parent: then let it have it
83+
- otherwise: lazy continuation of last list item
84+
- `current indent` >= `item content indent`: sub-item
85+
- `current indent` >= `list content indent`: list item
86+
87+
- Markdown:
88+
- Definitions/Defaults:
89+
- `ITEM_INDENT` = 4
90+
- `CODE_INDENT` = 8
91+
- `current indent` = `line indent`
92+
- Start List Conditions:
93+
- `current indent` < `ITEM_INDENT`: new list with new item
94+
- Continuation Conditions:
95+
- `current indent` >= `list indent` + `CODE_INDENT`:
96+
- if had blank line in item && have previous list item parent: then let it have it
97+
- otherwise: lazy continuation of last list item
98+
- `current indent` > `list indent`: sub-item
99+
- `current indent` == `list indent`: list item
100+
101+
Minor differences, are addressed with options applied on top of the `family` list behavior to
102+
tweak parser emulation:
103+
104+
- [x] bullet item can interrupt a paragraph {C} {GFM} {GFC}
105+
- [x] bullet item can interrupt a paragraph of a list item
106+
- [x] empty bullet item can interrupt a paragraph
107+
- [ ] empty bullet item can interrupt a paragraph of a list item
108+
- [x] ordered item can interrupt a paragraph
109+
- [x] ordered item can interrupt a paragraph of a list item
110+
- [x] ordered non 1 item can interrupt a paragraph
111+
- [x] ordered non 1 item can interrupt a paragraph of a list item
112+
- [ ] empty ordered item can interrupt a paragraph
113+
- [ ] empty ordered item can interrupt a paragraph of a list item
114+
- [ ] empty ordered non 1 item can interrupt a paragraph
115+
- [ ] empty ordered non 1 item can interrupt a paragraph of a list item
116+
- [x] mismatch item type continue same list type
117+
- [x] mismatch item type start new list
118+
- [x] mismatch item type start a sub-list
119+
- [x] bullet mismatch starts a new list
120+
- [x] ordered items only with `.` after digit, otherwise `)` is also allowed
121+
- [x] first ordered item prefix sets start number of list
122+
- [x] item is loose if it has trailing blank line in it or its last child
123+
- [ ] item is loose if previous item has trailing blank line in it or its last child
124+
- [x] item is loose if it or previous item is loose
125+
- [x] all items are loose if any in the list are loose

‎README.md‎

Lines changed: 32 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
![Flexmark Icon Logo](/assets/images/flexmark-icon-logo%402x.png) flexmark-java
22
===============================================================================
33

4-
flexmark-java is a fork of [commonmark-java] project, modified to generate an AST which reflects
4+
**flexmark-java** is a fork of [commonmark-java] project, modified to generate an AST which reflects
55
all the elements in the original source, full source position tracking for all elements in the
66
AST and easier JetBrains Open API PsiTree generation.
77

@@ -48,7 +48,36 @@ earlier versions of this project.
4848
- Android compatibility neglected for now
4949
- No attempt is made to keep API backward compatibility to the original project.
5050

51-
#### This is a work in progress with many API changes.
51+
#### Markdown Parser
52+
53+
Latest addition was a rewrite of the list parser to better control emulation of other markdown
54+
parsers as per [Markdown Processors Emulation](MarkdownProcessorsEmulation.md) and the addition of
55+
processor presets to emulate specific markdown processing behaviour of these parsers.
56+
57+
Some presets do a better better job of emulating their target than others. Most of the effort
58+
was directed at emulating how these processors parse standard Markdown. For processors that
59+
extend original Markdown, you will need to add those extensions that are already implemented in
60+
flexmark-java to the Parser/Renderer builder options.
61+
62+
Extensions will be modified to include their own presets for specific processor emulation, if
63+
that processor has an equivalent extension implemented.
64+
65+
If you find a discrepancy please open an issue so it can be addressed.
66+
67+
Major processor families with presets for their variants:
68+
69+
- [x] CommonMark (spec 0.27)
70+
- [ ] GitHub Comments
71+
- [ ] League/CommonMark
72+
- [ ] MultiMarkdown
73+
- [ ] Pegdown
74+
- [ ] Pandoc
75+
- [ ] Kramdown
76+
- [ ] GitHub Docs
77+
- [ ] Jekyll
78+
- [ ] Markdown
79+
- [ ] Php Markdown Extra
80+
5281

5382
### Feature Comparison
5483

@@ -325,6 +354,7 @@ Copyright (c) 2016, Vladimir Schneider,
325354

326355
BSD (2-clause) licensed, see LICENSE.txt file.
327356

357+
328358
[Markdown Navigator]: http://vladsch.com/product/markdown-navigator
329359
[Pegdown - Achilles heel of the Markdown Navigator plugin]: http://vladsch.com/blog/15
330360
[VERSION.md]: https://github.com/vsch/idea-multimarkdown/blob/master/test/data/performance/VERSION.md

‎VERSION.md‎

Lines changed: 34 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ flexmark-java
55

66
## Version History
77
- [Next Release To Do List](#next-release-to-do-list)
8-
- [(0.6.2)](#062)
8+
- [0.7.0](#070)
99
- [0.6.1](#061)
1010
- [0.6.0](#060)
1111
- [0.5.0](#050)
@@ -56,6 +56,7 @@ Next Release To Do List
5656
-----------------------
5757

5858
- [ ] Add: generated HTML element positions to `TagRanges` to allow mapping from source offset
59+
5960
to HTML offset for the element(s). This is needed to allow synchronization with source
6061
when using an attribute to hold the source information is not an option.
6162

@@ -65,27 +66,37 @@ Next Release To Do List
6566
- [ ] Fix: clean up and verify the Extensions wiki options lists for name changes, missing or
6667
extra entries. Update description for better understanding.
6768

68-
(0.6.2)
69-
-------
70-
71-
- [ ] Fix: HTML comment blocks should only be recognized as blocks if they have a blank line
72-
above. Otherwise HTML in a lazy continuation becomes an HTML block. Difference with spec
73-
since it probably does not understand the uses for HTML comments other than empty breaks.
74-
In IntelliJ comments are used for TODO markers and ability to embedd one in a paragraph is
75-
important.
76-
77-
- [x] Add: option `HTML_COMMENT_BLOCKS_INTERRUPT_PARAGRAPH` with `true` by default but
78-
when false then they require a blank line.
79-
80-
- [ ] Add: test for above option
69+
0.7.0
70+
-----
8171

82-
- [ ] Fix: clean up list processing options to be consistent and easily configured for
83-
processors with which flexmark-java can be made compatible.
84-
- [x] commonmark, obviously defaults
85-
- [ ] Markdown.pl
86-
- [ ] Fixed 4, MultiMarkdown, pandocs, pegdown
87-
- [x] github comments, common mark
88-
- [x] kramdown, Jekyll, github docs
72+
- [x] Add: final to all node visitors' node parameter
73+
- [x] Add: upgrade to CommonMark spec 0.27
74+
75+
- [x] Add: option `Parser.PARSE_JEKYLL_MACROS_IN_URLS` which allows any characters to appear
76+
between `{{` and `}}` in URLs, including spaces, pipes and backslashes.
77+
78+
- [x] Add: option `HTML_COMMENT_BLOCKS_INTERRUPT_PARAGRAPH` with `true` by default but when
79+
false then they require a blank line before, otherwise they become inline HTML.
80+
81+
- [x] Add: test for `HTML_COMMENT_BLOCKS_INTERRUPT_PARAGRAPH` option
82+
83+
- [ ] Fix: rewrite list handling and list handling options to allow for Markdown parser
84+
emulation based on major parser families, as described in
85+
[Markdown Parser Emulation](MarkdownProcessorsEmulation.md) with processor profile preset:
86+
- [x] CommonMark
87+
- [ ] GitHub Comments
88+
- [ ] League/CommonMark
89+
- [ ] FixedIndent
90+
- [ ] Pegdown
91+
- [ ] MultiMarkdown
92+
- [ ] Pandoc
93+
- [ ] Kramdown
94+
- [ ] GitHub Docs
95+
- [ ] Jekyll
96+
- [ ] Kramdown
97+
- [ ] Markdown
98+
- [ ] Markdown.pl
99+
- [ ] Php Markdown Extra
89100

90101
- Change: rename `Parser.LISTS_OVER_INDENTS_TO_FIRST_ITEM` to
91102
`Parser.LISTS_CONTENT_INDENT_OVER_MARKER_TO_FIRST_ITEM`
@@ -106,7 +117,7 @@ Next Release To Do List
106117

107118
- Fix: #19, ArrayIndexOutOfBounds while parsing markdown with backslash as last character of
108119

109-
text block
120+
text block
110121

111122
- Change: `SpecReader` to parse for headings only if spaces are present after leading `#`.
112123
Otherwise, leading github issue `#7` in the description is treated as a heading.
@@ -1106,7 +1117,7 @@ Next Release To Do List
11061117
Text[12, 13]
11071118
Reference[15, 40] refOpen:[15, 16, "["] ref:[16, 25, "*foo* bar"] refClose:[25, 27, "]:"] urlOpen:[0, 0] url:[28, 32, "/url"] urlClose:[0, 0] titleOpen:[33, 34, """] title:[34, 39, "title"] titleClose:[39, 40, """]
11081119
````````````````````````````````
1109-
1120+
11101121
- Convert all extension tests to spec.txt style driven testing to make generating tests easier
11111122
and to also test for the generated AST
11121123

‎flexmark-ext-abbreviation/src/main/java/com/vladsch/flexmark/ext/abbreviation/AbbreviationExtension.java‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -25,17 +25,17 @@ public class AbbreviationExtension implements Parser.ParserExtension, HtmlRender
2525
/**
2626
* A {@link DataKey} that is used to get the document's Node repository holding all the abbreviations defined in the current document.
2727
*/
28-
public final static DataKey<AbbreviationRepository> ABBREVIATIONS = new DataKey<>("ABBREVIATIONS", AbbreviationRepository::new);
28+
public static final DataKey<AbbreviationRepository> ABBREVIATIONS = new DataKey<>("ABBREVIATIONS", AbbreviationRepository::new);
2929

3030
/**
3131
* A {@link DataKey} that is used to set the behavior of the abbreviations repository when duplicates are defined. {@link KeepType}
3232
*/
33-
public final static DataKey<KeepType> ABBREVIATIONS_KEEP = new DataKey<>("ABBREVIATIONS_KEEP", KeepType.FIRST);
33+
public static final DataKey<KeepType> ABBREVIATIONS_KEEP = new DataKey<>("ABBREVIATIONS_KEEP", KeepType.FIRST);
3434

3535
/**
3636
* A {@link DataKey} that is used to set the use links option when true, default is false and abbr tag will be used in the rendered HTML.
3737
*/
38-
public final static DataKey<Boolean> USE_LINKS = new DataKey<>("USE_LINKS", false);
38+
public static final DataKey<Boolean> USE_LINKS = new DataKey<>("USE_LINKS", false);
3939

4040
public static Extension create() {
4141
return new AbbreviationExtension();

‎flexmark-ext-abbreviation/src/main/java/com/vladsch/flexmark/ext/abbreviation/AbbreviationVisitor.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,6 @@ static <V extends AbbreviationVisitor> VisitHandler<?>[] VISIT_HANDLERS(V visito
2525
};
2626
}
2727

28-
void visit(AbbreviationBlock node);
29-
void visit(Abbreviation node);
28+
void visit(final AbbreviationBlock node);
29+
void visit(final Abbreviation node);
3030
}

‎flexmark-ext-abbreviation/src/main/java/com/vladsch/flexmark/ext/abbreviation/internal/AbbreviationPostProcessor.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,13 +21,13 @@
2121
public class AbbreviationPostProcessor extends DocumentPostProcessor {
2222
private Pattern abbreviations = null;
2323
private HashMap<String, String> abbreviationMap = null;
24-
final private NodeVisitor myVisitor;
24+
private final NodeVisitor myVisitor;
2525

2626
AbbreviationPostProcessor(Document document) {
2727
myVisitor = new NodeVisitor(
2828
new VisitHandler<>(Text.class, AbbreviationPostProcessor.this::visit)
2929
);
30-
30+
3131
AbbreviationRepository abbrRepository = document.get(AbbreviationExtension.ABBREVIATIONS);
3232

3333
if (!abbrRepository.isEmpty()) {

‎flexmark-ext-anchorlink/src/main/java/com/vladsch/flexmark/ext/anchorlink/AnchorLinkExtension.java‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -19,13 +19,13 @@
1919
* </p>
2020
*/
2121
public class AnchorLinkExtension implements Parser.ParserExtension, HtmlRenderer.HtmlRendererExtension {
22-
final public static DataKey<Boolean> ANCHORLINKS_WRAP_TEXT = new DataKey<>("ANCHORLINKS_WRAP_TEXT", true);
23-
final public static DataKey<String> ANCHORLINKS_TEXT_PREFIX = new DataKey<>("ANCHORLINKS_TEXT_PREFIX", "");
24-
final public static DataKey<String> ANCHORLINKS_TEXT_SUFFIX = new DataKey<>("ANCHORLINKS_TEXT_SUFFIX", "");
25-
final public static DataKey<String> ANCHORLINKS_ANCHOR_CLASS = new DataKey<>("ANCHORLINKS_ANCHOR_CLASS", "");
26-
final public static DataKey<Boolean> ANCHORLINKS_SET_NAME = new DataKey<>("ANCHORLINKS_SET_NAME", false);
27-
final public static DataKey<Boolean> ANCHORLINKS_SET_ID = new DataKey<>("ANCHORLINKS_SET_ID", true);
28-
final public static DataKey<Boolean> ANCHORLINKS_NO_BLOCK_QUOTE = new DataKey<>("ANCHORLINKS_NO_BLOCK_QUOTE", false);
22+
public static final DataKey<Boolean> ANCHORLINKS_WRAP_TEXT = new DataKey<>("ANCHORLINKS_WRAP_TEXT", true);
23+
public static final DataKey<String> ANCHORLINKS_TEXT_PREFIX = new DataKey<>("ANCHORLINKS_TEXT_PREFIX", "");
24+
public static final DataKey<String> ANCHORLINKS_TEXT_SUFFIX = new DataKey<>("ANCHORLINKS_TEXT_SUFFIX", "");
25+
public static final DataKey<String> ANCHORLINKS_ANCHOR_CLASS = new DataKey<>("ANCHORLINKS_ANCHOR_CLASS", "");
26+
public static final DataKey<Boolean> ANCHORLINKS_SET_NAME = new DataKey<>("ANCHORLINKS_SET_NAME", false);
27+
public static final DataKey<Boolean> ANCHORLINKS_SET_ID = new DataKey<>("ANCHORLINKS_SET_ID", true);
28+
public static final DataKey<Boolean> ANCHORLINKS_NO_BLOCK_QUOTE = new DataKey<>("ANCHORLINKS_NO_BLOCK_QUOTE", false);
2929

3030
private AnchorLinkExtension() {
3131
}

0 commit comments

Comments
 (0)