Core Block Tokens

These block tokens are defined in the CommonMark specification.

Token

Description

Example

(ATX)Heading

Level 1-6 headings, denoted by number of #

 
### Heading level 3

SetextHeading

Underlined header (using multiple = or -)

 
Header
======

Paragraph

General block of text

 
any *text*

Quote

quoted block of text

 
> this is a quote

BlockCode

indented text (4 spaces or a tab)

 
    included as literal

CodeFence

enclosed in 3 or more backticks with an optional language name

 
```python
print('this is python')
```

ThematicBreak

Creates a horizontal line in the output

 
---

List

bullet points or enumerated.

 
- item
  - nested item
1. numbered item

LinkDefinition

A substitution for an inline link, which can have a reference target (no spaces), and an optional title (in “)

 
[key]: https://www.google.com "a title"

HTMLBlock

Any valid HTML (rendered in HTML output only)

 
<p>some text</p>

Document

class mistletoe.block_tokens.Document(*, children: List[mistletoe.base_elements.Token], link_definitions: dict = NOTHING, footnotes: Dict[str, mistletoe.base_elements.Token] = NOTHING, footref_order: list = NOTHING, front_matter: Optional[mistletoe.block_tokens.FrontMatter] = None)[source]

Bases: mistletoe.base_elements.BlockToken

Document container.

Parameters

  • children (List[mistletoe.base_elements.Token]) – Child tokens list

  • link_definitions (dict) – Mapping of keys to (url, title) (default: Factory(factory=<class ‘dict’>, takes_self=False))

  • footnotes (Dict[str, mistletoe.base_elements.Token]) – Footnote tokens mapped to their target names (default: Factory(factory=<class ‘dict’>, takes_self=False))

  • footref_order (list) – A set of footnote targets, in the order they are referenced in the document. (default: Factory(factory=<class ‘list’>, takes_self=False))

  • front_matter (mistletoe.block_tokens.FrontMatter, NoneType) – Front matter YAML block (default: None)

classmethod read(lines: Union[str, List[str], mistletoe.base_elements.SourceLines], reset_definitions: bool = True, skip_tokens: list = ('LinkDefinition', 'Footnote'), front_matter: bool = False)[source]

Read a document

Parameters

  • lines – Lines to parse

  • reset_definitions – remove any previously stored definitions in the global context (see ParseContext.reset_definitions()).

  • skip_tokens – do not store these token.name in the syntax tree. These are usually tokens that store themselves in the global context.

  • front_matter – search for an initial YAML block front matter block (note this is not strictly CommonMark compliant)

Atx Heading

class mistletoe.block_tokens.Heading(*, level: int, children: List[mistletoe.base_elements.Token], position: mistletoe.base_elements.Position = None)[source]

Bases: mistletoe.base_elements.BlockToken

Heading token. ([“### some heading ###n”])

Boundary between span-level and block-level tokens.

Parameters
classmethod start(line)[source]

Takes a line from the document as argument, and returns a boolean representing whether that line marks the start of the current token. Every subclass of BlockToken must define a start function (see block_tokenizer.tokenize_main).

classmethod read(lines, expand_spans=False)[source]

takes the rest of the lines in the document as an iterator (including the start line), and consumes all the lines that should be read into this token.

The default is to stop at an empty line.

Setext Heading

class mistletoe.block_tokens.SetextHeading(*, level: int, children: List[mistletoe.base_elements.Token], position: mistletoe.base_elements.Position = None)[source]

Bases: mistletoe.base_elements.BlockToken

Setext headings.

Not included in the parsing process, but returned by Paragraph.read.

Parameters
classmethod start(line)[source]

Takes a line from the document as argument, and returns a boolean representing whether that line marks the start of the current token. Every subclass of BlockToken must define a start function (see block_tokenizer.tokenize_main).

classmethod read(lines)[source]

takes the rest of the lines in the document as an iterator (including the start line), and consumes all the lines that should be read into this token.

The default is to stop at an empty line.

Paragraph

class mistletoe.block_tokens.Paragraph(*, children: List[mistletoe.base_elements.Token], position: mistletoe.base_elements.Position = None)[source]

Bases: mistletoe.base_elements.BlockToken

Paragraph token. ([“somen”, “continuousn”, “linesn”])

Boundary between span-level and block-level tokens.

Parameters
static start(line)[source]

Takes a line from the document as argument, and returns a boolean representing whether that line marks the start of the current token. Every subclass of BlockToken must define a start function (see block_tokenizer.tokenize_main).

classmethod read(lines, expand_spans=False)[source]

takes the rest of the lines in the document as an iterator (including the start line), and consumes all the lines that should be read into this token.

The default is to stop at an empty line.

Quote

class mistletoe.block_tokens.Quote(*, children: List[mistletoe.base_elements.Token], position: mistletoe.base_elements.Position = None)[source]

Bases: mistletoe.base_elements.BlockToken

Quote token. ([“> # headingn”, “> paragraphn”]).

Parameters
static start(line)[source]

Takes a line from the document as argument, and returns a boolean representing whether that line marks the start of the current token. Every subclass of BlockToken must define a start function (see block_tokenizer.tokenize_main).

classmethod read(lines)[source]

takes the rest of the lines in the document as an iterator (including the start line), and consumes all the lines that should be read into this token.

The default is to stop at an empty line.

BlockCode

class mistletoe.block_tokens.BlockCode(*, children: List[mistletoe.base_elements.Token], language: str = '', position: mistletoe.base_elements.Position = None)[source]

Bases: mistletoe.base_elements.BlockToken

Indented code.

Parameters
static start(line)[source]

Takes a line from the document as argument, and returns a boolean representing whether that line marks the start of the current token. Every subclass of BlockToken must define a start function (see block_tokenizer.tokenize_main).

classmethod read(lines)[source]

takes the rest of the lines in the document as an iterator (including the start line), and consumes all the lines that should be read into this token.

The default is to stop at an empty line.

CodeFence

class mistletoe.block_tokens.CodeFence(*, children: List[mistletoe.base_elements.Token], language: str = '', arguments: str = '', position: mistletoe.base_elements.Position = None)[source]

Bases: mistletoe.base_elements.BlockToken

Code fence. ([“`sh\n", "rm -rf /", ..., "`”])

Boundary between span-level and block-level tokens.

See <https://spec.commonmark.org/0.29/#fenced-code-blocks>

Parameters
classmethod start(line)[source]

Takes a line from the document as argument, and returns a boolean representing whether that line marks the start of the current token. Every subclass of BlockToken must define a start function (see block_tokenizer.tokenize_main).

classmethod read(lines)[source]

takes the rest of the lines in the document as an iterator (including the start line), and consumes all the lines that should be read into this token.

The default is to stop at an empty line.

List

class mistletoe.block_tokens.List(*, children: List[mistletoe.base_elements.Token], loose: bool, start_at: Optional[int], position: mistletoe.base_elements.Position = None)[source]

Bases: mistletoe.base_elements.BlockToken

List token (unordered or ordered)

Parameters
classmethod start(line)[source]

Takes a line from the document as argument, and returns a boolean representing whether that line marks the start of the current token. Every subclass of BlockToken must define a start function (see block_tokenizer.tokenize_main).

classmethod read(lines)[source]

takes the rest of the lines in the document as an iterator (including the start line), and consumes all the lines that should be read into this token.

The default is to stop at an empty line.

ListItem

class mistletoe.block_tokens.ListItem(*, children: List[mistletoe.base_elements.Token], loose: bool, leader: str, prepend: int, next_marker=None, position: mistletoe.base_elements.Position = None)[source]

Bases: mistletoe.base_elements.BlockToken

List items.

Not included in the parsing process, but called by List.

Parameters
classmethod parse_marker(line)[source]

Returns a pair (prepend, leader) if the line has a valid leader.

classmethod read(lines, prev_marker=None)[source]

takes the rest of the lines in the document as an iterator (including the start line), and consumes all the lines that should be read into this token.

The default is to stop at an empty line.

LinkDefinition

class mistletoe.block_tokens.LinkDefinition(*, definitions: list, position: mistletoe.base_elements.Position = None)[source]

Bases: mistletoe.base_elements.BlockToken

LinkDefinition token: [ref]: url “title”

These are stores in Document.link_definitions in the final syntax tree.

Parameters
classmethod start(line)[source]

Takes a line from the document as argument, and returns a boolean representing whether that line marks the start of the current token. Every subclass of BlockToken must define a start function (see block_tokenizer.tokenize_main).

classmethod read(lines: mistletoe.base_elements.SourceLines)[source]

takes the rest of the lines in the document as an iterator (including the start line), and consumes all the lines that should be read into this token.

The default is to stop at an empty line.

ThematicBreak

class mistletoe.block_tokens.ThematicBreak(*, position: mistletoe.base_elements.Position = None)[source]

Bases: mistletoe.base_elements.BlockToken

Thematic break token (a.k.a. horizontal rule.)

Parameters

position (mistletoe.base_elements.Position) – Line position in source text (default: None)

classmethod start(line)[source]

Takes a line from the document as argument, and returns a boolean representing whether that line marks the start of the current token. Every subclass of BlockToken must define a start function (see block_tokenizer.tokenize_main).

classmethod read(lines)[source]

takes the rest of the lines in the document as an iterator (including the start line), and consumes all the lines that should be read into this token.

The default is to stop at an empty line.

HTMLBlock

class mistletoe.block_tokens.HTMLBlock(*, content: str, position: mistletoe.base_elements.Position = None)[source]

Bases: mistletoe.base_elements.BlockToken

Block-level HTML token.

Parameters
classmethod start(line)[source]

Takes a line from the document as argument, and returns a boolean representing whether that line marks the start of the current token. Every subclass of BlockToken must define a start function (see block_tokenizer.tokenize_main).

classmethod read(lines)[source]

takes the rest of the lines in the document as an iterator (including the start line), and consumes all the lines that should be read into this token.

The default is to stop at an empty line.

Base Elements Core Span Tokens