Using markdocs to Python documentation (markdown) - Idea - WIP

markdocs_idea.md

UNDER DEVELOPMENT HERE: https://github.com/rochacbruno/markdocs

---

Python documentation

I still think that the problem of lack of good documentation in the Python ecosystem is also related to .rst format and the use of complicated tools like Sphinx.

I'm enjoying writing functional documentation using Markdown in Rustlang, so I'll do experiments to have the same functionality in Python. Take a look at rustdoc and here an example of documentation site generated for a Rust crate using markdown comments.

Information is extracted using python -m tokenize file.py https://docs.python.org/3.5/library/tokenize.html#examples

Markdown

How it works

NOTE: this is just an early stage idea, not implemented yet! if you like please comment.

The Markdocs extracts documentation from all .py files and outputs in a well organized documentation html site which can use the mkdocs.org to expose and deploy.

markdocs /path/to/project

If you dont want to generate full documentation you can easily generate a readme file for your repo

markdocs /path/project --readme README.md -k 'filter-oly-some-files-and-objects'

With the above a README.md is generated including only the filtered files and objects documentation, but you can also generate a single README for your whole project.

All .py files on that folder will be parsed for documentation blocks which are Python multiline comments starting in ! example:

NOTE: if you don't like mixing code and documentation, you can use a mymodule.md to document mymodule.py and the .md should be located in the same folder or in mdocs folder of the project. You can also write separated object files like in mymodule.myclass.mymethod.md which will be linked only to the mymethod of MyClass.

    """!
    # this is a documentation written in markdown
    As it has only one `!` at the top, it is considered the module documentation
    I can include module documentation along the file and will be merged in to the top level documentation
    """
    
    from foo import bar
    
    """!!
    # This is an object documentation, can be used for any object but most for functions and classes
    It is defined before the object and not on the `__doc__` docstring, as markdocs does not conflicts with it.
    
    ## What are the advantages
    - Markdown is easy to learn
    - More people will contribute to documentation because they already know the format
    - With simple commands like `markdocs /path --readme README.md` the readme for your repo is generated from markdocs
    - Markdocs will generate the output for http://www.mkdocs.org/
    - You can write bare `.md` files in a `mdocs` folder and they will be added to you documentation as well
    
    [[params
      # X is the single param of this function
      x: int | default 0
      # The return is a string with the x interpolated.
    ]] result: str
    """
    def a_function(x=0):
      """This regular docstring does not conflicts with the above markdoc"""
      return f'Hello {x}'
      
    """!!
    # This is a class documentation
    We can also define runnable and highlighted blocks of code.
    ```run
    obj = MyClass()
    ```
    """
    class MyClass:
        """the class docstring is not affected"""
        attr = 'foo'
        """!!!
        # this is a method documentation
        [[params
           x: str
        ]]
        """
        def method(self, x):
            """This is the regular docstring for method"""
            a = x
            """!!!!
            ## Here we increase the nesting level
            Markdown is amazing!
            """
            def inner_function(..):
                pass

As you can see the !! can be also used, in fact you can use as many !!!!! you want to define nesting.

Parser options are:

• https://github.com/miyuchina/mistletoe/
• https://github.com/lepture/mistune#lexers

Website output formats

• mkdocs.org
• https://github.com/rocadocs/rocadocs

test.py

#! /usr/bin/python
"""This is the regular module __doc__"""

"""!
# this is a documentation written in markdown
As it has only one `!` at the top, it is considered the module documentation
I can include module documentation along the file and will be merged in to the
top level documentation
"""

from os import path  # this comment is ignored

"""!!
# This is an object documentation
can be used for any object but most for functions and classes
It is defined before the object and not on the `__doc__` docstring, as markdocs
does not conflicts with it.

## What are the advantages
- Markdown is easy to learn
- More people will contribute to documentation because they already know the
  format
- With simple commands like `markdocs /path --readme README.md` the readme for
  your repo is generated from markdocs
- Markdocs will generate the output for http://www.mkdocs.org/
- You can write bare `.md` files in a `mdocs` folder and they will be added to
  you documentation as well

[[params
  # x is the single param of this function
  x: int | default 0
  # The return is a string with the x interpolated.
]] result: str
"""
@foo_bar  # this decorator is listed as part of function in docs (maybe linked)
def a_function(x=0):
    """This regular docstring does not conflicts with the above markdoc"""
    return f'Hello {x}'

"""!!
# This is a class documentation
We can also define runnable and highlighted blocks of code.
```run
obj = MyClass()
```
"""
class MyClass:
    """the class docstring is not affected"""
    attr = 'foo'
    """!!!
    # this is a method documentation
    [[params
      x: str
    ]]
    """
    @bla_bla_bla
    def method(self, x):
        """This is the regular docstring for method"""
        a = x
        """!!!!
        ## Here we increase the nesting level
        Markdown is amazing!
        """
        def inner_function(..):
            pass

    # This comment is ognored

"""!
> This is also the module documentation, will be included as a foot note
"""

tokenize_output.sh

# python3.6 -m tokenize -e test.py

0,0-0,0:            ENCODING       'utf-8'        
1,0-1,18:           COMMENT        '#! /usr/bin/python'
1,18-1,19:          NL             '\n'           
2,0-2,40:           STRING         '"""This is the regular module __doc__"""'
2,40-2,41:          NEWLINE        '\n'           
3,0-3,1:            NL             '\n'           
4,0-9,3:            STRING         '"""!\n# this is a documentation written in markdown\nAs it has only one `!` at the top, it is considered the module documentation\nI can include module documentation along the file and will be merged in to the\ntop level documentation\n"""'
9,3-9,4:            NEWLINE        '\n'           
10,0-10,1:          NL             '\n'           
11,0-11,4:          NAME           'from'         
11,5-11,7:          NAME           'os'           
11,8-11,14:         NAME           'import'       
11,15-11,19:        NAME           'path'         
11,21-11,46:        COMMENT        '# this comment is ignored'
11,46-11,47:        NEWLINE        '\n'           
12,0-12,1:          NL             '\n'           
13,0-34,3:          STRING         '"""!!\n# This is an object documentation\ncan be used for any object but most for functions and classes\nIt is defined before the object and not on the `__doc__` docstring, as markdocs\ndoes not conflicts with it.\n\n## What are the advantages\n- Markdown is easy to learn\n- More people will contribute to documentation because they already know the\n  format\n- With simple commands like `markdocs /path --readme README.md` the readme for\n  your repo is generated from markdocs\n- Markdocs will generate the output for http://www.mkdocs.org/\n- You can write bare `.md` files in a `mdocs` folder and they will be added to\n  you documentation as well\n\n[[params\n  # x is the single param of this function\n  x: int | default 0\n  # The return is a string with the x interpolated.\n]] result: str\n"""'
34,3-34,4:          NEWLINE        '\n'           
35,0-35,1:          AT             '@'            
35,1-35,8:          NAME           'foo_bar'      
35,10-35,79:        COMMENT        '# this decorator is listed as part of function in docs (maybe linked)'
35,79-35,80:        NEWLINE        '\n'           
36,0-36,3:          NAME           'def'          
36,4-36,14:         NAME           'a_function'   
36,14-36,15:        LPAR           '('            
36,15-36,16:        NAME           'x'            
36,16-36,17:        EQUAL          '='            
36,17-36,18:        NUMBER         '0'            
36,18-36,19:        RPAR           ')'            
36,19-36,20:        COLON          ':'            
36,20-36,21:        NEWLINE        '\n'           
37,0-37,4:          INDENT         '    '         
37,4-37,74:         STRING         '"""This regular docstring does not conflicts with the above markdoc"""'
37,74-37,75:        NEWLINE        '\n'           
38,4-38,10:         NAME           'return'       
38,11-38,23:        STRING         "f'Hello {x}'" 
38,23-38,24:        NEWLINE        '\n'           
39,0-39,1:          NL             '\n'           
40,0-40,0:          DEDENT         ''             
40,0-46,3:          STRING         '"""!!\n# This is a class documentation\nWe can also define runnable and highlighted blocks of code.\n```run\nobj = MyClass()\n```\n"""'
46,3-46,4:          NEWLINE        '\n'           
47,0-47,5:          NAME           'class'        
47,6-47,13:         NAME           'MyClass'      
47,13-47,14:        COLON          ':'            
47,14-47,15:        NEWLINE        '\n'           
48,0-48,4:          INDENT         '    '         
48,4-48,45:         STRING         '"""the class docstring is not affected"""'
48,45-48,46:        NEWLINE        '\n'           
49,4-49,8:          NAME           'attr'         
49,9-49,10:         EQUAL          '='            
49,11-49,16:        STRING         "'foo'"        
49,16-49,17:        NEWLINE        '\n'           
50,4-55,7:          STRING         '"""!!!\n    # this is a method documentation\n    [[params\n      x: str\n    ]]\n    """'
55,7-55,8:          NEWLINE        '\n'           
56,4-56,5:          AT             '@'            
56,5-56,16:         NAME           'bla_bla_bla'  
56,16-56,17:        NEWLINE        '\n'           
57,4-57,7:          NAME           'def'          
57,8-57,14:         NAME           'method'       
57,14-57,15:        LPAR           '('            
57,15-57,19:        NAME           'self'         
57,19-57,20:        COMMA          ','            
57,21-57,22:        NAME           'x'            
57,22-57,23:        RPAR           ')'            
57,23-57,24:        COLON          ':'            
57,24-57,25:        NEWLINE        '\n'           
58,0-58,8:          INDENT         '        '     
58,8-58,54:         STRING         '"""This is the regular docstring for method"""'
58,54-58,55:        NEWLINE        '\n'           
59,8-59,9:          NAME           'a'            
59,10-59,11:        EQUAL          '='            
59,12-59,13:        NAME           'x'            
59,13-59,14:        NEWLINE        '\n'           
60,8-63,11:         STRING         '"""!!!!\n        ## Here we increase the nesting level\n        Markdown is amazing!\n        """'
63,11-63,12:        NEWLINE        '\n'           
64,8-64,11:         NAME           'def'          
64,12-64,26:        NAME           'inner_function'
64,26-64,27:        LPAR           '('            
64,27-64,28:        DOT            '.'            
64,28-64,29:        DOT            '.'            
64,29-64,30:        RPAR           ')'            
64,30-64,31:        COLON          ':'            
64,31-64,32:        NEWLINE        '\n'           
65,0-65,12:         INDENT         '            ' 
65,12-65,16:        NAME           'pass'         
65,16-65,17:        NEWLINE        '\n'           
66,0-66,1:          NL             '\n'           
67,4-67,29:         COMMENT        '# This comment is ognored'
67,29-67,30:        NL             '\n'           
68,0-68,1:          NL             '\n'           
69,0-69,0:          DEDENT         ''             
69,0-69,0:          DEDENT         ''             
69,0-69,0:          DEDENT         ''             
69,0-71,3:          STRING         '"""!\n> This is also the module documentation, will be included as a foot note\n"""'
71,3-71,4:          NEWLINE        '\n'           
72,0-72,0:          ENDMARKER      ''             

Why do you think it's important to not conflict with __doc__'s content? I don't see a use case to mix the two, either people will use rst and sphinx, or markdown and whatever-the-tool-you-build. And if you did put the markdown docs in the __doc__ attribute, it would simplify the documentation extraction.

Also, some IDEs and linters might take issue with all those docstrings hanging around, not attached to anything (from the linter point of view)

@tonnydourado there are some projects which checks the documentation using make test-docs and also extracts test meta data from __doc__ there is also some projects using doctests.

reference: pypi/warehouse#869

pypa/readme_renderer#51