pawelsikora / mkdocs-with-confluence Goto Github PK

View Code? Open in Web Editor NEW

67.0 67.0 33.0 26 KB

MkDocs plugin for uploading markdown documentation to Confluence via Confluence REST API

License: MIT License

Python 100.00%

mkdocs-with-confluence's People

Contributors

Stargazers

Watchers

mkdocs-with-confluence's Issues

Submission to Confluence fails with error

Hello,

my upload to confluence API fails with error for a certain page in my set of markdown files, so I suspect there is a certain md content that is not properly converted. Is there a way to increase verbosity of the mkdocs run in order to get more details?

Here is what I have so far:

...
[2023-06-01T10:48:14.399Z] ERROR    -  Error reading page 'outbound_pipeline.md': 400 Client Error:  for url: https://XXXXXXXXX.net/confluence/rest/api/content/
...
[2023-06-01T10:48:14.400Z] INFO    -   * Mkdocs With Confluence: Using the Pipeline - *UPDATE*
[2023-06-01T10:48:14.400Z] INFO    -   * Mkdocs With Confluence: Using the Pipeline for DDP - *NEW PAGE*
[2023-06-01T10:48:14.400Z] Trying to ADD page 'Using the Pipeline for DDP' to parent0(FTP Links Automaton) ID: 2999877739
[2023-06-01T10:48:14.400Z] INFO    -   * Mkdocs With Confluence: The MFT Pipeline - *NEW PAGE*
[2023-06-01T10:48:14.400Z] Traceback (most recent call last):
[2023-06-01T10:48:14.400Z]   File "/usr/local/bin/mkdocs", line 8, in <module>
[2023-06-01T10:48:14.400Z]     sys.exit(cli())
[2023-06-01T10:48:14.400Z]              ^^^^^
[2023-06-01T10:48:14.400Z]   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
[2023-06-01T10:48:14.400Z]     return self.main(*args, **kwargs)
[2023-06-01T10:48:14.400Z]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
[2023-06-01T10:48:14.400Z]   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1055, in main
[2023-06-01T10:48:14.400Z]     rv = self.invoke(ctx)
[2023-06-01T10:48:14.400Z]          ^^^^^^^^^^^^^^^^
[2023-06-01T10:48:14.400Z]   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
[2023-06-01T10:48:14.400Z]     return _process_result(sub_ctx.command.invoke(sub_ctx))
[2023-06-01T10:48:14.400Z]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2023-06-01T10:48:14.400Z]   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
[2023-06-01T10:48:14.400Z]     return ctx.invoke(self.callback, **ctx.params)
[2023-06-01T10:48:14.400Z]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2023-06-01T10:48:14.400Z]   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 760, in invoke
[2023-06-01T10:48:14.400Z]     return __callback(*args, **kwargs)
[2023-06-01T10:48:14.400Z]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2023-06-01T10:48:14.400Z]   File "/usr/local/lib/python3.11/site-packages/mkdocs/__main__.py", line 250, in build_command
[2023-06-01T10:48:14.400Z]     build.build(cfg, dirty=not clean)
[2023-06-01T10:48:14.400Z]   File "/usr/local/lib/python3.11/site-packages/mkdocs/commands/build.py", line 308, in build
[2023-06-01T10:48:14.400Z]     _populate_page(file.page, config, files, dirty)
[2023-06-01T10:48:14.400Z]   File "/usr/local/lib/python3.11/site-packages/mkdocs/commands/build.py", line 177, in _populate_page
[2023-06-01T10:48:14.400Z]     page.markdown = config.plugins.run_event(
[2023-06-01T10:48:14.400Z]                     ^^^^^^^^^^^^^^^^^^^^^^^^^
[2023-06-01T10:48:14.400Z]   File "/usr/local/lib/python3.11/site-packages/mkdocs/plugins.py", line 520, in run_event
[2023-06-01T10:48:14.400Z]     result = method(item, **kwargs)
[2023-06-01T10:48:14.400Z]              ^^^^^^^^^^^^^^^^^^^^^^
[2023-06-01T10:48:14.400Z]   File "/usr/local/lib/python3.11/site-packages/mkdocs_with_confluence/plugin.py", line 334, in on_page_markdown
[2023-06-01T10:48:14.400Z]     self.add_page(page.title, parent_id, confluence_body)
[2023-06-01T10:48:14.400Z]   File "/usr/local/lib/python3.11/site-packages/mkdocs_with_confluence/plugin.py", line 541, in add_page
[2023-06-01T10:48:14.400Z]     r.raise_for_status()
[2023-06-01T10:48:14.400Z]   File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
[2023-06-01T10:48:14.400Z]     raise HTTPError(http_error_msg, response=self)
[2023-06-01T10:48:14.400Z] requests.exceptions.HTTPError: 400 Client Error:  for url: https://XXXXXXXX.net/confluence/rest/api/content/

I think the issue might be related to the content inside the md page I am submitting, but how can I know if I cannot see the error message coming with the HTTP 400 from the Confluence space.

Hence the first request would be: Is there a way to increase verbosity of the logs?
Second request: Is there a document that described the 'sensitive' characaters that should not appear on the .md page when attempting to convert to HTML with mkdocs?

Thanks for your consideration!
Warm regards

Carsten

Interaction with other plug-ins?

First, thanks a lot for publishing this plugin, it does work with confluence quickly and with no trouble at all.

My challenge is how to get it to interact with other mkdocs plugins for some more advanced rendering. When generating the confluence pages, it seems that the underlying renderer (md2cf/mistune) does not take them into account.

I am using this in the mkdocs.yml file:

[...]
# See https://squidfunk.github.io/mkdocs-material/reference/code-blocks/
theme:
  name: material
  features:
    - content.code.annotate

markdown_extensions:
  - tables
  - def_list
  - admonition
  - toc
  - markdown_inline_graphviz
  - plantuml_markdown
  - mdx_truly_sane_lists
# See https://squidfunk.github.io/mkdocs-material/reference/code-blocks/
  - pymdownx.highlight:
      anchor_linenums: true
      line_spans: __span
      pygments_lang_class: true
  - pymdownx.inlinehilite
  - pymdownx.superfences
  - pymdownx.smartsymbols
  - pymdownx.arithmatex:
      generic: true
  - pymdownx.snippets:
      check_paths: true

plugins:
  - monorepo
  - search
  # https://github.com/pawelsikora/mkdocs-with-confluence
  - mkdocs-with-confluence:
      host_url: https://flexport.atlassian.net/wiki/rest/api/content
      space: !ENV CONFLUENCE_SPACE
      parent_page_name: !ENV CONFLUENCE_PARENT_PAGE
      username: !ENV CONFLUENCE_USER
      password: !ENV CONFLUENCE_TOKEN
      enabled_if_env: MKDOCS_TO_CONFLUENCE
      verbose: false
      debug: false
      dryrun: false

And the rendered pages don't render correctly at least:

plantuml
snippets
admonitions
math (MathJax)

Which are all rendered O.K. when generating an html site.

Any ideas would be very appreciated.

AttributeError: 'MkdocsWithConfluence' object has no attribute 'dryrun'

First off, thank you for writing this plugin. This is going to help me greatly.

Unfortunately when trying to get this plugin running I am encountering the following error:

ERROR    -  Error reading page 'index.md': 'MkdocsWithConfluence' object has no attribute 'dryrun'
Traceback (most recent call last):
  File "/Users/theD1360/Library/Caches/pypoetry/virtualenvs/docs-project-A5zrhKJ8-py3.9/bin/mkdocs", line 8, in <module>
    sys.exit(cli())
  File "/Users/theD1360/Library/Caches/pypoetry/virtualenvs/docs-project-A5zrhKJ8-py3.9/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/theD1360/Library/Caches/pypoetry/virtualenvs/docs-project-A5zrhKJ8-py3.9/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/theD1360/Library/Caches/pypoetry/virtualenvs/docs-project-A5zrhKJ8-py3.9/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/theD1360/Library/Caches/pypoetry/virtualenvs/docs-project-A5zrhKJ8-py3.9/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/theD1360/Library/Caches/pypoetry/virtualenvs/docs-project-A5zrhKJ8-py3.9/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/theD1360/Library/Caches/pypoetry/virtualenvs/docs-project-A5zrhKJ8-py3.9/lib/python3.9/site-packages/mkdocs/__main__.py", line 250, in build_command
    build.build(cfg, dirty=not clean)
  File "/Users/theD1360/Library/Caches/pypoetry/virtualenvs/docs-project-A5zrhKJ8-py3.9/lib/python3.9/site-packages/mkdocs/commands/build.py", line 308, in build
    _populate_page(file.page, config, files, dirty)
  File "/Users/theD1360/Library/Caches/pypoetry/virtualenvs/docs-project-A5zrhKJ8-py3.9/lib/python3.9/site-packages/mkdocs/commands/build.py", line 177, in _populate_page
    page.markdown = config.plugins.run_event(
  File "/Users/theD1360/Library/Caches/pypoetry/virtualenvs/docs-project-A5zrhKJ8-py3.9/lib/python3.9/site-packages/mkdocs/plugins.py", line 520, in run_event
    result = method(item, **kwargs)
  File "/Users/theD1360/Library/Caches/pypoetry/virtualenvs/docs-project-A5zrhKJ8-py3.9/lib/python3.9/site-packages/mkdocs_with_confluence/plugin.py", line 274, in on_page_markdown
    self.update_page(page.title, confluence_body)
  File "/Users/theD1360/Library/Caches/pypoetry/virtualenvs/docs-project-A5zrhKJ8-py3.9/lib/python3.9/site-packages/mkdocs_with_confluence/plugin.py", line 571, in update_page
    if not self.dryrun:
AttributeError: 'MkdocsWithConfluence' object has no attribute 'dryrun'

Environment:

Software	Version
python	3.9
poetry	1.4.0
mkdocs	^1.4.2
mkdocs-material	^9.1.5
mkdocs-with-confluence	^0.2.7

mkdocs.yaml

plugins:
  - mkdocs-with-confluence:
      host_url: https://<confluence domain>/wiki/rest/api/content
      space: ""
      parent_page_name: Qualification Engine (Qual-E)
      username: [email protected]
      password: mypassword
      dryrun: true
      verbose: true
      debug: true

One interesting thing to note is that the error seems to be limited to just that config key.
All other config items seem to be loaded properly. It seems the self.dryrun property is not getting set properly in
plugin.py.

plugin.py Line 141

 if self.config["dryrun"]:
            print("WARNING -  Mkdocs With Confluence - DRYRUN MODE turned ON")
            self.dryrun = True
        else:
            self.dryrun = False

Cannot build minimal example

I have tried to publish the minimal example provided by mkdocs new . to confluence using mkdocs-with-confluence without success:

$ WITH_CONFLUENCE=1 mkdocs build
INFO    -  Mkdocs With Confluence: Exporting MKDOCS pages to Confluence turned ON by var {env_name}==1!
INFO     -  Cleaning site directory
INFO     -  Building documentation to directory: /Users/mkj/Developer/playgrounds/mkdocs/mkdocs-confluence-examples/site
Number of Files: 1
Traceback (most recent call last):
  File "/Users/mkj/.pyenv/versions/mkdocs-with-confluence/bin/mkdocs", line 8, in <module>
    sys.exit(cli())
  File "/Users/mkj/.pyenv/versions/3.9.7/envs/mkdocs-with-confluence/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/mkj/.pyenv/versions/3.9.7/envs/mkdocs-with-confluence/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/mkj/.pyenv/versions/3.9.7/envs/mkdocs-with-confluence/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/mkj/.pyenv/versions/3.9.7/envs/mkdocs-with-confluence/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/mkj/.pyenv/versions/3.9.7/envs/mkdocs-with-confluence/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/mkj/.pyenv/versions/3.9.7/envs/mkdocs-with-confluence/lib/python3.9/site-packages/mkdocs/__main__.py", line 187, in build_command
    build.build(config.load_config(**kwargs), dirty=not clean)
  File "/Users/mkj/.pyenv/versions/3.9.7/envs/mkdocs-with-confluence/lib/python3.9/site-packages/mkdocs/commands/build.py", line 287, in build
    nav = config['plugins'].run_event('nav', nav, config=config, files=files)
  File "/Users/mkj/.pyenv/versions/3.9.7/envs/mkdocs-with-confluence/lib/python3.9/site-packages/mkdocs/plugins.py", line 102, in run_event
    result = method(item, **kwargs)
  File "/Users/mkj/.pyenv/versions/3.9.7/envs/mkdocs-with-confluence/lib/python3.9/site-packages/mkdocs_with_confluence/plugin.py", line 45, in on_nav
    p = spaces + self.__get_page_title(n)
  File "/Users/mkj/.pyenv/versions/3.9.7/envs/mkdocs-with-confluence/lib/python3.9/site-packages/mkdocs_with_confluence/plugin.py", line 274, in __get_page_title
    return re.search("\\s*Page\\(title='(.*)',", section).group(1)
AttributeError: 'NoneType' object has no attribute 'group'

mkdocs.yml:

site_name: Publish from MkDocs to Confluence
plugins:
  - search
  - mkdocs-with-confluence:
      host_url: !ENV [HOST, "n/a"]
      space: !ENV [SPACE, "foo"]
      parent_page_name: !ENV [PARENT, "foo"]
      username: !ENV [USER, "foo"]
      password: !ENV [PW, "foo"]
      enabled_if_env: WITH_CONFLUENCE

Environment variables have been tested and work properly (using confluence api directly).

This happend with the following setup:

$ pip list
Package                Version
---------------------- ---------
certifi                2021.10.8
charset-normalizer     2.0.7
click                  8.0.3
colorama               0.4.4
formats                0.1.1
ghp-import             2.0.2
httpretty              1.1.4
idna                   3.3
importlib-metadata     4.8.1
Jinja2                 3.0.2
Markdown               3.3.4
MarkupSafe             2.0.1
md2cf                  1.0.3
mergedeep              1.3.4
mistune                0.8.4
mkdocs                 1.2.3
mkdocs-with-confluence 0.2.2
packaging              21.0
pip                    21.3.1
pyparsing              3.0.1
python-dateutil        2.8.2
PyYAML                 5.3.1
pyyaml_env_tag         0.1
requests               2.26.0
setuptools             57.4.0
six                    1.16.0
tortilla               0.5.0
urllib3                1.26.7
watchdog               2.1.6
zipp                   3.6.0

Unable to generate subpages

Prerequisites

Hi, I have generated a documentation with the structure presented in the image below:

When I try to uplopad the documentation to Confluence, only Home, Architecture, and Installation are getting published. Nothing under the Stacks subpage gets uploaded to Confluence as seen in the image below:

Config

This is my mkdocs.yml configuration file:

site_name: Monitoring
theme:
  name: windmill
plugins:
  - search
  - mkdocs-with-confluence:
          host_url: https://confluencebt.host/rest/api/content
          space: CONFLUENCE_SPACE
          parent_page_name: Monitorizare
          username: confluence_user
          password: confluence_password
          enabled_if_env: MKDOCS_TO_CONFLUENCE
          verbose: true
          debug: true
          dryrun: false
nav:
  - Home: "index.md"
  # - Introduction: "index.md"
  - Architecture: "architecture.md"
  - Installation: "installation.md"
  - Stacks:
    - Metrics: 
      - Architecture: "stacks/metrics/architecture.md"
      - Installation and Operation: "playbooks/metrics/README.md"
    - Logs:
      - Architecture: "stacks/logs/architecture.md"
      - Installation and Operation: "playbooks/logs/README.md"
    - Traces:
      - Architecture: "stacks/traces/architecture.md"
      - Installation and Operation: "playbooks/traces/README.md"

Log entry

INFO    -  Mkdocs With Confluence: Exporting MKDOCS pages to Confluence turned ON by var MKDOCS_TO_CONFLUENCE==1!
Number of Files in directory tree: 10
DEBUG    - SECTION title: Section(title='Stacks')
DEBUG    - SECTION title:     Section(title='Metrics')
DEBUG    - SECTION title:     Section(title='Logs')
DEBUG    - SECTION title:     Section(title='Traces')

DEBUG    - Handling Page 'Home' (And Parent Nav Pages if necessary):

DEBUG    - Get section first parent title...: 
DEBUG    - WRN(list index out of range): No first parent! Assuming DEBUG    - Monitorizare...
DEBUG    - None
DEBUG    - Get section second parent title...: 
DEBUG    - ERR(list index out of range) No second parent! Assuming second parent is main parent: Monitorizare...
Monitorizare
DEBUG    - ONLY ONE PARENT FOUND. ASSUMING AS A FIRST NODE after main parent config Monitorizare
DEBUG    - PARENT0: Monitorizare, PARENT1: Monitorizare, MAIN PARENT: Monitorizare

#################################################

INFO    -   * Mkdocs With Confluence: Find Page ID: PAGE NAME: Monitorizare
URL: https://confluencebt.host/rest/api/content?title=Monitorizare&spaceKey=CONFLUENCE_SPACE&expand=history
ID: 129549192
DEBUG    - JUST ONE STEP FROM UPDATE OF PAGE 'Monitorizare' 
DEBUG    - CHECKING IF PARENT PAGE ON CONFLUENCE IS THE SAME AS HERE
INFO    -   * Mkdocs With Confluence: Find PARENT OF PAGE, PAGE NAME: Monitorizare
INFO    -   * Mkdocs With Confluence: Find Page ID: PAGE NAME: Monitorizare
URL: https://confluencebt.host/rest/api/content?title=Monitorizare&spaceKey=CONFLUENCE_SPACE&expand=history
ID: 129549192
PARENT NAME: Documentation
DEBUG    - ERR, Parents does not match: 'Monitorizare' =/= 'Documentation' Aborting...

DEBUG    - Handling Page 'Installation and Operation' (And Parent Nav Pages if necessary):

DEBUG    - Get section first parent title...: 
DEBUG    - SECTION title: Section(title='Logs')
DEBUG    - Logs
DEBUG    - Get section second parent title...: 
DEBUG    - SECTION title: Section(title='Stacks')
Logs
DEBUG    - PARENT0: Logs, PARENT1: Stacks, MAIN PARENT: Monitorizare

Question

How can I change the configuration so the plugin will create the required subpages?

Collaboration and password security - do not embed username and password

This seems like a dream come true, but there are problems here with usability and password security. I may not store my password in clear-text in the mkdocs.yml file. Since a piece of code is not my personal possession but something I wish to share with a team, I would also not want to place my username in the mkdocs.yml file.

upload via md2cf

Hello, what about using md2cf APIs also for uploading to confluence? It's an alive project, it would eliminate lot of duplicated efforts and expose some neat feature like the on_changed machinery which only upload changed stuff.

I think that would be a great addition to your super cool plugin! If you think we could discuss that I could definitely work on that, so what do you think?

Thank you, regards

Packaging - modeling dependencies in setup.py and/or move to newer project builders

setup.py should declare the packages you depend on so that pip will install these.

Error: ModuleNotFoundError: No module named 'requests'

Dear @pawelsikora
First : thanks for your plugin.
I'm trying to use it with Gitlab CI/CD.

I've got some questions (some basic usage).
But, before, let me present my configuration

Current configuration I used (and led to my issue)

CI/CD : Gitlab
docker image use : image python:3.8-buster

Issue

When running, I've got the following pipeline error message

ModuleNotFoundError: No module named 'requests'

Complementary informations:

I've put my configuration for the plugin into the mkdocs.yml

Is is the good place to put the configuration ?
Why the "import requests" from the python plugin is not working ? Is it necessary for me to update my docker image with this dependancy ?
For password with special character : should I put it with some mask ? (refer : https://docs.gitlab.com/ee/ci/variables/index.html#mask-a-cicd-variable) or with quote ?

plugins:
  - search:
      lang:
        - fr
        - en
  - macros
  - mkdocs-confluence: #For conversion of mkdocs into confluence
      host_url: https://my_confluence.fr/rest/api/content?type=page&spaceKey=bac&expand=ancestros&limit=100
      space: bac
      parent_page_name: Test_Florent
      username: myuser_name
      password: "MYPASSWORD!*ù$^4_with_special_character"
      verbose: true
      dryrun: true

AND LOT : thanks for your support.

Enhancement - About page versus parent page

It should be possible to manage whether the index markdown file becomes an about page or populates the content of the parent page.
It seems desirable at least that some part of the index.md file become part of the parent page, but I understand why the default should be to create an About page. I think what I'm asking for should be considered an enhancement - some people would prefer the parent page to have TOC via the Children display macro.

Support Bearer Token

Hi,

as far as I can see, this plugin only supports username/password authentication.
For instance for ci-jobs I don't want to expose my password. Instead I want to use a Personal Access Token.
Is it possible to support Access Tokens too?

Warning

Installed plugin and plugin block in mkdocs.yml . I keep getting this error:

INFO - Your documentation should shortly be available at: https://symplr-software.github.io/innovation-documentation-pipeline/
WARNING - Mkdocs With Confluence: Exporting MKDOCS pages to Confluence turned OFF: (set environment variable MKDOCS_TO_CONFLUENCE to 1 to enable)
Number of Files in directory tree: 12

Please advise

Save Confluence files

Hi,
Thank you for your package, it looks awesome !
We use a private confluence website using SSO (with 2FA) to access it. It seems quite difficult to upload directly using the provided username / password arguments so we would like to generate the confluence pages and upload them manually. We didn't find in the doc any way to save the pages before upload. Is that something currently available ?
Thank you for your help

SSL CACERT missing

Dear,

I've got an SSL error message when I try to use your excellent code from a gitlab pipeline.
It seems that I need to pass cacert as parameter.
Does this feature exist ?

Test done :

Without your library and directly with CURL I can reach my confluence. So I think that my issue root cause is the absence of SSL/CACERT in your example.
For instance :

curl  --cacert curl-ca-bundle.crt  --url "https://myconfluence.com/rest/api/content/251548690" -H "Authorization: Bearer fjhs_my_tokenskjd"

This curl request send my well informations. Whereas, if I suppress the --cacert option, I've got the same SSL error message.

Error message from my gitlab pipeline :

requests.exceptions.SSLError: HTTPSConnectionPool(host='myconfluence.com', port=443): Max retries exceeded with url: /rest/api/content?title=Home&spaceKey=TDF&expand=history (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))

A lot of thanks for your support.