From 14a6c4a352798a1371011c9345d4d3a6172d1992 Mon Sep 17 00:00:00 2001
From: Tim Vink <vinktim@gmail.com>
Date: Thu, 29 Aug 2024 13:37:17 +0000
Subject: [PATCH] Update documentation on macros compatibility

---
 README.md                |   2 +-
 docs/howto/use_jinja2.md |  39 +++---
 docs/readers.md          | 276 ++++++++++++++++++++++++++++++++++++---
 mkdocs.yml               |   2 +-
 4 files changed, 281 insertions(+), 38 deletions(-)

diff --git a/README.md b/README.md
index 1e6ed7e..7cba439 100644
--- a/README.md
+++ b/README.md
@@ -41,7 +41,7 @@ In your markdown files you can now use:
 Where the path is relative to the location of your project's `mkdocs.yml` file, _or_ your project's `docs/` directory, _or_ the location of your markdown source file (all 3 possible locations will be searched, in that order).
 
 - There are [readers](https://timvink.github.io/mkdocs-table-reader-plugin/readers/) available for many common table formats, like `.csv`, `.fwf`, `.json`, `.xls`, `.xlsx`, `.yaml`, `.feather` and `.tsv`. There is also the `read_raw()` reader that will allow you to insert tables (or other content) already in markdown format.
-- `table-reader` is compatible with [`mkdocs-macros-plugin`](https://mkdocs-macros-plugin.readthedocs.io/en/latest/), which means you can [dynamically insert tables using jinja2 syntax](https://timvink.github.io/mkdocs-table-reader-plugin/howto/use_jinja2/).
+- `table-reader` is compatible with [`mkdocs-macros-plugin`](https://mkdocs-macros-plugin.readthedocs.io/en/latest/). This enables further automation like filtering tables or inserting directories of tables. See the documentation on [compatibility with macros plugin](howto/use_jinja2.md) for more examples.
 
 ## Documentation and how-to guides
 
diff --git a/docs/howto/use_jinja2.md b/docs/howto/use_jinja2.md
index 02cbe8f..f1adc10 100644
--- a/docs/howto/use_jinja2.md
+++ b/docs/howto/use_jinja2.md
@@ -1,4 +1,5 @@
-# Use jinja2 for automation
+# Compatibility with mkdocs-macros-plugin to enable further automation
+
 {% raw %}
 
 `table-reader` supports [`mkdocs-macros-plugin`](https://mkdocs-macros-plugin.readthedocs.io/en/latest/), which enables you to use jinja2 syntax inside markdown files (among other things).
@@ -11,28 +12,19 @@ plugins:
     - table-reader
 ```
 
-Now you can do cool things like:
-
-## Dynamically load a list of tables
-
+Everything will work as before, _except_ indentation will not be retrained. This means components that rely on indentation (like [Admonitions](https://squidfunk.github.io/mkdocs-material/reference/admonitions/) and [Content tabs](https://squidfunk.github.io/mkdocs-material/reference/content-tabs/#usage)) will break.
 
-```markdown
-# index.md
+The solution is to use the custom _filter_ `add_indendation` (a filter added to `macros` by `table-reader` plugin, see the [readers](../readers.md)). For example:
 
-{% set table_names = ["basic_table.csv","basic_table2.csv"] %}
-{% for table_name in table_names %}
+```jinja
+!!! note "This is a note"
 
-{{ read_csv(table_name) }}
-
-{% endfor %}
+    {{ read_csv("basic_table.csv") | add_indentation(spaces=4) }}
 ```
 
+The upside is you now have much more control. A couple of example use cases:
 
-## Insert tables into content tabs
-
-If you inserted content has multiple lines, then indentation will be not be retained beyond the first line. This means things like [content tabs](https://squidfunk.github.io/mkdocs-material/reference/content-tabs/#usage) will not work as expected.
-
-To fix that, you can use the custom _filter_ `add_indendation` (a filter add to `macros` by `table-reader` plugin). For example:
+## Dynamically load a specified list of tables into tabs
 
 === "index.md"
 
@@ -68,7 +60,6 @@ To fix that, you can use the custom _filter_ `add_indendation` (a filter add to
     ``` 
 
 
-
 ## Recursively insert an entire directory of tables
 
 [`mkdocs-macros-plugin`](https://mkdocs-macros-plugin.readthedocs.io/en/latest/) enables you to define additional functions (called _macros_) that you will be able to use within your markdown files.
@@ -104,4 +95,14 @@ Now you could do something like:
 {% endfor %}
 ```
 
-{% endraw %}
\ No newline at end of file
+## Filter a table before inserting it
+
+When you enable the `macros` plugin in your `mkdocs.yml`, `table-reader` will add additional _macros_ and _filters_ (see the [readers overview](../readers.md)).
+
+You can use the `pd_<reader>` variants to read a file to a pandas dataframe. Then you can use the pandas methods like [`.query()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.query.html). For example:
+
+```
+{{ pd_read_csv("numeric_table.csv").query("column_name >= 3") | convert_to_md_table }}
+```
+
+{% endraw %}
diff --git a/docs/readers.md b/docs/readers.md
index 901997c..50ed597 100644
--- a/docs/readers.md
+++ b/docs/readers.md
@@ -5,11 +5,20 @@ hide:
 
 # Readers
 
+## Basic readers
+
 The following table reader functions are available:
 
-## read_csv
+### `read_csv`
+
+Use {% raw %}`{{ read_csv() }}`{% endraw %} to read a comma-separated values (csv) and output as a markdown table.
 
-{% raw %}`{{ read_csv() }}`{% endraw %} passed to [pandas.read_csv()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html). Example:
+1. Arguments are parsed safely and then passed to corresponding functions below
+2. File is read using [pandas.read_csv()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html)
+3. The `pd.DataFrame` is then converted to a markdown table using [`.to_markdown()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_markdown.html)
+4. The markdown table is fixed to match the indentation used by the tag in the markdown document (only when _not_ used with `mkdocs-macros-plugin`. See [compatibility with macros plugin](howto/use_jinja2.md))
+
+Example:
 
 === "Input"
 
@@ -24,9 +33,16 @@ The following table reader functions are available:
     {{ read_csv('assets/tables/basic_table.csv') | add_indentation(spaces=4) }}
 
 
-## read_fwf
+### `read_fwf`
+
+Use {% raw %}`{{ read_fwf() }}`{% endraw %} to read a table of fixed-width formatted lines and output as a markdown table.
 
-{% raw %}`{{ read_fwf() }}`{% endraw %} passed to [pandas.read_fwf()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_fwf.html). Example:
+1. Arguments are parsed safely and then passed to corresponding functions below
+2. File is read using [pandas.read_fwf()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_fwf.html)
+3. The `pd.DataFrame` is then converted to a markdown table using [`.to_markdown()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_markdown.html)
+4. The markdown table is fixed to match the indentation used by the tag in the markdown document (only when _not_ used with `mkdocs-macros-plugin`. See [compatibility with macros plugin](howto/use_jinja2.md))
+
+Example:
 
 === "Input"
 
@@ -41,9 +57,16 @@ The following table reader functions are available:
     {{ read_fwf('assets/tables/fixedwidth_table.txt') | add_indentation(spaces=4) }}
 
 
-## read_yaml
+### `read_yaml`
+
+Use {% raw %}`{{ read_yaml() }}`{% endraw %} to read a YAML file and output as a markdown table.
 
-{% raw %}`{{ read_yaml() }}`{% endraw %} is parsed with [yaml.safe_load()](https://pyyaml.org/wiki/PyYAMLDocumentation#loading-yaml) and passed to [pandas.json_normalize()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.json_normalize.html). Example:
+1. Arguments are parsed safely and then passed to corresponding functions below
+2. File is read using [yaml.safe_load()](https://pyyaml.org/wiki/PyYAMLDocumentation#loading-yaml) and then passed to [pandas.json_normalize()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.json_normalize.html)
+3. The `pd.DataFrame` is then converted to a markdown table using [`.to_markdown()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_markdown.html)
+4. The markdown table is fixed to match the indentation used by the tag in the markdown document (only when _not_ used with `mkdocs-macros-plugin`. See [compatibility with macros plugin](howto/use_jinja2.md))
+
+Example:
 
 === "Input"
 
@@ -58,9 +81,16 @@ The following table reader functions are available:
     {{ read_yaml('assets/tables/yaml_table.yml') | add_indentation(spaces=4) }}
 
 
-## read_table
+### `read_table`
 
-{% raw %}`{{ read_table() }}`{% endraw %} passed to [pandas.read_table()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_table.html). Example:
+Use {% raw %}`{{ read_table() }}`{% endraw %} to read a general delimited file and output as a markdown table.
+
+1. Arguments are parsed safely and then passed to corresponding functions below
+2. File is read using [pandas.read_table()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_table.html)
+3. The `pd.DataFrame` is then converted to a markdown table using [`.to_markdown()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_markdown.html)
+4. The markdown table is fixed to match the indentation used by the tag in the markdown document (only when _not_ used with `mkdocs-macros-plugin`. See [compatibility with macros plugin](howto/use_jinja2.md))
+
+Example:
 
 === "Input"
 
@@ -74,9 +104,16 @@ The following table reader functions are available:
 
     {{ read_table('assets/tables/basic_table.csv', sep = ',') | add_indentation(spaces=4) }}
 
-## read_json
+### `read_json`
+
+Use {% raw %}`{{ read_json() }}`{% endraw %} to read a JSON string path and output as a markdown table.
+
+1. Arguments are parsed safely and then passed to corresponding functions below
+2. File is read using [pandas.read_json()](https://pandas.pydata.org/docs/reference/api/pandas.read_json.html)
+3. The `pd.DataFrame` is then converted to a markdown table using [`.to_markdown()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_markdown.html)
+4. The markdown table is fixed to match the indentation used by the tag in the markdown document (only when _not_ used with `mkdocs-macros-plugin`. See [compatibility with macros plugin](howto/use_jinja2.md))
 
-{% raw %}`{{ read_json() }}`{% endraw %} passed to [pandas.read_json()](https://pandas.pydata.org/docs/reference/api/pandas.read_json.html). Example:
+Example:
 
 === "Input"
 
@@ -90,15 +127,22 @@ The following table reader functions are available:
 
     {{ read_json('assets/tables/data.json', orient='split') | add_indentation(spaces=4) }}
 
-## read_feather
+### `read_feather`
+
+Use {% raw %}`{{ read_feather() }}`{% endraw %} to read a feather-format object and output as a markdown table.
 
-{% raw %}`{{ read_feather() }}`{% endraw %} passed to [pandas.read_feather()](https://pandas.pydata.org/docs/reference/api/pandas.read_feather.html). Example:
+1. Arguments are parsed safely and then passed to corresponding functions below
+2. File is read using [pandas.read_feather()](https://pandas.pydata.org/docs/reference/api/pandas.read_feather.html)
+3. The `pd.DataFrame` is then converted to a markdown table using [`.to_markdown()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_markdown.html)
+4. The markdown table is fixed to match the indentation used by the tag in the markdown document (only when _not_ used with `mkdocs-macros-plugin`. See [compatibility with macros plugin](howto/use_jinja2.md))
+
+Example:
 
 === "Input"
 
     {% raw %}
     ```markdown
-    {{ read_json('assets/tables/data.feather') }}
+    {{ read_feather('assets/tables/data.feather') }}
     ```
     {% endraw %}
 
@@ -107,10 +151,16 @@ The following table reader functions are available:
     {{ read_feather('assets/tables/data.feather')  | add_indentation(spaces=4) }}
 
 
-## read_excel
+### `read_excel`
 
-{% raw %}`{{ read_excel() }}`{% endraw %} passed to [pandas.read_excel()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html). Example:
+Use {% raw %}`{{ read_excel() }}`{% endraw %} to read an Excel file and output as a markdown table.
 
+1. Arguments are parsed safely and then passed to corresponding functions below
+2. File is read using [pandas.read_excel()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html)
+3. The `pd.DataFrame` is then converted to a markdown table using [`.to_markdown()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_markdown.html)
+4. The markdown table is fixed to match the indentation used by the tag in the markdown document (only when _not_ used with `mkdocs-macros-plugin`. See [compatibility with macros plugin](howto/use_jinja2.md))
+
+Example:
 
 === "Input"
 
@@ -135,11 +185,17 @@ The following table reader functions are available:
     ```
     {% endraw %}
 
-## read_raw
+### `read_raw`
+
+Use {% raw %}`{{ read_raw() }}`{% endraw %} to insert the contents from a file directly. 
 
-{% raw %}`{{ read_raw() }}`{% endraw %} inserts contents from a file directly. This is great if you have a file with a table already in markdown format. 
+This is great if you have a file with a table already in markdown format. 
 It could also replace a workflow where you use the [snippets extension to embed external files](https://squidfunk.github.io/mkdocs-material/reference/code-blocks/#embedding-external-files).
 
+1. Only the first argument is read. This should be the file path.
+2. File is read using python
+4. The markdown table is fixed to match the indentation used by the tag in the markdown document (only when _not_ used with `mkdocs-macros-plugin`. See [compatibility with macros plugin](howto/use_jinja2.md))
+
 Example:
 
 === "Input"
@@ -154,3 +210,189 @@ Example:
 
     {{ read_raw('assets/tables/markdown_table.md')  | add_indentation(spaces=4) }}
 
+
+## Macros
+
+When you use `table-reader` with [`mkdocs-macros-plugin`](https://mkdocs-macros-plugin.readthedocs.io/en/latest/), in next to all the readers, the following _additional_ macros will be made available:
+
+### `pd_read_csv`
+
+Use {% raw %}`{{ pd_read_csv() }}`{% endraw %} to read a comma-separated values (csv) using [pandas.read_csv()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html).
+
+=== "Input"
+
+    {% raw %}
+    ```markdown
+    {{ pd_read_csv('assets/tables/basic_table.csv').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }}
+    ```
+    {% endraw %}
+
+=== "Output"
+
+    {{ pd_read_csv('assets/tables/basic_table.csv').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }}
+
+
+### `pd_read_fwf`
+
+Use {% raw %}`{{ pd_read_fwf() }}`{% endraw %} to read a table of fixed-width formatted lines using [pandas.read_fwf()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_fwf.html)
+
+Example:
+
+=== "Input"
+
+    {% raw %}
+    ```markdown
+    {{ pd_read_fwf('assets/tables/fixedwidth_table.txt').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }}
+    ```
+    {% endraw %}
+
+=== "Output"
+
+    {{ pd_read_fwf('assets/tables/fixedwidth_table.txt').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }}
+
+
+### `pd_read_yaml`
+
+Use {% raw %}`{{ pd_read_yaml() }}`{% endraw %} to read a YAML file using [yaml.safe_load()](https://pyyaml.org/wiki/PyYAMLDocumentation#loading-yaml) and [pandas.json_normalize()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.json_normalize.html).
+
+Example:
+
+=== "Input"
+
+    {% raw %}
+    ```markdown
+    {{ pd_read_yaml('assets/tables/yaml_table.yml').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }}
+    ```
+    {% endraw %}
+
+=== "Output"
+
+    {{ pd_read_yaml('assets/tables/yaml_table.yml').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }}
+
+
+### `pd_read_table`
+
+Use {% raw %}`{{ pd_read_table() }}`{% endraw %} to read a general delimited file using [pandas.read_table()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_table.html).
+
+Example:
+
+=== "Input"
+
+    {% raw %}
+    ```markdown
+    {{ pd_read_table('assets/tables/basic_table.csv', sep = ',').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }}
+    ```
+    {% endraw %}
+
+=== "Output"
+
+    {{ pd_read_table('assets/tables/basic_table.csv', sep = ',').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }}
+
+
+
+### `pd_read_json`
+
+Use {% raw %}`{{ pd_read_json() }}`{% endraw %} to read a JSON string path using [pandas.read_json()](https://pandas.pydata.org/docs/reference/api/pandas.read_json.html)
+
+Example:
+
+=== "Input"
+
+    {% raw %}
+    ```markdown
+    {{ pd_read_json('assets/tables/data.json', orient='split').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }}
+    ```
+    {% endraw %}
+
+=== "Output"
+
+    {{ pd_read_json('assets/tables/data.json', orient='split').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }}
+
+
+
+### `pd_read_feather`
+
+Use {% raw %}`{{ pd_read_feather() }}`{% endraw %} to read a feather-format object using [pandas.read_feather()](https://pandas.pydata.org/docs/reference/api/pandas.read_feather.html)
+
+
+Example:
+
+=== "Input"
+
+    {% raw %}
+    ```markdown
+    {{ pd_read_feather('assets/tables/data.feather').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }}
+    ```
+    {% endraw %}
+
+=== "Output"
+
+    {{ pd_read_feather('assets/tables/data.feather').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }}
+
+
+### `pd_read_excel`
+
+Use {% raw %}`{{ pd_read_excel() }}`{% endraw %} to read an Excel file using [pandas.read_excel()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html)
+
+Example:
+
+=== "Input"
+
+    {% raw %}
+    ```markdown
+    {{ pd_read_excel('assets/tables/excel_table.xlsx', engine='openpyxl').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }}
+    ```
+    {% endraw %}
+
+=== "Output"
+
+    {{ pd_read_excel('assets/tables/excel_table.xlsx', engine='openpyxl').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }}
+
+
+!!! info "Reading xlsx files"
+
+    You might get a `XLRDError('Excel xlsx file; not supported',)` error when trying to read modern excel files. That's because `xlrd` does not support `.xlsx` files ([stackoverflow post](https://stackoverflow.com/questions/65254535/xlrd-biffh-xlrderror-excel-xlsx-file-not-supported)). Instead, install [openpyxl](https://openpyxl.readthedocs.io/en/stable/) and use:
+
+    {% raw %}
+    ```markdown
+    {{ pd_read_excel('assets/tables/excel_table.xlsx', engine='openpyxl') }}
+    ```
+    {% endraw %}
+
+
+## Filters
+
+When you use `table-reader` with [`mkdocs-macros-plugin`](https://mkdocs-macros-plugin.readthedocs.io/en/latest/), in next to all the readers, the macros, the following _additional_ filters will be made available:
+
+### `add_indentation`
+
+Adds a consistent indentation to every line in a string. This is important when you are inserting content into [Admonitions](https://squidfunk.github.io/mkdocs-material/reference/admonitions/) or [Content tabs](https://squidfunk.github.io/mkdocs-material/reference/content-tabs/).
+
+Args:
+    text (str): input text
+    spaces (int): Indentation to add in spaces
+    tabs (int): Indentation to add in tabs
+
+
+Example usage: 
+
+{% raw %}
+```markdown
+!!! note "this is a note"
+    {{ pd_read_csv('assets/tables/basic_table.csv').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }}
+```
+{% endraw %}
+
+### `convert_to_md_table`
+
+Converts a pandas dataframe into a markdown table. Arguments are passed to [`.to_markdown()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_markdown.html). By default, `tablefmt='pipe'` and `index=False` are used.
+There is also an additional fix to ensure any pipe (`|`) characters in the dataframe are properly escaped ([python-tabulate#241](https://github.com/astanin/python-tabulate/issues/241)).
+
+Example usage: 
+
+{% raw %}
+```markdown
+{{ pd_read_csv('assets/tables/basic_table.csv') | convert_to_md_table  }}
+```
+{% endraw %}
+
diff --git a/mkdocs.yml b/mkdocs.yml
index 1439a67..80b134c 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -10,7 +10,7 @@ nav:
     - How to:
         - howto/customize_tables.md
         - howto/preprocess_tables.md
-        - howto/use_jinja2.md
+        - Macros plugin compatibility: howto/use_jinja2.md
         - howto/project_structure.md
         - howto/docker.md
         - howto/alternatives.md