Following #32, we can add a children() method that returns a Vec<Either<&BlockElement, &InlineElement>>

By default with the LocatedElement and individual element structs and enums, there is no concept of looking back up the tree for a parent. You can only explore deeper and must know the structure of each element in order to traverse its children.

For graph-based searching and location-based searching, it will be useful for results to include the ability to traverse up to the parent element.

From my thinking, there are two main categories of elements: block and inline. Both of these are enums that encompass all explicit block and inline element concrete types. This makes it a bit easier to traverse up with a parent() method as we only have to worry about returning the generic enum for one of those two types since we wouldn't know if a tag was contained within a paragraph, a non-blank line, a header, or something else.

This also opens the door for a siblings() or next_sibling() method that could return one of those two types.

We could use an Either type to contain Either<BlockElement, InlineElement> with the actual parent method being fn parent(&self) -> Option<Either<BlockElement, InlineElement>>. This also makes the GraphQL implementation a bit easier as we can write an object to represent the either choice:

#[async_graphql::Object]
impl Parent {
    async fn block_element(&self) -> Option<BlockElement> { ... }
    async fn inline_element(&self) -> Option<InlineElement> { ... }
}

For the graph setup, nodes could be one of these options:

1. An Either<&BlockElement, &InlineElement> so we have a pointer

2. Some usize id where we have built a HashMap<usize, Either<&BlockElement, &InlineElement>>

3. Some usize id where we have built a HashMap<usize, LinkedElement> where LinkedElement is

   pub struct LinkedElement {
       root: Option<&LinkedElement>,
       parent: Option<&LinkedElement>,
       element: Either<&BlockElement, &InlineElement>,
   }

Can use Juniper to build a query model instead of making my own routing layer. Supports being called without an HTTP server, although this also means we could support both providing queries via stdin and HTTP (web app in future?).

Task to keep track of differences between what vimwiki supports vs pandoc:

1. Headers
   a. Pandoc supports bold, italic, link, etc.
   b. vimwiki does not (according to help doc)
2. Line comments
   a. Pandoc supports at any point in a line
   b. vimwiki supports at start of line (according to help doc)

Provide the main page documentation for the vimwiki_macros crate. This involves writing markdown with examples, explanation of the library, and more. This can be used both in the README and in a doc string within lib.rs.

Part of the reason for trying out GraphQL with Juniper is requiring less glue code between an action (modify a page) and an API to trigger the action. One of the pain points I discovered building out a CLI for over there was the amount of glue code. I eventually decided to parse JSON directly and pass that as a data stream. Can achieve similar by accepting graphql queries, mutations, and subscriptions.

There are some clippy warnings about some variants being much larger than others and the recommendation is to wrap in a Box<...>. Worth investigating and cleaning up.

Given that we will be supporting regions to know where each component lies within a file/buffer, we should be able to send syntax highlighting information to vim so it can highlight following the rules of the vimwiki parser rather than needing to align a separate syntax highlighting file and deal with all of those nuances.

An example of another vim plugin that does this is vim-lsp-cxx-highlight, which - to my knowledge - uses an lsp server to provide syntax highlighting.

It would be nice to be able to build up an index of all pages, tags, links, and more so they are more easily searchable. Caching this to a file such that we can load it, maybe pinning a timestamp to each page and comparing to invalidate entries whose files/paths have changed.

It's an unnecessary type and floods the Page type with noise. The parser checking for a blank line would need to do so sooner. That or we filter out blank lines somewhere. Unsure what the best option is here.

Might move this one to a later milestone, but it would be incredibly useful to have a macro that can parse vimwiki language, report errors, and spit out a complete Page component at compile time. Could use this for tests where we can generate this and test actions like moving tasks between pages, etc.

// Parses the syntax at compile time and generates the 
// `Page(vec![Header, Paragraph, List(vec![Item1, Item2])])` object. 
//
// If it fails, would highlight where in the syntax below the error is
let page = vimwiki! {
= Some header =

A paragraph of text.

- List item 1
- List item 2
};

It's a pain to collect the raw text for an inline element or an inline element container.

If we have Some *bold* text with [[link]] and [[link|other link]] with $math formula$, I'd like to get "Some bold text with link and other link", which is what would be shown to a human (minus the stylings). This would also be a good GraphQL field to be exposed at the InlineElement and container levels.

Similar to what was done for Header and DefinitionList, other data models like List should support content and content_elements versus just the elements portion.

When providing a template, support sending a syntax error or some sort of other diagnostic to help vim highlight when a template does not exist. Would depend on #21 being done, first.

Comments, while easy to parse in themselves, make it a bit difficult in the overall handling of vimwiki parsing.

Parse twice

The first thought is to do a first pass over the input, looking only for comments. We would capture their regions just like with regular components, but after being done we would go through and remove all comments from the input prior to parsing for everything else. This would enable us to query against comments for information and properly glue stuff together like:

== Head%%+
er+%%==

would become

== Header ==

The problem with this approach is that the region for the header is altered compared to what would be reported from vim or any other tool inspecting a vimwiki file. We might be able to have some sort of adjustment by adding all comment regions to every component that starts on or after them, but we don't know how long lines are and this would probably be more error prone.

Support inline

According to the vimwiki docs, a single line comment would start at the beginning of the line, which is fine. We can capture that at the block level. As for the multi-line comment, that would is more difficult.

We could add it to an inline component and allow multi-line to span across lines, which would enable later inline components to continue on a single line like usual. This would keep locality to my knowledge.

The challenge is when a multi-line comment appears in a block-level component unexpectedly.

== Head%%+
er+%%==

The above would fail as the header parser would begin and then see the comment as additional characters that would result in the parser failing.

Custom input type

We do the twice parsing where the first time we find all of the comments and their regions. From there, we have a custom input type around the span that knows to skip over regions that are commented. The advantage is that this would seamlessly allow comments that cut over across another line.

== Head%%+
er+%%==

For the above, the parser would consume from span input like usual. When we take slices, we'd need to support hiding the commented segments while keeping the line and column information.

This seems like the best solution as we don't have to do custom logic within parsers.

We would also want to put comments at the top of the depth stack when building the RTree as inspecting on top of a comment within a component should first yield the comment.

Reading through nom's error management, there's discussion on creating a custom error type. Given that errors can be created with input - in my case Span - and the author of nom mentioned Span provides location information, it might provide extra support such as enabling alt to keep track of the branch that made the most progress.

Top-level issue for writing a vimwiki parser. Can test out with a couple of my own pages, but needs to support parsing the entire vimwiki language, supporting hierarchy and proper testing.

• Typefaces (vimwiki-syntax-typefaces)
  • Bold text
  • Italic text
  • Bold italic text
  • Strikeout text
  • Code (no syntax) text
  • Super script
  • Sub script
• Links (vimwiki-syntax-links)
  • Plain link
  • Link with description
  • Subdirectory link
  • Relative link
  • Absolute link
  • External local box links (// prefix)
  • Subdirectory links
  • Interwiki links
  • Diary links
  • Anchors (under links and vimwiki-anchors)
  • Raw URLs
  • External files (file: or local:)
  • Transclusion (wiki-include) links
  • Thumbnail links
• Headers (vimwiki-syntax-headers)
  • Normal header with levels
  • Centered header (spaces in front)
• Paragraphs (vimwiki-syntax-paragraphs)
  • Group of lines starting in column 1 (no indentation)
  • Blank line separating paragraphs
• Lists (vimwiki-syntax-lists)
  • Unordered
  • Ordered
  • Nested (for either of above)
  • Spanning multiple lines
  • Spanning multiple lines with sub items inbetween
  • Definition lists
  • Todo lists (vimwiki-todo-lists)
• Tables (vimwiki-syntax-tables and vimwiki-tables)
  • Plain text version
  • Formatting inside each cell
  • Divider rows
• Preformatted text (vimwiki-syntax-preformatted)
  • Plain block
  • Optional information after {{{
• Mathematical formulae (vimwiki-syntax-math)
  • Inline math
  • Block display
  • Block environment
• Blockquotes (vimwiki-syntax-blockquotes)
  • Four or more spaces
  • Prefixed with >
• Comments (vimwiki-syntax-comments)
  • Single line using %%
  • Multi-line using %%+ and +%%
• Horizontal line (vimwiki-syntax-hr)
  • Four or more dashes at start of line
• Tags (vimwiki-syntax-tags)
  • Single tag :name:
  • Multi-tag :name1:name2:...
  • Mapping to file (first two lines of file)
  • Mapping to header (first two lines below header)
  • Standalone mode
• Placeholders (vimwiki-placeholders)
  • Title (vimwiki-title)
  • No HTML (vimwiki-nohtml)
  • Template (vimwiki-template)
  • Date (vimwiki-date)

Would involve building a hashmap of tag name to some id and then using the id to connect to pages. The pages themselves would also be stored in some hashmap, maybe? So we can connect ids properly.

Originally, my concern was that I could hash and look up definitions by terms. If #30 is completed, this will be a non-issue as we'll just convert to a raw string first with no frills.

*Bold* Term:: _Italic_ definition with [[links]]
:: $Math formula$ as part of definition

While I want to use Juniper as the library to define schema, operations, etc; I don't want vim to have to talk over web. There is no built-in HTTP requests for vimscript and calling out to curl (or equivalent) may not be ideal. Additionally, for realtime operations, HTTP may not be the best due to its overhead.

Rather, it would be nice to be able to launch a vimwiki daemon in "vim mode" similar to many LSPs where it listens on stdin for input rather than a socket. Queries, mutations, and subscriptions can still be issued and responses received. Main challenge would be that a callback id needs to be associated with requests and returned with responses so that vim can map responses async rather than blocking for a response, especially if other responses arrive inbetween.

Each request/response would be a single line of stdin/stdout. As described in GraphQL's HTTP documentation, a GraphQL query/mutation/subscription can be represented as encoded JSON. So we may be able to parse a custom id field at the top level, maintain that id through a request, and add it to the top level of a response when using stdin/stdout. For web/HTTP, there is no need to do that as the mapping information is encoded in the protocol already.

I'm not a fan of the env_logger family. flexi_logger appears to be more my style. I can write to stdout/stderr and files at the same time. I can specify logging levels from CLI without exposing an environment variable to override, and I can still get color.

Rather than having a separate repo for the vim plugin, we can include it as a subdirectory here. Something to consider.

Top-level issue for writing a vimwiki data model for language components. This is the precursor to #8 as we need the data model to put the parser content into.

• Typefaces (vimwiki-syntax-typefaces)
  • Bold text
  • Italic text
  • Bold italic text
  • Strikeout text
  • Code (no syntax) text
  • Super script
  • Sub script
• Links (vimwiki-syntax-links)
  • Plain link
  • Link with description
  • Subdirectory link
  • Relative link
  • Absolute link
  • External local box links (// prefix)
  • Subdirectory links
  • Interwiki links
  • Diary links
  • Anchors (under links and vimwiki-anchors)
  • Raw URLs
  • External files (file: or local:)
  • Transclusion (wiki-include) links
  • Thumbnail links
• Headers (vimwiki-syntax-headers)
  • Normal header with levels
  • Centered header (spaces in front)
  • Table of Contents (vimwiki-toc) (no difference in data model)
• Paragraphs (vimwiki-syntax-paragraphs)
  • Group of lines starting in column 1 (no indentation)
  • Blank line separating paragraphs
• Lists (vimwiki-syntax-lists)
  • Unordered
  • Ordered
  • Nested (for either of above)
  • Spanning multiple lines
  • Spanning multiple lines with sub items inbetween
  • Definition lists
  • Todo lists (vimwiki-todo-lists)
• Tables (vimwiki-syntax-tables and vimwiki-tables)
  • Plain text version
  • Formatting inside each cell
  • Divider rows
• Preformatted text (vimwiki-syntax-preformatted)
  • Plain block
  • Optional information after {{{
• Mathematical formulae (vimwiki-syntax-math)
  • Inline math
  • Block display
  • Block environment
• Blockquotes (vimwiki-syntax-blockquotes)
• Comments (vimwiki-syntax-comments)
  • Single line using %%
  • Multi-line using %%+ and +%%
• Horizontal line (vimwiki-syntax-hr)
  • Four or more dashes at start of line
• Tags (vimwiki-syntax-tags)
  • Single tag :name:
  • Multi-tag :name1:name2:...
• Placeholders (vimwiki-placeholders)
  • Title (vimwiki-title)
  • No HTML (vimwiki-nohtml)
  • Template (vimwiki-template)
  • Date (vimwiki-date)

Make a workspace for the vimwiki parser and language components. Rather than trying to cram all of the feature separation that includes server functionality, I think this is an example where a separate, dedicated crate for the vimwiki language would be a cleaner idea.

Responsibilities of the vimwiki crate:

1. Provide rust structures and enums representing the vimwiki language
2. Provide means to parse a string representing the vimwiki language
3. Serialize vimwiki components as vimwiki language
4. Feature to enable juniper graphql so we can generate code for language components (juniper dependency)
5. Feature to enable lookup of vimwiki components by line & column (rstar dependency)

Responsibilities of the vimwiki-server crate:

1. Manipulation of vimwiki data structures? Or should that be in the vimwiki crate? Depends on what specifically, but thinking of re-ordering tasks (e.g. cut & paste components to preserve formatting), inserting new components, deleting components
2. Provide graphql query layer (implement mutation, query, and eventually subscription)
3. Feature to enable caching of vimwiki page structure to a file
4. Feature to enable web layer to serve graphql
5. Feature to enable stdin to serve graphql
6. Feature to watch files for changes (notify dependency)

Will make it possible to return mix of elements. This could also be where we add functionality like to_html().

impl Element for BlockElement {}
impl Element for Header {}
...

Similar to MathInline and MathBlock, it'd be nice to have a standalone inline code element given that it behaves differently than the other typefaces.

https://github.com/fflorent/nom_locate provides location information through a span. This can be used to know where a component starts and - if we encapsulate all components including a blank line - its end based on the start of the next element.

This will enable looking up a component under cursor, providing a line and column to use as a search.

Once we have a location, we want to construct an RTree that can be used to look up assoicated locations. Leveraging the rstar crate, we can build the RTree following the documentation. Will need to define a custom RTreeObject, which will have the line and column start position. Additionally, will implement PointDistance where the distance is based on the start position. The library indicates that a distance of 0 is used if within an object (like a circle). In our case, we have have an irregular shape.

To use either juniper or async-graphql, we have to augment our components with various macros that generate the needed code to make our components become GraphQL objects.

This will be gated behind a feature called graphql and will compile the vimwiki crate with graphql support. The vimwiki-server crate will pull in this dependency with the feature enabled.

ListItem just needs an attributes field. Additionally, rather than a hashmap of attributes, just make another struct that is ListItemAttributes that contains a single, optional field that holds the todo enum. We can move over all of the functionality associated with todo over to the listitem struct and the associated attribute struct.

Allow examples like these:

== My *bold* header with [[links]] and $forumlas$ and more ==

https://github.com/tantivy-search/tantivy

Provide a query that takes in search terms and spits out results that include the wiki, page, and element matched.

When building out the vimwiki crate's components and parsers, it wasn't clear what the structure should be. Now, having properly bundled everything in good nesting structure for vimwiki-server, it makes sense to go back and move things around for the vimwiki crate. This shouldn't affect anything as the components are exposed externally at the top level.

https://github.com/JelteF/derive_more

Leverage rlua as an optional feature in vimwiki-server to provide the entrypoint into a custom GraphQL call. Because I don't think we can dynamically create GraphQL schemas, we will instead provide a single type (query, mutation, and maybe subscription?):

{
    lua_query(key: "some identifier", input: "some input") {
        key
        output
    }
}

If dynamic schemas were possible as described in async-graphql/async-graphql#30, then lua could be used to make the schemas, although from what was described there it would be a high performance cost.

This is something that org mode supports and is even used to write an emacs config file using org mode with elisp code blocks.

This might belong more in a vim extras feature that could spawn a floating window with the result, but the process of making sure that we can select the code under cursor to evaluate would be from the server.

Provide the main page documentation for the vimwiki-server crate. This involves writing markdown with examples, explanation of the binary, and more. This can be used both in the README and in a doc string within lib.rs.

tokio has time module and I'm familiar with it, versus std-async.

Provide the main page documentation for the vimwiki crate. This involves writing markdown with examples, explanation of the library, and more. This can be used both in the README and in a doc string within lib.rs.

Due to the upfront cost of parsing vimwiki files, indexing, and more, want to have a friendly way to show progress. https://github.com/mitsuhiko/indicatif seems to be a good choice.

Rather than always bringing in the parser code, which includes dependencies like nom, bytes, and bytecount, it would be nice to make that entire section optional. This would be useful in scenarios where other tooling wants to either a) write its own parser, b) build vimwiki elements by hand, or c) load up vimwiki elements that were serialized elsewhere for it to operate on.

Having written SpanFactory and leveraging other hacks on top of nom_locate, the performance of reading a file has massively dropped. For the pandoc example, the performance prior to incorporating SpanFactory was 1.5s (already not good) and post incorporation is 30~35s!

My current thoughts are:

1. Parser combinators do not require the input to be copy. Rather, they take in the input and provide it back, which I use. Try to avoid a copy with our hack struct as my guess is copying around the struct is causing massive slowdowns
   a. We need to update scan(...) to not consume the input
2. Remove nom_locate in favor of writing my own span struct. To make things easier (and more efficient), leverage the Bytes crate. This crate maintains multiple reference counts and allows overlapping slices, etc. Main draw for me is the ability to chain non-consecutive slices. Have the wrapping struct containing an instance of Bytes implement all traits needed by nom input and NOT be a copy to see how well it performs.
   a. With the struct, we would maintain a master Bytes reference and the shortened Bytes chain based on removed slices, just like we are now with the hack approach.
   b. We would re-implement the get_utf8_column without the need for unsafe as we can leverage the master Bytes instance to get the proper start (not needing a consecutive slice). nom_locate uses bytecount, which counts over a byte slice. If needed, I'm assuming I could count of multiple byte slices and then sum up in the same way.

One advantage of having a server power vimwiki inspection is that we can do fancy tasks that would be impractical in vimscript. One such task that could be done after #14, would be to evaluate a link to indicate:

1. If a local file/directory, does it exist? Can be used with #14 to highlight a link in different colors (or use virtual text for details) just like with rendered wiki pages
2. If a remote URL, does it return okay? Same as above, could highlight in a different color (or use virtual text) to indicate if successful
3. Alongside verification, could use a floating window to show a preview of the page (if a wiki file) or HTML content (if a webpage) triggered by some command on hover

Inspired by Option<T>, provide support for exposing the inner value of a LocatedElement.

This involves renaming the interior from element to inner and then providing these four methods, which will be useful in #12 and all of the graph-based issues where I need references to elements.

pub fn as_ref(&self) -> LE<&T>;
pub fn as_mut(&mut self) -> LE<&mut T>;
pub fn as_inner(&self) -> &T;
pub fn as_mut_inner(&mut self) -> &mut T;
pub fn into_inner(self) -> T;

vimwiki

• Core: language components and utilities like LocatedComponent, Region, and Point
• Features:
  • "parse": expose the parsers module and include dependencies like nom
  • "serde": derive serde serialize & deserialize on components
  • "juniper": derive juniper graphql object on components (some may need custom definitions, scalars, etc)
    • NOTE: No custom scalar is needed for NaiveDate, but a custom feature on juniper is needed
    • NOTE: If we go the scalar route for uriparse's URI, we'd need to make a newtype wrapper as described in graphql-rust/juniper#429
  • "locate": add module that has ability to transform an LC<Page> into an rstar tree where we can query for a component by line & column
    • NOTE: Would also expose ability to walk up the tree. For instance, finding a list item and walking up to the list, etc.
  • "tag-graph": add module that has ability to extract a graph of tags to parent components using petgraph
    • NOTE: Tag is hashable and can be converted to a string, so we want the graph to be able to search by tag, string, and &str
  • "convert-json": add method to LC that does serde_json's to & from json
  • "convert-html": add method to LC that converts to html
  • "debug-timekeeper": add our timekeeper code for context (and include lazy_static dep)

vimwiki_macros

• Core: produce rust code to construct components

vimwiki-server

• Core: graphql server endpoint (juniper, warp, and clap deps)
• "watch": Watches a directory for changes and automatically re-loads those files (notify)
• "multi-threaded": Supports multi-threaded execution of parsers (tokio workers for pages); may need something like this or "async" in vimwiki crate for parsers

Convert decorated text into an enum instead of the structs setup, migrate code out of enum, and simplify content.

To support vimwiki URIs, I was originally using the url crate, but it turns out that this has a different spec that results in file:../../images/img.jpg getting translated to file:/images/img.jpg as described in servo/rust-url#641 and vimwiki/vimwiki#989 (comment).

The uriparse crate now supports stable rust as of version 0.6.2, although it does not support serde serialization/deserialization yet. sgodwincs/uriparse-rs#9 is open to do this and I may be able to write up some sort of serialization for it or at least a wrapper struct that contains a String (what was to be parsed) that can be serialized (skipping the uri) and deserialized (re-parsing). Question would be how to fill in the uri when deserializing and throwing an error if failing to parse the string upon deserializing.

If we have a data structure to represent a page and all of the elements it contains, we could use a library like https://github.com/petgraph/petgraph to connect pages together (for the bi-directional linking) to make it easier to navigate internally.

The above library supports serde for serialization/deserialization, which means we could save the graph links alongside the page data as discussed in #4.

Each template could look like:

     <html>
     <head>
         <link rel="Stylesheet" type="text/css" href="%root_path%style.css" />
         <title>%title%</title>
         <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
     </head>
     <body>
         <div class="content">
         %content%
         </div>
         <p><small>Page created on %date%</small></p>
     </body>
     </html>

where
%title% is replaced by a wiki page name or by a |vimwiki-title|
%date% is replaced with the current date or by |vimwiki-date|
%root_path% is replaced by a count of ../ for pages buried in subdirs:
if you have wikilink [[dir1/dir2/dir3/my page in a subdir]] then
%root_path% is replaced by '../../../'.
%wiki_path% Path to current wiki-file.` The file path to the current wiki
file. For example, if you are on page a/b.wiki %wiki-path% contains
"a/b.wiki". Mostly useful if you want to link the to raw wiki page from
the rendered version.

%content% is replaced by a wiki file content.

The default template will be applied to all wiki pages unless a page specifies
a template. Consider you have wiki page named 'Maxim.wiki' and you want apply
'person.html' template to it. Just add: >
%template person
to that page.