Giter Club home page Giter Club logo

mathdown's Introduction

Online collaborative markdown with math. Main features:

  • Markdown is styled in-place, no source/preview separation.
  • Edits are synced in real time.
  • Access control is simply by sharing the secret URL. No sign up needed to collaborate.
  • LaTeX-syntax formulas rendered in-place when cursor is outside formula.

Powered by CodeMirror, MathJax and Firebase's Firepad. I'm using "CM" = CodeMirror, "MJ" = MathJax abbreviations a lot in the project.

Alpha quality – will eat your math, burn your bookmarks & expose your secrets. I mean it. See for example #85 — saving would sometimes be silently broken, for half a year! I'm working to make it more robust (and tested) but for now, be careful.

Issues: mathdown HuBoard CodeMirror-MathJax issues

License

My code is under MIT License. Exception: font/ contains a subsetted Bitstream Charter font under a permissive license - see fonts/LICENSE.

Dependencies:

  • CodeMirror is also MIT.

  • MathJax is under Apache License 2.0.

  • My CodeMirror-MathJax glue is also MIT.

  • The collaborative editor Firepad is MIT. It calls firebase javascipt API.

  • Firebase is a proprietary service; their client-side javascipt API firebase.js is also proprietary, though apparently fine to distribute in practice — (#4). [firbease.js has been accidentally MIT-licensed for a time but I've upgraded to newer versions so this doesn't apply.]

    I'm not including firebase.js directly but using it as a git submodule.

Document hosting and privacy(?) on Firebase

All user data is stored in Firebase, now owned by Google. Their privacy policy. Documents access (read AND edit) is by secret document id which is part of the url. This is grossly unsecure unless using HTTPS.

The downside is users can't really control their data. Running a "self-hosted" copy of the site still leaves all data in the hands of Firebase. See #4 for more discussion.

The upside is all forks interoperate; you can change the design or tweak the editor and still access same documents. E.g. https://mathdown.net/index.html?doc=demo and http://rhythmus.be/mathdown/index.html?doc=demo look different but access the same doc -- and real-time collaboration between them works!

I'm so far on the free Firebase plan - 100 devices (not sure if 1:1 with users), 1GB Data Storage (used < 100MB). => Will need 49USD/mo plan as soon as I get non-negligible usage. https://mathdown.firebaseio.com/?page=Analytics (only visible to me)

Deletion is impossible

The current Firebase security rules make document history append-only. That's a nice safety feature but it means that once a document's URL gets out, it's full history is forever accessible to the the world. This must change eventually (#92).

Browser support

Basically whatever CodeMirror supports: IE8+ and about everything else. But mobile is currently almost unusable (#81).

JavaScript is required (and this includes running the non-Free firebase.js in your browser). You can't even read documents without JavaScript; reading won't be hard to fix (#7) — but editing documents without JavaScript is implausible (I plan to settle for append-only form).

Cookies

The only cookies I'm aware of:

  • mjx.menu cookie set for a year(?) if you manually change MathJax settings.

I'm not sure Firebase never sets cookies. Things will change once I implement login (#50).

Installing dependencies

Dependency Status devDependency Status Greenkeeper badge

  1. After checking out, run this to materialize client-side dependencies:

    git submodule update --init --recursive

Append --remote to upgrade to newest versions of all submodules (need to commit afterwards if anything changed). Known constraints on updating all deps:

* firepad only includes pre-built dist/firepad.js in tagged versions (after every release they strip it back).
* [CodeMirror-MathJax currently doesn't support MathJax 2.5](https://github.com/cben/CodeMirror-MathJax/issues/33).

(I'm directly working in gh-pages branch without a master branch. GH Pages automatically resolves https://... submodules. It's no longer the primary hosting but it's still useful to test the static version works.)

  1. To install server-side dependencies (and devDependencies) listed in package.json run:

    npm install
    

(But when deploying to RHcloud or Heroku, npm install might run in --production mode and devDependencies won't be available.)

To see whether any updates are needed/possible, run npm outdated. To update run:

npm update --save
npm shrinkwrap

Then commit the new `package.json` and `npm-shrinkwrap.json`.

TODO: find way to use same node.js version in dev and prod?

Test(s)

Travis test runner Saucelabs browser tests

test/browser-on-saucelabs.spec.coffee runs tests on several browsers using free browser testing courtesy of Sause Labs. There are pathetically few tests.

To run the tests:

npm install  # once
npm test

To run only some tests and/or browsers, use:

./node_modules/.bin/mocha --grep firefox

The test runs automatically on any commit and pull request. I've tried several free services for this, and currently prefer Travis:

  • Travis - works, open source code. Controlled by .travis.yml.
  • Wercker - works. Controlled by wercker.yml.
  • Drone - Docker-based, open source rewrite in progress. Alas, always timed out during test. Test config on the web.
  • Shippable - builds history only accessible by me? Bad, I want public. Controlled by .travis.yml.
  • Codeship - same, dashboard is private. Test config on the web.

Where it's deployed and how to run your fork

The main deployment currently runs on Heroku. See deployment/ subdirectory for details. I'm interested at going back to static hosting.

However you run it, you can open the same document ids (doc=...) and real-time collaboration will work!

Quick ways to run:

Deploy on Heroku:

heroku create my-mathdown --remote heroku-my-mathdown
git push heroku-my-mathdown gh-pages:master

some other ways to deploy on Heroku might not work due to my use of submodules (?)

Run local server (server.coffee):

npm install  # once
env PORT=8001 npm start  # Prints URL you can click

(you can choose any port of course. Ctrl+C when done.)

This app mostly works as pure static pages, and I intend to keep it this way.

  • From a checkout, just open index.html in your browser.

  • Github Pages serves the gh-pages branch at https://cben.github.io/mathdown.

  • For other branches/commits, there is no trivial solution - rawgit.com would be great but doesn't currently support submodules.

  • The easiest way to run (and share) uncommitted modifications is probably Cloud 9. TODO: test, details.

The only benefits the dynamic server is going to bring (not implemented yet) will be:

  1. Including the document text in the HTTP response for search engines (#7).
  2. Prettier mathdown.net/foobar instead of mathdown.net/?doc=foobar URLs (#59).

Other things called "mathdown":

I should really talk to these folk whether it's OK that I'm using the name and the domain...

mathdown's People

Contributors

cben avatar dependabot[bot] avatar greenkeeper[bot] avatar greenkeeperio-bot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mathdown's Issues

Support RTL text

Codemirror supports bidi well enough, but all lines are left-aligned.
Want gedit-like automatic paragraph direction. (for starters, per-line direction would be good.)

URL tracking first line spams browser history

Every keypress in first line creates new entry in browser history, because the URL fragment changes.

Should somehow delay/debounce it. Note that simple constraining it to cursor out of first line won't work in co-author's browsers which similarly track the first line but have cursor elsewhere.

Mar 5, 2014
\newcommand{\Z}{\mathbb{Z}} | mathdown
mathdown.net

Mar 4, 2014
\newcommand{\Z}{\mathbb{Z}} | mathdown
mathdown.net

Mar 4, 2014
\newcommand{\Z}{\mathbb{Z} | mathdown
mathdown.net

Mar 4, 2014
\newcommand{\Z}{\mathbb | mathdown
mathdown.net

Mar 4, 2014
\newcommand{\Z}{\mathb | mathdown
mathdown.net

Mar 4, 2014
\newcommand{\Z}{\math | mathdown
mathdown.net

Mar 4, 2014
\newcommand{\Z}{\mat | mathdown
mathdown.net

Mar 4, 2014
\newcommand{\Z}{\ma | mathdown
mathdown.net

Mar 4, 2014
\newcommand{\Z | mathdown
mathdown.net

Mar 4, 2014
\newcommand{ | mathdown
mathdown.net

Mar 4, 2014
\newcomband | mathdown
mathdown.net

Mar 4, 2014
\newcomand | mathdown
mathdown.net

Mar 4, 2014
\newcoman | mathdown
mathdown.net

Mar 4, 2014
\newcoman | mathdown
mathdown.net

Mar 4, 2014
\newcom | mathdown
mathdown.net

Mar 4, 2014
\newco | mathdown
mathdown.net

Mar 4, 2014
\newc | mathdown
mathdown.net

Mar 4, 2014
\new | mathdown
mathdown.net

Mar 4, 2014
\ne | mathdown
mathdown.net

Mar 4, 2014
\ned | mathdown
mathdown.net

Mar 4, 2014
\n | mathdown
mathdown.net

Mar 4, 2014
| mathdown
mathdown.net

In-document anchors

Decode #foo in URL and scroll to # Foo heading inside document.
(Doable with pure id="foo" on line DOM? Unlikely if CodeMirror handles scrolling.)

Make # in headings clickable and result in such permalink.

LaTeX export

  • Either use Docverter or host pandoc myself on heroku.
  • One-click button to then import into writelatex and/or sharelatex.

Far away:

  • Markdown lint that warns about suspicious markdown (e.g. list not preceded by newline).

Read and write offline

  • A way to even open cached mathdown when you're offline.
    Look into app manifest / chrome extension / firefox OS app?
  • Cache last seen content of every document in localStorage.
    Bonus if having any mathdown open does background updating on all known docs.
  • Be able to edit offline and sync on reconnect.
  • index.html?doc=... prevents caching of index.html for new documents.
    history.pushState() might help, assuming you have one tab open.

Offline debugging mode

Currently mathdown doesn't work offline.
It seems that firepad would initialize and allow pasting text if only firepad.js was available. But it's proprietatry so not sure I can (or want) to put firepad.js in my git. And I'm not keen on starting to use npm/bower and losing the conveniece of Just a Bunch of Files...

Let CodeMirror do scrolling

Many things (including math) perform better if CM does scrolling and doesn't have to create DOM for invisible parts of doc. Blockers:

Security: hide secret doc id from Referer header

Almost not an issue now but blocker for #9:
Since the URL is secret, directly linking to external sites would expose the URL in Referer: header.

Should probably use an extra redirect. Is there a way to keep the link structure search-engine friendly? Links on public pages should count as links. OTOH, if they're publicly editable they probably shouldn't to deter spam?

Alternitive: I'm again tempted to keep secret portion in #fragment, which should not (though sometimes did) leak via Referer.

Stronger tests

Yesterday I pushed ad146e1 which updated CodeMirror but not Firepad, resulting in a combination where firepad doesn't sync any changes ("pos exceeds the bounds of the AnnotationList").
I knew all about it and even prepared a correct update on a new-deps branch but forgot about it, and only tested the CM commit in my dirty checkout which also had new firepad... FAIL! I should fire myself.

But my main point here is that my laughable smoke test didn't catch that I completely broke mathdown (pages did load but edits weren't saved! That's even worse that being visibly down!)

I really need a functional test opening same page from 2 instances and seeing edits propogate.
Should check if I can reuse existing firepad tests...

Also, JS console errors should be blockers. See https://code.google.com/p/selenium/issues/detail?id=148

Spiderable documents (content in HTTP response)

Search engines, or Javascript-less frontends (links, curl) etc. should be getting a at least a plain-text version of the document content (in a text area). Bonus points for:

  • rendering links as links.
  • full server-side CodeMirror styling.

Either way needs node.js on the server.

Support pandoc-style citations

This probably means:

  • a way for user to supplies a bibtex file (global file per user? research Zotero / Mendeley / citeulike etc integration...)
  • autocomplete of citation keys
  • tooltip showing full citation, behave like a clickable link (#9)

presentation mode

  • mode to show one section at a time
    • readonly vs edit modes?
  • beamer export
  • http://slid.es/ or similar export? Better yet, edit markdown here, use reveal.js or similar for beautiful view?

tables

  • pick "supported" table format(s)
  • use monospaced font in tables
  • align help, a-la Org mode

Hide all formatting chars except for currently edited?

http://mangomarkdown.com/ does this. I'm not quite convinced it's a good idea — it becomes too close to a WYSIWYG editor (with an obscure way to format things) to my taste.

But here is a very simple idea for implementation that might just work: enable styleActiveLine option, use markdown mode's option to style formatting chars, hide them with CSS except for current line!

Incremental math rendering

The initial render on document load is horribly unacceptably slow (saw 1 minute on a math-heavy page).
The unrendered text does show very fast (#15) but it's not very useful since the editor is paralyzed until all math is rendered.

Cache list of needed mathjax extensions in firebase

MathJax does all typesetting in-order (TODO: could be because I'm queueing one by one, check with one Typeset call on long string).
Which means any extra extension to be loaded delays the whole process.

We could save the list of mathjax extensions actually needed for a given document in Firebase, and pre-load them all at start. Could then give up the -full config.

Login => List of user's documents

The basic need is straightforward, the question is what kind of login do I want (cf #41 for the other major use case for logging-in users).

  • Simplest: login required to remember visited documents.
    Should start with this, easiest to implement.
  • Better (a-la writeLaTeX): when anonymous, still remember documents in localStorage (or firebase with anon auth cached in cookies/localStorage?)
  • Fancy partial login: first just ask for email, send secret URL for accessing your doc list.
    This effectively validates email, but asynchronously — link only needed for 2nd login.
    If link is lost, can resend it — similar to password reset.
    After using secret link, can optionally set a password?

Also, should allow search.
Firepad indexing is hard, but something as dumb as loading ALL docs into the doc list page for browser search to work might be acceptable if sufficiently lightweight (just <pre> with no highlighting, load snapshot instead of firepad reconstruction...)

Should probably expose doc title in firebase, without reconstructing firepad text.

Default text may OVERWRITE existing text

I created a new document, which was surprisingly empty.
Typed several lines.
Then saw them disappear and the default # Untitled ... text replaced them :-(
Luckily, Ctrl+Z twice brought my text back.

Anat also said it has happened to her.

Probably a race condition, may be related to slow loading (cold cache + bittorrent running).

Formatting inside quotes

> quote
# Header 0123456789

Renderend now as a header inside a continued quote.

(The non-portability here is one of these things where having a "markdown lint" would help.)

Made worse by 2 bugs => only Header is colored green as a quote.

  • The # is bright-bluish - wrong CSS cascading?
  • 0123456789 is blue, somehow due to GFM styling it as github SHA link and my kludge unstyling it.

Should be

But here things are

Markup for theorems

Find and recommend some way to write theorems, lemmas etc. in MD and get them converted to \begin{theorem}... in latex.

Pandoc supposedly added arbitrary <div> and <span> but I haven't been able to reproduce yet — probably need a newer version.

Security: HTTPS

Github pages doesn't serve HTTPS with a valid certificate.

  • register mathdown.{com,net}
  • get a cert (StartSSL offer free ones, and cheap recursive ones — which might come in handy for staging new versions / forks etc.)
  • find free/cheap hosting with easy deploy from github.

P.S. Cloud9 workspaces like https://c9.io/cben/codemirror-mathjax/workspace/demo.html support HTML. Great for editing / sending others a demo, not good enough for the main site.


It's tempting to use #xyz (as in Firebase demo) instead of ?doc=xyz,
so that secret document ids are not sent in the plain (they'd only be sent to Firebase over https).
But this doesn't really make HTTP hosting OK — a man in the middle can easily inject malicious JS that would leak the document id.
A stronger point is that ?doc=xyz would be stored on server's log, even with HTTPS. Not sure it's evil, collecting URL-level statistics is standard and useful.

What's more important, I want ?doc=xyz to preserve disting documents having distinct URLs from search engine POV. I'd use /xyz but then I'd need .htaccess or similar.
And I want to use #heading for linking to # Heading inside doc (#5).

Persistent undo / history

Per-user undo seems to work nicely (thanks to firepad).

But once you refresh the page you can't undo older changes, probably because you got a new random user id.

  • When I implement any kind of login, make sure to assign a fixed firepad user id.
  • Sticky user ID in a cookie. (Is there any anonymity/tracking concern?).
    This is useful even after login is implemented

Additionally, look into other ways to expose the firepad history (e.g. etherpad-like slider?). It's all there, just no way to access it.

AsciiMath

It seems someone wants AsciiMath support, as evidenced by this attempt on the front page:

Suppose $$ax^2+b x+c=0$$ and `a!=0`. We first divide by `a` to get `x^2+b/a x+c/a=0`. 

Then we complete the square and obtain `x^2+b/a x+(b/(2a))^2-(b/(2a))^2+c/a=0`. 
The first three terms factor to give `(x+b/(2a))^2=(b^2)/(4a^2)-c/a`.
Now we take square roots on both sides and get `x+b/(2a)=+-sqrt((b^2)/(4a^2)-c/a)`.

Finally we subtract `$$b/(2a)$$d` from both sides and simplify to get the two solutions: 
`x_(1,2)=(-b+-sqrt(b^2 - 4a c))/(2a)`
  1. I could easily enable AsciiMathML in mathjax, but not with backtick syntax - it's already taken by literals in markdown.
  2. Markdown with AsciiMathML wouldn't be very useful; the only converter I know that understands it was MultiMarkdown 2, replaced by LaTeX math since version 3.
    This does suggest the syntax should be MMD2-compatible: << e^(i pi) + 1 = 0>>.

I see how AsciiMath is attractive for easier input (e.g. a/b compared to \frac{a}{b}) and some degree of Do-What-I-Mean magic for beginners.
In the long run, I believe LaTeX math is the right syntax for stored documents, for power and interoperability.
The benefits of AsciiMath could be realized if you could type a/b and Mathdown would suggest to replace it with $\frac{a}{b}$ (cf. #36). But these are very far-off ideas, whereas enabling AsciiMath is doable now.

FIREBASE WARNING: set at /firepads/.../checkpoint failed: permission_denied

Hard to reproduce — only after a lot of writing, when firepad tries to checkpoint.

/checkpoint writes are conditioned on "author of checkpoint is the same as the author of the revision they're checkpointing."

A couple times seen with these network errors:

Failed to load resource https://s-dal5-nss-8.firebaseio.com/.lp?start=t&ser=14891542&cb=547&v=5&ns=mathdown
Failed to load resource https://mathdown.firebaseio.com/.lp?start=t&ser=19417576&cb=548&v=5

Today seen with "undefined has no .from property." exceptions.

Author-attributed text

Needed for comments, chat, threaded responses, everything.

An obvious direction is Etherpad-like magic coloring of all text by author. But I'd like to steer cleer of magic text properties and keep the document model be plain text [until proven that it doesn't work].
And there are really 2 scenarious:

  1. Collaborative writing, where the end result should be unattributed (though a special history/blame mode is occassionally useful). In Etherpad, you nuke all coloring when it gets annoying.
  2. Discussion (including comments on writing), where who wrote what should be part of the text.

Ideas:

  • <user> text attributed quotation a-la IRC logs.
  • @username and/or email@domain. I'm leaning towards email with compact rendering as a gravatar and autocomplete for fast typing.
    • Potential conflict between @user notification ("to") semantics and attribution ("from") semantics I need here.
    • Conflict between @ and pandoc citations.
    • Email makes more portable documents. [Unless I adopt 1:1 login-with-twitter usernames.]

List assist

Getting lists right is tricky for markdown newcomers.

  • Enter should probably continue list item; second Enter would start new item; third Enter would close list.
  • List items should auto-indent so text (not bullet) is aligned to multiple of 4 spaces, so continuation lines can line up nicely. Reindent when typing - ("electric"), on Enter, or both?
  • Tab/Shift+Tab when notheng is selected should affect whole paragraph / list item (incl. sub-items) etc.

Steal ideas from Stack Exchange?

http://meta.stackexchange.com/a/56430/204832
Actually Shift+Enter is a continue formatting command: I.e. it doesn't continue extra indenting within code, but it does add the first 4 needed spaces if you're already in code. It also adds the next numeric value for a numbered list, a copy of the first mark for an unnumbered list, and for both of the latter it "fixes" the other numbers/marks to be sequential/the same. It also continues > blockquotes. – Mark Hurd Jul 10 '10 at 3:16

Only consider doing all that on unshifted Enter?

Activity indication

  • Unseen activity indication in favicon. This is easy.
  • Some kind of transient highlighting of what changed since you last looked? This is hard — when exactly have you seen something? Track your viewport? Cursor motion perhaps is good middle ground (but then if 2 people are doing append-only chat, are their messages never considered read)?

This may overlap with #41 but I think I do want in-text names while side-channel magic for unread activity is more acceptable as a projection of the history of the text?

Math not highlighted when selected by others

Firepad uses markText with className to color selections by other users.
Turns out widgets don't get any classes applied, even when fully covered by a marker. Marijn says it's intentional: https://groups.google.com/d/msg/codemirror/L7VPuMrul78/l_iicpP97N4J

So should apply selection styles automatically to math widgets.
Should (1) apply existing selection classes to new widgets, (2) apply new selections to existing widgets.
Both should probably use recently added markerAdded, markerCleared events.
The place to do it is probably firepad, but with these events also can do it myself.
Or maybe CM addon?

Doc load performance

Initializing Firepad takes over 40s for long documents!
This is not result of MathJax, slow even if disabled.
It seems that Firepad fails to create snapshots due to security rules. Perhaps I must use simple login even for anonymous users?

Firebase is proprietary

The Firebase backend is proprietary so you can't run your own instance of the full stack.
At this stage, I would not trust myself with running a secure & reliable backend, so using Firebase is better for Mathdown end-users.

OTOH, there is little need to change anything about Firebase and being able to develop as a purely client-side code makes forking and testing much easier - a user can try any fork without exporting/importing user data.
"Running" a fork is as simple as loading the static HTML from anywhere - local checkout, github pages, cloud9 workspace etc.

Any free solution should preserve the ability to fork and use without migrating data.


Best free-backend alternatives to Firepad:

For other firebase-like functionality (if we add any), meteor.com looks most similar in spirit, and even provides free hosting for server-side part of the app (arbitrary node.js?).

Printing

Shows a scrollbar instead of full text.

\symbol autocomplete

Should offer autocomplete when typing \foo.

  • Autocomplete dollars when outside math: \al -> $\alpha$ (with cursor before closing $ if possible).
  • Bag-of-words search & spelling, ideally \arow should offer \rightarrow etc.
  • Display symbols in completions list.

Big question: should I autocomplete to unicode chars (e.g. \beta -> β) to encourage Mathdown's "the source is the document" ethos? It's seems to be supported well enough by MathJax(?). Should limit to only chars that work with \usepackage[utf8]{inputenc} on pdflatex. Anyway there must be an option to export back to ascii.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.