Flesh out starting details about paranext HOT 15 CLOSED

✅Repo structure

Discussion:

• Monolith
  • Easier development
  • Easier CI/CD
  • Easier to keep compatibility between parts, share code
  • Don’t want to contribute - could organize well and keep self-contained. Does that reduce intimidation?
    • Subrepositories?
• Split into many
  • More annoying to develop
  • Harder to do CI/CD
  • Harder to keep compatibility between parts, share code
  • Easier to contribute
  • Cleaner divisions and separation
  • Submodules
    • Allow full PT dev to get all of PT together with everything in the right place
    • Allow extension devs to get just what they want
    • Create extension template repo for internal and external use

Original Decision:

Split into many repositories with submodules.

1. paranext - master repo, has all the submodules and such
2. paranext-core? - contains frontend and backend. Not sure if we need this
3. paranext-backend
4. paranext-frontend
5. paranext-extension-template
6. paranext-core-extensions? - contains all built-in Paranext extensions as submodules
7. Each extension gets its own repo

Follow-up: should we make our own paranext organization?
Yes

New Decision:

Split into many repositories. One repository holds "Paranext-wide" workflows, actions, and such

1. paranext - Paranext-wide workflows, actions, dev instructions, etc
2. paranext-core - contains frontend and backend
3. paranext-extension-template
4. Each extension gets its own repo

from paranext.

✅How will PT9 and Paranext interoperate?

Discussion:

• Paranext and PT9 are required to work side-by-side with no extra step required
• Conversion on S&R server
  • Client doesn't have to worry about details of conversion or what version the project is on
  • Less extra data stored on client
  • Faster for the client in sense of not having to import/export
  • Don't have to preserve PN files in PT9 project folder (can have two separate repos for one project)
• Conversion on client - import a temp folder of a PT9 project, then export when appropriate
  • Client converts, so we can convert anytime (automatic on save or whatever)
  • Can be done offline. Users can swap between PT9 and Paranext freely (don't open at same time)
  • Using familiar languages (C#, JS) to do the import/export. May even have some code that helps already like writing data to file
  • All the PN data can be stored in PT9 folder

Decision:

Paranext will have a different dataset, and it will have some synchronization step with PT9 at some point in the future.

from paranext.

✅How will we leverage existing PT9 code?

Discussion:

• Use ParatextData.dll
  • Get code fixes, new features, data format changes, etc.
  • Harder to fix bugs, features in ParatexetData.dll esp for people who don't have access to Paratext repo
  • New Paranext features would be harder to pull forward
  • Harder to encapsulate things that are not currently well-encapsulated
• Pull ParatextData.dll code forward and modify
  • More open source code
  • Easy to maintain, add features
  • Hard to keep in sync with ParatextData.dll fixes, features, data format changes, etc.

Decision:

Use ParatextData.dll for now, and pull code in later as we need.

from paranext.

✅C# Extensions?

Discussion:

• Pros
  • Supports existing plugins better like Transcelerator (Tom Bogle will kill us if we don't), but would need some reworking either way since Plugin API for WinForms is going away
• Cons
  • No way to sandbox if we were to go with security

Decision:

No C# extensions. The new software is so different that almost everything on an existing plugin will probably need to be rewritten anyway.
Note: there will be some C# code in core that responds to extension API calls.

from paranext.

✅New API endpoints vs extend existing API endpoints

Discussion:

• New API endpoints
  • Lock us into a certain API technology like REST or gRPC or something
  • More familiar API for devs
  • More flexibility in how an endpoint works
• Extend existing endpoints
  • Hide implementation behind our own functions
  • Two-way messages are easier
  • Keeping data up-to-date is easier for extension devs (don't have to implement their own network layer)

Decision:

Extend existing endpoints through our APIs

from paranext.

✅Node vs Deno

Discussion:

• Node
  • Widely used, lots of tooling, powerful
  • Familar. We know it can do what we need. Other devs will know how to use it
  • Not built for security
  • Established
• Deno
  • Unfamiliar. Don't know much about this. Probably can do what we need. Other devs may not know how to use it
  • Not widely used
  • Has built-in webpack-y and dev tool kinds of things, but not necessarily as flexible
  • Built for security, but can't allow extension host to have network access but extension doesn't (can't change process permissions when forking)
  • Not established

Decision:

Go with Node. If Deno replaces Node one day, we can try to swap over, but extensions will likely also have to swap over unless we decided to serve them through a CDN that packages it all up to use in either environment.

from paranext.

✅How much core Paranext will be made of extensions?

Discussion:

• Could go to the extreme and make everything an extension like even internet access or file access extensions
  • Then we have to release access to some extension
• APIs for core to access internet and files
  • Then no extension has direct internet access itself

Decision:

• API features that are built into core - fundamental to PT or every extension needs - Resource stuff, internet access, file system access
• API features that are not built into core - things specific to an extension - implementing a ResourceView, setting up a command

from paranext.

✅RTE - starting point and data format

Discussion:

• Chris Klapp's USFM editor
  • Nice compatibility - inputting USFM sounds convenient, uses data we will probably want to serve whether or not we keep this editor around
  • Primary data is string, not tokenized as anything. Don't have paths for updating, annotations, etc
• Proskomma editor?
  • Know very little about this
• POC Slate editor
  • Keeps the round-tripping to a minimum
  • Power of a RTE library
  • Lots of work to do - If we end up scrapping this for another RTE library, we will have to do different data conversion

Decision:

Push off the decision regarding Proskomma and editor library. Not as critical as others. Let's go with USFM for the moment and see what happens as we move forward.

from paranext.

✅How to deal with project-scoped extensions?

Discussion:

• Project admin can assign extensions to projects, and new project files can't be added by an extension that isn't assigned to a project
• Users are prompted to install assigned extensions that they don't have installed
• Each extension has its own project data folder and others disallowed from messing with it
• Users can still use anything they want that displays things or edits existing data in whatever way they want, but they can't add new data with their own random extensions
• If a consultant wants to add some kind of checking data with some extension, they don't want to have to get the project admin to add their extension so they can work
  • Allowing any extension to add project data in their own individual folder does add risk of extension developers adding way too much data in the project folder that is then S/R'ed around everywhere. Not worth worrying about right now

Decision:

Projects can require and recommend extensions, but we don't need to restrict non-approved extensions from creating files. Maybe consider file protection in the future.

from paranext.

✅Extension ecosystem - how are extensions found without curation?

Discussion:

• We host an extension marketplace
  • Could implement an API feature at some point that allows people to add their own downloadable extension lists (extend our marketplace with their own marketplace integrated with ours)? Or allow them to add their own URL like Nuget
• Orgs can verify extensions for their members to see in the marketplace
• Users can select a menu item and type (sign) their name in a prompt telling them they accept responsibility and can see unverified extensions
• Extensions display lots of stats like number of downloads, reviews, tests passed, etc. Maybe some info about their functionality based on the manifest.json?

Decision:

We host an extension marketplace for anyone to contribute. Organizations can verify extensions for their members to see in the marketplace. Otherwise, only verified extensions are show in the marketplace. If a user wants to see unverified extensions in their extension marketplace in Paranext, they can do some action to confirm that they assume responsibility and can see all extensions on the marketplace. Extensions possibly display some stats for people to make informed decisions.

from paranext.

✅gRPC vs REST/Web sockets vs Web sockets

Discussion:

• Want some kind of two-way communication, so prefer not long-polling
• gRPC seems like it isn't working too well with web implementation
• Everything going through web sockets
  • Simpler
  • Higher potential for bottle-neck
  • Lots of languages have libraries for web sockets but not necessarily as simple as REST
• Most everything going through REST, web socket messages alert clients about updates, get updates through REST
  • Bit more clunky, complex than going through one avenue
  • More points of communication, so less bottle-neck
  • REST is very widely supported in many languages

Decision:

Websockets. Expecting connection overhead to be a larger issue than data throughput for now. We can optimize large data transfer later as needed since this is all internal.

from paranext.

✅How to keep UI consistent?

Discussion:

• Web components?
  • Supports multiple languages, more flexible
  • More work, more technologies to learn
• React components?
  • What we are already using
  • Can wrap into web components later
• Well-documented CSS classes
  • Not sure if we need these if we have React components already

Decision:

React components! Try to allow using directly from extension webviews. Make separate npm package if not? May or may not need CSS classes passed down at some point.

from paranext.

✅What extension developer experience features are we aiming for?

Discussion:

• Debug mode - allows running unzipped extensions?
  • Menu item and command-line argument to do debug mode
  • Run unzipped extension
  • Enable DevTools shortcuts for easier access (prevent easy access outside debug mode since that would confuse normal users)
• Debugging extension code?
  • Very good to have. Pretty much need this
  • Need some way to hook into extension host code and html that is delivered to the frontend
• Hot reloading extension code?
  • Good to have. Worth looking into but don't absolutely need it
• Unit and UI tests for extensions in Paranext? https://github.com/microsoft/vscode-test
  • Want some kind of straight-forward way to do tests on extensions. Probably in template
• Logging to Paranext log
  • Absolute must

Decision:

Debug mode with above specifications. Need to find a way to debug extension code on extension host and in html. Need some kind of unit testing on extensions (may provide testing framework as part of template). Need logging to Paranext log. Try to get hot reloading as it would be nice.

from paranext.

✅C# or Node as primary webserver?

Discussion:

• C#
  • Pretty familiar (ASP.NET is not, but C# itself is)
  • Only one websocket to go through to hit C# code
• Node
  • Somewhat familiar (webserver technology is not, and JS is only to TJ)
  • Might go this way in the future as we will likely migrate more and more to JS over time
  • Only one websocket to go through to hit JS code (extensions)
  • Server IS the extension process manager
  • C# code could be in an "internal executable" extension to use extension API routes a bit better (but then we would be tying ourselves to the idea of an internal executable, which seems significantly less secure/robust)
  • Encapsulate and divide C# code into individual extensions (C# code for an extension can be packaged together with JS code for that extension)
  • Don't have the weight of an ASP.NET server running
• Are we limiting ourselves using Node websocket as opposed to a C# websocket?
• How do startup and resource use look between C# and Node?

Decision:

See Web server or Electron-Edge. Try Electron-Edge first. Otherwise consider further. Likely going with Node server as we will likely be phasing out C# over time.

from paranext.

✅Web server or Electron-Edge?

Discussion:

• What code would this decision impact? Can we put an anti-corruption layer around it to be flexible?
  • All website code would be served from Electron vs from webserver. Implementation detail, but this could be significantly complex
  • Calling to C# code through Edge instead of web socket messages.
    • In JS, can theoretically use Edge either way. But could make C# calling service that handles implementation details. Anti-corruption layer
    • In C#, could make a web socket client that calls the C# methods in a very similar structure to how Edge would call the methods (basically split them off like a dll)
  • Messaging between backend and frontend web sockets vs electron ipc can be through anti-corruption layer
• Web server
  • Need to connect frontend to the web server
  • Async full support
  • Great architecture for future-proofing with browser support, headless
  • Takes more advantage of extension architecture in calling C# through commands and such
  • Dog-food web socket external executable API
• Electron-Edge
  • Relying on seemingly precarious technologies maintained independently
  • Frontend and backend connected through ipc automatically
  • Don't have built-in web socket delay time (Seems to be ~.5 ms per message per direction on TJ's computer with just one message at a time. 1 request from frontend to epm and back takes ~2ms total. 1000 requests fired all at once take 123ms total with average 77ms. 10000 requests take 1171ms total with average 686ms.)
  • Easier to connect to C# code
    • supports synchronous main-process-blocking execution of C#
    • Have to fork Node process to run async C# code as edge blocks the current process while running whether or not the C# is async. Could fire pool of processes to do this, fire up more as needed. Not sure of performance/memory implications
  • Not future-proofing for browser support; would require significant work. Probably need to do a lot of work to support headless without electron. Alternatively, could run normal electron main process but not launch renderer process.
  • Dog-food internal extension API but not external executable API which would still need to use web socket or something similar
• Web server with Edge
  • Relying a bit but less on seemingly precarious technologies maintained independently
  • Need to connect frontend to the web server
  • Async full support on frontend, but backend still gets blocked by C# code and would need to be placed on forked process
  • Easier to connect to C# code as in Electron-Edge above
  • Great architecture with future-proofing with browser support, headless
  • Dog-food web socket external executable API
  • Doesn't completely eliminate web-socket delay time but maybe halves it since calls to JS and C# go through same amount of web socket

Decision:

Create a POC for async coding in Electron-Edge with forking process and possibly threading in C# side. If that goes well, probably start with Electron-Edge. Let's consider further otherwise.

from paranext.