Notes regarding creating a GitHub Discussions Forum on the opendp/opendp repository.

• Not ideal in that it's at the repository rather than org level.
• The preference is to only have a single forum for the opendp org
  • This would replace Gitter for now. (Gitter is moving/being merged into Matrix, e.g. we may look at that in the near future)
• Given team resources, it's more manageable to keep everything within GitHub

Tasks:

• Add GitHub Discussion + initial topics
  • OpenDP Library
  • Differential Privacy (Theory/Practice)
  • SmartNoise Library, SQL, synthesizers
  • Getting Involved
• Update repository README page to link to the discussion. (Also remove Gitter links when appropriate)
  • docs.opendp.org - pull request #39
  • smartnoise-sdk
  • smartnoise-core
  • smartnoise-core-python
  • smartnoise-samples
  • opendp
  • dpcreator
• Gitter
  • Link to the new discussions forum

• sphinx documentation generated from pypi
• initial doc generation
• manual "top" page similar to https://fairlearn.github.io/v0.5.0/api_reference/index.html
• notebooks integration
• versioning
• integrate with CI

• Add a deprecation warning to the top page of https://docs.smartnoise.org

• Switch all channels to the OpenDP workspace from the TwoRavens workspace.

@raprasad and I talked about how it would be nice to be able to preview the HTML from pull requests online while reviewing them. This would save reviewers the trouble of building the branch locally.

I asked about this in Write the Docs Slack ( https://www.writethedocs.org/slack/ ) today and was told that Netlify is a good option and the free tier might be enough for us.

Here's a pretty compelling gif from https://www.netlify.com/blog/2016/07/20/introducing-deploy-previews-in-netlify/

More details at https://www.netlify.com/products/deploy-previews/

The Toolsday podcast is always saying good things about Netlify, most recently at https://spec.fm/podcasts/toolsday/wKcuYe6l

• The main OpenDP library docs will live in opendp/opendp and be built via CI
• "Downsize" these docs to only the SmartNoise section
• Point to the docs.smartnoise.org url
• Add a link from docs.opendp.org to docs.smartnoise.org - see opendp/opendp#240
• Rename repository to smartnoise-docs docs.smartnoise.org

• For people using OpenDP, SmartNoise, etc. what is the proper citation format, e.g. for academic papers

Include:

• SmartNoise: Mayana intro presentation for DAO (check with Mayana/Scott)
• Ethan/Mayana: Classroom lesson on DP
• Danny: DV Community presentation
• Community Meeting Dates
• OpenDP Fellow program (has started)
  • Check with Annie re: listing projects/names
• Theory/Practice
  • Programming Framework paper;
  • Ask Andy re: slides

• OpenDP library (check with Andy/Mike)

Announce

• mailing list
• tweet

Today @raprasad brought up to me and @anniewu332 that it might be nice to list some resources on differential privacy (not specific to OpenDP). For now, Raman and I are thinking there could be a "Resources" page in the User Guide.

Please feel free to edit the following list:

• Differential privacy: A primer for a non-technical audience: https://salil.seas.harvard.edu/publications/differential-privacy-primer-non-technical-audience
• The Algorithmic Foundations of Differential Privacy (recommended by a potential contributor): https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf
• ...

First, the reason the fail on warnings is that otherwise various content can go missing without anyone realizing it. Tables are especially prone to this. In #32 we noticed that two autogenerated pages were empty.

Local builds (make html) already fail on warnings.

Builds from GitHub Actions (make versions) don't.

Note that even with builds failing on warnings we still have the problem of not knowing if pull requests will break the build or not. #24 is about having an HTML preview of pull requests so that one can visually inspect changes. I suppose we could roll the concept of "doesn't the PR break the build?" into that issue. Or perhaps we could have a separate GitHub Actions job just to know if the build is broken or not (without an HTML preview)?

For now, by switching the build to fail on warnings we'll at least prevent oddness on the live site after a bad pull request is merged. It a step forward.

This week we had a meeting to discuss the vetting process for contributing to the OpenDP library. Of particular concern is contribution of new algorithms that would need to be vetted by someone with the appropriate expertise in mathematics and proofs.

We decided to review the contribution guidelines for a few open source projects that we feel are similar in spirit, that also have the need for extended vetting not just of code but of mathematical formulas. Specifically, we had in mind libraries that implement cryptography.

Using a list from Wikipedia as a starting point, I looked at three cryptography libraries I'd heard of and one (Crypto++) that caught my eye.

To summarize my findings:

• From what I can tell, Bouncy Castle doesn't strongly encourage contributions. (I couldn't find a contributing guide on the Bouncy Castle website.)
• Both OpenSSL and NSS encourage contribution but don't put particular emphasis on cryptographic code being different than other parts of the code. (See the OpenSSL contributing guide and the NSS code review checklist.)
• Crypto++ at least indicates that contributions of features and enhancements can be "time consuming because algorithms and their test cases need to be reviewed and merged."1 (See "Source Code and Contributions" on the Crypto++ website.)

(If we'd like to review additional libraries, I started a spreadsheet for the ones above.)

In the course of this review, I came across FIPS 140 which is an official form of vetting from the U.S. government for cryptography. I don't believe we are looking for anything this formal.

Like Crypto++, OpenDP should communicate that contributing algorithms will get extra scrutiny. Other projects like OpenSSL don't seem to put emphasis on this point but it doesn't appear to be necessary.

• update notebooks to work with data available via url
• update converstion to percent format with validation
• https://mybinder.org/

• initial script to deploy the shell site to the gh-pages branch
• opendp.org domain such as: test-docs.opendp.org

example at https://github.com/opendp/smartnoise-core/blob/develop/.github/workflows/rust.yml

Move emails as shown:
https://docs.google.com/spreadsheets/d/1YnOtvQkql4880kk2jhFXlW5isj3NXcqiLEs90RAJ8Is/edit#gid=0

(admin task, doesn't actually belong to this repo)

• add logo to opendp github org (can you forward an org)
• [email protected] -> .org
• [email protected] -> .org
• add [email protected] email!
• remove [email protected]
  • requires PyPI package update for: https://pypi.org/project/opendp-smartnoise-core/
• Twitter handle and website link
  • opendp, twitter icon link
  • account email
• youtube
  • about section with website link
  • account email

Distinct from "ability to preview HTML from pull requests online" (#24) we should have the ability to know ahead of time if a pull request is going to build or not. That is, do all the requirements install? Does make html work?

Currently reviewers of pull requests do these checks manually, which is tedious.

I would think we'd be able to tweak our existing GitHub Actions script added in #9 to run the build on all branches but only push to GitHub Pages if it's the "latest" branch? That is, if the pull request has been merged or if the default branch has been updated directly.

I'm open to other ideas, of course!

Just like how Google lists OpenDP under "related projects" here...

https://github.com/google/differential-privacy#related-projects

... we should list their project and other open source DP projects.

I'm thinking that "Related Projects" fits nicely in the User Guide, like this:

Add this to the documentation site

• Make the case for rust and memory safety.
  • Initial document: https://docs.google.com/document/d/16LFjllHI6jAtgURweasJ733X4l-XI8Fq3Ryy0b98Y0w/edit
  • Recent android article with nice summary toward the end (Prioritizing Prevention): https://security.googleblog.com/2021/04/rust-in-android-platform.html

I had Danny try the README but he didn't have Python 3.8 installed, only 3.9. So we should explain more.

• basic menus
• initial content
  • quickstart: https://github.com/opendp/smartnoise-core-python
  • api guide (link or embed). (#2)
  • notebooks embed + binder link. (#4)
    • https://github.com/opendp/smartnoise-samples/blob/master/analysis/basic_data_analysis.ipynb

We just renamed it to https://twitter.com/opendp_org and we'd like to link to it in the header after pull request #39 is merged.

This relates to the automated building of Python docs added in pull request #13.

I'm not sure where the QUAILSynthesizer error comes from. I just did a fresh install and here are the versions I'm using:

Successfully installed Jinja2-3.0.1 MarkupSafe-2.0.1 Pygments-2.9.0 alabaster-0.7.12 antlr4-python3-runtime-4.8 babel-2.9.1 beautifulsoup4-4.9.3 certifi-2021.5.30 chardet-4.0.0 docutils-0.16 greenlet-1.1.0 idna-2.10 imagesize-1.2.0 isodate-0.6.0 msrest-0.6.21 numpy-1.20.3 oauthlib-3.1.1 opendp-smartnoise-0.1.4 opendp-smartnoise-core-0.2.2 packaging-20.9 pandas-1.2.4 pandasql-0.7.3 patsy-0.5.1 protobuf-3.17.3 pydata-sphinx-theme-0.6.3 pyparsing-2.4.7 python-dateutil-2.8.1 pytz-2021.1 pyyaml-5.4.1 requests-2.25.1 requests-oauthlib-1.3.0 scipy-1.6.3 six-1.16.0 snowballstemmer-2.1.0 soupsieve-2.2.1 sphinx-3.5.2 sphinx-multiversion-0.2.4 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-2.0.0 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.5 sqlalchemy-1.4.17 statsmodels-0.12.2 urllib3-1.26.5

I poked around in https://github.com/opendp/smartnoise-sdk/commits/main and https://pypi.org/project/opendp-smartnoise/#history but I'm not sure what changed.

Here's the full output of the error:

(venv) HMDC-beamish:opendp-documentation pdurbin$ make html
sphinx-build -W -D 'html_sidebars.**'=search-field.html,sidebar-nav-bs.html source build/html
Running Sphinx v3.5.2
*****************************************
/private/tmp/sdafssda/opendp-documentation/source/..
/private/tmp/sdafssda
/private/tmp/sdafssda/opendp-documentation/venv/bin
/usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python39.zip
/usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9
/usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload
/private/tmp/sdafssda/opendp-documentation/venv/lib/python3.9/site-packages
*****************************************
ModuleSpec(name='opendp.smartnoise', loader=<_frozen_importlib_external.SourceFileLoader object at 0x10e6dffd0>, origin='/private/tmp/sdafssda/opendp-documentation/venv/lib/python3.9/site-packages/opendp/smartnoise/__init__.py', submodule_search_locations=['/private/tmp/sdafssda/opendp-documentation/venv/lib/python3.9/site-packages/opendp/smartnoise'])
*****************************************
making output directory... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 28 source files that are out of date
updating environment: [new config] 28 added, 0 changed, 0 removed
reading sources... [ 50%] smartnoise/api-reference/opendp.smartnoise.core.componreading sources... [ 64%] smartnoise/api-reference/opendp.smartnoise.synthesizerreading sources... [ 67%] smartnoise/api-reference/opendp.smartnoise.synthesizerreading sources... [100%] user/related-projects                                 

Warning, treated as error:
autodoc: failed to import class 'QUAILSynthesizer' from module 'opendp.smartnoise.synthesizers'; the following exception was raised:
Traceback (most recent call last):
  File "/private/tmp/sdafssda/opendp-documentation/venv/lib/python3.9/site-packages/sphinx/util/inspect.py", line 393, in safe_getattr
    return getattr(obj, name, *defargs)
AttributeError: module 'opendp.smartnoise.synthesizers' has no attribute 'QUAILSynthesizer'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/private/tmp/sdafssda/opendp-documentation/venv/lib/python3.9/site-packages/sphinx/ext/autodoc/importer.py", line 111, in import_object
    obj = attrgetter(obj, mangled_name)
  File "/private/tmp/sdafssda/opendp-documentation/venv/lib/python3.9/site-packages/sphinx/ext/autodoc/__init__.py", line 320, in get_attr
    return autodoc_attrgetter(self.env.app, obj, name, *defargs)
  File "/private/tmp/sdafssda/opendp-documentation/venv/lib/python3.9/site-packages/sphinx/ext/autodoc/__init__.py", line 2604, in autodoc_attrgetter
    return safe_getattr(obj, name, *defargs)
  File "/private/tmp/sdafssda/opendp-documentation/venv/lib/python3.9/site-packages/sphinx/util/inspect.py", line 409, in safe_getattr
    raise AttributeError(name) from exc
AttributeError: QUAILSynthesizer

make: *** [html] Error 2
(venv) HMDC-beamish:opendp-documentation pdurbin$ 

• List of features in the code such as other ways to input datasets, extra arguments that allow re-sampling with amplification, etc.
• This can be a non-exhaustive list to start

Add introductory paragraphs, links to repositories and quickstart with links to notebooks

• when needed, add assertion statements to notebooks
  • e.g. some notebooks have "expected error messages", such as the steps 5, 6 and 7 of the basic data analysis notebook
  • Add logic/assertions to make sure these error messages are appearing
• replicate the SDK notebook testing workflow in the samples repository

It's fairly common for software projects to provide documentation for previous versions like this:

• https://docs.python-requests.org/en/v3.0.0/
• https://docs.python-requests.org/en/v2.9.1/
• https://docs.python-requests.org/en/v1.2.3/

Likewise, often you can see the most recent unreleased version of the docs at "master" (indicating the default branch):

• https://docs.python-requests.org/en/master/

It's also nice to have "stable" indicate the most recent release:

• https://docs.python-requests.org/en/stable/

Finally, it's nice to have translations available (if you have translators):

• https://docs.python-requests.org/es/latest/
• https://docs.python-requests.org/de/latest/

All of this makes me think that we should add "en" and "main" (for our default branch) for now like this:

• https://test-docs.opendp.org/en/main

That way, we have a namespace in which to grow for versions and languages.

My thinking is heavily influenced by Read the Docs, which has a nice write up about versions at https://docs.readthedocs.io/en/stable/versions.html and languages at https://docs.readthedocs.io/en/stable/guides/manage-translations.html

Read the Docs also has a nice picker that lets your switch between versions and languages:

For now, I don't think we should worry about languages apart from using "en" to give us room the the URL for other languages. I did play a bit with sphinx-intl but actually getting it all set up seems like overkill right now.

For versions I've been playing with sphinx-multiversion and it seems promising.

• Create @opendp.org email to flag security issues
• Add the email to docs

cc @pdurbin

We now have a new "Contact" page at https://docs.opendp.org/en/latest/contact/index.html that looks like this:

@raprasad and I discussed briefly if we should go ahead and add the "opendp-community" Google Group to this page. This: https://groups.google.com/a/g.harvard.edu/g/opendp-community

The question I have is how to explain the difference between GitHub Discussions and Google Groups. When should one use one or the other? Does it matter?

For now I wrote, "A great way to get in contact with the OpenDP community is GitHub Discussions."

A simple fix would be mention both, as in, "A great way to get in contact with the OpenDP community is GitHub Discussions or the opendp-community Google Group."

Alternatively, we could try to explain when it's best to post to one or the other.

I should also note the following:

• Once a security email (and process) has been set up in #6 we should add the address to this Contact page. (For now, the Contact page suggests sending security issues to the "info" address.)
• We don't currently suggest contacting us through GitHub Issues, but we could.
• https://groups.google.com/a/g.harvard.edu/g/opendp-community says, "If you wish to send a private message to the OpenDP Executive Committee, please email [email protected] instead." We could add this to the Contact page as well.

Thoughts and suggestions are certainly welcome! 😄

The link to docs.opendp.org on this page is broken

Add version compatibility to these repos:

• smartnoise-core-python - python 3.6+
• smartnoise-sdk - python 3.8+ (check with Joshua)
• opendp-documentation - add to SmartNoise QuickStart section

When I introduced the sphinx-multiversion extension in e2f5e5f as part of #15, it had the unintended consequence of preventing us from previewing edits that haven't been committed yet.

That is to say, the extension works by checking out each branch or tag so you only see what's been committed, not what you just edited.