Create an feature proposal in the upstream kopia repo for StreamedDirectory

Determine the data layout for corso data in kopia. Things to hash out:

• structure of data items for each user
• data encoding (json, binary, etc)
• where to store data like attachmens

The current OSS license checks merged in #50 are not strict enough and only fails if one of the forbidden licenses is used.

We would like to be stricter and fail the restricted set as well.

Create a new kopia interface that allows one to incrementally return children of the current directory and wire it up to the uploader

• Add a github action yml with a nightly cron to purge test data from corso-testing account buckets.
• Remove post-test bucket cleanup in ci.yml.

corso repo init s3 requires a handful of configuration and credential properties to succeed. Some of these requirements can be enforced by the cobra cli package (ex: the s3 --bucket flag). Others, especially env-sourced credentials (and config file sourced properties, later), must be validated within corso code.

• Validate presence of required config properties in storage configurations.

Create a standard GitHub action for code check-in

• Checks Markdown for spelling, typos, grammar
• Checks code for adhering to gofmt standard or ensures that all code is put into that standard at check-in
• Annotates code coverage onto root directory READMe

This tracks setup of AWS S3 environment (bucket(s), credentials) that we can use in our test pipeline (Github actions).

https://github.com/aws-actions/configure-aws-credentials appears to be the way to go here.

This will involve:

• Creating an AWS account that will be used for Corso testing
• Setup a OIDC provider in that AWS account configured to allow the Corso repository GH actions to authenticate
• Create an AWS IAM role that the Corso Github action can be configured to assume and provision temporary session credentials as environment variables (AWS_ACCESS_KEY, AWS_SECRET_KEY, AWS_SESSION_TOKEN) that our tests can use

An alternative is to deploy and use MinIO but I'd like to explore if we can leverage S3 (assuming we can keep things locked down and avoid any credential leaks, etc via rouge PRs).

Create a Data Connect prototype to compare to the msgraph api

• Install the application with permissions to our developer environment
• Create a document that shows the current way to create a new application as current documentation is confusing
• Demo should be simple to see how the level of granularity that Connect exposes.

Related to Issue #17
Create an automated process to increase email traffic between team development accounts.
The ticket can be closed when the following is complete:

• Messages being sent should not trigger the spam folder
• Generated messages should not be hardcoded to a single user
• Increase total tenant emails to 100
• Increase total tentant emails to 1000
• Increase total tenant emails to 3000
• Test Suite included

1. Ensure a test repository has been initialized.
2. Call repository.Connect(), expecting success.

Kopia accepts configurable options on the initialization and connection of a repo. Corso isn't exposing those controls to users yet.

Init Options:
https://github.com/kopia/kopia/blob/master/repo/initialize.go#L37

Connect Options:
https://github.com/kopia/kopia/blob/master/repo/connect.go#L16

Developers should be able to run integration tests from their local machines. Currently they can't.

Options:

1. Figure out how to get nektos/act to play nice with aws-actions/configure-aws-credentials.
2. Curate a new github action yml file that allows devs to run integration tests using local credentials.
3. (preferred) Run all integration tests with go test ./.... This also requires object cleanup to occur with *_test.go files, as opposed to the aws cli commands currently in github action ymls.

We need to select a framework for logging so that we have structure and consistency in our logging. Here are some requirements and alternatives:

Requirements:

• Must support JSON format
• Must allow for different levels of logging

Nice to have:

• Allow dynamic adjustment to logging level

Alternatives:

• https://github.com/uber-go/zap
• https://github.com/sirupsen/logrus
• https://github.com/rs/zerolog

MSFT document discusses that throttling limits the number of concurrent calls to resources. For engineering purposes, a concrete understanding of the limiters that are placed on the application needs to be investigated.
This overall ticket can be closed when the following tickets are closed.

• Issue #18
• Issue #19
• Issue #20

Currently there's no way for the user to specify an encryption passphrase. That should be handled alongside other credential info, with an env var at minimum.

Related to Issue #17
Create an automated process to increase event occurrences for team development accounts.
The ticket can be closed when the following is complete:

• Events should be made in the future
• Generated events should include several different members to check for corner casess
• Increase total tenant events to 100
• Increase total tenant events to 1000
• Increase total tenant events to 5000
• Test Suite included

Corso defaults the s3 endpoint to 's3.amazonaws.com'. Currently no option exists to override that value.

Add an --endpoint flag for s3 initialization.

Create a minimal program that can backup two in-memory data items to a remote kopia repository as a single snapshot and then retrieve them

This tracks the implementation of corso repo init

Produce the initial design of the Corso CLI. The document should declare which commands are usable in the initial release, what analogous properties are supported (config files, env vars, etc), and provide clear description of the command and their usage. This document is not intended to duplicate or replace internal documentation such as the terminal help output or man page.

Upon completion, the finalized CLI.md document will be added to the /docs directory.

Msgraph-sdk-go recently updated to new version. Upgrades break most of our object stores. Upgrade is important to us as it solves several bugs and does add more functionality to the library.

• Switch core, abstractions, and msgraph to the newest versions -> Go.mod, go.sum
• Resolve library failures
• Freeze versioning until next stable release

Switch to the fork until upstream supports the functionality we require

Related to Issue: https://github.com/alcionai/graph-sandbox/issues/6

A calendar can be retrieved by Graph; however, the events are not able to be retrieved. It is unclear if the documentation for the API is current or not. CalendarView, a list of events for a certain time frame, has an API that suggests that operation is permitted.

SDK-level discussion on the subject.

The repository --prefix option in Kopia allows multiple repositories in the same bucket to be created. We should support this option in corso repo commands.

There are a bunch of misc. changes that need to be made to the Coros docs system:

• Add a logo for dark mode
• Set up dark mode to auto-detect system settings and remove selector
• Add site metadata for SEO
• Add Mermaid for diagrams - https://github.com/sjwall/mdx-mermaid
• Swap link from Discourse to GitHub Discussion
• Add style/spell/format checks for platform README
• Add development requirements in the platform README

Adds the following to the CLI:

• backup command set.
• create subcommand for backup.
• exchange application subcommands for backup create.
• Able to automatically connect to a previously connected repo.

Initial struct and api design can be found here.

Should cover:

• api and usage
• execution flow

Take a first pass at the BackupWriter code. The code should:

• connect to a remote s3 kopia repo
• use mock data
• have fake emails and attachments in the proper directory structure in kopia

On CI/CD, a github action should run an integration test for corso repo init s3.

Request

Ensure that corso CLI documentation is automatically maintained as a CI action. Changes to CLI structure should automatically result in an update to any relevant markdown files or other documentation. Example: a new CLI command is added- as a result a new markdown file documenting that command should be auto-generated and added to the commit changes.

Any markdown document changes, upon merger, should then automatically generate a new docusaurus release, so that the online corso documentation stays up to date.

Requirements

• Ensure help is available/displayed as expected.
• Auto-generate CLI markdown as a CI action.
• New MR for changes instead of inline commit
• Docusaurus docs build.

For clarity, the team will create a visualization of Corsito goals for the initial release. Visualizations of use cases.
It could be as simple as deleting a few emails and events to show what the user/administrator will be attempting to achieve.

• Should show potential CLI interface
• Show the interface
  • List messages system-wide
  • List of events
• Visualization can be Powerpoint or Whiteboard sketch

The ParseNode is the abstraction that all Outlook objects are transformed into. Document the creation process as well as how to retrieve individual fields from this object.

• Create Documentation in repo
• Code snippet for writing streamed data into object
• Document the test cases and why regular failures do not necessarily show up with this object

Figure out how to get kopia to connect to a remote repository (ex. S3) using code instead of the kopia CLI

Corso needs to track release versions, and provide those version details with the corso --version command. Whatever we use for versioning should work with https://goreleaser.com/.

Possible packages:

• https://github.com/christopherhein/go-version

Add a README.md to corso/src briefly explaining the code layout and design intents.

Create a PR/start discussions with the kopia maintainers about upstreaming StreamedDirectory

Currently fails with error or "group already exists" due to
how Mac uses low group numbers

At the time of initialization, the GraphConnector component should initialize.

• Intake configuration params set to clientId, tenantId, and client_secret
• Component design created
  Related to Issue #25

• unit tests
• integration tests
• ignorefs tests

The testing package provided by Go give us a lot of functionality and we want to be able to use go test ... to run our tests because that allows integration with other systems (e.g. VS code)

In addition to that - what is typically useful is:

• Assertion/Compare utilities to validate results
• The ability to group tests into suites which allows for easy suite selection and shared setup/teardown for a set of tests

We need to figure out where Corso design docs/discussion should go.

We could put this in a docs directory in the repo (in markdown) for example. It's also worth looking into what Github functionality can be used here.

Create an automated process to evaluate the amount of throttling that exists from the vantage point of our tenant.

This issue can be closed when the following have been tasks have been completed:

• Create a generic application that can pull mail from all of the members in the tenant dev account.
• Expand the application to pull events from all of the members in the tenant dev account.
• Application should display basic telemetry data.
  • messages per second
  • average wait time between responses
  • any forced disconnects from the server

• Github actions on CI/CD
• Investigate additional unit testing.

We want to integrate https://vale.sh/ with Docusaurus.