🚀 Top level project management for conda-forge CZI grant
Home Page: https://github.com/orgs/Quansight-Labs/projects/10
License: BSD 3-Clause "New" or "Revised" License
Integrate a timeline view for events in conda-forge (such as uploads) to the Quetz maintainers' dashboard.
The Quetz instance will accumulate rich logs of the conda-forge and bioconda ecosystems.
The data collected can be useful to provide information about estimated usage, popularity, growth and other metrics.
This is a stretch goal. There are no estimates for its completion.
Primarily focused on performance and reliability.
Improve the artifact verification and validation workflows as they are moved from cf-staging to conda-forge.
conda-forge feedstocks are repositories equipped with the build machinery provided by conda-smithy.
When a PR is merged to a feedstock branch, the resulting conda packages are not uploaded directly to conda-forge.
They are first placed in a staging channel named cf-staging.
The artifact validation bot hosted in a Heroku instance downloads the artifacts, runs some analysis and if successful, then they are copied to the actual conda-forge channel.
The analysis includes checks like:
Depending on the size of the package, this causes some strains on the already overworked Heroku dyno.
This task should be finished in the first 18 months.
conda-forge/conda-forge-webservices: the Heroku app deployed to run conda-forge admin commands and lintingconda-forge/artifact-validation: code and workflows to validate conda-forge artifactsMerge and mix
The content you are editing has changed. Please copy your edits and refresh the page.
Investigate and design prototype mockups with the desired features for the maintenance board
conda-forge.org/status offers a "maintainers dashboard" with information about:
It could also show how "healthy" the feedstocks (conda-forge) and package-creating repositories (bioconda) are.
To inform what content should be displayed in the dashbaord, we would like ti survey the community for any other features that could be considered usegul, such as:
This task should be finished in the first six months.
conda-forge/status repo)Integrate feedstock data (GitHub repositories) representation in the Quetz maintainer's dashboard.
With 20K+ feedstocks, it's difficult to assess the state of each repository in a timely manner.
Maintainers are often drowned in notifications, mentions and review requests, which hinders the ability of members to help out other volunteers.
We'd like to provide better ways to assess where help is needed as part of the maintainer's dashboard.
This needs to account for both multi-repository organizations like conda-forge and monolith-based orgs like bioconda.
At this point, we should have input from the community about which metrics are relevant and informative.
The UI/UX team will have provided modern mockups that account for accessibility.
This task should be finished in the first 18 months.
See main dashboard mission issue.
Initial improvements to the conda-forge infrastructure as detailed in the public
roadmap focusing on critical performance and security risks.
After reviewing the audit report and approving work items, we will focus on the highest priority ones:
conda-forge resources (onboarding and offboarding)This task should be finished in the first 12 months.
Deploy a minimal Quetz instance for conda-forge and bioconda
Quetz is an open-source server for conda packages and channel metadata.
Its UI frontend can be extended with components.
The idea is to have a running instance that the team can use to iterate on the planned features and receive feedback from the community.
This task should be finished in the first six months.
Audit conda-forge infrastructure to generate a roadmap that can be followed over the course of the project to improve the long-term sustainability of the ecosystem.
Since its emergence in 2015, the conda-forge project has seen explosive growth in contributors, maintainers, repositories, artifacts, and packages served.
To serve such a vast ecosystem (and around 300M downloads per month), the core team has heavily relied on automation, Continuous Integration and Delivery platforms and in-kind donations from multiple infrastructure providers.
Current conda-forge's infrastructure and tooling are distributed across many GitHub repositories, external CI services (Azure DevOps, GitHub Actions, TravisCI, Drone.io, CircleCI), Heroku "dynos" and AWS instances.
Many were built as ad-hoc fixes and currently lack documentation or risk mitigation plans.
We plan to migrate the configuration and infrastructure provisioning to reproducible, vendor-agnostic tools such as Terraform, complemented with rigorous testing, vulnerability detection, and documentation strategies to enable better security, reliability, and recovery from adverse events.
See issues labeled as
mission: infra 🛠
This task consists of building a sqlite database with all the package metadata. Equivalent to the deprecated regro/libcfgraph:/artifacts repository.
Outline risks and mitigation plans for the transition to the proposed infrastructure design.
The audit of the existing infrastructure link will generate a report of recommended changes.
Once evaluated and approved, we will compile a public roadmap that specifies how to apply and transition to the new infrastructure.
We aim to minimize unplanned disruptions and let conda-forge operate as usual.
Infrastructure testing and validation after Terraform migration.
TBD.
This is a stretch goal. There are no estimates for its completion.
Provide a way for users to find which package(s) provide a certain file (e.g. a header, or a library, or an executable), similar to what portals like pkgs.org do.
We do have the info in the database designed in https://github.com/Quansight-Labs/conda-forge-db, but we need to serve it somewhere, preferably serverless or close-to-zero maintenance (e.g. one-click deployment). This is tricky because populating the database from scratch has a non-negligible overhead.
The content you are editing has changed. Please copy your edits and refresh the page.
Develop end-user and maintainer documentation as needed, including security best practices for supply chain security and Quetz dashboard use.
At this point, a Quetz instance providing services for conda-forge and bioconda should be ready and running.
To ensure its long-term sustainability, thorough documentation should be available, covering user- and developer-focused topics.
This task should be done by the end of the project (24 months).
We need a documentation site to keep track of the audits, infra docs, and other docs or references generated through the grant.
This can then be moved to conda-forge org.
Work on bots to eliminate long-lived credentials, improve performance and
reliability, and develop end-user and maintainer's documentation.
The term "conda-forge bots" encompasses several pieces of automated infrastructure key to the operating status of the organization.
It has grown organically, with improvements, additions and hotfixes being made on an "as-needed" basis.
As a result the documentation has some gaps that need to be filled.
Since there was no initial design for its current state, no systematic review of its bottlenecks or risks has been performed.
This makes it difficult to maintain, and given the lack of a testing infrastructure, scary to even try if unfamiliar.
The audit report from the first year will have included security recommendations, performance improvement suggestions and reliability measures.
On top of that, we will make it easier to for newcomers to contribute to the valuable automation ecosystem in conda-forge.
This task should be finished in the first 18 months.
The goal is to audit and document existing conda-forge's infrastructure.
This will be done in public.
We need to ensure no critical security details such as credentials are exposed.
To propose technical debt reduction measures and ensure the long-term viability of the project and its growing ecosystem, we first need to understand and document the current status of the platform.
Expected challenges:
The content you are editing has changed. Please copy your edits and refresh the page.
This task should be finished in the first six months.
Create an itemized list of all the pieces of infrastructure that we can find on conda-forge.
Use IaC vocabulary to identify providers, services, stateful properties and access control (credentials, default access level), as well as purpose and utility.
Make conda and mamba support OCI registries as repodata and packages sources.
Having an OCI mirror is only the archival part.
The tooling has to learn how to "speak" to it.
conda and mamba should be able to download repodata and packages from OCI-backed servers.
The content you are editing has changed. Please copy your edits and refresh the page.
Extend the deployed Quetz dashboard with the newly designed interfaces for maintainers.
This is the frontend counterpart of "Review and add needed Quetz plugins for the backend".
The needed UI elements need to be designed and implemented in the Quetz frontend, following the style guide proposed in the first milestone.
This task should be finished in the first 12 months.
This in the only source of data that we can't serve via OCI-backed or Anaconda.org-backed info/ metadata directly because it requires a global package search. We need to figure out how to present these maps without relying on libcfgraph-like setups (i.e. not another JSON in a repo?).
One way is to reutilize the files-to-artifacts mapping and compute the import-maps on the spot, because they can be inferred by the location of __init__.py modules, mostly.
Work with the broader packaging community to complete the implementation of oras-py.
oras-py is the Python client of the ORAS project, which aims to provide an opinionated abstraction for the storage of arbitrary artifacts on OCI services.
In order to leverage OCI-based artifacts in the conda ecosystem, this library will be needed by many tools.
This task should be finished in the first 12 months.
oras-py: a Python library to interact with an OCI registryPrepare the conda ecosystem for OCI-based storage compatibility.
There is no straightforward way to monitor the operational status of conda-forge's infrastructure.
conda-forge.org/status offers a "maintainers dashboard" with information about:
Unfortunately, this is far from being comprehensive view of ongoing maintenance tasks, bottlenecks, or the overall health of the many bots and infrastructure pieces.
Having a detailed picture of the infrastructure and automation tools will significantly improve the maintainers' workflow and aid with identifying critical risks— which is essential to keeping up with the increasing growth and demand from the community.
Quetz is chosen as an open-source server for hosting conda packages, thus allowing for increased transparency and extensibility.
This would have the added benefit of centralizing the currently scattered-across-repositories packaging metadata in a canonical, API-first, performant-at-scale database, laying the foundation for further infrastructure automation and improvements to the building processes.
See issues labeled as
mission: dashboard 🎛
oras-py: a Python library to interact with an OCI registrychannel-mirrors organization: OCI-based mirror of conda-forge and biocondaMake Quetz understand OCI-backed servers as a packages source.
Having an OCI mirror is only the archival part.
The tooling has to learn how to "speak" to it.
Quetz should be able to list packages uploaded to OCI servers as done with Anaconda.org.
This task should be finished in the first 12 months.
This list collects upstream contributions related to security. Keeping here for reporting purposes.
This is a security-sensitive task so it is being worked on in a separate repository.
Prepare the conda ecosystem for OCI-based storage compatibility.
OCI (Open Container Initiative) registries are a well-defined standard for versioned blob storage already implemented by many public cloud providers (i.e. GitHub).
We plan to leverage the OCI registry on GitHub Packages to develop a community-maintained mirror for bioconda and conda-forge.
Besides, adopting an OCI-based strategy will provide additional flexibility for metadata expansion.
With this in mind, we aim to produce and capture more comprehensive package metadata, which we will later use for: maintenance dashboards, package search functions, and security checks.
A prototype mirror is already available at the
channel-mirrorsorganization.
See issues labeled as
mission: mirror 🪞
oras-py: a Python library to interact with an OCI registrychannel-mirrors organization: OCI-based mirror of conda-forge and biocondaEvaluate the feasibility of integrating sigstore into the conda/mamba package managers.
sigstore is the open-source standard for signing packages and containers.
This is a stretch goal. There are no estimates for its completion.
Migration to Terraform for critical infrastructure parts.
At this point, the infrastructure should have accumulated enough improvements with or without IaC approaches.
Finalizing its migration to Terraform or similar will help us with provisioning, scalability, recovery and risk aversion.
Having this goal in the 24 months milestone does not mean it should only start in the last six months of the project,
rather than it should be completed by this time.
This task should be done by the end of the project (24 months).
Use IaC to document and manage who has access to each infrastructure piece.
We have decided to use Pulumi mostly due to its Python scripting support (helpful for custom providers like anaconda.org) and the included cloud dashboard.
The content you are editing has changed. Please copy your edits and refresh the page.
The content you are editing has changed. Please copy your edits and refresh the page.
The content you are editing has changed. Please copy your edits and refresh the page.
The idea is to use git object database to manipulate the repo instead of doing it via the filesystem. The prototypes:
Investigate and design prototype mockups with the desired features for the maintenance board
conda-forge.org/status offers a "maintainers dashboard" with information about:
This panel has grown organically over the years without significant UX or accessibility analysis.
We will provide mockups that offer the same functionality with a consistent UI informed by best practices in UX and accessibility.
This task should be finished in the first six months.
conda-forge/status repo)Further improvement and adoption of DevSecOps practices across the conda-forge/bioconda workflows
The audit report should have revealed gaps in the infrastructure where security could be improved.
We will tackle this by adopting practices borrowed from DevSecOps approaches.
This task should be done by the end of the project (24 months).
Document the onboarding and off-boarding process, with the aim of eventually automating most steps.
When a user joins any of the conda-forge teams, members of the core team need to manually provision the permissions across the required services.
The list of services and the required permissions are not fully enumerated,
and members sometimes acquire the needed credentials by manually requesting it to other members, as needed.
This also means there's no comprehensive list or way of knowing who has access to each service.
The content you are editing has changed. Please copy your edits and refresh the page.
This task should be finished in the first six months.
Migration of existing data sources into a purpose-built database, including a suitable
data schema and schema validation.
The metadata supporting the diverse automations at conda-forge is made of a number of repositories that serve JSON files in a given directory structure.
Depending on the tool, the metadata will be used to build a graph.
This data is downloaded, and the graph recreated, every time the job runs, adding to substantial overhead as conda-forge grows.
Other tools depend on a different presentation of the metadata, and are supported a different repository.
This means that the same data is duplicated just to be presented in a different way.
This is not sustainable or considerate with the available resources, and we would prefer a single source of truth that runs in a performant way.
regro/cf-graph-countyfair: maintained and used by regro/cf-scripts (autotick bot). Also powers the status page and other things.regro/libcfgraph: maintained by regro/libcflib and used by several pieces. Most important pieces: run_exports data and the import maps.conda-forge/feedstock-outputs: maps package names to feedstocks that are allowed to generate them.conda-forge/artifact-validation: map of file paths to originating artifacts. Currently not available in a global way, only for some special packages.libcfgraph's artifacts/ can be fetched from GHCR OCI mirror: regro/conda-forge-metadata#6
feedstock-outputs: regro/conda-forge-metadata#7
run_exports can be served next to repodata.json files:
--run-exports and serve those files (example run_exports.json)regro/conda-forge-feedstock-check-solvable - regro/conda-forge-feedstock-check-solvable#17This task should be finished in the first 12 months.
Quansight-Labs/conda-metadata-app#7 shows a snippet where a Docusaurus page could access Anaconda.org to query the .conda metadata live. Sadly this is not possible for .tar.bz2 artifacts, so those would need to be retrieved from either OCI or an intermediate API server (e.g. https://condametadata-1-n5494491.deta.app/).
It would use URLs like the current /status dashboard does for migrations (see https://deploy-preview-12--cf-infra-docs.netlify.app/status/).
https://conda-forge.org/packages lists all conda-forge packages and their feedstocks. We could also generate links to conda-metadata-app from that table so folks can explore the metadata in a detailed way.
In the future, we could also use a client-side implementation to query the backend directly. xref #51
Create and maintain a mirror of conda-forge and bioconda on Github packages
Using the OCI capabilities of Github Packages, we can store conda artifacts.
This can be used as a mirror for archival purposes, at the very least.
The content you are editing has changed. Please copy your edits and refresh the page.
This task should be finished in the first 12 months.
#34 added some preliminary diagrams but these are not final. Diagrams are great to create a visual image of what's out there, which seems handy in conda-forge, given the amount of moving pieces we have.
I started a mindmap diagram with mermaid to provide this mental model of all the infrastructure (without relationships, just the inventory). You can see it here:
Once we have that I guess we can start breaking it down in chunks, with each chunk having the relationships between elements when needed.
After reviewing the feedback received from the community, implement new plugins or repurpose existing ones in Quetz.
The UI/UX mockup tasks will have provided a set of needs and features requests for the Quetz dashboard.
In this task we will review which ones are already available in Quetz, which ones need some work, and which ones need to be implemented from scratch.
The plugins might need new data sources and integrations not yet available at the time of implementation.
This task should be finished in the first 12 months.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.