more cleanly extract publication HTML, pre-sanitize HTML

Such a file could be identified by the web importer to make things easier maybe?

Idea

Develop common format so that most future data additions won't require updating @cougargrades/importer.

"Patch" system would theoretically have individual objects which:

• contain a field like "target" that denotes which Firestore objects should be affected by their absolute document path (ex: catalog/CHEM 1331)
  • archetype: 'document' | 'collection'
  • path: path to what is being referred to
• contain a field like "actions" that denotes which actions should be taken such as
  • write: write the provided object to the Firestore document at the location (will overwrite existing objects) (only for document)
  • merge: merge the provided object with the object at the existing location (will fail if object doesn't exist) (only for document)
  • append: appending to an array-type with a provided amount (any) (only for document)
  • increment: incrementing to a field by a provided amount (number) (only for document)
  • create: write the provided object to a new document in the provided collection (only for collection)

Distinction

Patches are NOT intended to be flexible or programmatic. They should be discrete ways to modify a specific Firestore document in an explicit or "hard-coded" way. Patch files should be used as a medium of interchange and NOT as a method of describing data updates. Ideally, an intelligent tool will have generated a "patch" file that it is certain will be correct.

"Patch" files should be mostly idempotent.

Usage

This "Patch" system would allow the inclusion of supplementary data such as extra fields from various data in this repository. This is especially applicable to the use of:

• edu.uh.core_by_component
  • usage of core component across Firestore
• com.collegescheduler.uh.subjects
  • usage of subjectName across Firestore
• edu.uh.publications.courses
  • linking to the official UH catalog
• possible RateMyProfessor integration in the future 👀

Possible models

write Example

{
  "format": "io.cougargrades.publicdata.patch",
  "target": {
    "archetype": "document",
    "path": "catalog/CHEM 1331"
  },
  "actions": [
    {
      "operation": "write",
      "payload": {
        "department": "CHEM",
        "catalogNumber": "1331",
        "description": "Fundamentals of Chemistry"
      }
    }
  ]
}

merge Example

{
  "format": "io.cougargrades.publicdata.patch",
  "target": {
    "archetype": "document",
    "path": "catalog/CHEM 1331"
  },
  "actions": [
    {
      "operation": "merge",
      "payload": {
        "publicationUrl": "http://publications.uh.edu/preview_course_nopop.php?catoid=34&coid=165599"
      }
    }
  ]
}

append Example

{
  "format": "io.cougargrades.publicdata.patch",
  "target": {
    "archetype": "document",
    "path": "catalog/CHEM 1331"
  },
  "actions": [
    {
      "operation": "append",
      "arrayfield": "instructors",
      "datatype": "firebase.firestore.DocumentReference",
      "payload": "/instructors/Zaitsev, Vladimir G"
    }
  ]
}

increment Example

{
  "format": "io.cougargrades.publicdata.patch",
  "target": {
    "archetype": "document",
    "path": "catalog/CHEM 1331"
  },
  "actions": [
    {
      "operation": "increment",
      "field": "sectionCount",
      "payload": 1
    }
  ]
}

create Example

{
  "format": "io.cougargrades.publicdata.patch",
  "target": {
    "archetype": "collection",
    "path": "users"
  },
  "actions": [
    {
      "operation": "create",
      "payload": {
        "name": "John Doe",
        "score": 9001
      }
    }
  ]
}

Usage varies depending on how groups will be implemented in the future.

as seen in:

cougargrades/types#16

relevant code:

https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

• Possibly scraping http://www.uh.edu/catalog-undergraduate

sitemap.txt includes /g/10 but not /g/10-15888

Suggested changes:

• courses.json and instructors.json
  • add totalEnrolled and totalSections properties
• Transpose edu.uh.publications.subjects/subjects.json to ./subjects.json:

  {
    "MATH": {
      "description": "Mathematics",
      "totalCourses": 123,
      "totalEnrolled": 123,
      "totalSections": 123
    }
  }

• https://stackoverflow.com/a/36445821

Dependent on: cougargrades/types#26

Generate "append" operation Patchfiles for adding existing Courses to their appropriate Group

Described in greater detail here: cougargrades/types#12

Some excerpts:

Referenced here: https://uh.edu/academics/courses-enrollment/course-number-updates/index.php

In an effort to make transferring credit for students easier, several University of Houston course names and numbers are changing to align with the TCCNS. These changes go into effect starting in Fall 2021.
Below is a chart outlining all course number changes, organized alphabetically by current UH course number. For more information related to these course changes please see the FAQ page.
The Texas Common Course Numbering System (TCCNS) is a cooperative effort among Texas community colleges and universities to facilitate transfer of freshman- and sophomore-level general academic courses.

I'm curious how this can be addressed in the cougargrades dataset.

Some initial thoughts:

• Is this even a problem? Can't we just leave them be and start a new "course"?
• If we were to implement some kind of new "reference" system, what would be the best way to do this?
• Is it possible for a new course number to conflict with an old course number for a different course?
  • Example:
  • "calc 1 (math 1500)" and "geometry (math 1300)" are courses.
  • "geometry" was discontinued under the course number "math 1300" many years ago, but still exists in old sections
  • "calc 1" has its course number changed to "math 1300"
  • the course number "math 1300" now means 2 completely different courses across a large time period