any call for example
/da/list?dept=CIS
will return ""

This would involve going to each quarter's previous data and using Beautiful Soup to find gather all the departments, all of the child courses, and all of the child sections. Then, using the same format as the current data scraper, make a database for every quarter.

The resulting database should be named using this schema:

[year][quarter][campus]_database.json

201812_database.json

    201812: The 2017-2018 academic year runs from July 1, 2017 to June 30, 2018. Therefore the ending year, 2011 is used to refer to the entire year.

    201812: The fifth digit represent the quarter:
        1 for Summer
        2 for Fall
        3 for Winter
        4 for Spring

    201812: The last digit refers to the college, 1 for Foothill and 2 for De Anza. 

This would make our local database very consistent with the one found on MyPortal, as the GET request made for the tabular course data follows this same format.

This is an enhanced version of /list that should list all departments, all courses, and a url for hitting each course. can be structured however as long as all this info is present.

Waypoints.js is not triggering when sidebar links are clicked.
In this example, the mouse just clicked single but the active status did not change on the menu item. This behaviour is controlled with Waypoints.js but does not work on some motions through the menu.

See this link for an example on how to write the documentation.

Both single/ and batch/ should have their responses explained since they are different.

This is a continuation upon issue #18 to address mobile bug

Note that this is only a fix for mice/trackpads, as it uses the wheel event. Touch events will require a separate, similar listener.
-- @joshuaptfan

When the modal for the API response is opened, scrolling in the modal will also trigger a scroll on the body. We need to find a way to freeze the scrolling on body while the modal is triggered.

As this project becomes more public, the code needs to be easier to understand.
Inline comments on confusing lines would help too.

I want to record along with the normal classes, a log of when seat data changes overtime. This will be crucial for getting some of the analytics engine's stats.

Add filters TDB to the backend batch POST endpoint.

As discussed with @TryExceptElse, we will move to an easier to traverse schema for the database. As we move away from a course first system, we will decouple from the section data being buried under their departments and course keys.

Comparison between old and new format:

.
├── dept
│   ├── course
│   │   ├── section 

.
├── section

This switch allows for the easy filtering of section data like teacher, time, and room data. Advanced multi-step queries can be constructed using this method as well. Additionally, the new data format will not require tables to be used for each department.

• Teachers contact info
• Map and Room data
• Calendar data for the next quarter

The documentation for deploying OwlAPI (server setup / nginx config / etc.) needs to be updated with the latest details.

In this example, each interact element has an ID of modal where there should only be one ID of modal.

<span class="modal" id="modal" aria-hidden="true">
  <span class="modal-background" onclick="toggleModal(this.parentElement, false)"></span>
  <span class="modal-content"></span>
</span>

This is an abstract feature with spec:

• Run a timed update of the MyPortal data
• Account for downtime while refreshing database

TODO: redefine API spec

This may require the frontend to be more modular to accommodate for more changes in the future

While working on #46 I realized that we need to move away from all the api scripts in root. This caused some difficult while trying to get Travis setup to run automatically.

Specifically the modules found in root should be moved to a folder named OwlAPI.
With an __init__.py in that folder, it should be easier to call OwlAPI modules without the need of an __init__.py in the root.

This issue should be tackled after the merge of @TryExceptElse PR

Should accept up to 5 sections from various departments as a key value list then return back a list of the CRNs and their data.

Would be useful to visualize /quantify when most classes take place, and when most people are free.

A way to filter the most common classes as well.

Would be useful for planning the best times for events/workshops/etc. Would be able to see when ppl are most likely available without having to do surveys .

Graphs

Going to update this with more ideas while more ideas are thought of. Can be used later when making graphs.

Data Sets

• S_all : set of all classes
• S_common : set of "common" classes, ie. those required for major and ge

X-axis

• X_axis = minute 0 -> minute max (maybe 1 week? 5-days?)
• x = t = iterator of 5 minute increments

Y-axis

• Y_classes_in_session : number of classes in set S that are 'in session' at time t
• Y_students_in_session : sum of (number of students currently enrolled in (each class that is currently in session))

let x_axis be an iterator where t_next = t + 5
let class[start] be the start time of the class in minutes since t = 0
let class[end] be the end time of the class in minutes since t = 0 

for each value t in x_axis : 
    classes_in_session[t] = {class ∈ (Set of all classes) such that (class[start] < t < class[end])}
    y[t] 🡐 0
    for each class in classes_in_session[t]:
        y[t] 🡐 y[t] + class[students_enrolled]

Note that this does not account for students who didn't show up to class ;)

programs

There could be an Updater, which fetches the data from the API and saves it into a Json file with filename based on date/time, into a folder called cache

There could be another program called data pre proccessor, which does any intermediate calculations. It outputs a new file with the data in such a way that it can be used directly by the grapher.

File_renamer could be a program that generates a list of all the cache and data filenames, which could be used by the HTML file to determine where all the data is. This might be helpful because filenames would be based on date/times, and it will be hard to predict what the next one is called. (Alternatively, make the filenames easily predictable, like data1, data2, data3 ... , and save the timestamp in the content)

The graph.html can then simply load the prepared data, and display it.

how to use

One possibility is to just have the updater call the data proccessor directly.

./program <cache directory> <processed data directory> 

Externally, you would just call the program and give the directory names where you want to store the output files (the filenames will be automatically generated).

For https://floof.li/fh/single?dept=POLI
Data returned where there are spaces before the number in units:

[
  {
    "1" : {
      "42082" : [
        {
          "room" : "ONLINE",
          "units" : "  5.00",
          "end" : "06\/29\/2018",
          "start" : "05\/21\/2018",
          "campus" : "FH",
          "desc" : "POLI SCI:INTRO AMERICAN GOVN\/P",
          "CRN" : "42082",
          "course" : "POLI F001.14W",
          "status" : "Waitlist",
          "wait_cap" : "20",
          "instructor" : "Callow",
          "days" : "TBA",
          "time" : "TBA",
          "seats" : "0",
          "wait_seats" : "15"
        }
      ]
    }
  }
]

It's also prevalent in other calls https://floof.li/da/single?dept=JOUR
Not that it really matters but letting you know.

Also, something new I discovered if you didn't know already
https://floof.li/da/single?dept=NURS

There apparently are possible locations out of campus- DO refers to DO HOSP in myPortal under Location.. and it crashes my application because I hardcoded locations. wuu. ;_;

Change 201841 and 201842 to 201911 and 201912.

Data from the archive scraper (#59) needs to be properly cleansed and converted into a common format between the current term scraper and the archive scraper.

Resolve this: Warning: Your Pipfile requires python_version 3.x, but you are using 3.8.2

This seems to be caused by defining the python version as 3.x in the Pipefile.

With such a massive amount of data (100,000 courses) being added with the #59 archive scraper, we need a way to check if all the correct keys for section data exist. An automated test should be made with pytest. You can find some of the examples that we've made in tests/test_server.py.

Here's a snippet to get you started:

class TestGenerateURL(TestCase):
    def test_sample_url_can_be_generated(self):
        self.assertEqual(
            test_data.test_sample_url_can_be_generated_data,
            generate_url('test_dept', 'test_course')
)

This class calls the generate_url() function with two example parameters and check to see if the result is equal to that found in the test_data file's copy. assertEqual will return 0 if it passes, and and 1 if it fails. All failed tests will be flagged with pytest.

Post here if you have any questions regarding how to start on this task. The archival data will not be ready until #59 has been merged.

Miscellaneous, for future, still brewing

Basic concept

• Each course is a node.
• Each node contains some logic about it's prerequisites, in the form of relations to another nodes.
• Create graph automatically (should look something like a web)

How to interact with it

• Highlight what you've finished
• Pick a target node to see the paths to get there.
• Map out class plan, or discover new classes

How it COULD be used

• Add in courses from other sources
• Create learning plans that extend outside of regular courses
• Basically a skill tree like in a game, but more practical, and has real data from real courses

Configure travis.yml

This is an issue with the use of Quart. A ticket has been opened here to diagnose this. If the solution cannot be achieved I might have to switch back to flask.

Currently, some of the API responses are not consistent. For example, fetching a non-existent course such as /fh/single?dept=MATH&course=2D returns a response as plaintext, instead of being valid JSON:

Error! Could not find given selectors in database

These instances should be fixed and the API should consistently return valid JSON .

UPDATE: Our course key parsing logic is indeed broken. Fixing that will fix this issue as well.

Querying with course modifiers included does not currently work. Searching for courses with certain modifiers, such as H for Honors, is a common use case and should be accounted for. A spike to investigate which modifiers should be queryable and which shouldn't, along with how feasible it is to do with or without creating special cases, is necessary.

For example, the ENGL 1BH only shows up at /fh/single?dept=ENGL&course=1B but not at /fh/single?dept=ENGL&course=1BH.

[
  {
    "CRN": "21132",
    "campus": "FH",
    "course": "ENGL F01BH1HW",
    "days": "TBA",
    "desc": "HONORS COMP,CRITICAL READ & TH",
    "end": "12/11/2020",
    "instructor": "Armerding",
    "room": "ONLINE",
    "seats": "29",
    "start": "09/21/2020",
    "status": "Open",
    "time": "TBA",
    "units": "5.00",
    "wait_cap": "25",
    "wait_seats": "25"
  }
]

With the introduction of the advanced scraper in #59, we may not need the original '5 minute scraper'. This issue has been brought up in #60 as we try and consolidate to a streamlined data model.

The new data from the advanced scraper could be served up with the api, and give us a consistent format for accessing the data. Right now, the two different data formats make the methods for accessing data from new and old data a pain to manage.

For this research to be done, we need to see if My Portal records incremental seat changes in the advanced dataset. If we can monitor those changes overtime, then we can verify that the advanced scraper can be used as a replacement for the old '5 minute scraper'

A style guide would be a helpful aid for people contributing to the project, in order to help maintain a consistent style, and therefore readability, throughout the project.