wgbh-mla / chowda Goto Github PK
View Code? Open in Web Editor NEWCLAMS processing app
Home Page: https://wgbh-mla.github.io/chowda/
CLAMS processing app
Home Page: https://wgbh-mla.github.io/chowda/
Hide sync button from non admins
Create IngestFlow
to ingest all assets from SonyCi
TLDR: poetry came before the python standard for metadata in pyproject.toml, so they do things in a (now) non-standard way.
This will allow us to make use of python's [project.optional-dependencies]
, so we can install "extra" dependencies with pip, and not need an extra package manager for normal (non-developer) images.
e.g:
pip install chowda[test]
As a Developer
I can use a migration tool to
In order to
SQLModel.metadata.create_all()
ERROR tests/test_app.py - sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "pg_type_typname_nsp_index"
DETAIL: Key (typname, typnamespace)=(mediatype, 2200) already exists.
Users story is satisfied (check boxes)
Most sources say that Alembic is the best tool for SQLAlchemy, and thus also SQLModel.
Add sync button to dashboard.
Does not need to work.
(Copied from #76)
After some research, a fully automated k8s migration process seems more complicated than we initially anticipated:
n
podsn
podsThis simplistic strategy has some drawbacks:
bash
script
kubectl
Metaflow
argo-cd
Sync
or PreSync
As an Admin
When I view the dashboard
I can a list of entries where
As a Clammer,
When I have defined a batch and I am just about to "Start" it and I am warned that what I am about to do may cost money,
I am given some very rough estimate (either in dollar amount or just relative to the cost of other batches) of how expensive the batch would be to run.
Rationale: Some batches will be larger than others, and pipelines vary a lot in terms of how computationally intensive they are. It would be easy to be about to run a batch that was crazily enormous. It would be very helpful to know how enormous.
The Chowda interface presents some representation of the expense at the appropriate moment.
A rough estimate, or even a bit of information that the Clammer can use to make a rough estimate, would be the most valuable thing.
As a Clammer, when I am looking at a list view
I can select the number of rows displayed: 10, 25, 100, 500, 2000, all
In order to more efficiently navigate the section.
On the Batch list page
Sort the batches by id
Descending order
As a developer
I can use the session to temporarily store flash and error messages for UI display
in order to avoid passing them with URL params
which keeps URLs cleaner
and allows flash/error messages to be seen just once
/admin
shows Dashboard page
Done When
Done when mmif is pretty.
As a Chowda Admin
I can view Clams Events
and for every Media File I can see:
in order to monitor individual media files as they go through a pipeline
As a User, when I select the number of rows to show in a section and then later return to that section,
I can still have selected the number I previously selected (e.g., 100)
In order to not have to change that every time.
Forthcoming: jowilf/starlette-admin#298
As a Chowda Admin
I can create a new Batch from a subset of items in an existing batch
in order to
re-run failed items
or re-run a batch with some additional items
or re-run a batch that succeeded with different parameters
etc.
They aren't all duplicates, and the first one isn't always the media.
https://chowda.wgbh-mla.org/admin/media-file/detail/cpb-aacip-00d35d49377
This media file has 12 assets:
As a Chowda Admin
When a media file has been processed by a pipeline
I can see the results including:
In order to check status of of individual media files
and access output mmif
From a batch detail page
When I click on a media-file GUID
It redirects me to media-file/detail/
It should redirect me to media-file/detail/{{ media.guid }}
Pydantic 2.0 was released last week.
https://docs.pydantic.dev/latest/migration/
A code transformation tool is available:
pip install bump-pydantic
Done when
Chowda Batch can be created from a list of GUIDs from uploading a text file
As a Chowda Admin (like Kevin)
When viewing Media Files list interface
I can filter by a list of GUIDs
In order to run exports on my custom GUID list
As a Chowda admin
when a batch is running
I can abort it
in order to recover from problems
and save time and resources.
As a user, when viewing any page,
I should be able to see the title of that page in the browser title
So I can see my browsing history better.
Create /api/sync to resync all assets from SonyCi
As an Admin
When I go to /admin
And click Sync Now
And there was an error starting the Sync
Then I am redirected back to /admin
And I can see a user-frinedly error message.
Done When
Each batch accurately displays the pipeline run information from Mario.
Statuses are
As a clammer, I can:
The /batches/list
page errors with:
DataTables warning: table id=dt - Requested unknown parameter 'media_files' for row 0, column 4. For more information about this error, please see http://datatables.net/tn/4
IngestFlow
having been run yet./admin
See a message indicating that no Sync has yet been run
Error
SQS set up
Start Batch should add events to the queue, which get pulled off as the cluster is ready for more jobs.
When a new asset is added, modified, or deleted in SonyCi,
It should fire a webhook request to chowda.wgbh-mla.org/api/sonyci
And update the chowda database.
When I edit a MediaFile attached to a batch,
the edit page does not include the batch relationship.
As a Clammer
When I am in any section and I choose “Export”
I can get the full export, not limited to the number of rows (e.g., 10, 100, etc.) currently shown In order to have exports suitable for large batches.
Done When
As a developer, I can set up user accounts with admin role
Migrate from SQLite -> Postgres for all environments, including test
pydantic-factories has rebranded to polyfactory
Migrate to new name in all usage files.
Done When
As a clammer, when I go to the view batch page I can see a list of media files with their link and current status within the metaflow run.
Create db backups
Add Metaflow code that generates Argo Events Sensor that triggers a Whisper workflow
When a clammer edits a Batch with existing MediaFiles, we get an Internal Server Error
with:
File "/Users/harpo/gbh/clams/chowda/.venv/lib/python3.11/site-packages/sqlalchemy/orm/identity.py", line 151, in add
raise sa_exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: Can't attach instance <MediaFile at 0x108274b90>; another instance with key (<class 'chowda.models.MediaFile'>, ('cpb-aacip-000043a51d5',), None) is already present in this session.
This does not happen when editing a Collection with media_files, which uses the starlette-admin default RelatedField
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.