Giter Club home page Giter Club logo

mallard's Introduction

Developing

If you are interested in developing MALLARD, please see these instructions.

Deploying Locally

MALLARD can be deployed locally using Docker. The provided configuration contains everything you need to run a local instance of MALLARD, including a MariaDB database for metadata, and a MinIO object store for image data.

Prerequisites:

Compose files for MALLARD are generated interactively based on your chosen configuration. To generate a new compose file, run:

cookiecutter cookiecutter-docker/

You will be given the chance to specify the following options:

  • config_name: A human-readable name for this configuration.
  • mode: The mode to run MALLARD in. Development mode means that the local filesystem will be mounted inside the containers instead of copying the code over, so that you can test new code without rebuilding Docker images. If you are not testing new code, use production mode.
  • proxy_config: Specifies the configuration to use for MALLARD's built-in reverse proxy. By default, it will use SSL. Disabling SSL could break things, because MALLARD uses features of modern browsers that only run in a secure context. However, there are legitimate reasons for disabling SSL. For instance, you might wish to run MALLARD behind your own reverse proxy and handle SSL there.
  • host_port: The port that MALLARD will listen on for connections.
  • object_location: Specifies the path where data uploaded to MALLARD will be stored.
  • metadata_location: Specifies the path where the MALLARD database will be stored.
  • backup: If true, it will enable the containers necessary for backing up data. Enable if you are planning to use MALLARD's built-in backup capabilities

A docker-compose file will be created in the repository root, named based on the value you set for config_name. For instance, if you set config_name to "default", it will generate a docker-compose.default.yml file. To generate multiple configurations, you can re-run the cookiecutter command with different config_names.

Once you have the compose file, you can run the following to build the images and start the MALLARD services:

docker compose -f docker-compose.default.yml build
docker compose -f docker-compose.default.yml up

Initializing the Database

The first time you start MALLARD with docker-compose up, it creates a blank database. To actually use MALLARD, this database needs to be initialized with the proper tables. A script for doing this is conveniently included.

First, you will need to figure out the name of the gateway service container, using docker ps. Most likely, it will be mallard_gateway_1. Then, to initialize the database, run:

docker exec mallard-gateway-1 /init_db.sh

This only has to be done once, unless you delete the MALLARD Docker volumes. On subsequent restarts, you should be able to use MALLARD normally without this step.

Accessing the Application

The MALLARD application should be accessible on your local machine at https://localhost:8081 (or whatever you set host_port to). The MALLARD API should be accessible at https://localhost:8081/api/v1. To access a convenient interface that allows you to test the API manually, visit https://localhost:8081/api/v1/docs.

Backing Up

Any production deployment of MALLARD is likely going to need some mechanism to back up the data. MALLARD includes some functionality designed to simplify this. Currently, it only works with the default MinIO+MariaDB backend. To enable backups, use the following procedure:

  1. Generate your compose file with the backup configuration option set to true. (See this section.)
  2. Start the MALLARD server. After initialization completes, you should see a new sub-directory in the current folder called backups/. This directory allows for access to the data in the object store through s3fs. Additionally, the contents of the metadata DB will be automatically dumped to this folder once per day.
  3. Point your favorite backup tool at the backups/ directory.

Restoring

Since the object store is mirrored through S3FS, restoring it should be trivial: just restore all the files into the original folder. Restoring the metadata involves accessing the latest DB dump (located in /backups/metadata/) and restoring it manually to MariaDB.

Authentication

In the event that MALLARD is exposed on a public network, you will probably want some control over who has access to it. MALLARD currently supports user authentication through Fief. Note that, going forward, we will assume that you have installed and configured Fief and it is running at https://example.fief.dev/.

MALLARD Configuration

The MALLARD config file must be updated to enable authentication. For a default (production) deployment, the local_config.yaml file will be used. Modify this file and rebuild the Docker container to change the configuration.

Under the security section, there is an enable_auth option that defaults to false. Instead, you will want to set it to true, and add the additional fief section that is present in the example config. Fill out the base_url and client_id parameters appropriately for your Fief configuration. The client_id can be found on the Fief admin page: fief_client_id

Fief Configuration

For authentication to work in Fief, you have to configure your client to allow all of the MALLARD redirect URIs. This can be done on the Fief admin page, in the "Clients" section, by clicking on the client you are using. It should open up a panel on the right with a list of redirect URIs. Click the "Edit Client" button at the bottom. fief_edit_client

This will open a modal where you can change the client settings. First of all, under "Type", select "Public". fief_client_type

Also, under "Redirect URIs", add the following new entries:

  • https://mallard.example.com/auth_callback
  • https://mallard.example.com/api/v1/docs/oauth2-redirect

Note that you should change mallard.example.com to whatever domain your MALLARD deployment is served at.

mallard's People

Contributors

djpetti avatar wjtw9802 avatar

Watchers

James Cloos avatar  avatar

mallard's Issues

Responsive redesign for details page

Right now it doesn't really look good on portrait screens. This can be fixed by using a vertical layout for these screens, where the metadata cards are below the image instead of beside it.

Allow zooming in details view

Zooming in and out with the mouse wheel would be a nice feature to have. See Google Photos for a good implementation of this feature.

Add a user menu

This should show the currently logged in user and provide options to log out or change their profile.

Better handling of upload failures

When an upload fails, I'm thinking that it should just show an "X" icon next to that file in the uploader. Ideally, the user could hover over the icon and see an option to retry the upload.

Support displaying videos on the frontend

This doesn't actually have to be complete video support. All that needs to be present is the following:

  • Video thumbnails are displayed on the homepage, much like image thumbnails.
  • Clicking on a video thumbnail takes you to a details page where videos can be played in the browser or downloaded.

Image view doesn't update after first upload

I think this is because theclearImageView action doesn't actually do anything, since there are initially no uploaded items. The InfiniteScrollingElement update cycle is triggered by a size change in the element, and since nothing gets cleared, the size doesn't change, and this update never triggers. Consequently, the user has to manually reload to see newly-uploaded data after the first upload.

Handle duplicate names when downloading.

When downloading multiple files, it will produce weird results if some of the images have the same name. The MALLARD frontend should automatically detect this condition and disambiguate the names.

Show overall progress of uploads

The upload workflow should have an overall progress bar that reflects the percentage of the total number of images that are uploaded. This should be very easy to implement, although some thought needs to go into the UI.

Back button fails to work in the details view

I've seen this off and on forever, but until now, I didn't have a reliable way to reproduce it. Here's how to do that:

  • Click on a video thumbnail.
  • Press the play button to play the video.
  • Pause the video
  • Press the back button

The back button will disappear, but it will not actually go back to the home page.

Upload fails to handle difficult image types

I have some directories that contain large TIFF orthophotos. If you try to upload these to MALLARD, it will just hang. I don't think it's even getting to the preprocessing stage. No errors appear in the console, so I suspect this is failing somewhere in an action creator and the error is getting eaten by Redux.

image

These types of images don't necessarily need to be supported, but it should at least handle them gracefully, possibly by ignoring them as it does with other non-image files.

Notes are cut off on the details page

For some media, especially wide ones like videos, part of the notes card can be cut off, at least on certain screen sizes. This issue might plausibly be merged with #12, since fixing that should fix this too.

Site sometimes fails to load

This was observed on Firefox running on Windows 11, on a machine connected to the VPN where I hadn't tried loading the site before. I see the following error in the console:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://bsailn2.ad.ufl.edu:8085/.well-known/openid-configuration. (Reason: CORS request did not succeed). Status code: (null).

This probably just means that Fief needs to be added to the list of allowed origins.

Expanding a section while selected hangs the UI

This does not work the way it should. Instead, it dispatches a whole bunch of selectImages actions, each for only a single item. I think something weird is going on here with the selection logic in artifact-thumbnail.

Preserve scroll location

When you click on a thumbnail, enter the details view, and then go back to the main view, it will scroll back to the top of the page. It would be nice to preserve and restore the users' scroll position instead.

Implement Basic Auth

This does not need to be fancy. We just need a stop-gap authentication method so that our data isn't completely exposed on the UF network.

Switch video player to Vidstack

Vidstack seems like a good recent project. After fighting for hours with Video.js and still not getting it to work, I appreciate that it supports web components natively. I think that this could significantly improve the experience of watching videos within MALLARD,

Increase speed of upload thumbnail generation

Currently, thumbnail generation is bottlenecking uploads with high-resolution images.

I think Pica supports some options to do nearest-neighbor downsampling and stuff like that, but I haven't really investigated it.

Support live previews for videos

This goes beyond showing a mere video thumbnail on the homepage. The thumbnail will be styled in such a way as to make clear that it's a video (similar to Google Photos). Additionally:

  • Hovering over the thumbnail starts playing the video.
  • Moving the mouse horizontally while hovering over the thumbnail scrubs through the video.

Multiple queries should result in field-wise OR-ing instead of unions.

This is less intuitive, but allowing users to request unions of arbitrary queries has some non-negligible performance implications. Furthermore, the only place where this capability is actually used right now is for searching, and this would work just as well (and be much more efficient) if we made this change.

Wait to close upload modal until `batch_update` completes.

With really big uploads, batch_upload can take a non-trivial amount of time. If you close the modal before it's complete, the user will see incorrect data. If the user then reloads the page, batch_upload will be cancelled and that incorrect data will become permanent.

So... we should probably wait for batch_upload to complete before closing the modal. This can be implemented similar to the bulk deletion action.

Implement CI Pipeline

This project is long overdue for some sort of CI, possibly using Github actions. While we're at it, we can build Docker images and push them to Docker Hub.

Create some workflow for backups

This is especially important as the amount of data stored in MALLARD continues to grow.

  • MySQL backups are pretty easy. We can mostly copy the workflow from eLabFTW.
  • MinIO backups are harder, but we might be able to cook something up using s3fs.

Uploading with an active search doesn't work

If you search for something, and then upload new data while searching, when you finish the upload and it reloads the image view, it will use the incorrect default query instead of the search query.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.