iodepo / oih-ui Goto Github PK

Description:

On search.oceaninfohub.org, copy/pasting and sharing a URL to a search result page works. On oceaninfohub.org, it doesn't.

• Fix this function

This may be linked to #49

All CreativeWork patterns and their derivatives have a license property that can and should be leveraged for filtering.

• Coordinate with ODIS-Arch developers to make best use of the license field.
• Coordinate with any CDIF progress on the issue.

xref: Cross-Domain-Interoperability-Framework/licenses-and-restrictions#1

iodepo/oih-ui-docker#13

to reproduce:

• single left-click once on the top-right 'burger button' menu
• close the menu (click X)
• now try to re-click (single click) the burger button
  • section/scrollbar seems to "flash" but the menu does not open

Can reproduce this with Firefox and Chrome (on staging and production instances).

iodepo/oih-ui-docker#12

@jmckenna to investigate:

Current state:

• The QGIS file in the regions directory controls how queries are restricted based on the dropdown on the main search bar.
• This QGIS file will only filter records that have geospatial information in them.
• A list of UNSD country::region codes is available in the same folder for textual filtering.

Desired state:

• The QGIS regions should be redrawn to include the landmasses and their neighbouring waters
• The dropdown names should be updated to include the landmasses, e.g. "Africa and coastal waters"
• When a region is chosen, the filtering should use the QGIS spatial filter as well as a textual filter based on the UNSD codes (which map countries etc to regions) provided here
• Thus, a textual query "cetaceans" and a selection of "Africa and coastal waters" from the dropdown should recover all records which either 1) have geospatial information in the QGS pockets we define or (inclusive) 2) have any African nation or sub-region listed in the UNSD document in their textual location fields (and related fields, like keywords and titles)

In future, we will consider if and how to parse out region/sub-region/country/city ... names from textual queries to dynamically restrict records returned.

Is there any reason we have a COPY in the frontend Dockerfile, as we already have the volume mounted at /app?

An advanced search allowing selection of / mashup of facets and properties to formulate more precise queries.

Options open, to be drafted here.

The advanced search would allow:

• multi-facet queries
• Query by (one or more) partners
• selection of properties to query
  • selection of regions to query
  • search in a selected spatial region (x, y, z) on a map interface
  • search in a selected temporal range
• ...

Application case: searching for Marine Spatial Planning assets

• my test server is solr main branch built from source (unreleased Solr 10.0.0)
• logs show warnings of several settings in /solr/conf/solrconfig.xml
• at some point we should tackle these (for example, the jmx warning can be avoided, as jmx was deprecated since Solr 7.0 release, by commenting out line #260 of solrconfig.xml, so it uses the new settings in solr.xml instead)
• to tackle later on a rainy day
  

OIH partners can and do add provenance information in their records. This is / should be distinct from the provenance information that OIH adds to these records when they are harvested.

• Examine the support pattern for provenance and ensure this is used correctly in the UI/UX
• Consider if further uses for provenance can be easily implemented.

@jmckenna to develop dashboard check in sync with progress here.

@fils and @pbuttigieg to help settle provenance pattern with bearing on CDIF.

• On the landing page, some elements to help users construct valid Boolean searches (AND, OR, XOR, NOT)
• On the advanced search (#48) - allow Boolean query construction with dropdowns.

May only be feasible in the advanced search, as we don't want to overcomplicate the landing page. Perhaps offer a syntax cheat sheet in a tool tip.

the main README file points to the README files in the subdirs, but there is no such file in /solr

add GA4 analytics so we know what people have been looking for and in what region.
add analytics to see what links they have clicked after a search

All lists of partners should be derived from a query to the OIH graph - this is not a manually edited list.

• An element links off to a list of all OIH partners (with a URL to their human-friendly site, and the last time the partner was indexed) should be added to the landing page and/or the ribbon. This may be done as a link to the dashboard.
• Partner logos should be harvested from their ODIS records schema:logo, see this example.
  • these images should come from ODISCat records (there will be a field in ODISCat that allows the submitter to add a URL to their logo)
• Logos and partner names to appear on a visual element (like a carousel) on the OIH landing page.
  • the start point of the carousel rotation should be randomised.
• All lists of partners should be alphabetical UNLESS there's a good reason not to

• add single link, somewhere on the opening interface, to query the ODIS graph directly: http://graph.oceaninfohub.org/blazegraph/#query

When running the docker version of the loader I tried

 docker run -ti --rm -v "/data/imports/data:/opt/data" -e "SOLR_URL=http://solr:8983/solr/ckan" 5b4155878485 indexer.py

Where the 5b4.... is my local image build from the Dockerfile.
This fails to connect to the solr endpoint

For reference, with the docker ps we see

CONTAINER ID   IMAGE                            COMMAND                  CREATED       STATUS       PORTS                                                      NAMES
a1c284d2ffcd   fils/dagster_oih:0.0.38          "dagster-daemon run"     6 days ago    Up 6 days                                                               oih-dagster-dagster-daemon-1
a21265452938   fils/dagster_oih:0.0.38          "dagit -w workspace.…"   6 days ago    Up 6 days    0.0.0.0:3000->3000/tcp                                     oih-dagster-dagster-dagit-1
d684ba3b6255   postgres:13.3                    "docker-entrypoint.s…"   6 days ago    Up 6 days    0.0.0.0:5432->5432/tcp                                     oih-dagster-dagster-postgres-1
c1a29cd136ed   minio/minio:latest               "/usr/bin/docker-ent…"   6 days ago    Up 6 days    0.0.0.0:54321->54321/tcp, 0.0.0.0:9001->9000/tcp           oih-index-support-s3system-1
a93cff110cbd   nawer/blazegraph                 "docker-entrypoint.s…"   6 days ago    Up 6 days    0.0.0.0:9999->9999/tcp                                     oih-index-support-triplestore-1
6999ce530992   chromedp/headless-shell:latest   "/headless-shell/hea…"   6 days ago    Up 6 days    0.0.0.0:9222->9222/tcp                                     oih-index-support-headless-1
e2be0455ecc5   portainer/portainer-ce           "/portainer"             6 days ago    Up 6 days    0.0.0.0:8000->8000/tcp, 0.0.0.0:9000->9000/tcp, 9443/tcp   portainer
9c682f7c7402   nginxproxy/acme-companion        "/bin/bash /app/entr…"   12 days ago   Up 12 days                                                              oih-ui-docker_letsencrypt-nginx-proxy-companion_1
7e7527e3b758   oih-ui-docker_web                "docker-entrypoint.s…"   12 days ago   Up 12 days                                                              oih-ui-docker_web_1
d84a2a4a6a52   oih-ui-docker_api                "uvicorn api.main:ap…"   12 days ago   Up 12 days   8000/tcp                                                   oih-ui-docker_api_1
b71640a3f34e   nginxproxy/nginx-proxy:latest    "/app/docker-entrypo…"   12 days ago   Up 12 days   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                   oih-ui-docker_nginx-proxy_1
d4dfefdbe093   solr:8                           "docker-entrypoint.s…"   12 days ago   Up 12 days                                                              oih-ui-docker_solr_1

The questions are:

• The SOLR_URL wont work will it (I get a connection error) since that is an internal docker network address?
• Can I connect a docker container to the established network that container (solr) is part of? Which I think in our case is oih-ui-docker_solr so the connection can be made?
• Do I need just need to expose solr to a host port to enable this indexing (I'm sure that would work, but would be nice to avoid)
• Is there an oih_api proxied endpoint that routes me back into solr?

Thanks

the UNSD Methodology CSV file is included in both the indexer and the regions (with different filenames), this will lead to problems in the future whenever the list is update and not both files are updated. Should only be one file.

no info about rewrite_prov.py and how to use it

the main README file points to the README files in the subdirs, but there is no such file in /api

• Indonesia NODC node leverages JSON-LD with GeoSPARQL values in Well Known Text (WKT) format, such as

"geosparql:hasGeometry":  {
  "@type": "http://www.opengis.net/ont/sf#Polygon",
  "geosparql:asWKT": {
    "@type": "geosparql:wktLiteral",
    "@value": "POLYGON((80.0 -20.0, 80.0 20.0, 160.0 20.0, 160.0 -20.0, 80.0 -20.0))"
  },
  "geosparql:crs": { 
    "@id":"http://www.opengis.net/def/crs/OGC/1.3/CRS84"
  }
}

• however that WKT is not used in the region dropdown:
  • see https://oih.staging.gatewaygeo.ca/results/Dataset?search_text=%22moored+Buoy%22&page=0&region=Global

the general README file should contain information on all the software that needs to be installed to run the complete suite

till now I had to:

 pip3 install requests
pip3 install shapely
pip3 install conversions
pip3 install python-dateutil

• Convert "TRY" to title case
• Update list of Try examples / query terms to return instructive results in terms of UX

The "Spatial Data" label is reportedly confusing some users: the content shown is any ODIS record that has spatial metadata.

• Relabel "Spatial Data" to "Spatial search"
• Consider - with IODE team - alternative visual representation of the spatial UI (rather than being "just another" facet)

xref #38

e.g. queries/dups.py

In addition to a number of CSS issues which are probably covered by #31, I noticed that the category icons on the home page do not have the correct aspect ratio. This has been introduced here (max-width on .category-icon): aa6c4ba.

Ensure responsive design is implemented.

• Create a UI element that allows users to resort dropdown lists according to:
  • Alphabetical order
  • Number of records provided
  • ...

The UI element can be near / experientially equivalent to the "Clear" button.

the indexer README file ends with the line ## To Run, no information on how to run this

the hardcoded paths
BASE_DIR='./../../jsonld/120822/summoned'
PROV_DIR='./../../jsonld/120822/re-prov'
should be coming from a setting or argument

iodepo/oih-ui-docker#14

The issue

Some of the schema.org keys are given very generic labels, which can be misunderstood on the UI. For example, "provider" is meant to indicate the provider of the metadata record to OIH rather than the provider of the entity it describes (courses, vessels, services...). Therefore we should relabel ambiguous fields (functionality to do this already exists).

• provider --> "metadata from" or "metadata provided by"
• ... (more to be added)

We expect that partners will host copies of digital assets, leading to multiple identical or near-identical results. On the UI and in the UX, these should not appear as independent results but - collectively - in a grouped result. That is, one result element with a visual sub-element stating that "[0-9] versions available". Clicking on the sub-element

• IODE to decide whether the identification of (near-)identical records occurs in the ODIS Graph or in the OIH SOLR index.
  • If ODIS-level detection is chosen, update Dashboard to report on degree of redundancy.
• Determine which record should be the "displayed" record (metadata completeness?)
• implement grouping on the schema:sameAs property
• define grouping rules based on other keys (e.g. name, description, identifier, ...)
• implement UI/UX elements to handle grouping and display/expand groups on click, noting different providers.

The terms in the drop down on the landing page (see figure) should pre-filter the corpus by 1) subsetting records that have geospatial coordinates within a set of OIH-defined GeoJSON shapes associated with each category and 2) subsetting records which have an schema.org address which includes a term associated with the region category in our UNSD-derived file.

• Assess whether regional categories and their associated GeoJSON files are representative of the corpus
  • Confirm whether the link to the UNSD administrative region classification is functioning as expected.
• Assess whether regions from marineregions.org or another provider (IHO regions?) can be used in addition to what we have
• Update shapefiles defining regions
• Update the UNSD file in our repo to include other regions (documenting clearly that these are not UNSD)
• Update regional selection
• Enable multi-select (xref #48)

• example screen capture, today, on a Google phone (Android), using its Chrome browser
• the burger-button flashes at top and then disappears on first load (in other words, the page loads without any burger-button menu), and then the facet counts are all mangled together
  

• this causes odd whitepace in header, sidebar, and footer, after performing a search
• see: https://search.oceaninfohub.org/results/?search_text=turtle&page=0

(might be related to recent changes through aa6c4ba ?

• the current search interface only queries for type: ResearchProject when a user selects Project
• however some partners leverage type: Project in their JSON-LD, which then causes those records not to be found

There are several unwieldy UX issues that would be good to smooth out during development.

Growing list, here:

Minor

• In some cases, clicking on facets resets any other filters (e.g. provider) - fix: the UX should offer a more intuitive way to manage the retention and removal of filters.
• In some cases, results are not shown when dropdown filters are used (hard to reproduce, encountered with searching for a provider within Institutions pre-filtered for the Africa region)
• Add a "report this result" button to each search result - clicking on this will autogenerate a report that links directly to the offending result and prompts the user to fill in more detail. The report will be sent to a designated email address for triage to this tracker.
• Update the Contact element on the site to include feedback and optimisation mechanisms.
• Add number of OIH sources searched per query (xref #59) and the number of sources in the result set

Major

• Sort function after search completes: similar to the OBPS (example, here, not the "Sort" function on right). Essentially we should be able to leverage all used / populated metadata fields to sort results.

• WorldOceansDatabase (WOD) partner potentially has ~18 millions records
• OIH search interface must handle results, efficiently
• related to iodepo/odis-arch#126

• Logo not centered and insufficient spacing
• Lack of contrast in search button
• Incorrect sizing of search button
• Large white space above footer
• Insufficient spacing in footer

• CIOOS partner's JSON-LD has multiple languages for many of its properties, here is a snippet:

        "keywords": [
            {
              "@language": "en",
              "@value": "watershed"
            },
            {
              "@language": "fr",
              "@value": "bassin versant"
            }
        ],
        "name": [
          {
            "@language": "fr",
            "@value": "Projet REKEAU - Entreprises manufacturi\u00e8res (2019)"
          },
          {
            "@language": "en",
            "@value": "REKEAU Project - Manufacturing (2019)"
          }
        ],
  

• both our back-end indexing process, as well as our front-end UI should be adapted to handle multiple languages
  • the ODIS graph could be adapted to handle multiple languages, and then those changes could be pushed into the Solr index
  • likely the front-end UI should allow users to filter results by language also

xref: iodepo/odis-arch#287

• Explore how to implement a subscription service / continuous search / push notification when an offer (approximately) matches a demand.
• Consider implementing a UI page where offers and demands can be created and pushed into the OIH SOLR backend.
• Consider registering this matchmaking service and its corpus in a dedicated ODIS node
• Consider requiring an OceanExpert login and authentication to use the service, and identify where to send notifications.
  • Explore leveraging "knows about" and other properties to boost matchmaking

On the UI/UX dev side, this should be a question of creating a form-like interface. IODE to provide backend elements.

• only happens on Staging
  • https://oih.staging.gatewaygeo.ca/results/Person?search_text=Xinmei+Huang&page=0
  • versus (correct) :
  • https://search.oceaninfohub.org/results/Person?search_text=Xinmei+Huang&page=0
• likely related to changes in #33

@fils @jmckenna

On the UI, results pointing back to vessels coming from the Africa node (here) link directly to the JSON-LD, rather than an landing page in the partner's site.

I think this is because the @id is used correctly, or perhaps the url is the same as the @id to the JSON-LD.

The expected behaviour is that clicking on the result's main link will take you to the partner's landing page for that vessel.

when running the indexer :

Exception on bare upsert: HTTPConnectionPool(host='localhost', port=8983): Max retries exceeded with url: /solr/ckan/update?commit=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3ae1f17a60>: Failed to establish a new connection: [Errno 111] Connection refused'))

error: invalid path 'indexer/test/prov:wasAttributedTo/dest/52ba2db7a5.json'
fatal: unable to checkout working tree

git version 2.38.0

./indexer.py
Traceback (most recent call last):
File "/Users/arno/PhpstormProjects/oih-ui/indexer/./indexer.py", line 424, in
index_all(['../all_files.v2.txt'])
File "/Users/arno/PhpstormProjects/oih-ui/indexer/./indexer.py", line 344, in index_all
with open(path) as files:
FileNotFoundError: [Errno 2] No such file or directory: '../all_files.v2.txt'

The issue

A hard quoted string "foo bar" may not have any exact or only a few exact matches. There should be a fall back mechanism which will show results for "foo" and "bar" if so.

• Ensure OIH search function has a fall back as described
• Add visual element informing user that their literal search string has been modified.

• the oceaninfohub.org page, the facet pages, result pages, and any other page that does not require human editing (e.g. the news page) should be transferred to REACT
• Investigate and fix odd behaviour of the back button on browsers (the links change, but the UI / webpage is static)

python3 main.py
Traceback (most recent call last):
File "/Users/arno/PhpstormProjects/oih-ui/api/api/main.py", line 14, in
from api.util.solr_query_builder import SolrQueryBuilder
ModuleNotFoundError: No module named 'api.util'

should this be api.api.util???