david-allison / manx-corpus-search Goto Github PK

Requested by Max

You could spell out what happens when you put just * in the Search box, either on the Main page, or on the Result-Browse page.

I feel we should show this as placeholder text below the search box. Remove or hide it when results are shown.

It's a little too tall on mobile (dominating compared to the objective of the site).

Make it wider (or redo it completely).

I noticed that it won't be able to find a definition for goll-mygeayrt but it can for goll and mygeayrt

Dictionary definitions were tested as a 'nice to have' - it seems they're useful enough to stay (I've had a few positive comments on them).

Currently, we don't provide the server with enough context to provide a translation. We work on the 'word' level, but the client side does not have enough context to either:

• break a word into morphemes
• expand a word to a known phrase/idiom

The request should be: (context, selection) rather than (word). From here, on the server we can process the selection and more accurately determine the correct phrase/word to return.

Test case above (could also be handled at the dictionary-level), but we lose out on phrase selection, and defining recursive/contextual structures.

I have added ManxCorpus to the Manx-English dictionaries accessible via Multidict

https://multidict.net/multidict/?sl=gv&tl=en&dict=MxCorp&word=snaue 

Very easy, since the URL

https://manxcorpus.com/?q=snaue

will search for the Manx word “snaue”.

However, I have bother doing things English searches, since I can’t find any URL which will search ManxCorpus for the English word “crawl”, for example. What I would hope for is something like

https://manxcorpus.com/?q=crawl&lang=en

It would useful to have some clear marking when documents are the original text (as in most of the eighteenth century religious texts which are translations), and when they are new translations from the Manx (Ned Beg for example).

If I search for 'lumlane' and nothing appears, suggest lum-lane (2 results)

Fix the commandline in publish.yml

Make the file executable

The main aim is to allow anyone to deploy a fork of the site to DigitalOcean

• remove folders in home
• Add jq install

https://www.imuseum.im/Olive/APA/IsleofMan/get.res?id=page.Scripts&kind=script&uq=20210325071104&for=%7E%2Fdefault.aspx&mode=group

version:'2.7.102.0',appVersion:'5.2.6',uq:'20210325071104',viewPointVersion:'4.28.18017.37845',lastModificationDateString:'2021-03-25-19-11-04'

COM:"Calf of Man Bird Observatory Report",
CEC:"Camp Echo",
CHU:"Camp Humor",
KCZ:"Camp Zeitung",
CTG:"Castletown Gazette",
DSC:"Das Schleierlicht",
MGN:"German Gymnastics Association",
GFL:"Green Final",
HNS:"Holiday News",
IDT:"Isle of Man Daily Times",
IME:"Isle of Man Examiner",
IMT:"Isle of Man Times",
WAC:"Isle of Man Weekly Advertising Circular",
IWG:"Isle of Man Weekly Gazette",
JMM:"Journal of The Manx Museum",
LAE:"Lager Echo",DLA:"Lager Laterne",
LAZ:"Lager Zeitung",LAU:"Lager-Ulk",
MNA:"Manks Advertiser",
MNM:"Manks Mercury",
TMC:"Manx Cat",
TFP:"Manx Free Press",
MNB:"Manx Liberal",
MMNT:"Manx Museum and National Trust Report",
MNP:"Manx Patriot",
MRS:"Manx Rising Sun",
MNS:"Manx Star",
TMS:"Manx Sun",
TMN:"Manxman",
MDP:"Mona Daily Programme",
MNH:"Mona's Herald",
PCG:"Peel City Guardian",
PSL:"Peel Sentinel",
QUT:"Quousque Tandem",
RCE:"Ramsey Chronicle",
RYC:"Ramsey Courier",
RWN:"Ramsey Weekly News",
TRS:"Rising Sun",
TTS:"TT Special",
UNU:"Unter Uns",
WER:"Werden"

A newspaper image seems to be in the format: https://www.imuseum.im/Olive/APA/IsleofMan/get/image.ashx?kind=block&href=MNH%2F1833%2F10%2F25&id=Ar0010000&ext=.png
&id=Ar0080001&ext=.png

Where MNH%2F1833%2F10%2F25 = MNH/1833/10/25

==Problems (currently)==

Unknown if these are solvable

• Need to link to an explorable version of a newspaper
  • This can be obtained by the "email" functionality, but these include a sk link which is unique to each page
    • https://www.imuseum.im/Olive/APA/IsleofMan/sharedpages/SharedView.Page.aspx?sk=FD92EBFD&href=MNH%2F1833%2F10%2F25&page=1
    • https://www.imuseum.im/Olive/APA/IsleofMan/sharedpages/SharedView.Page.aspx?sk=DFD5F928&href=MNH%2F1833%2F10%2F25&page=2
    • sk seems to be generated client-side, so we can hopefully reverse-engineer it
  • PDF download
    • Checks: https://www.imuseum.im/Olive/APA/IsleofMan/get/checkPdf.ashx?pdf=RrqaycCoWIe%2BFdbaSGrTeWM%3D
    • Download: https://www.imuseum.im/Olive/APA/IsleofMan/get/pdf.ashx?pdf=RrqaycCoWIe%2BFdbaSGrTeWM%3D
• Would be ideal to link to an explorable version of a clipping

JH would prefer for notes to not be visible while reading the corpus.

I agree. It would be sensible to hide notes by default and toggle them if [1] is clicked.

If a linkage between notes and the text can't be found, then the notes should be shown as-is.

We only have 1 trivial warning: might as well get to 0 and keep it that way

'Dhelby' returns results.
'Clague' seems case insensitive

TODO: Add Notes to the main page of the document

Originally posted by @david-allison-1 in david-allison/manx-search-data#475 (comment)

aall -> results

aall -> no results

Helps merchandise features

Takes up RAM, more finnicky and verbose than using C#, and we no longer need it now we've moved to Lucene

Maybe make the help more prominent if nobody has searched, or provide example queries

Requested by Max

Done in the backend - just needs a UI

Initially: en + gv

Then: Different editions (such as in Carrey yn Pheccagh): Two English sources and 1 Manx

We have a lot of corpus information from the Newspaper archives on https://www.imuseum.im/newspapers/.

Ideal:

• Map from a text-based reference in manifest.json.txt to a data structure. Add an arbitrary validation check on manx-search-data
• Map from this data structure to a URL on the corpus search site
• Redirect from this URL to the appropriate page on the iMuseum site
• Add a link on the public-facing site

This is extremely inefficient and low-hanging fruit, can lead to timeouts when searching for a wildcard on a given work

https://github.com/david-allison-1/manx-corpus-search/blob/68b97e517f22c5f91c0427e0eed3d7ec556ccf45/CorpusSearch/Dependencies/Lucene/LuceneIndex.cs#L121-L137

TODO: Add iMuseum link

Originally posted by @david-allison-1 in david-allison/manx-search-data#475 (comment)

Now we have a domain, we should focus more on SEO. Lots of low hanging fruit here:

• Add details to the HEAD of the main corpus search page
• Provide HTML versions of works
• Backlinks (for later, long-term domains requested from govt)

I can't see how to initiate a new issue - but some Ned Beg pieces have '???' in the text - indicating that I wasn't sure of the translation and wanted to discuss with other contributors. When I use ??? in the search field I find other items, and the markers I was looking for are not highlighted.

Originally posted by @robteare in david-allison/manx-search-data#322 (comment)

This needs a test (may be hard if we don't have dictionaries on disk)

Originally posted by @david-allison in #149 (comment)

Request by Rob: Just use the slider in the Advanced Options

Requested by Max

Currently, manifest.json.txt refers to a single work.

In some cases, we'll have works translated by two different people (eg: Psalms1610).

These may want to be in the same CSV for alignment and comparison (?).

If this is the case, then we'll need to expand manifest.json.txt

Currently: dotnet publish --framework net8.0

Highlighting likely won't be perfect, but we can make it a lot better:

• Add highlighting to the main page
• Obtain the positions of the token that match in Lucene
  • See if we can match these to the 'real text' which is displayed to the user - likely possible in 98% of cases - word at index 5 in Lucene will likely be word 5 in the real text.
  • Question: Server-side or client side? Both are equally as powerful, and it seems I'm avoiding Client-side because I'm avoiding JavaScript
• Translations - likely going to be harder, main aim is to fix punctuation

https://github.com/david-allison-1/manx-corpus-search/runs/2550569740?check_suite_focus=true

2021-05-10T23:29:10.8274884Z Determining projects to restore...
2021-05-10T23:29:11.7157857Z Restored /var/www/manx-corpus/CorpusSearch/CorpusSearch.csproj (in 472 ms).
2021-05-10T23:29:12.5325316Z Restored /var/www/manx-corpus/CorpusSearch.Test/CorpusSearch.Test.csproj (in 800 ms).
2021-05-10T23:29:24.2855022Z CorpusSearch -> /var/www/manx-corpus/CorpusSearch/bin/Debug/net5.0/CorpusSearch.dll
2021-05-10T23:29:24.2866213Z CorpusSearch -> /var/www/manx-corpus/CorpusSearch/bin/Debug/net5.0/CorpusSearch.Views.dll
2021-05-10T23:29:24.4409503Z v15.7.0
2021-05-10T23:29:24.4426333Z Restoring dependencies using 'npm'. This may take several minutes...
2021-05-10T23:53:16.3627646Z ##[error]The operation was canceled.

Unsure what's happening here. Works when I ssh into the machine

• Need to re-run to see if this is flaky, or broken
• Might want to consider using a local npm repo mirror on the box

I like it that the Search throws up Cregeen entry and dictionary entry first. Presumably dictionary here means Phil Kelly’s dictionary, while “Dictionary” top right main page = Cregeen Aa-orderit. It would be helpful to make this clear.

From Max

Using the same terminology as http://bible.learnmanx.com/

If set, send the query with *query*

Initially on YouTube

Requested by Max