denshoproject / encyc-mw Goto Github PK

Mediawiki source, themes, and extensions

PHP 50.04% Makefile 0.02% Shell 0.03% HTML 0.54% CSS 5.73% JavaScript 41.65% Python 0.06% Lua 1.00% Ruby 0.04% PLpgSQL 0.18% Perl 0.04% Less 0.62% Gherkin 0.01% Handlebars 0.01% Batchfile 0.01% EJS 0.01% Mustache 0.03% Vue 0.01%

encyc-mw's Introduction

TODO

encyc-mw's People

Contributors

Watchers

encyc-mw's Issues

Error when uploading image from encyc-psms in stage

When adding a new Source record to encyc-psms, after clicking Save the request spins for awhile and then fails with a 502 Bad Gateway.

The relevant parts of the encyc-psms log (/var/log/encyc/psms.log) do not indicate a problem that I can see:

2020-08-28 11:30:47,685 DEBUG    [models.wiki_sync]  wiki_sync(Source object (2540))
2020-08-28 11:30:47,686 DEBUG    [wiki._login]  initializing
2020-08-28 11:30:47,686 DEBUG    [wiki._login]  no http passwd
2020-08-28 11:30:47,690 DEBUG    [connectionpool._new_conn]  Starting new HTTP connection (1): encycmw.local:80
2020-08-28 11:30:48,141 DEBUG    [connectionpool._make_request]  http://encycmw.local:80 "GET /api.php?meta=siteinfo%7Cuserinfo%7Cuserinfo&siprop=general%7Cnamespaces&uiprop=groups%7Crights%7Cblockinfo%7Chasmsg&continue=&action=query&format=json HTTP/1.1" 200 None
2020-08-28 11:30:48,142 DEBUG    [wiki._login]  <Site object 'encycmw.local/'>
2020-08-28 11:30:48,142 DEBUG    [wiki._login]  logging in
2020-08-28 11:30:48,172 DEBUG    [connectionpool._make_request]  http://encycmw.local:80 "GET /api.php?meta=tokens&type=login&action=query&format=json HTTP/1.1" 200 None
2020-08-28 11:30:48,999 DEBUG    [connectionpool._make_request]  http://encycmw.local:80 "POST /api.php HTTP/1.1" 200 None
2020-08-28 11:30:49,000 WARNING  [client.handle_api_result]  Main-account login via action=login is deprecated and may stop working without warning. To continue login with action=login, see [[Special:BotPasswords]]. To safely continue using main-account login, see action=clientlogin.
2020-08-28 11:30:49,026 DEBUG    [connectionpool._make_request]  http://encycmw.local:80 "GET /api.php?meta=userinfo%7Cuserinfo&uiprop=groups%7Crights%7Cblockinfo%7Chasmsg&continue=&action=query&format=json HTTP/1.1" 200 None
2020-08-28 11:30:49,027 DEBUG    [wiki._login]  <Site object 'encycmw.local/'>
2020-08-28 11:30:49,027 DEBUG    [wiki._login]  done
2020-08-28 11:30:49,119 DEBUG    [connectionpool._make_request]  http://encycmw.local:80 "GET /api.php?prop=info%7Cimageinfo&titles=File%3Addr-testing-123-456.jpg&inprop=protection&iiprop=timestamp%7Cuser%7Ccomment%7Curl%7Csize%7Csha1%7Cmetadata%7Carchivename&continue=&meta=userinfo&uiprop=blockinfo%7Chasmsg&action=query&format=json HTTP/1.1" 200 None
2020-08-28 11:30:49,120 DEBUG    [models.wiki_sync]      keys: file:----:----:----
2020-08-28 11:30:49,120 DEBUG    [models.wiki_sync]      functions: ['upload', 'update', 'link']
2020-08-28 11:30:49,121 DEBUG    [models.wiki_sync]  _wiki_upload
2020-08-28 11:30:49,121 DEBUG    [models._wiki_upload]  _wiki_upload(Source object (2540))
2020-08-28 11:30:49,121 DEBUG    [models._wiki_upload]          /var/www/encycpsms/media/sources/N/None/startrek-lowerdecks_3g9jtge.jpg
2020-08-28 11:30:49,121 DEBUG    [models._wiki_upload]          > /var/www/encycpsms/media/sources/N/None/ddr-testing-123-456.jpg
2020-08-28 11:30:49,121 DEBUG    [wiki.upload_file]  upload_file(/var/www/encycpsms/media/sources/N/None/ddr-testing-123-456.jpg)
2020-08-28 11:30:49,121 DEBUG    [wiki.upload_file]  page_name ddr-testing-123-456.jpg
2020-08-28 11:30:49,143 DEBUG    [connectionpool._make_request]  http://encycmw.local:80 "GET /api.php?prop=info%7Cimageinfo&titles=File%3Addr-testing-123-456.jpg&inprop=protection&iiprop=timestamp%7Cuser%7Ccomment%7Curl%7Csize%7Csha1%7Cmetadata%7Carchivename&continue=&meta=userinfo&uiprop=blockinfo%7Chasmsg&action=query&format=json HTTP/1.1" 200 None
2020-08-28 11:30:49,144 DEBUG    [wiki.upload_file]  wf <Image object 'b'File:Ddr-testing-123-456.jpg'' for <Site object 'encycmw.local/'>>
2020-08-28 11:30:49,144 DEBUG    [wiki.upload_file]  f <_io.BufferedReader name='/var/www/encycpsms/media/sources/N/None/ddr-testing-123-456.jpg'>
2020-08-28 11:30:49,164 DEBUG    [connectionpool._make_request]  http://encycmw.local:80 "GET /api.php?prop=info%7Cimageinfo&titles=File%3Addr-testing-123-456.jpg&inprop=protection&iiprop=timestamp%7Cuser%7Ccomment%7Curl%7Csize%7Csha1%7Cmetadata%7Carchivename&continue=&meta=userinfo&uiprop=blockinfo%7Chasmsg&action=query&format=json HTTP/1.1" 200 None
2020-08-28 11:30:49,182 DEBUG    [connectionpool._make_request]  http://encycmw.local:80 "GET /api.php?meta=tokens&type=csrf&action=query&format=json HTTP/1.1" 200 None

Note that encyc-psms is using http://encycmw.local, an internal domain set in /etc/hosts, instead of https://mwstage.densho.org, because mwclient cannot handle both TODO and HTTP Basic passwords. We cannot simply use an IP address here because Nginx would have no way to determine which vhost to send the request to. I suppose we could use the IP:PORT combo (http://127.0.0.1:8082).

The MediaWiki log (/var/log/nginx/encycmw.log) mostly looks normal::

127.0.0.1 - - [28/Aug/2020:11:30:48 -0700] "GET /api.php?meta=siteinfo%7Cuserinfo%7Cuserinfo&siprop=general%7Cnamespaces&uiprop=groups%7Crights%7Cblockinfo%7Chasmsg&continue=&action=query&format=json HTTP/1.1" 200 3565 "-" "mwclient/0.10.1 (https://github.com/mwclient/mwclient)"
127.0.0.1 - - [28/Aug/2020:11:30:48 -0700] "GET /api.php?meta=tokens&type=login&action=query&format=json HTTP/1.1" 200 111 "-" "mwclient/0.10.1 (https://github.com/mwclient/mwclient)"
127.0.0.1 - - [28/Aug/2020:11:30:48 -0700] "POST /api.php HTTP/1.1" 200 451 "-" "mwclient/0.10.1 (https://github.com/mwclient/mwclient)"
127.0.0.1 - - [28/Aug/2020:11:30:49 -0700] "GET /api.php?meta=userinfo%7Cuserinfo&uiprop=groups%7Crights%7Cblockinfo%7Chasmsg&continue=&action=query&format=json HTTP/1.1" 200 1040 "-" "mwclient/0.10.1 (https://github.com/mwclient/mwclient)"
127.0.0.1 - - [28/Aug/2020:11:30:49 -0700] "GET /api.php?prop=info%7Cimageinfo&titles=File%3Addr-testing-123-456.jpg&inprop=protection&iiprop=timestamp%7Cuser%7Ccomment%7Curl%7Csize%7Csha1%7Cmetadata%7Carchivename&continue=&meta=userinfo&uiprop=blockinfo%7Chasmsg&action=query&format=json HTTP/1.1" 200 418 "-" "mwclient/0.10.1 (https://github.com/mwclient/mwclient)"
127.0.0.1 - - [28/Aug/2020:11:30:49 -0700] "GET /api.php?prop=info%7Cimageinfo&titles=File%3Addr-testing-123-456.jpg&inprop=protection&iiprop=timestamp%7Cuser%7Ccomment%7Curl%7Csize%7Csha1%7Cmetadata%7Carchivename&continue=&meta=userinfo&uiprop=blockinfo%7Chasmsg&action=query&format=json HTTP/1.1" 200 418 "-" "mwclient/0.10.1 (https://github.com/mwclient/mwclient)"
127.0.0.1 - - [28/Aug/2020:11:30:49 -0700] "GET /api.php?prop=info%7Cimageinfo&titles=File%3Addr-testing-123-456.jpg&inprop=protection&iiprop=timestamp%7Cuser%7Ccomment%7Curl%7Csize%7Csha1%7Cmetadata%7Carchivename&continue=&meta=userinfo&uiprop=blockinfo%7Chasmsg&action=query&format=json HTTP/1.1" 200 418 "-" "mwclient/0.10.1 (https://github.com/mwclient/mwclient)"
127.0.0.1 - - [28/Aug/2020:11:30:49 -0700] "GET /api.php?meta=tokens&type=csrf&action=query&format=json HTTP/1.1" 200 110 "-" "mwclient/0.10.1 (https://github.com/mwclient/mwclient)"
127.0.0.1 - - [28/Aug/2020:11:31:17 -0700] "POST /api.php HTTP/1.1" 499 0 "-" "mwclient/0.10.1 (https://github.com/mwclient/mwclient)"

But then in the MediaWiki error logs (/var/log/nginx/encycmw.err):

2020/08/28 11:30:48 [error] 19975#19975: *3 FastCGI sent in stderr: "PHP message: PHP Warning:  "continue" targeting switch is equivalent to "break". Did you mean to use "continue 2"? in /opt/encyc-mw/htdocs/includes/json/FormatJson.php on line 297" while reading response header from upstream, client: 127.0.0.1, server: editors.densho.org, request: "GET /api.php?meta=siteinfo%7Cuserinfo%7Cuserinfo&siprop=general%7Cnamespaces&uiprop=groups%7Crights%7Cblockinfo%7Chasmsg&continue=&action=query&format=json HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "encycmw.local"
2020/08/28 11:30:48 [error] 19975#19975: *3 FastCGI sent in stderr: "PHP message: PHP Warning:  "continue" targeting switch is equivalent to "break". Did you mean to use "continue 2"? in /opt/encyc-mw/htdocs/languages/LanguageConverter.php on line 773" while reading response header from upstream, client: 127.0.0.1, server: editors.densho.org, request: "POST /api.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "encycmw.local"
2020/08/28 11:30:49 [error] 19975#19975: *3 FastCGI sent in stderr: "PHP message: PHP Warning:  count(): Parameter must be an array or an object that implements Countable in /opt/encyc-mw/htdocs/includes/api/ApiQueryBase.php on line 267" while reading response header from upstream, client: 127.0.0.1, server: editors.densho.org, request: "GET /api.php?prop=info%7Cimageinfo&titles=File%3Addr-testing-123-456.jpg&inprop=protection&iiprop=timestamp%7Cuser%7Ccomment%7Curl%7Csize%7Csha1%7Cmetadata%7Carchivename&continue=&meta=userinfo&uiprop=blockinfo%7Chasmsg&action=query&format=json HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "encycmw.local"
2020/08/28 11:30:49 [error] 19975#19975: *3 FastCGI sent in stderr: "PHP message: PHP Warning:  count(): Parameter must be an array or an object that implements Countable in /opt/encyc-mw/htdocs/includes/api/ApiQueryBase.php on line 267" while reading response header from upstream, client: 127.0.0.1, server: editors.densho.org, request: "GET /api.php?prop=info%7Cimageinfo&titles=File%3Addr-testing-123-456.jpg&inprop=protection&iiprop=timestamp%7Cuser%7Ccomment%7Curl%7Csize%7Csha1%7Cmetadata%7Carchivename&continue=&meta=userinfo&uiprop=blockinfo%7Chasmsg&action=query&format=json HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "encycmw.local"
2020/08/28 11:30:49 [error] 19975#19975: *3 FastCGI sent in stderr: "PHP message: PHP Warning:  count(): Parameter must be an array or an object that implements Countable in /opt/encyc-mw/htdocs/includes/api/ApiQueryBase.php on line 267" while reading response header from upstream, client: 127.0.0.1, server: editors.densho.org, request: "GET /api.php?prop=info%7Cimageinfo&titles=File%3Addr-testing-123-456.jpg&inprop=protection&iiprop=timestamp%7Cuser%7Ccomment%7Curl%7Csize%7Csha1%7Cmetadata%7Carchivename&continue=&meta=userinfo&uiprop=blockinfo%7Chasmsg&action=query&format=json HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "encycmw.local"
2020/08/28 11:30:49 [error] 19975#19975: *3 FastCGI sent in stderr: "PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP War
2020/08/28 11:30:49 [error] 19975#19975: *3 FastCGI sent in stderr: "e 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: P
2020/08/28 11:30:49 [error] 19975#19975: *3 FastCGI sent in stderr: "1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860PHP message: PHP
...

Some strange errors relating to editors.densho.org which should not be anywhere in the configs for the app as it's installed in stage.
This error is repeated over and over ad nauseum for ~76 MB:

PHP message: PHP Warning:  preg_replace(): Compilation failed: invalid range in character class at offset 4 in /opt/encyc-mw/htdocs/includes/media/FormatMetadata.php on line 1860

FormatMetadata.php's job is to "[f]ormat Image metadata values into a human readable form". The error occurs in sanitizeKeyForAPI. This is strange because it appears that the contents of the image file are being fed to this function. Some abuse of the error_log() function reveals this to be the case.

This is MediaWiki 1.27.7, which you'd think would have had the bugs worked out...

On the Python side we're using mwclient current stable version 0.10.1, which claims to support "MediaWiki 1.16 and above". There are a couple closed issues that might seem to relate but don't really.

MW will not accept certain file types for upload

LocalSettings.php is not properly configured to accept several filetypes used in the encyc-psms app.

$wgFileExtensions was not set correctly in LocalSettings.php after the MW upgrade. Compare config file on master at: https://github.com/denshoproject/encyc-mw/blob/master/conf/LocalSettings.php-1.24#L165

Reference: https://www.mediawiki.org/wiki/Manual:Configuring_file_uploads#Configuring_file_types

Related to: denshoproject/encyc-psms#18

APIError when uploading files from encyc-psms

Binaries for Source records in encyc-psms are uploaded to the editors' MediaWiki so they can see the files in context.
However, since installing encycfront-master_4.2.1~deb10 on the server MediaWiki has been throwing the following error for many files.

APIError at /admin/sources/source/add/
(
'verification-error',
'The file is a corrupt or otherwise unreadable ZIP file. It cannot be properly checked for security.',
'See https://editors.densho.org/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce&gt; for notice of API deprecations and breaking changes.'
)

Many of the files which are now causing the error were used for testing yesterday without incident.

A search for the error message returns two open MediaWiki project issues:

T201374 Investigate verification-error "'The file is a corrupt or otherwise unreadable ZIP file. It cannot be properly checked for security.'"
T33930 Image upload fail with error bad-zip due to being misdetected as a broken zip irrespective of mimetype
These issues are marked low priority and have not received attention since Aug 2018.

Comments to T33930 suggest that the error can be ignored by enabling $wgAllowJavaUploads which is disabled by default.

This error message is from encyc-psms but that app is simply reporting the error it received from MediaWiki.
APIError at admin_sources_source_add.zip

Upgrade to current MediaWiki LTS

We are currently using v1.24, which was EOL'd at the end of 2015. Current LTS version is 1.31 which is good til June 2021.

MediaWiki Version lifecycle: https://www.mediawiki.org/wiki/Version_lifecycle/en#Versions_and_their_end-of-life
Upgrading MediaWiki: https://www.mediawiki.org/wiki/Manual:Upgrading
MediaWiki on Debian/Ubuntu: https://www.mediawiki.org/wiki/Manual:Running_MediaWiki_on_Debian_or_Ubuntu

Note: running an out-of-date MediaWiki also affects encyc-psms and encyc-core.

Update: It would be nice to use the packaged MW in Debian.

stretch (oldstable): 1:1.27.7-1
stretch-backports: 1:1.31.2-1
buster (stable): 1:1.31.6-1
OTOH our customizations may not get upgraded along with Debian.

Upgrade MediaWiki

encyc-mw is currently running MediaWiki 1.31.8, which hits end-of-life in June 2021. This might be a good time to consider switching to a Debian package.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.