Provide our own JSON-LD Schema subset for deployments

Related to #56

What?

Giancarlo had the amazing idea of instead of depending on JSON-LD files that come and go, we provide for new deployments a smaller subset that can be used directly in

/admin/structure/strawberry_keynameprovider/schema_org/edit

We can also add a link in a README.md file inside strawberryfield module

This requires some group effort to select the best properties. Of course this is a starting point and people can extend or replace. But its a good and stable starting point

This base @context file will be hosted in this repo and will have its own versioning.
Given the flexible nature archipelago people then can add other plugin instances via the UI to expand on further vocabularies

Thanks @giancarlobi

Patch Drupal google_api_client 3.x on deployment through our composer.json patch command

What?

New module. An AMI dependency so we do not have to (grrmrmr ... such a bad idea, i should code all from scratch always...anyways) code GOOGLE API OAUTH2 access to Google sheets like we do on IMI. But, of course the module has bugs!

See this one i reported https://www.drupal.org/project/google_api_client/issues/3172577

The fix is like super crazy simple but it will take forever to be applied by the owner so we better do this.

Of course i will need a reminder of this by Friday =(

Archipelago 2020 Roadmap:1.0.0-RC1

Archipelago 2020 Roadmap

See also #5 and #35 and #79 for the whole historic recreation. @alliomeria I will work on this tomorrow.

This is our working enumeration of concrete tasks until end of 2020 (the year of the void()), per Component and Service, for public evaluation (critics and comments welcome).

All tasks listed here are leading our full feature release somewhere this first half of the year. 2020 has been weird, hurtful but strangely productive and Archipelago sailing, floating steady in the current and full of suprises. Checked tasks are ready, unchecked are in progress or planned. Priority is not given by this order.

Please feel free to comment, request more info or ask for clarification. Feature requests are also highly appreciated and taken in account (always, please!).

Strawberryfield

Field Property exposure to Drupal strategies
- JSON KEY Provider (flattener)
- JSON Flatten Keys
- JSONPATH/JMESPATH
- Entity Reference Casting Provider (Using UUID loading and configurable entity type) using JSON based hints to expose any semantic relationship to Search API
- JSON stored Service Endpoints with extended logic (e.g HOCR) - A.k.a Strawberry Flavor Data Source.
- Multi Map/ join: many properties to single. e.g All keys - Authorities- referring to creators, contributors etc unified as Agents keys. This leads to Fractal Ontologies and our Buckets approach.
File downloads and streaming
Ranged Request Streamer with back-to-front S3 managment and buffer/memory managment. For any exposed Binary Endpoint. Also streaming. This was some not sleeping much!

JSON representation and enrichment

Webforms integration

Media Displays

Field Formatters

API Ingest, Migration and backup

Service Architecture (Strawberry Runners)

SEO and API

Allow Media displays output to be embeded in HTML head for SEO
Test/Develop nested DATA VIEWS integration for OAI-ORE and OAI-PMH (See Format Strawberryfield and API builder)
Create (TWIG, metadata displays) and expose as endpoints full set of IIIF API JSON outputs.
Add helper methods and twig extensions to allow Metadata displays to access pre existing views (like object listings for a collection) to help build those lists.

ACL / Permissions

Integrate custom ACL with JSON Paths into per NODE ACL. Allowing this way to apply permissions to individual metadata elements/paths.
Same but needs better UI for referenced Services and Media
Allow Metadata (rule) to trigger ACL permissions. e.g if embargo_date == bla bla = remove public access
Allow for ACL inheritance (from parent, recursive) without hard copies.

Deployment and DevOPS

Batch Operations

Future roadmap

[x ] Solr Cloud/ Consortial ensemble
Native Wikibase/Wikidata publishing

Documentation:

Archipelago on Drupal9 without docker.

Just to share some settings here about this specific installation.

Bump drupal/config_inspector from 1.x@beta to dev-1.x for Beta2

What is new?

Config inspector, a contributed module we ship as a facility to debug missing config schemas is passing a wrong template variable and failing to render its main inspector page on Drupal 8.8.1. Moving to dev version and fixing composer.lock to a certain hash fixes the problem. This issues was discovered by @marlo-longley while testing beta2

Archipelago Roadmap 2019: Beta1

Archipelago 2019 Roadmap

This is our working enumeration of concrete tasks, per Component and Service, for public evaluation (and comments).

All tasks listed here are leading our full feature release end of 2019. Checked tasks are ready, unchecked are in progress or planned. Priority is not given by this order.

There are a lot more things involved and many are already done and coded, but i wanted to have a list that was closer to features checklist than the actual code milestones we are managing internally. Please feel free to comment, request more info or ask for clarification. Feature requests are also highly appreciated and taken in account (always, please!).

Strawberryfield

Field Property exposure to Drupal strategies
- JSON KEY Provider (flattener)
- JSON Flatten Keys
- JSONPATH/JMESPATH
- Entity Reference Casting Provider (Using UUID loading and configurable entity type)
- JSON stored Service Endpoints with extended logic (e.g HOCR)
- Multi Map/ join: many properties to single. e.g All keys - Authorities- referring to creators, contributors etc unified as Agents keys. This leads to Fractal Ontologies and our Buckets approach.

JSON representation and enrichment

Webforms integration

Media Displays

Add expected mime/type output to Media displays. Allows to tag media displays as JSON, XML or HTML only.
- React to mime type to allow JSON or XML output to be downloaded too.
Add new Data Views Plugin integration to allow Media Displays to preprocess values on views exposed as API endpoints
Version Media Displays
Provide example Twig templates for
- MODS
- DC and
- JSON-LD

Field Formatters

API Ingest, Migration and backup

Service Architecture (Strawberry Runners)

SEO and API

Allow Media displays output to be embeded in HTML head for SEO
Test/Develop nested DATA VIEWS integration for OAI-ORE and OAI-PMH
Create (TWIG, metadata displays) and expose as endpoints full set of IIIF API JSON outputs. UPDATED
- Add helper methods and twig extensions to allow Metadata displays to access pre existing views (like object listings for a collection) to help build those lists.

ACL / Permissions

Integrate custom ACL with JSON Paths into per NODE ACL. Allowing this way to apply permissions to individual metadata elements/paths.
Same but needs better UI for referenced Services and Media
Allow Metadata (rule) to trigger ACL permissions. e.g if embargo_date == bla bla = remove public access
Allow for ACL inheritance (from parent, recursive) without hard copies.

Deployment and DevOPS

Sync Configurations and remove non used ones for minio branch / periodic for each Drupal release
Site-build and remove orphan blocks
Add more utility views
Enable JSONAPI by default on minio branch
Create jsonapi user with jsonapi credentials for minio branch
Create basic scripts to automate Docker/Bash operations
Update AWS deployer to match minio including docs and Cloud Services integration

Batch Operations

Future roadmap

D9 readyness 😄
[x ] Solr Cloud/ Consortial ensemble
Native Wikibase/Wikidata publishing

Documentation:

Explore rate-limiting connections so we do not suffer from bots and learn from old mistakes

What?

Bots are the worst. They crawl in masses and many ignore robots.txt. They over use our resources and generally end in making us having to restart services just to close open connections (cantaloupe, Solr, etc)

For archipelago (at least the default deployment) we have
https://www.nginx.com/blog/rate-limiting-nginx/

This way we can limit certain endpoints we know may suffer from rate limit issues leading to Denial of Service

@giancarlobi @dmer what we want is to have a list of possible pages (patterns) that can suffer from this

e.g

/user/logic
/search
/cantaloupe/iiif/2
/webform_strawberryfield/
/webform_strawberry/auth_autocomplete/*
/webform_strawberry/nominatim/*
/do/{node}/iiif/{uuid}/full/full/0/{format}

Real challenge is to think in terms of a user here. We want normal operations to happen correctly, we do not want to block access.

Any ideas?

Drupal 9 Readiness check before June 3rd

What? Drupal 8 is death (or will be by June 3rd.. ok it's not like that really), long live Drupal 9 (also, will be a just born one, full of bugs...so)

Yep. It took us some time, like almost 2 years from idea to what we have now and we designed Archipelago to be as compliant to Drupal 9 as we could with such a tiny team and too little time. Still, codebase has grown, we have so much good and complex too code, so we need to pass a general review on each Module we coded but also on each module we suggest should be used. This Issue will be a placeholder for this Friday, when that work will be passed. Idea is to check here what passed and open the issues needed to make the other ones pass. And then have (yes!) a 9.x-1.0 branch for our very own and of course more code to maintain... help!.

@giancarlobi @mitchellkeaney

Schema.org/Book.jsonld is gone. We need a replacement

What is this?

As an example deployment configuration of our basic JSON Key Name provider we were using https://Schema.org/Book.jsonld to filter a HUGE list of properties of the main Schema.org Context size to a manageable size. So yesterday i saw it go away. And it was hard!
Probably it has been gone for longer and since we do cache it i never noticed it. Schema.org is also inconsistent, why would you remove a public jsonld file???
So we need another subset of properties for the next deployment. Maybe even another strategy so filtering a larger context is simpler and more granular.

@giancarlobi. This can affect you too. even when not using docker.
Current copy is here https://github.com/esmero/archipelago-deployment/blob/8.x-1.0-beta2/web/sites/default/files/jsonld/c20db5b6a8d641e98c0570d1ef89017f.jsonld

Upgrade to PHP 7.4.3+ and Drupal 8/9 does not support officially PHP 8

Hey. Memory leaks and garbage collection in PHP.

So. During my super production tests and while allowing a HUGE file (Gbyte+ to be delivered on demand to a viewer) to deal with a Range-IF request I saw a few PHP Out of memory messages.
Guzzle was leaking memory through Symfony (binaryresponse).

NOTICE: PHP message: PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 989388968 bytes) in /var/www/html/vendor/guzzlehttp/psr7/src/Stream.php on line 225

Digging deeper and deeper happened to be a PHP bug (not our first one but also means we are stretching things quite a bit!)
See php/php-src#5014

The actual solution was merged in PHP 7.4.3+. So. Time to move forward if we want to support large large files. Related to this exploratory commit.

esmero/format_strawberryfield@f53adb4

What will happen?

New Docker container for our PHP (esmero/7.4-fpm:1.0.0-RC1) which will also include (building as we talk) some new performance improvements (and done as I finished)

> exporting to image                                                                                                                                                                                         0.0s
 => => exporting layers                                                                                                                                                                                        0.0s
 => => writing image sha256:c88c3e29abfe9e657aa37c51ee123be73ca6d67ceb70512e25c21b0aba4f0a8b                                                                                                                   0.0s
 => => naming to docker.io/esmero/php-7.4-fpm:1.0.0-RC2

modified docker-compose for all 1.0.0-RC1 options using this

But now some testing first!!

8.x-1.0-beta2: Moving to Drupal 8.8.1

This is an internal note. WIP as we move the whole stack to 8.x-1.0-beta2.

Today, Dec 4th 2019 Drupal 8.8.0 was released (yay! or nay?)

Full list of changes: https://www.drupal.org/project/drupal/releases/8.8.0
(read it, i did not.. and here i am)

Since 8.x-1.0-beta2 is coming i wanted to deliver by default that version. But again, so much has changed. Of course i tried the "do it, then solve it" way and i broken one of my local Archipelagos because of a Contributed Module (mentioned in the release notes, but who reads that right? jajaja). The Module is Drupal/pathalias. Drupal 8.8.0 breaks on anything less than 1.6 and it needs to have 1.6 installed first before moving forward because of a Database/entity that needs to be deployed.

Site note: will have to use my Time machine tonight, i had some stuff i liked in that repo!

So until now: i managed to find this set of composer commands to move forward.

New Archipelago Deployment (note, people won't do this, i will provide a composer.lock)

docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer require 'drupal/pathauto:^1.6'"
docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer remove webflo/drupal-core-require-dev"
docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer require drupal/core-dev:^8.8"
docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer require typo3/phar-stream-wrapper:3.1.3 pear/archive_tar:^1.4.8 drupal/core:8.8.0"
docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer update"

Updating existing Archipelago Deployment (from beta01)

docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer require 'drupal/pathauto:^1.6'"
docker exec -ti esmero-php bash -c "drush updatedb"
docker exec -ti esmero-php bash -c "drush cr"
docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer remove webflo/drupal-core-require-dev"
docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer require drupal/core-dev:^8.8"
docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer require typo3/phar-stream-wrapper:3.1.3 pear/archive_tar:^1.4.8 drupal/core:8.8.0"
docker exec -ti esmero-php bash -c "drush cr"
docker exec -ti esmero-php bash -c "drush en path_alias"
docker exec -ti esmero-php bash -c "drush updatedb"
docker exec -ti esmero-php bash -c "drush cr"

Updates: new core module that is needed (given by 8.8.0) drupal/path_alias.
Will document this, but gist is: needs to be either enabled on existing machines via drush, or on new ones our config/sync/core.extension.yml needs to have that one added. If not, BUM!

How lovely dependencies are....

As you also noticed i'm using php -dmemory_limit=-1 /usr/bin/composer instead of composer directly. Newer version of composer is using more than 512Mbytes we assigned to PHP. We will deal with that too.

So this are the first steps: i'm now testing a from scratch deployment. Once i'm ok with that i will move to an update and will provide also a new update script for the new release.

We also need to update config/sync settings and add new configurations. A lot do folks!

Update default "Descriptive Metadata" webform with titles, descriptions, order of operations, etc. that feel better

What? Feelings??

This issue has been discussed offline and in meetings time and time again: the webform fields, their names and their purposes are not always clear.

With the development of Beta 2, let's consciously address some of these concerns and create a new Descriptive Metadata webform to be deployed as the default webform for Archipelago Digital Object (ADO) ingesting.

First things that come to mind to be updated:

Object Title vs. Title ( see issue #15 )
The order of the ingest wizard
The names of the fields ( i.e. "Owner", "date published", "Place or Geographic Location" )
Utilizing the help boxes to support to support users through their ingesting

@DiegoPino @marlo-longley thoughts?

Upgrade Cantaloupe to 4.1.6 so we have info.json for pdf pages and friends

What is needed here?

This question led to a fix from Cantaloupe (so thankful) and now we just need to update our Cantaloupe Docker container to 4.1.6 once that is out.

cantaloupe-project/cantaloupe#359

This will go into Beta3. Thanks

IIIF Server Base URL Settings field fails validity check within Settings Form

Following the installation directions for Installing Archipelago on Ubuntu (actually on a RHEL 7.6) for Beta3, seemed to have encountered the same issue as with Beta2. The difficulty appears when setting up Archipelago on a remote server (i.e. you are not using "http://localhost:8183/iiif/2/" as the IIIF Server base URL, but instead need to specify an actual base name.)

One should be able to specify the "Base URL of your IIIF Media Server public accesible [accessible] from the Outside World." from the IIIF Server Settings Form. This is not the case, since the form validation does not accept a valid URL of the IIIF server. A guess, is that the form validation for that field, might be querying for a IIIF media server at that URL, but it is querying on an INTERNAL docker network and not on the outward looking (non-virtual) network. The example given below, the URL provided does resolve in the web browser, but is not seen as valid in the form.

My current work around, is to modify the configuration files by following the steps located at: https://groups.google.com/d/msg/archipelago-commons/N5D0M6h7DUI/6xLYWfKUAwAJ
But that might be asking to much, for a first time Archipelago user.

Enhance SBF Audio Formatter's dimensions 💪🍓

What's the issue?

Right now SBF Audio Formatter ships with the dimensions 720x240. This is fine and does not affect the functionality of the player, but it looks a bit funky when it loads for folks viewing an object.

Let's get swole...

...by making the height dimension smaller 🤸‍♀️
I found that 720x50 pixels makes a nice and compact-looking player that also allows for maximum user-friendly scrubbing of the file.

Broken links in demo content documentation

I found some broken links in the demo documentation and they are already fixed.

#64

Add needed infrastructure for NLP/ENTITY extraction in our docker-compose

What flavor of ice cream is AI?

For Natural Language Processing and AI analysis of extracted Corpus of text from Files, metadata Description fields or similar textual bodies i started building a few Search API Solr Post Processors that can deal with this, all under the umbrella of the Strawberryfield Runners module for 1.0.0. PS: Ping me for a showcase of the proof of concept i have running in AWS

But to make this an option for people testing-using or just admiring archipelago (yeah.. not sure how many) i want to dockerize these needs through an extra additional docker-compose.yml that can be appended via the -f option when doing and docker-compose up -d

What i need?

This

Our own local dbpedia-spotlight (for Semantic annotation which include immediately links to WIKIDATA too) exposed on port 2231 internally it runs on 80. i think we should allow people a few options
- https://hub.docker.com/r/dbpedia/spotlight-multilingual-databus Which is a multi lingual version but can be quite heavy on storage, or
- https://hub.docker.com/r/dbpedia/spotlight-spanish (no fear there is also an english one..) based on your primary language
https://github.com/web64/nlpserver dockerized (have not found an image so we have to built our own based on Alpine LInux to make it small and fast) and put/tested here https://github.com/esmero/archipelago-docker-images . This runs on port 6400 so would love to have it exposed at :2331

The idea here is basically not everyone needs fancy robots and AI getting names and links from their metadata, but some could need. I'm working on formalizing (and generalizing) the life code i have. I like the fact that it integrates good into the SBF runners idea but also forced me to make some changes there, like how/when things run and i found myself quite happy seeing that actually a lot can go directly to Solr instead of being saved on metadata but other things need to go into permanent storage.

Not the place (i know) but this also open a chance of working again on my idea that AI/NPL generated data needs to be tagged/classified and exposed to the world as such. Means i'm adding provenance, how it was processed, version of the software, etc to an upper structure of the JSON. That way we know and we can keep a separation of human generated classification v/s machine one. Machines are not always quite aware of context so the level of trust we can put on this data will vary a lot, so good we can pass that back to the end user.

@alliomeria because of metadata discussion and @giancarlobi since i think you will like this. Should be able to add code to https://github.com/esmero/strawberry_runners/tree/ISSUE-4 soon with the full refactor and the new plugins.I do have high hopes here!

osx.md Step 1 docker-compose.yml filename problem

~/Development/archipelago-deployment $ docker-compose up -d
ERROR:
Can't find a suitable configuration file in this directory or any
parent. Are you in the right directory?
    Supported filenames: docker-compose.yml, docker-compose.yaml

I renamed the file (mv docker-compose-nginx.yml docker-compose.yml) and everything continues as expected.

Give Nginx a larger buffer when using Archipelago as Development platform

What is needed?

if you are using like myself Archipelago deployment to develop and you are putting a lot of data out, changes are that nginx will complain and give you a few 502 errors. Reason, buffer is just too small for Development (when you are outputting JSON and stuff out).

Solution? (no, it is not stop developing..)

Add

 fastcgi_buffers 16 16k; 
 fastcgi_buffer_size 32k;

To nginxconfigford8/default.conf , restart and all good!

Pull coming!

Update Archipelago Subtheme settings to match new schema

What is needed?

Relating to esmero/archipelago_subtheme#14 in the subtheme, we pushed an updated schema.
This means some of the old settings were of the wrong type to match the new schema.

What to do?

Update /config/sync/archipelago_subtheme.settings to match new schema types.

FYI, FWIW; install broken b/c https://schema.org/docs/jsonldcontext.json give a 503

Hi,

I was trying to follow the directions @ https://github.com/esmero/archipelago-deployment/blob/8.x-1.0-beta1/docs/osx.md#step-1-docker-deployment

I get

In RequestException.php line 113:
                                                                                                                    
  Server error: `GET https://schema.org/docs/jsonldcontext.json` resulted in a `503 Service Unavailable` response:  
                                                                                                                    
  <html><head>                                                                                                      
  <meta http-equiv="content-type" content="text/html;charset=utf-8">                                                
  <title>503 Over Quota</title>                                                                                     
  <style><! (truncated...)

when I run

archipelago-deployment % docker exec -ti -u www-data esmero-php bash -c "cd web;../vendor/bin/drush -y si --verbose config_installer  config_installer_sync_configure_form.sync_directory=/var/www/html/config/sync/ --db-url=mysql://root:esmerodb@esmero-db/drupal8 --account-name=admin --account-pass=archipelago -r=/var/www/html/web --sites-subdir=default --notify=false install_configure_form.enable_update_status_module=NULL install_configure_form.enable_update_status_emails=NULL;drush cr;chown -R www-data:www-data sites;"

Provide SSL/AWS container specific configurations and docs

What is needed?

As good, nice, people start to deploy archipelago on production, or at least, public facing instances, having an base AWS deployment strategy that involves a real domain name and SSL using docker and hopefully AWS S3 (for a cheaper experience than super fast min.io) is crucial.

What will this provide?

A new docker-compose-production yaml file which includes cert bot, etc.
A new ngninx config for production
A new config/yaml files for drupal and for cantaloupe to work from AWS S3.
Documentation (advanced, this is not meant for people testing locally, its meant to run things on actual load)

Switch View Mode was not enabled by default:

Switch View Mode is used to allow to decide how a DIGITAL Objects is going to be displayed, in other words that Field Formatter combo, Also known as Display Mode will apply over the metadata. The default one is always a IIIF Viewer using OpenSeadragon.

This 1660619
restores that functionality by default. If you have that functionality missing, please go to "Extend"
and enable the Display Suite Switch View Mode submodule as seen here:

Alternatively, you can do a git pull and sync the configurations again at http://localhost:8001/admin/config/development/configuration
which gives you also a good chance to learn how Drupal manages configurations, live and stored on file.

Drupal Solr Search API v3.5+ breaks everything and makes me mad/sad

What is wrong?

I'm super confused by this bug, but something in Drupal Search API after version 3.4 is breaking everything in our ensemble. Everything means, every time a Solr Index is called, either on a search result, an autocomplete, a config form or even a simple task like Seeing the http://localhost:8001/admin/config/search/search-api/index/default_solr_index to find out who is tracked or not we get a nasty (white page) error like this:

NOTICE: PHP message: Uncaught PHP Exception LogicException: "The database connection is not serializable. This probably means you are serializing an object that has an indirect reference to the database connection. Adjust your code so that is not necessary. Alternatively, look at DependencySerializationTrait as a temporary solution." at /var/www/html/web/core/lib/Drupal/Core/Database/Connection.php line 1546

Which basically means, something (and i'm looking at you MR. SOLR SEARCH API 👀 ) is injecting not an entityTypeManager container, but probably an entityTypeManager->getStorage as a property in a constructor, or, a specific logger, or something that opens a connection to the DB and keeps it in a property initialized, as i already said, in the constructor.
Since many many things, like plugins, etc are serialized by the Symfony Kernel, we end with this quite huge error. Normally, if someone (not us) needs that, you either use a special trait that removes the container on __sleep() and restores it on __wake() (up), but i don't see the why this is now happening.

That said, it can be something we do/did, and since Solr Search API is now, maybe? trying to cache something or being smarter, it gets stuck in the way. I checked all our source code, like everything, and could not find a single place where we commit such Dependency Injection heresy (well one, in a config form, but that one is never called in a normal request stack and its this weeks code)

I spend 18 hours (with a small pause for Subway and lunch) and i can not find the culprit. 🍭

Removed all our own dependencies, changed the index to only use Pages and Articles, removed our Formatters, our Events and of course all fields. And nothing! Rolled back to 3.4 and all is working, no errors, no messages, nothing, pure Archipelago beauty.

After this, i have to say it could be also any other contributed module interaction or the fact that we are trying to run Drupal core 8.8.1 too.

What is needed?

Fresh, well slept eyes. I would suggest, downloading 3.4 and 3.8 (latest), and doing and full release comparison to understand what changed. Only way we can deal with this since we will want eventually be able to move forward with the newer solarium library too.
Also, probably a from scratch deployment with 3.8 and only basic packages could give us a hint. Both options are quite time consuming (days, centuries) so i will move forward on beta2 with 3.4, but this task can not wait longer than beta3.

I have spoken. This is the way 😃

@marlo-longley @giancarlobi thoughts?

Color css does not load with new Archipelago deployment

When deploying an Archipelago from scratch, the new Archipelago base theme does not render right away.
This appears to be a known issue with the color module: https://www.drupal.org/project/drupal/issues/2995825

Quick fix for Archipelago users is to go to /admin/appearance/settings/archipelago_subtheme, click Save at the very bottom, clear your cache and refresh. Theme should then load and be good to go!

Beta3: Core updates and module dependencies

What is this?

Preparing for Beta3 (finally, gosh!)

General updates (2020 SA updates)

@giancarlobi in case you want to keep in the meantime your archipelagos updated

docker exec -ti esmero-php bash -c 'php -dmemory_limit=-1 /usr/bin/composer require drupal/core:8.8.6'
docker exec -ti esmero-php bash -c 'php -dmemory_limit=-1 /usr/bin/composer require drupal/webform:^5.13;drush updatedb; drush cr'

Also if using Workflow, this allows content moderation to work with batch publish/unpublish

docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer require 'drupal/moderated_content_bulk_publish:1.x-dev';drush en moderated_content_bulk_publish; drush updatedb;"

And for emails i'm using this

docker exec -ti esmero-php bash -c 'php -dmemory_limit=-1 /usr/bin/composer require drupal/smtp:^1.0'

I pretend for beta3 to deliver also a script that does this and enables via drush new modules to allow people to update without having to run a full composer install (if e.g they have custom modules)

Fully automatized Docker compose deployment on AWS ECS

What is this?

This is really my own exploration. I'm thinking (while I think and code and do other stuff) on Fargate Tasks and fully automatic installations of Archipelago. I know what I'm doing and it feels Ok. Won't share details here until I have it running and solved. This does not apply to non Cloud based deployments and I do not think this is a good match for larger installations, but it does work for many smaller, medium use cases where every parts of the Stack (except SOLR which can be shared by many of even be a cloud service, Cantaloupe same idea via S3 Bucket separations) can be totally encapsulated, self sustained Containers that are totally regenerated in case of Software upgrades.

All this to serve low budget communities but also avoid spending my time ssh-ing every server in the world.

More later on this.

Object Title vs. Title in Webform is unclear

What's up?

These 2 fields are labeled so similar, it is confusing some users.

Fix

Change the labels. Suggestion:
Object Title becomes: Object Label (to be used in the metadata)
Title becomes: Object Title (display label)

Additional Issue with this input

Right now when I enter an Object Title on step 1 of the webform wizard, and hit save to go to step 2, the title I entered is erased or at least not retrieved.
Not sure if others can reproduce that, or if there is some background about this I don't know yet.

Bump Cantaloupe top 4.1.5 for Beta2

See esmero/archipelago-docker-images#8

What is new?

New Cantaloupe container using Java 13 (OpenJDK) and Dec 29th 2019 Graphicsmagic + new settings.
We need to research if the new S3 settings i will add make things slower or not, i'm using for the first time chunked requests, could be fast on AWS S3, but slower on Mini.io. Time (or unhappy users) will tell.

Also moved some processors around, now PDF is dealt via PDFBOX, since Graphics magic uses the gs binary directly (or i got the compiling wrong..)

Ubuntu 18.04 documentation updated to accommodate Alpine Linux

What is new

We did a few retests with our Deployment docs for Ubuntu and found some inconsistencies between our new Alpine Containers (in specific the FPM PHP 7.3 one, esmero-php) and Ubuntu's user ids.

After some cleaning up and spinning some machines up, we got it right again. Previous docs where fine for all-around-ubuntu, but we(i) wanted smaller containers right? 😄

Fix is up and docs/ubuntu.md was proof read, steps retraced and corrected. I also added some comments and explanations of why we do what we do there. Please give it a try.

By the way, On AWS on a M5.large EC2 machine, deployment took 5 minutes (including copy pasta and scrolling) and runs fast, blazing fast.

This is the the actual fix 7b811cf

Will keep this issue open until beta2 rolls.

Upgrading from a pretty old Min.io? Your bucket is empty? fear? panic?

Ok, this is new:

Since Jan 2020, if you have a quite old min.io and you are updating your archipelago to beta2 and your docker-compose by pulling latest containers, you could see (its a could, not an always) on your Archipelago (like no images and a lot of stress? You log into localhost:9000 and there is nothing there! You call your parents and nobody answers, panic!). Fear not! Probably your old container version did not upgrade cleanly.

This is what you would see if things are not well (other than missing assets which is what will alert you)

I'm running right now
minio/minio:RELEASE.2020-02-20T22-51-23Z

archipelago-deployment (8.x-1.0-beta2) $ docker logs esmero-minio

 You are running an older version of MinIO released 1 week ago 
 Update: docker pull minio/minio:RELEASE.2020-02-20T22-51-23Z 


Server startup failed with 'Unable to initialize config system: madmin: invalid AEAD algorithm ID'
Server switching to safe mode
Please use 'mc admin config' commands fix this issue
Endpoint:  http://172.20.0.3:9000  http://172.21.0.3:9000  http://127.0.0.1:9000

Solution is simple, from your archipelago-deployment folder do:

rm -rf miniodata/.minio.sys

Which will basically reset your configs to default.

Restart your docker ensemble

docker-compose restart

All good now.

Archipelago 2020 Roadmap: Beta3

Archipelago 2020 Roadmap

See also #5 and #35 for the whole historic recreation. @alliomeria this is for you so we can catch up with old business

This is our working enumeration of concrete tasks for this year, per Component and Service, for public evaluation (critics and comments welcome).

All tasks listed here are leading our full feature release somewhere this first half of the year. We went more ambitious (we planned for end of 2019) so code has been moving along and we feel we are quite there! Checked tasks are ready, unchecked are in progress or planned. Priority is not given by this order.

Please feel free to comment, request more info or ask for clarification. Feature requests are also highly appreciated and taken in account (always, please!).

Strawberryfield

Field Property exposure to Drupal strategies
- JSON KEY Provider (flattener)
- JSON Flatten Keys
- JSONPATH/JMESPATH
- Entity Reference Casting Provider (Using UUID loading and configurable entity type)
- JSON stored Service Endpoints with extended logic (e.g HOCR)
- Multi Map/ join: many properties to single. e.g All keys - Authorities- referring to creators, contributors etc unified as Agents keys. This leads to Fractal Ontologies and our Buckets approach.

JSON representation and enrichment

Webforms integration

Media Displays

Field Formatters

API Ingest, Migration and backup

Service Architecture (Strawberry Runners)

SEO and API

Allow Media displays output to be embeded in HTML head for SEO
Test/Develop nested DATA VIEWS integration for OAI-ORE and OAI-PMH
Create (TWIG, metadata displays) and expose as endpoints full set of IIIF API JSON outputs.
Add helper methods and twig extensions to allow Metadata displays to access pre existing views (like object listings for a collection) to help build those lists.

ACL / Permissions

Integrate custom ACL with JSON Paths into per NODE ACL. Allowing this way to apply permissions to individual metadata elements/paths.
Same but needs better UI for referenced Services and Media
Allow Metadata (rule) to trigger ACL permissions. e.g if embargo_date == bla bla = remove public access
Allow for ACL inheritance (from parent, recursive) without hard copies.

Deployment and DevOPS

Batch Operations

Future roadmap

[x ] Solr Cloud/ Consortial ensemble
Native Wikibase/Wikidata publishing

Documentation:

Archipelago Roadmap 2020: Beta2

Archipelago 2020 Roadmap

Strawberryfield

Field Property exposure to Drupal strategies
- JSON KEY Provider (flattener)
- JSON Flatten Keys
- JSONPATH/JMESPATH
- Entity Reference Casting Provider (Using UUID loading and configurable entity type)
- JSON stored Service Endpoints with extended logic (e.g HOCR)
- Multi Map/ join: many properties to single. e.g All keys - Authorities- referring to creators, contributors etc unified as Agents keys. This leads to Fractal Ontologies and our Buckets approach.

JSON representation and enrichment

Webforms integration

Media Displays

Field Formatters

API Ingest, Migration and backup

Service Architecture (Strawberry Runners)

SEO and API

Allow Media displays output to be embeded in HTML head for SEO
Test/Develop nested DATA VIEWS integration for OAI-ORE and OAI-PMH
Create (TWIG, metadata displays) and expose as endpoints full set of IIIF API JSON outputs.
Add helper methods and twig extensions to allow Metadata displays to access pre existing views (like object listings for a collection) to help build those lists.

ACL / Permissions

Integrate custom ACL with JSON Paths into per NODE ACL. Allowing this way to apply permissions to individual metadata elements/paths.
Same but needs better UI for referenced Services and Media
Allow Metadata (rule) to trigger ACL permissions. e.g if embargo_date == bla bla = remove public access
Allow for ACL inheritance (from parent, recursive) without hard copies.

Deployment and DevOPS

Batch Operations

Future roadmap

D9 readyness 😄 -- we need second pass
[x ] Solr Cloud/ Consortial ensemble
Native Wikibase/Wikidata publishing

Documentation:

Fine tune Cantaloupe for Huge Images

What is this?

Q: who has huge images? A: basically everyone!

Our cantaloupe custom Docker Container works well but there are some issues we may need to tune:
1.- When serving Files from S3, in specific large > 100 Mbytes TIFFS, the processor Streaming Strategy tends to fail randomly. It safer the Cache Strategy (means download and keep around) but also slower and requires more physical space on the server to keep the caches around. Also for these Java2D and Jair totally fail (like badly) and give me a strange Memory error. Using GraphicsMagic is safer but also slower and works. We can experiment and maybe do a test suit to know our limits/fine tuning needs?
2.- I created a new Docker Container Version that allows XMX and XMS to be passed as arguments and also include TurboJPEG and a newer version of GraphicsMagic.

Serials Digital Objects and Digital Object Collections

What is needed?

Additional options and configurations for Serials (newspapers, magazines, etc.) Digital Objects and Digital Object Collections

Webform for Serials Digital Object Collections
Default Webform adjustments for information related to Serials
- such as Volume/Issue information
- any other common info to include? frequency/publication patterns?
Sets of additional options
- RDF Types
  - For Digital Objects: PublicationIssue, NewspaperIssue
  - For Digital Object Collections: Newspapers, Periodicals
- Another others?
- Display formatting?
Config/setup that binds all together

Large File uploads via Drupal and NGINX, multipart and improvements for 1.0.0-RC1

Everybody loves uploading 1Gbyte+ via a webform

Ok, not everybody but I just found myself doing it and it was not fun.

Changes we need to make this happen (this is a mix of documentation but also testing new things)

https://github.com/esmero/archipelago-deployment/blob/1.0.0-RC1/nginxconfigford8/default.conf#L82
and https://github.com/esmero/archipelago-deployment/blob/1.0.0-RC1/nginxconfigford8/default.conf#L4 needs to be updated to a large number. I set this to 2048M for testing. This allows us to avoid changing the PHP.ini directives
esmero-web needs to mount a separate cache folder here https://github.com/esmero/archipelago-deployment/blob/1.0.0-RC1/docker-compose-nginx.yml#L25 adding

  -  ${PWD}/persistent/ngixncache:/var/cache/nginx:cached

Webforms need their elements updated. This may be needed for Webarchives, Datasets and Video

We may want multipart upload too. With resuming capabilities: there are a number of JS libraries that can handle this but I also like the nginx native approach, here:

https://www.nginx.com/resources/wiki/modules/upload_progress/ which exposes a special JSON endpoint that a webform element could, via JQUERY consult in certain intervals and display realtime info on uploads
http://www.grid.net.ru/nginx/upload.en.html which allows multi part/resumable uploads (chunks)

@giancarlobi you use Apache I know, but this is needed so maybe you have backend independent suggestions we may want to add

Also:

All this implies changes to our docker deployment. Right now we are using the generic Nginx docker container but also, for SSL I moved to staticfloat/nginx-certbot which includes not only base system but also automatic renewal and request of SSL certs (free ones, fully valid) and non interactive which means just a few more seconds startup time but no copying .well-known hashes around and sending prayers. I feel if we are going this route, we may need to extend our own Docker Container for nginx.

admin user view

It looks like logging in as admin user makes it impossible to see the site as a regular user, you are only exposed to admin tools. Maybe Drupal has some 'switch user' functionality that can help with this. If yes, this would be good to add to documentation

"demo" and "jsonapi" users can not be created at the same time in Terminal

In the OSX deployment documentation, Step 4 provides two scripts for creating users "demo" and "jsonapi":

docker exec -ti esmero-php bash -c 'drush ucrt demo --password="demo"; drush urol metadata_pro "demo"'
docker exec -ti esmero-php bash -c 'drush ucrt jsonapi --password="jsonapi"; drush urol metadata_pro "jsonapi"'

I made the mistake of running both lines at the same time, which Terminal cannot complete and was only able to create "demo". I found that the user must copy each script separately and run individually in order for both the "demo" and "jsonapi" users to be created successfully.

TIFF default processor (Java2D) forCantaloupe is not working on 4.1.5 version

What?

we used to use Java2D for tiffs, now after the update to the latest container and in Beta2, people don't see the TIFFs (sad).

Fix is to change to Graphicsmagic and then figure out why. I have suspects, like a bad Java2D processor but also maybe the streaming strategy for TIFFS not working on 2D.

Anyway. bad, bad.

Check if Drupal's Core AJAX state display bug needs to be patched automatically by us

See https://www.drupal.org/project/drupal/issues/1091852 and
See https://www.drupal.org/files/issues/2019-11-19/1091852-104.patch

Basically there is a Core bug in Drupal 8.x and 9.x that affects #state driven forms when loaded/updated via AJAX. I remember (slightly so much code i have added in the last year) having to work double shifts to have our Ajax driven forms work top notch, but as we advance to more Complex webforms (the EAD 2002 one i'm doing is quite heavy on conditional logic) it could be cool to test if the patch here applies clean on 8.9.2 and if we really need it.

Docker on production and Beta3 upgrade paths

We have seen what "production" can mean this week...more (huge) and strange than many have ever seen!

So we need to start not only thinking of making our docker-compose AWS smarter but actually our Docker Daemon 🙈 tuning smarter
https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file

E.g I like to run my /root devices slim and fast. Means we need default volumen storage go into second drives.
And so on and so on.

NOTE: updated!

Let's add

Multiple Cantaloupe's and backend routing
MYSQL fine tuning and DB replication: Moving to MYSQL 8 (including DB migration from 5.7)
PostgreSQL option for Super fast data
NGINX caching and PHP 8
Solr: Testing 8.5.x (performance concerns, never liked very last Solr versions but we need to try)
Include NLP64 container we added, used by strawberry_runners
Azure storage option
WASABI storage option

Drupal Search API 1.18 breaks indexing when multiple Data Sources are inside the same index

Drupal bug

This was bothering Giancarlo for a few days. We are running 1.17 but the upgrade in his D9 instance killed any Field based Updates because of bad code in Search API.

More here https://www.drupal.org/project/search_api/issues/3178941#comment-13899506
I found an additional bug so had to make our own patch (c'mon Drupal..)

Make Archipelago ready for Drupal 9.1

We have to check if all Archipelago modules RC1 version are ok running onto Drupal 9.1

Identify not running on draft ADOs

See esmero/strawberryfield#120

Riding the Thumbnail bicycle, no hands while coding Twig, into the Thumbnails sunset (Font-Awesome music fading away)

What is this?

A bad title for a good show off of our Twig/IIIF capabilities.

Actually this is something that came through a shared use case and my constant avoidance of harcoding things we do not want to hardcode because everyone will want a different need. Thumbnails. Yes, old fashioned ones.

We provide a simple IIIF Image formatter (and configured by default for thumbnails) that takes the first image in any ADO and renders a thumbnail using Cantaloupe. Great!. But what about a PDF and no image? Or Objects with just ZIP files where the rdf type is more important than the MEDIA itself. And Movies? Well. We can, i mean this is ARCHIPELAGO with folks coding on a friday at 21:24, so it is possible (actually since 2018) 🥰

But what if the solution instead of more specific formatters is just the existing Metadata formatter using a Twig template one?
Well, here it is! Including gloriously inline docs @giancarlobi @alliomeria @tomadams (the one that explicitly asked for the use case) .

How to test?

Please copy and paste into a new /metadatadisplay/add, set as HTML, Save. I named it "Many thumbnails" but you are all smarter and will give it a better name.
Modify your Search results View Mode (first, then you can also do that for the collection listing too) at
/admin/structure/types/manage/digital_object/display/strawberryfield_processed_metadata
and set this formatter instead of the IIIF Image one

Customize?

Things you can change (everything really)

The width of thumbnails
Which page of the PDF (if any) it takes
Which timecode of a video is used
Which Font based icons (if nothing matches or conditions fails or nothing is there) can be used
Add more conditions (like only trust a few pronom IDs, like only pdf 1.3 and 1.4 but never 2.0)
Size of the icons?
Alt? Title? Rotation? You want all in B&W? =)
Want many? You can

Basically control, the love of the icon and display is back to you dear user.

What can go wrong. Basically same that can go wrong with unknown media, missing media, etc. Good thing is no recoding/asking for a new release is needed. Just add more test/checks in the conditionals (flv:exif has a lot you can use to decide if processing will be even possible) and learn.

Don't forget to clear your caches after the change(s)

The Template

{# This Template generates Thumbnails for various Media representations

Structures passed from Archipelago are
- node
  The full Node entitya
- data
  The Full JSON Structure converted in an Array
- iiif_server
  The publicly available IIIF Server (Global Setting)

Since JSON was decoded into the 'data' values are valid PHP but could not
be valid JSON when outputting. So We use data.description|json_encode|raw
Without Double quotes.
#}
{# <---- SETUP ---> #}
{# -- The Desired width -- #}
{% set width = 240 %}
{# -- The IIIF Server URLS -- #}
{% set Webserverurl =  url('<front>')|render|replace({':8001/':''}) ~ ':8183/iiif/2/' %}
{% set IIIFserverurl = iiif_server ? iiif_server ~ '/': Webserverurl %}
{# -- Node Info -- #}
{% if node.id %}
  {% set nodeurl = url('entity.node.canonical', {'node': node.id}, {'absolute': true}) %}
{% else %}
  {% set nodeurl = '<current>' ~ '/iiif/manifest' %}
{% endif%}
{# -- Language -- #}
{% set len = language.getId() ? language.getId() : "en" %}
{# --THIS WILL BE THE IMAGE IIIF URL. CAN HAVE A DEFAULT VALUE IN CASE OF FAILURE THERE IS ALWAYS A GENERIC THUMB-- #}
{# -- NOTE: for loops are scoped so if we onlt define source inside one it will not be accesible from the outside- #}
{% set source = '' %}
{# <---- END SETUP --> #}
{#  We will test for 3 main options
1.- Images
2.- PDF
3.- Movies
#}
{% apply spaceless %}    
{% spaceless %}
{% if attribute(data, 'as:image')|length > 0 %}  
  {% set document  = data['as:image']|first %} 
  {% set cantaloupeid  = IIIFserverurl ~ document['url']|replace({'s3://':''})|replace({'private://':''})|url_encode %}
  {% set source = cantaloupeid ~ "/full/" ~ width  ~ ",/0/default.jpg" %}
{% elseif attribute(data, 'as:document')|length > 0 %}  
  {% set break = false %}
  {% for document in data['as:document'] if not break %}
  {# this is just an example, we  allow every PDF here but we can also filter by pronom id "info:pronom/fmt/17" or info:pronom/fmt/18 #}
    {% if document['flv:exif'].MIMEType == "application/pdf" and document['flv:exif'].PageCount > 0 %}   
      {% set cantaloupeid  = IIIFserverurl ~ document['url']|replace({'s3://':''})|replace({'private://':''})|url_encode %} 
      {% set source = cantaloupeid ~ "/full/" ~ width  ~ ",/0/default.jpg?page=1" %}
      {% set break = true %}
    {% endif%}
  {% endfor %}
{% elseif attribute(data, 'as:video')|length > 0 %}  
  {% set break = false %}
  {% for document in data['as:video'] if not break %}
  {# this is just an example, we process here only MP4 quicktimes #}
    {% if document['flv:pronom'].mimetype == "video/quicktime" %}
      {% set cantaloupeid  = IIIFserverurl ~ document['url']|replace({'s3://':''})|replace({'private://':''})|url_encode %} 
      {% set source = cantaloupeid ~ "/full/" ~ width  ~ ",/0/default.jpg?time=00:00:01" %}
      {% set break = true %}
    {% endif %}
  {% endfor %}  
{% endif%}
{# <----START OUTPUT --> #}
<a href="{{ nodeurl }}" alt="{{ node.label.value }}" hreflang="{{ len }}">
{% if source|length > 0 %}
<img class="field-iiif image-iiif img-fluid rounded" id="thumb_iiif-field_descriptive_metadata-{{ node.uuid.value }}-0-image1" src="{{ source }}" width="240" alt="Thumbnail for {{ node.label.value }}" typeof="foaf:Image">
{% else %}
{# -- IF we have no source we can e.g use font awesome -- #}
{# 
Options we took from /admin/structure/webform/config/options/manage/schema_org_creative_works/source
Article: Article 
Book: Book
Collection: Collection
Dataset: Dataset
DigitalDocument: 'Digital Document'
Map: Map
Movie: Movie
MusicRecording: 'Music Recording'
Painting: Painting
Photograph: Photograph
Sculpture: Sculpture
VisualArtwork: VisualArtwork
Thesis: Thesis
Podcast: Podcast
PodcastEpisode: 'Podcast Episode'
Panorama: 'Panorama (http://ns.google.com/photos/1.0/panorama)'
PanoramaTour: 'Panorama Tour (http://ns.google.com/photos/1.0/panorama)'
3DModel: '3D Model'
WebPage: 'Web Page'
AudioObject: 'Audio Object'
VideoObject: 'Video Object'
Conversation: 'Conversation (Oral History)'
ShortStory: 'Short Story'
#}
  {% if data.type in ['DigitalDocument', 'Article','Thesis'] %}
<i class="fas fa-file-alt fa-9x"></i>
  {% elseif data.type in ['Photograph','Painting','VisualArtwork','VideoObject'] %}
<i class="fas fa-photo-video fa-9x"></i>
 {% elseif data.type in ['Movie'] %}
 <i class="fas fa-film fa-9x"></i>
  {% elseif data.type in ['MusicRecording', 'AudioObject'] %}
 <i class="fas fa-headphones-alt fa-9x"></i>
  {% elseif data.type in ['Podcast','PodcastEpisode'] %}
 <i class="fas fa-podcast"></i>
  {% elseif data.type in ['ShortStory','Conversation',] %}
<i class="fas fa-comments fa-9x"></i>
  {% elseif data.type in ['Sculpture','3DModel'] %}
<i class="fas fa-cubes fa-9x"></i>
  {% elseif data.type in ['Book'] %}
<i class="fas fa-book fa-9x"></i>
  {% elseif data.type in ['WebPage'] %}
<i class="fas fa-sitemap fa-9x"></i>
 {% elseif data.type in ['PanoramaTour'] %}
<i class="fas fa-vr-cardboard fa-9x"></i>
 {% elseif data.type in ['Dataset'] %}
<i class="fas fa-table fa-9x"></i>
 {% elseif data.type in ['Map'] %}
<i class="fas fa-map-marked-alt fa-9x"></i>
  {% else %} 
<i class="fas file-contract fa-9x"></i>
  {% endif %}
{# -- ENDIF fontawesome -- #}
{% endif %}
</a>
{% endspaceless %}
{% endapply %}

Preview here with a normal image source, a PDF and a VIDEO

Please let me know if this works or you feel it needs more logic/options. If good i will add as alternative to this deployment. Also feel free to ask about the Twig statements and things i did here if something is unclear. I tried to add as many inline comments as possible.

Good night

Create docker-compose.dev.yml file

Why.

Well, I am referencing docker-compose.dev.yml here. It helps to launch the development php image and is used as an override file in docker-compose up during development or debugging sessions. So I need to add it to the repo for testing and approval.

I am open to a different approach but for now, wish for you @DiegoPino to have access to the file.

Investigate drupal/config_split for Dev environments

What is this?

We ship our deployment with a full set of configurations to get started. But sometimes, specially during development we want/you want to have some configs to be sync/able and others to be local to your own reality. Config Managment (import/export) does not allow that. So there is this
https://www.drupal.org/project/config_split
Handy module that says it can do that

There are of course old Features module, but i kinda like the idea we are not mixing things up here and are mostly relaying on core for config managment. Just too risky.

@giancarlobi wonder if you could look at this module some of these days and give me your opinion.

Conditional logic for multiple values.

It would be great to have a system in Webforms for conditional logic like "Value is in" so that I can check and see if, for instance, the media type is a "3D Model", "Photograph", or "Video" instead of having 3 separate "Value is" for each media type.

Add a simple Archipelago Packages update script for running instances

What?

While testing, fixing and debugging our beta releases (now) and future full releases, it would be nice to have a super simple bash script that runs composer update on anything strawberryfield (Archipelago packages) so people have a single point of interaction with a deployed Archipelago repository.

How?

Super simple update.sh bash script that runs composer update strawberryfield/* and our theme of course too and, if Drupal is already installed, runs a drush cr. Its nothing more than a facility.

Then add this to documentation.

Docker for Mac and new features + AFPS case insensitive File system. DOCUMENT!!

GOSH!

Yes. I spend a few hours of my free day dealing with random Docker issues. Short story is i had to change my decaying Mac for a new one and was quite happy with new speed and was coding like hell! (this happened a few weeks ago)

But I had a few strange issues with Solr indexes going to hell after a docker-compose down (assumed it was a badly made backup..) but now deploying a fresh version of 1.0.0-RC1 using Solr 8.7 and MYSQL 8.0.22 nothing worked. Lots of DB errors, constant restart and lost hope and countless hours. (4)

What i learned:

Docker Desktop for Mac enables now (I have 2.5.0.1 with Docker Engine 19.03.13) by default an experimental file sharing option (same as for windows) which is not only buggy but quite buggy. Issues with async writing, lots of GitHub issues with people being annoying all that added to the fact that I was in the presence of a CASE INSENSITIVE filesystem.

Solutions:

Disable the god forsaken gRPC FUSE file sharing option when using bind/cached mounts. (works)
Maybe for development we can use an actual docker volumen for the DB which is native Linux and case sensitive
Provide a specific OS X docker-compose.yml file
Always check your OS X drives when you buy a new one. Make it case sensitive. Please, trust me!

All is working now (I mean testing but all starting and running). But we need to document this use case/reality/issue and also provide an alternative docker-compose file for OS X users if we do not change the /persistent/db storage via a bind mount.

GOSH!! (which is not the same as 'gosh')

NOTE:

this is the type of errors I was getting:

esmero-db | 2020-11-11T20:35:35.238363Z 1 [Warning] [MY-012637] [InnoDB] 16384 bytes should have been read. Only 0 bytes read. Retrying for the remaining bytes.
esmero-db | 2020-11-11T20:35:35.238978Z 1 [Warning] [MY-012638] [InnoDB] Retry attempts for reading partial data failed.
esmero-db | 2020-11-11T20:35:35.241630Z 1 [ERROR] [MY-012642] [InnoDB] Tried to read 16384 bytes at offset 49152, but was only able to read 0
esmero-db | 2020-11-11T20:35:35.242429Z 1 [ERROR] [MY-012592] [InnoDB] Operating system error number 95 in a file operation.
esmero-db | 2020-11-11T20:35:35.242921Z 1 [ERROR] [MY-012596] [InnoDB] Error number 95 means 'Operation not supported'
esmero-db | 2020-11-11T20:35:35.244192Z 1 [ERROR] [MY-012646] [InnoDB] File ./undo_001: 'read' returned OS error 195. Cannot continue operation
esmero-db | 2020-11-11T20:35:35.246773Z 1 [ERROR] [MY-012981] [InnoDB] Cannot continue operation.
esmero-db | 2020-11-11 20:35:36+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.22-1debian10 started.

esmero / archipelago-deployment Goto Github PK

archipelago-deployment's People

Contributors

Stargazers

Watchers

Forkers

archipelago-deployment's Issues

What?

What?

Archipelago 2020 Roadmap

Strawberryfield

JSON representation and enrichment

Webforms integration

Field Formatters

API Ingest, Migration and backup

Service Architecture (Strawberry Runners)

SEO and API

ACL / Permissions

Deployment and DevOPS

Batch Operations

Future roadmap

Documentation:

What is new?

Archipelago 2019 Roadmap

Strawberryfield

JSON representation and enrichment

Webforms integration

Field Formatters

API Ingest, Migration and backup

Service Architecture (Strawberry Runners)

SEO and API

ACL / Permissions

Deployment and DevOPS

Batch Operations

Future roadmap

Documentation:

What?

What? Drupal 8 is death (or will be by June 3rd.. ok it's not like that really), long live Drupal 9 (also, will be a just born one, full of bugs...so)

What is this?

Hey. Memory leaks and garbage collection in PHP.

This is an internal note. WIP as we move the whole stack to 8.x-1.0-beta2.

New Archipelago Deployment (note, people won't do this, i will provide a composer.lock)

Updating existing Archipelago Deployment (from beta01)

What? Feelings??

What is needed here?

What's the issue?

Let's get swole...

What flavor of ice cream is AI?

What i need?

What is needed?

Solution? (no, it is not stop developing..)

What is needed?

What to do?

What is needed?

What will this provide?

What is wrong?

What is needed?

What is this?

General updates (2020 SA updates)

Also if using Workflow, this allows content moderation to work with batch publish/unpublish

What is this?

What's up?

Fix

Additional Issue with this input

What is new?

What is new

Archipelago 2020 Roadmap

Strawberryfield

JSON representation and enrichment

Webforms integration

Field Formatters

API Ingest, Migration and backup

Service Architecture (Strawberry Runners)

SEO and API

ACL / Permissions

Deployment and DevOPS

Batch Operations

Future roadmap

Documentation:

Archipelago 2020 Roadmap