They need to be deleted.

Related: artefactual-labs/ansible-archivematica-src#145.

The common settings doesn't seem available in current qa/1.x

Currently, they are added to the dockerfile. We should use
mk-install-deps -i debian/control

The error is:

Downloading/unpacking cryptography>=1.9 (from pyopenssl->-r ./../dashboard/src/requirements/base.txt (line 23))
  Running setup.py (path:/src/src/archivematica/src/MCPClient/debian/archivematica-mcp-client/usr/share/python/archivematica-mcp-client/build/cryptography/setup.py) egg_info for package cryptography
    error in cryptography setup command: Invalid environment marker: python_version < '3'
    Complete output from command python setup.py egg_info:
    error in cryptography setup command: Invalid environment marker: python_version < '3'

Two AVC denial errors are thrown on CentOS 7 with SELinux enabled when attempting to install the rpm archivematica-storage-service.x86_64 0:0.8.0-0.beta.1, with the consequence that the user 'archivematica' is not created:

SELinux is preventing /usr/sbin/useradd from setattr access on the directory archivematica.
SELinux is preventing /usr/sbin/useradd from create access on the file .bash_logout.

Error 1:
***** Plugin catchall_labels (83.8 confidence) suggests *******************

If you want to allow useradd to have setattr access on the archivematica directory
Then you need to change the label on archivematica
Do

semanage fcontext -a -t FILE_TYPE 'archivematica'

where FILE_TYPE is one of the following: alsa_home_t, antivirus_home_t, audio_home_t, auth_home_t, cache_home_t, chrome_sandbox_home_t, config_home_t, cvs_home_t, data_home_t, dbus_home_t, fetchmail_home_t, gconf_home_t, git_user_content_t, gkeyringd_gnome_home_t, gnome_home_t, gpg_secret_t, gstreamer_home_t, home_bin_t, home_cert_t, httpd_user_content_t, httpd_user_htaccess_t, httpd_user_ra_content_t, httpd_user_rw_content_t, httpd_user_script_exec_t, icc_data_home_t, iceauth_home_t, irc_home_t, irc_tmp_t, irssi_home_t, kismet_home_t, krb5_home_t, krb5kdc_var_lib_t, local_login_home_t, mail_home_rw_t, mail_home_t, mail_spool_t, mandb_home_t, mozilla_home_t, mpd_home_t, mpd_user_data_t, mplayer_home_t, mysqld_home_t, openshift_cgroup_read_tmp_t, openshift_cron_tmp_t, openshift_rw_file_t, openshift_tmp_t, openshift_var_lib_t, polipo_cache_home_t, polipo_config_home_t, procmail_home_t, pulseaudio_home_t, rlogind_home_t, rssh_ro_t, rssh_rw_t, sandbox_file_t, screen_home_t, selinux_config_t, selinux_login_config_t, semanage_store_t, semanage_tmp_t, smsd_var_lib_t, spamc_home_t, speech-dispatcher_home_t, ssh_home_t, stapserver_var_lib_t, svirt_home_t, systemd_home_t, telepathy_cache_home_t, telepathy_data_home_t, telepathy_gabble_cache_home_t, telepathy_logger_cache_home_t, telepathy_logger_data_home_t, telepathy_mission_control_cache_home_t, telepathy_mission_control_data_home_t, telepathy_mission_control_home_t, telepathy_sunshine_home_t, texlive_home_t, thumb_home_t, tvtime_home_t, uml_ro_t, uml_rw_t, user_fonts_cache_t, user_fonts_config_t, user_fonts_t, user_home_dir_t, user_home_t, user_tmp_t, useradd_var_run_t, virt_content_t, virt_home_t, vmware_conf_t, vmware_file_t, wine_home_t, wireshark_home_t, xauth_home_t, xdm_home_t.
Then execute:
restorecon -v 'archivematica'

***** Plugin catchall (17.1 confidence) suggests **************************

If you believe that useradd should be allowed setattr access on the archivematica directory by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:

grep useradd /var/log/audit/audit.log | audit2allow -M mypol

semodule -i mypol.pp

Additional Information:
Source Context unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023
Target Context system_u:object_r:var_lib_t:s0
Target Objects archivematica [ dir ]
Source useradd
Source Path /usr/sbin/useradd
Port
Host xxxx
Source RPM Packages shadow-utils-4.1.5.1-18.el7.x86_64
Target RPM Packages
Policy RPM selinux-policy-3.13.1-60.el7_2.7.noarch
Selinux Enabled True
Policy Type targeted
Enforcing Mode Enforcing
Host Name xxxx
Platform Linux xxxx 3.10.0-327.28.3.el7.x86_64 #1 SMP Thu
Aug 18 19:05:49 UTC 2016 x86_64 x86_64
Alert Count 1
First Seen 2016-08-25 08:33:26 CDT
Last Seen 2016-08-25 08:33:26 CDT
Local ID 0eb7d8aa-6b19-4401-926a-1d28646c18f9

Raw Audit Messages
type=AVC msg=audit(1472132006.845:1735): avc: denied { setattr } for pid=7142 comm="useradd" name="archivematica" dev="dm-0" ino=73548767 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir

type=SYSCALL msg=audit(1472132006.845:1735): arch=x86_64 syscall=chmod success=no exit=EACCES a0=7fff45a0d773 a1=1c0 a2=0 a3=3f items=0 ppid=7140 pid=7142 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=181 comm=useradd exe=/usr/sbin/useradd subj=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 key=(null)

Hash: useradd,useradd_t,var_lib_t,dir,setattr

Error 2:
***** Plugin catchall_labels (83.8 confidence) suggests *******************

If you want to allow useradd to have create access on the .bash_logout file
Then you need to change the label on .bash_logout
Do

semanage fcontext -a -t FILE_TYPE '.bash_logout'

where FILE_TYPE is one of the following: alsa_home_t, antivirus_home_t, audio_home_t, auth_home_t, cache_home_t, chrome_sandbox_home_t, config_home_t, cvs_home_t, data_home_t, dbus_home_t, default_context_t, etc_runtime_t, etc_t, fetchmail_home_t, file_context_t, gconf_home_t, git_user_content_t, gkeyringd_gnome_home_t, gnome_home_t, gpg_secret_t, gstreamer_home_t, home_bin_t, home_cert_t, httpd_user_content_t, httpd_user_htaccess_t, httpd_user_ra_content_t, httpd_user_rw_content_t, httpd_user_script_exec_t, icc_data_home_t, iceauth_home_t, irc_home_t, irc_tmp_t, irssi_home_t, kismet_home_t, krb5_home_t, krb5kdc_var_lib_t, local_login_home_t, mail_home_rw_t, mail_home_t, mail_spool_t, mandb_home_t, mozilla_home_t, mpd_home_t, mpd_user_data_t, mplayer_home_t, mysqld_home_t, openshift_cgroup_read_tmp_t, openshift_cron_tmp_t, openshift_rw_file_t, openshift_tmp_t, openshift_var_lib_t, passwd_file_t, polipo_cache_home_t, polipo_config_home_t, procmail_home_t, pulseaudio_home_t, rlogind_home_t, rssh_ro_t, rssh_rw_t, sandbox_file_t, screen_home_t, selinux_config_t, selinux_login_config_t, semanage_store_t, semanage_tmp_t, shadow_t, smsd_var_lib_t, spamc_home_t, speech-dispatcher_home_t, ssh_home_t, stapserver_var_lib_t, svirt_home_t, systemd_home_t, telepathy_cache_home_t, telepathy_data_home_t, telepathy_gabble_cache_home_t, telepathy_logger_cache_home_t, telepathy_logger_data_home_t, telepathy_mission_control_cache_home_t, telepathy_mission_control_data_home_t, telepathy_mission_control_home_t, telepathy_sunshine_home_t, texlive_home_t, thumb_home_t, tvtime_home_t, uml_ro_t, uml_rw_t, user_fonts_cache_t, user_fonts_config_t, user_fonts_t, user_home_t, user_tmp_t, useradd_var_run_t, virt_content_t, virt_home_t, vmware_conf_t, vmware_file_t, wine_home_t, wireshark_home_t, xauth_home_t, xdm_home_t.
Then execute:
restorecon -v '.bash_logout'

***** Plugin catchall (17.1 confidence) suggests **************************

If you believe that useradd should be allowed create access on the .bash_logout file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:

grep useradd /var/log/audit/audit.log | audit2allow -M mypol

semodule -i mypol.pp

Additional Information:
Source Context unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023
Target Context system_u:object_r:var_lib_t:s0
Target Objects .bash_logout [ file ]
Source useradd
Source Path /usr/sbin/useradd
Port
Host xxxx
Source RPM Packages shadow-utils-4.1.5.1-18.el7.x86_64
Target RPM Packages
Policy RPM selinux-policy-3.13.1-60.el7_2.7.noarch
Selinux Enabled True
Policy Type targeted
Enforcing Mode Enforcing
Host Name xxxx
Platform Linux xxxx 3.10.0-327.28.3.el7.x86_64 #1 SMP Thu
Aug 18 19:05:49 UTC 2016 x86_64 x86_64
Alert Count 1
First Seen 2016-08-25 08:33:26 CDT
Last Seen 2016-08-25 08:33:26 CDT
Local ID ba067c58-b010-4624-b385-78566310cf4c

Raw Audit Messages
type=AVC msg=audit(1472132006.846:1737): avc: denied { create } for pid=7142 comm="useradd" name=".bash_logout" scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_lib_t:s0 tclass=file

type=SYSCALL msg=audit(1472132006.846:1737): arch=x86_64 syscall=open success=no exit=EACCES a0=7fbb8aef64b0 a1=241 a2=1a4 a3=6165726373662f72 items=0 ppid=7140 pid=7142 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=181 comm=useradd exe=/usr/sbin/useradd subj=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 key=(null)

Hash: useradd,useradd_t,var_lib_t,file,create

We're adding in artefactual/archivematica#782 and artefactual/archivematica-storage-service#256 the possibility to populate logging configuration via a configuration file:

• /etc/archivematica/dashboard.logging.json
• /etc/archivematica/clientConfig.logging.json
• /etc/archivematica/serverConfig.logging.json
• /etc/archivematica/storageService.logging.json

The same PR includes a sample for each of the files that mimics the old AM17 behaviour. DEB/RPM package should populate these files because the desire is to keep logging as it was before in this type of installation.

It depends on artefactual/archivematica#782 that hasn't been merged at the time this issue was reported.

Rng-tools is a daemon that gathers entropy, and makes gpg key generation faster. Is listed as a build-depend here:

https://github.com/artefactual-labs/am-packbuild/blob/master/debs/trusty/archivematica-storage-service/debian-storage-service/control#L14

But it should be in the Depends: list instead

#80 was fixed but we had a related convo in that issue that I wanted to capture somewhere. This description should be updated at some point with a better description of the problem - but I don't have time for that right now!

It seems it's a legacy db change, that isn't need anymore:

https://github.com/artefactual/archivematica/blob/qa/1.x/src/dashboard/debian/postinst#L18

While running

pip download -d lib --no-binary :all: -r requirements.txt

I got this:

  Could not find a version that satisfies the requirement metsrw==0.2.0 (from -r requirements/base.txt (line 23)) (from versions: 0.1.0, 0.1.1)
No matching distribution found for metsrw==0.2.0 (from -r requirements/base.txt (line 23))

It's not working with yum-builddep for some reason. Look into that at some point.

Before 1.7 , transfer-browser.js and appraisal-tab.js were part of the am repo, now they are built as part of the deployment process.

For xenial and centos packages, we should tell systemd to re-read the init scripts

Easy to reproduce using rpm-testing. One solution is to stream to stdout/stderr and let systemd capture the events, but it'd be nice to figure out why the python handler can't write to disk. The log files are created but they are always left empty.

The ss repo carries install/storageService.logging.json, but the debs expect install/storageService.logging.conf

https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/install/storageService.logging.json

https://github.com/artefactual-labs/am-packbuild/blob/master/debs/trusty/archivematica-storage-service/debian-storage-service/archivematica-storage-service.install#L4

@sevein what does the docs say?

Changes are being made in SS 0.11 that will likely affect the way packages are built, e.g. new environment variables, WhiteNoise, etc...

Find all the details in the following links:

• artefactual/archivematica-storage-service#217
• artefactual/archivematica-storage-service#218
• artefactual-labs/ansible-archivematica-src#119

In AM16 we had the following repos:

• https://packages.archivematica.org/1.6.x/centos-extras
• https://packages.archivematica.org/1.6.x/ubuntu-externals

We haven't created the AM17 externals repos yet. This should be done at some point so we know that the dependencies are installed and the right versions are used, like siegfried v1.7.6. Maybe that's the only change needed?

Related question that I'm trying to squeeze here hoping that you can respond. I don't really understand the mechanics of deb/rpm repositories. What's in the externals repo that needs to be namespaced as /1.6.x or /1.7.x? Couldn't a single repository contain multiple versions of a package? E.g. I see how multiple versions of the siegfried package coexist under the same https://packages.archivematica.org/1.6.x/centos-extras/ repository: v1.5.0, v1.6.7 and v1.7.3? The archivematica-mcp-client package depends on siegfried but we don't define a version number constraint. If we started doing this, could we move toward a situation where we'd only need one externals repository that doesn't need to be versioned? Thank you in advance.

As a developer I want the packages available in the master branch to be accessible via public repositories. If a package is deleted, updated or added to master I want the public repositories to reflect the change as soon as possible.

SS packages are now built with mysql and postgresql support. Those are build dependencies, but they shouldn't be needed in a default install

When building with psycopg requirements, there is a bug:

"strip: debian/archivematica-storage-service/usr/share/python/archivematica-storage-service/lib/python2.7/site-packages/psycopg2/stlS8Kcz: Not enough room for program headers, try linking with -N"

The bug is similar to this one: numpy/numpy#7570

This repo builds a new version of siegfried (version 1.7.3). The package is uploaded to https://packages.archivematica.org/1.6.x/centos-extras.

The version of this package on that server right now can't be installed on a Centos 7.x machine. This has been tested on 5 machines so far - same error each time. Whether it is a fresh installation or an upgrade from siegfried 1.6.7, the same errors show:

warning: group 1000 does not exist - using root warning: user 1000 does not exist - using root warning: group 1000 does not exist - using root error: unpacking of archive failed on file /usr/share/siegfried/release-notes.xml;599d92c7: cpio: read

I am guessing that user 1000 and group 1000 means the 'archivematica' user? I think it should be possible to install siegfried without first installing Archivematica. The release-notes.xml, however, appear to just be missing from the package?

@scollazo is aware of this problem but I realize no issue has been submitted yet.

Our deb packages are using virtualenvs (yay!) but unlike rpms/ansible/docker they're not installing all the pip requirements as listed below. This seems to be a challenge but I still don't quite understand yet the reason - I hope @scollazo can add a comment with more details.

The approach of patching the requirements feels wrong, e.g. see debs/trusty/archivematica/debian-MCPClient/patches/mcp-client-extra-requirements.diff or debs/xenial/archivematica/debian-MCPClient/patches/mcp-client-extra-requirements.diff. These files are going to become obsolete as soon as the requirements changes in upstream. Also there are missing dependencies, e.g. agentarchives is not included in that diff and many more. They're maybe not always needed but some workflows will surely break, e.g. MCPClient has a client script used to upload to different access systems. This client script depends on agentarchives.

MCPClient

pip install -r src/archivematicaCommon/requirements/production.txt
pip install -r src/dashboard/src/requirements/production.txt
pip install -r src/MCPClient/requirements/production.txt

MCPServer

pip install -r src/archivematicaCommon/requirements/production.txt
pip install -r src/dashboard/src/requirements/production.txt
pip install -r src/MCPServer/requirements/production.txt

Dashboard

pip install -r src/archivematicaCommon/requirements/production.txt
pip install -r src/dashboard/src/requirements/production.txt

It won't be hard to fix and it could be wired with Travis perhaps?

@scollazo maintains a script here: https://gist.github.com/scollazo/4bd044ced115a09370274e157cc5536a - not the best venue perhaps but it's work in progress.

https://github.com/artefactual-labs/am-packbuild/blob/dev/rpm-packaging/rpm/fido/package.spec#L12

This spec file should be updated and specify python-six 1.10.0, or else fido will throw an error when used.

As the packaging configuration is keep outside of the main repo, we should fill the buildrequires and dependencies fields based on the osdeps.yml from the main repo.

This avoids duplicated work, as new package dependencies only need to be added in one place (osdeps.yml files), and not in each debian/control and rpm spec files.

There are some configuration parameters that are read as environment varaibles in Archivematica 1.7 and above, that are missing from the /etc/default/archivematica-mcp-client.default file.

ARCHIVEMATICA_MCPCLIENT_CLIENT_HOST=
ARCHIVEMATICA_MCPCLIENT_CLIENT_DATABASE=
ARCHIVEMATICA_MCPCLIENT_CLIENT_USER=
ARCHIVEMATICA_MCPCLIENT_CLIENT_PASSWORD=
ARCHIVEMATICA_MCPCLIENT_DISABLE_SEARCH_INDEXING=

ARCHIVEMATICA_MCPCLIENT_MCPCLIENT_MCPARCHIVEMATICASERVER=localhost:4730
ARCHIVEMATICA_MCPCLIENT_MCPCLIENT_CLIENTASSETSDIRECTORY=/usr/lib/archivematica/MCPClient/assets/
ARCHIVEMATICA_MCPCLIENT_MCPCLIENT_CLAMAV_SERVER=/var/run/clamav/clamd.ctl
ARCHIVEMATICA_MCPCLIENT_MCPCLIENT_ARCHIVEMATICACLIENTMODULES=/usr/lib/archivematica/MCPClient/archivematicaClientModules
ARCHIVEMATICA_MCPCLIENT_MCPCLIENT_CLIENTSCRIPTSDIRECTORY=/usr/lib/archivematica/MCPClient/clientScripts/

We can update wsgi.py as we do here

Instead of one dockerfile for each rpm, we should have a common one, and manage build dependencies using the BuildRequires: spec tag.

For things like golang 1.6, we might need to rebuild the package for centos7 from one of the multiple .src.rpms avaliable in rpmfind

HOUSEKEEPING! 🚿

Outside this repository:

• https://github.com/artefactual/archivematica/tree/qa/1.x/src/dashboard/debian
• https://github.com/artefactual/archivematica/tree/qa/1.x/src/MCPClient/debian
• https://github.com/artefactual/archivematica/tree/qa/1.x/src/MCPServer/debian
• https://github.com/artefactual/archivematica/tree/qa/1.x/src/archivematicaCommon/debian
• https://github.com/artefactual/archivematica-storage-service/tree/qa/0.x/debian

Mainly:

• debs/fido
• rpm/fido
• rpm/olefile
• rpm/six

Other possible cleanups:

• files/ss-lib
• packbuild.py
• packbuild-internal.py

Currently, our rpms are unsigned (but served over https). We should build signed rpms, as we build signed debian packages.

Both dashboard and storage service install gunicorn and Nginx in the same node so it's possible to serve both static and dynamic content. As we assume that gunicorn and Nginx services always run in the same node we could use unix sockets as well. It's a small optimization but it's also very easy to set up.

https://github.com/artefactual/archivematica/blob/stable/1.6.x/src/dashboard/install/dashboard.gunicorn-config.py

This is a config file that should get copied into /etc/archivematica/dashboard-gunicorn-config.py. The systemd service definintion for the dashboard should be copied from this file in the rpm:
https://github.com/artefactual/archivematica/blob/stable/1.6.x/src/dashboard/install/archivematica-dashboard.conf

Also, the nginx config file is not getting used (from https://github.com/artefactual/archivematica/blob/stable/1.6.x/src/dashboard/install/dashboard.conf)

On an rpm install of 1.6.1, the systemd config file has the timeout value and number of workers value hardcoded, and the dashboard-gunicorn-config.py file is missing. The same isue exists for the storage service 0.10.1 rpm.

Perhaps other packages from extra need to be updated too.

Currently we have VERSION RELEASE and BRANCH, and they have different meanings for archivematica, and for the external packages.

Add a template file with all availabe configuration variables in form of environment vars into /usr/share/doc/archivematica-

Hi,

With forcing SSL on the CentOS Repo, I am unable to access it via the yum proxy for a computer that does not have direct access to the internet. Would it be possible not force SSL for https://packages.archivematica.org/1.6.x/centos/ and sign the packages as in Issue #38?

Thank you,

Tom

Currently, we build the am repo as 4 different debian source and binary packages, we should go towards having one source package (archivematica) that provides 4 binary packages , as we do with rpms.

It's a dependency of rpm/mcpclient but that wouldn't be enough for someone only installing SS.

Currently, the gunicorn config files define their own environment vars as a raw_env param, they should honour the ones in /etc/default/archivematica-* (or /etc/sysconfig/archivematica-* for rh/centos )

As we have a default vars file for distros that use systemd, we should use it for all environment related configurations

This could be also filed in the archivematica-docs repo but I want to make sure this is also captured here for more visibility.

For AM17 we need docs that explain thinks like setting up the rng service (#73), using the new environment variables, etc...

@scollazo is maintaining the installation scripts here: https://gist.github.com/scollazo/4bd044ced115a09370274e157cc5536a. That could be used as a reference to make sure that our docs are up to date.

Current package files are in the SS repo (see branch qa/0.8.x). We use packbuld.py in this repo to build the packages in Launchpad. Can we ship our packages via our packages.archivematica.org domain instead? If this is doable and it's not going to take a lot of time then let's do it.

I'd like to be able to install SS as follows:

$ sudo add-apt-repository "deb https://packages.archivematica.org/storage-service/0.8.x/debian stable main"
$ sudo apt-get update && sudo apt-get --yes install archivematica-storage-service

That should provide Nginx and deploy the gunicorn upstart service. Support for ubuntu:trusty is a must. ubuntu:xenial or debian:jessie (systemd) are nice to have.

We could migrate the database from postinst. We didn't do that in our RPM package but we could. For now, SS uses sqlite, locally, so I think it's okay to make use of postinst.

The package should contain the sources of a tag v0.8.0. I can create a v0.8.0-rc1 for now until we are fully ready or you can use qa/0.8.x temporary.

The debian/ directory in qa/0.8.x may not be needed anymore if this repo become the authoritative way to build packages in 0.8.0 and newer.

The current packages are using https://github.com/spotify/dh-virtualenv, sounds like we could still make use of them, but we don't want to install deps in postinst, we rather include the whole virtualenv in the package like we do in our rpm packages?

Packages for ubuntu trusty are built using the debian/ folder from the archivematica and archivematica-storage-service repositories. We should move those debian folders to this repo, and build the package from them,

We can create the log files in advance, as we did in the past.

Currently, our Vagrantfile uses locally built rpm packages, but it should be also able to use the ones from the repo avaliable at packages.archivematica.org

I've tried the 1.7.x externals repo in #798 (artefactual/archivematica@a14cd23 - trusty-based). It seems to be fine excepting for jhove because it says it depends on openjdk-8-jre-headless which is not available.

root@d35561864e55:/# apt install jhove
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 jhove : Depends: openjdk-8-jre-headless but it is not installable
E: Unable to correct problems, you have held broken packages.

By default, CentOS 7 enables requiretty for sudo access, and that make some mcp-client scripts fail early.

It can be disabled per-user, with !requiretty.

Only 1.7.3 is installed but AM already expects 1.7.6 to be available.

There are build-dependencies that are not declared, and instead they are used as Depends:

Since now we ship the full binary virtualenvironment, he -dev packages shouldn't be needed in the installation environment.