suse / rmt Goto Github PK
View Code? Open in Web Editor NEWRPM repository mirroring tool and registration proxy for SUSE Customer Center.
License: Other
RPM repository mirroring tool and registration proxy for SUSE Customer Center.
License: Other
I am receiving:
> sudo ./bin/rmt-cli mirror
./bin/rmt-cli:19: syntax error, unexpected '.'
group = Settings&.[](:cli)&.[](:group) || RMT::DEFAULT_GROUP
^
./bin/rmt-cli:20: syntax error, unexpected '.'
user = Settings&.[](:cli)&.[](:user) || RMT::DEFAULT_USER
^
because .ruby-version has 2.4.1 so we can run on SLES12.
I did not find a way to enforce a ruby version syntax in rubocop.
Looking at the output of /connect/systems/activations
:
[
{
"id": 15,
"system_id": 30,
"service": {
"id": 624,
"name": "SUSE_Linux_Enterprise_Server_x86_64",
"url": "http://rmt:4224/services/624?credentials=RMT-http_rmt_4224",
"obsoleted_service_name": "",
"product": {
"id": 1421,
"name": "SUSE Linux Enterprise Server",
"description": "SUSE Linux Enterprise offers a comprehensive suite of products built on a single code base. The platform addresses business needs from the smallest thin-client devices to the world's most powerful high-performance computing and mainframe servers. SUSE Linux Enterprise offers common management tools and technology certifications across the platform, and each product is enterprise-class.",
"friendly_name": "SUSE Linux Enterprise Server 12 SP3 x86_64",
"shortname": "SLES12-SP3",
"former_identifier": "SLES",
"product_type": "base",
"product_class": "7261",
"release_type": null,
"release_stage": "released",
"identifier": "SLES",
"version": "12.3",
"arch": "x86_64",
"eula_url": "https://updates.suse.com/SUSE/Products/SLE-SERVER/12-SP3/x86_64/product.license/",
"free": false,
"cpe": "cpe:/o:suse:sles:12:sp3"
}
}
}
]
"eula_url":"https://updates.suse.com/SUSE/Products/SLE-SERVER/12-SP3/x86_64/product.license/"
-- hostname should have been replaced with RMT hostname;"free":false"
-- should always be true
Seems like the issue is caused by has_one
relationship in ActiveSerializer.
It should be possible to provide multiple product IDs to enable or disable for mirroring on the command line.
rmt-cli repos enable 1 2 3 4 5
should be supported.
As reported by @keichwa, when export directory isn't writable export command produced a traceback.
It would be better to have a nicer error message suggesting to make sure that the export directory is writable by RMT.
sudo rmt-cli export data /var/tmp/usb-ex
I, [2018-03-07T10:20:36.949116 #11559] INFO -- : Exporting data from SCC to /var/tmp/usb-ex
I, [2018-03-07T10:20:36.949174 #11559] INFO -- : Exporting products
Traceback (most recent call last):
20: from /usr/bin/rmt-cli:40:in `<main>'
19: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/base.rb:444:in `start'
18: from /usr/share/rmt/lib/rmt/cli/base.rb:22:in `dispatch'
17: from /usr/share/rmt/lib/rmt/cli/base.rb:33:in `handle_exceptions'
16: from /usr/share/rmt/lib/rmt/cli/base.rb:22:in `block in dispatch'
15: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor.rb:369:in `dispatch'
14: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/invocation.rb:126:in `invoke_command'
13: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/command.rb:27:in `run'
12: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor.rb:242:in `block in subcommand'
11: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/invocation.rb:115:in `invoke'
10: from /usr/share/rmt/lib/rmt/cli/base.rb:22:in `dispatch'
9: from /usr/share/rmt/lib/rmt/cli/base.rb:33:in `handle_exceptions'
8: from /usr/share/rmt/lib/rmt/cli/base.rb:22:in `block in dispatch'
7: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor.rb:369:in `dispatch'
6: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/invocation.rb:126:in `invoke_command'
5: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/command.rb:27:in `run'
4: from /usr/share/rmt/lib/rmt/cli/export.rb:5:in `data'
3: from /usr/share/rmt/lib/rmt/cli/base.rb:83:in `needs_path'
2: from /usr/share/rmt/lib/rmt/cli/export.rb:6:in `block in data'
1: from /usr/share/rmt/lib/rmt/scc.rb:52:in `export'
/usr/share/rmt/lib/rmt/scc.rb:52:in `write': Permission denied @ rb_sysopen - /var/tmp/usb-ex/organizations_products_scoped.json (Errno::EACCES)
https://www.novell.com/documentation/open-enterprise-server-2018/inst_oes_lx/data/bmu0fem.html
If SCC logins are supported, will NCC logins and repositories be supported as well?
guid
, secret
and target
in systems
table are empty and unused, drop them! ๐ฅ๐ฅ๐ฅ
From journalctl -fu rmt
:
Apr 03 16:36:18 rmt systemd[1]: rmt.service: Supervising process 21822 which is not our child. We'll most likely not notice when it exits.
I looked into this a bit, and it seems that we're not using the recommended way for starting the Rails server (puma) with systemd. We use type=forking
instead of type=simple
.
This is the recommended configuration according to the Puma docs.
Is there a reason we do it differently?
See also https://mikewilliamson.wordpress.com/2015/08/26/running-a-rails-app-with-systemd-and-liking-it/
Please forgive me if this is obvious, but I need to clarify something;
Adding a non-SCC repo mirror: For example, if we add the non-SCC rmt repo for OpenSuse Leap 42.3, we run: rmt-cli mirror custom https://download.opensuse.org/repositories/systemsmanagement:/SCC:/RMT/openSUSE_Leap_42.3/ opensuse/rmt - this is only a one time event, correct? Subsequent runs of just 'rmt-cli mirror' will only mirror SCC product repos, non-SCC custom repos are not mirrored automatically at this time. Is this correct? If not correct what am I missing. If it is correct, is there any reason custom repos can't be added to the list of maintained repos for scheduled rmt-cli mirror runs by default (or perhaps enable future mirroring of the custom repo via a command in rmt-mirror)?
I believe I'm using the latest build, however I receive a permission denied using rmt-cli mirror. Obviously I'm doing something wrong, but what? RPM version status (from zypper), sample rmt-cli mirror error, and permissions checks are below;
Repository : systemsmanagement:SCC:RMT (SLE_12_SP3)
Name : rmt-server
Version : 0.0.3-23.1
Arch : x86_64
Vendor : obs://build.opensuse.org/systemsmanagement
Installed : Yes
Status : up-to-date
Source package : rmt-server-0.0.3-23.1.src
server:/var/lib/rmt # rmt-cli mirror
WARNING: Nokogiri was built against LibXML version 2.9.1, but has dynamically loaded 2.9.4
Mirroring repository SLE-12-GA-Desktop-NVIDIA-Driver to /usr/share/rmt/public/repo
I, [2018-02-19T14:16:22.481536 #14874] INFO -- : No product license found
I, [2018-02-19T14:16:22.491409 #14874] INFO -- : โ repomd.xml
I, [2018-02-19T14:16:22.494395 #14874] INFO -- : โ repomd.xml.key
I, [2018-02-19T14:16:22.497288 #14874] INFO -- : โ repomd.xml.asc
I, [2018-02-19T14:16:22.504016 #14874] INFO -- : โ f210b539c39142308b91f14b82200bbb5e0cc31d9ab2bb6aad2d77da3a4d9d47-filelists.xml.gz
I, [2018-02-19T14:16:22.514136 #14874] INFO -- : โ 1f6d42da5f7bcf8d6d1e6b1563afa5f02dbacc1b4fe45e5be6913cfaea389327-other.xml.gz
I, [2018-02-19T14:16:22.520069 #14874] INFO -- : โ b37db1bf66214660a90df2e0220ca726f6f0adefa662ef5ea9baffe9d9ab113e-primary.xml.gz
Traceback (most recent call last):
20: from /usr/bin/rmt-cli:40:in `<main>'
19: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/base.rb:444:in `start'
18: from /usr/share/rmt/lib/rmt/cli/base.rb:22:in `dispatch'
17: from /usr/share/rmt/lib/rmt/cli/base.rb:33:in `handle_exceptions'
16: from /usr/share/rmt/lib/rmt/cli/base.rb:22:in `block in dispatch'
15: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor.rb:369:in `dispatch'
14: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/invocation.rb:126:in `invoke_command'
13: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/command.rb:27:in `run'
12: from /usr/share/rmt/lib/rmt/cli/main.rb:26:in `mirror'
11: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/activerecord-5.1.4/lib/active_record/relation/delegation.rb:39:in `each'
10: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/activerecord-5.1.4/lib/active_record/relation/delegation.rb:39:in `each'
9: from /usr/share/rmt/lib/rmt/cli/main.rb:26:in `block in mirror'
8: from /usr/share/rmt/lib/rmt/cli/base.rb:90:in `mirror!'
7: from /usr/share/rmt/lib/rmt/mirror.rb:37:in `mirror'
6: from /usr/share/rmt/lib/rmt/mirror.rb:169:in `replace_llectory'
5: from /usr/lib64/ruby/2.5.0/fileutils.rb:460:in `mv'
4: from /usr/lib64/ruby/2.5.0/fileutils.rb:1461:in `fu_each_src_dest'
3: from /usr/lib64/ruby/2.5.0/fileutils.rb:1479:in `fu_each_src_dest0'
2: from /usr/lib64/ruby/2.5.0/fileutils.rb:1463:in `block in fu_each_src_dest'
1: from /usr/lib64/ruby/2.5.0/fileutils.rb:471:in `block in mv'
**/usr/lib64/ruby/2.5.0/fileutils.rb:471:in `rename': Permission denied @ rb_file_s_rename - (/usr/share/rmt/public/repo/novell/sle12/repodata, /usr/share/rmt/public/repo/novell/sle12/.old_repodata) (Errno::EACCES)**
server:/usr/share/rmt # ll
total 60
<...>
**lrwxrwxrwx 1 _rmt nginx 19 Feb 19 14:08 public -> /var/lib/rmt/public**
<...>
server:/var/lib/rmt # ll
total 8
<...>
**lrwxrwxrwx 1 _rmt nginx 15 Feb 2 14:10 public -> /nfs_share/public**
<...>
server:/var/lib/rmt #
I.e. if HTML is returned instead of directory.yast
in one of the repos, that breaks the whole mirroring process:
I, [2018-02-12T12:32:16.925724 #22884] INFO -- : โ directory.yast
~/.rvm/rubies/ruby-2.4.1/lib64/ruby/2.4.0/uri/rfc3986_parser.rb:67:in `split': bad URI(is not URI?): <!DOCTYPE html> (URI::InvalidURIError)
from ~/.rvm/rubies/ruby-2.4.1/lib64/ruby/2.4.0/uri/rfc3986_parser.rb:73:in `parse'
from ~/.rvm/rubies/ruby-2.4.1/lib64/ruby/2.4.0/uri/rfc3986_parser.rb:117:in `convert_to_uri'
from ~/.rvm/rubies/ruby-2.4.1/lib64/ruby/2.4.0/uri/generic.rb:1099:in `merge'
from ~/.rvm/rubies/ruby-2.4.1/lib64/ruby/2.4.0/uri/rfc3986_parser.rb:89:in `inject'
from ~/.rvm/rubies/ruby-2.4.1/lib64/ruby/2.4.0/uri/rfc3986_parser.rb:89:in `join'
from ~/.rvm/rubies/ruby-2.4.1/lib64/ruby/2.4.0/uri/common.rb:269:in `join'
from ~/work/rmt/lib/rmt/downloader.rb:98:in `make_request'
from ~/work/rmt/lib/rmt/downloader.rb:38:in `block in download'
Ideally, the affected repo should be skipped and mirroring would continue.
In the README you say:
You can create a MySQL/MariaDB user with the following command:
mysql -u root -p <<EOFF
GRANT ALL PRIVILEGES ON \`rmt\`.* TO rmt@localhost IDENTIFIED BY 'rmt';
FLUSH PRIVILEGES;
EOFF
===================================================================
But this results in this error msg on my installation:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysql/mysql.sock' (2 "No such file or directory")
I tried to start it first with systemctl start mysql
, but creating the DB user still fails with:
GRANT ALL PRIVILEGES ON \`rmt\`.* TO rmt@localhost IDENTIFIED BY 'rmt';
FLUSH PRIVILEGES;
EOFF
Enter password:
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
Can you please enhance the README?
After installing the package the following message is shown:
(1/1) Installing: rmt-server-0.0.7-0.x86_64 .................................................................................................................................................................[done]
Additional rpm output:
W, [2018-05-09T15:06:32.260289 #4241] WARN -- : Creating scope :test. Overwriting existing method Subscription.test.
During installation we are calling rails
binary, that's probably where this output is produced:
Lines 222 to 223 in 359e1e8
Nice project! I decided to test rmt on SLES 12 SP3. After configuring rmt.conf, starting rmt with no errors, and verifying the web server is accessible on port 4224, rmt-cli fails. I have double checked all settings in my rmt.conf. Am I missing a ruby dependency or something silly? Here's the error;
kevin:/etc # rmt-cli scc sync /usr/share/rmt/lib/rmt/scc.rb:13:in
sync': undefined method username' for "username:xxxxxx password:xxxxxxxx":String (NoMethodError) from /usr/share/rmt/lib/rmt/cli/scc.rb:10:in
sync'
from /usr/lib64/rmt/vendor/bundle/ruby/2.4.0/gems/thor-0.20.0/lib/thor/command.rb:27:in run' from /usr/lib64/rmt/vendor/bundle/ruby/2.4.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in
invoke_command'
from /usr/lib64/rmt/vendor/bundle/ruby/2.4.0/gems/thor-0.20.0/lib/thor.rb:387:in dispatch' from /usr/share/rmt/lib/rmt/cli/base.rb:24:in
block in dispatch'
from /usr/share/rmt/lib/rmt/cli/base.rb:35:in handle_exceptions' from /usr/share/rmt/lib/rmt/cli/base.rb:24:in
dispatch'
from /usr/lib64/rmt/vendor/bundle/ruby/2.4.0/gems/thor-0.20.0/lib/thor/invocation.rb:115:in invoke' from /usr/lib64/rmt/vendor/bundle/ruby/2.4.0/gems/thor-0.20.0/lib/thor.rb:238:in
block in subcommand'
from /usr/lib64/rmt/vendor/bundle/ruby/2.4.0/gems/thor-0.20.0/lib/thor/command.rb:27:in run' from /usr/lib64/rmt/vendor/bundle/ruby/2.4.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in
invoke_command'
from /usr/lib64/rmt/vendor/bundle/ruby/2.4.0/gems/thor-0.20.0/lib/thor.rb:387:in dispatch' from /usr/share/rmt/lib/rmt/cli/base.rb:24:in
block in dispatch'
from /usr/share/rmt/lib/rmt/cli/base.rb:35:in handle_exceptions' from /usr/share/rmt/lib/rmt/cli/base.rb:24:in
dispatch'
from /usr/lib64/rmt/vendor/bundle/ruby/2.4.0/gems/thor-0.20.0/lib/thor/base.rb:466:in start' from /usr/bin/rmt-cli:38:in
Based on the report from QA: current nginx config uses /var/lib/rmt/public
as a root directory.
rmt-cli
commands download the files into /usr/share/rmt/public
, which is a symlink to /var/lib/rmt/public
.
If the user needs to download files somewhere else, they would change the /usr/share/rmt/public
symlink to point somewhere else -- nginx, however, will still serve files from /var/lib/rmt/public
.
I have split off the code that parses repository metadata into a separate gem -- repomd_parser
[1] [2], mostly to make it possible to use in repo-tools.
I've done some refactoring in the gem and added access to some more metadata attributes. It would be cool to move RMT over to repomd_parser
as well, so that it doesn't have its own implementation of those things.
Vote/comment on this issue if you think it's a good idea to do this ๐ฌ
We decided that 100% test coverage is valuable to the team, that's why we introduced
https://github.com/SUSE/connect/blob/master/spec/spec_helper.rb#L8
Here we should consider the same
If repo files already exist on the disk during mirroring (e.g., they were copied over from SMT), they are simply skipped during mirroring, no entries are created in downloaded_files
table as it would normally happen if files were downloaded.
This is not an issue. It's a usability question:
If an admin testing rmt has mirrored so many products the server is now reaching dangerous space limitations, what to do? ;-) Is it possible to safely change the path of the rmt-server mirror repo to another directory, after a one-time rsync copy to say a NFS share?
$ rmt-cli repo disable 1
/.../rmt/lib/rmt/cli/repos.rb:51:in `change_repository_mirroring': undefined method `change_mirroring!' for nil:NilClass (NoMethodError)
It would make sense to check all of the commands and make sure invalid input is correctly handled everywhere.
After running RMT for some time:
ls -ld /tmp/d20180* | wc -l
7147
Running make dist
fails with the following error:
~/.rvm/rubies/ruby-2.5.0/lib64/ruby/2.5.0/rubygems/resolver.rb:231:in `search_for': Unable to resolve dependency: user requested 'did_you_mean (= 1.2.0)' (Gem::UnsatisfiableDependencyError)
๐ฑ
Currently the log level is set to :debug
in production.rb
, which generates large amount of log messages:
rmt/config/environments/production.rb
Line 22 in ee5802c
It would be good to have the log level setting in /etc/rmt.conf
and have a lower verbosity level by default.
With SLE15 we introduced recommended modules. It makes sense to enable them by default when a product is enabled for mirroring.
An option --without-recommeded
should exist to disable this feature.
ParameterMissingTranslated
probably isn't loaded for some reason. However, used only once in the code and probably should be replaced with TranslatedError
.
[f0841d43-d932-484c-86c0-308fb69e316b] Started GET "/connect/repositories/installer" for 127.0.0.1 at 2018-03-28 14:57:05 +0000
[f0841d43-d932-484c-86c0-308fb69e316b] Processing by Api::Connect::V4::Repositories::InstallerController#index as JSON
[f0841d43-d932-484c-86c0-308fb69e316b] Parameters: {"installer"=>{}}
[f0841d43-d932-484c-86c0-308fb69e316b] Completed 500 Internal Server Error in 9ms (ActiveRecord: 0.0ms)
[f0841d43-d932-484c-86c0-308fb69e316b]
[f0841d43-d932-484c-86c0-308fb69e316b] NameError (uninitialized constant ActionController::ParameterMissingTranslated
Did you mean? ActionController::ParameterMissing):
[f0841d43-d932-484c-86c0-308fb69e316b]
[f0841d43-d932-484c-86c0-308fb69e316b] app/controllers/api/connect/base_controller.rb:23:in `require_params'
[f0841d43-d932-484c-86c0-308fb69e316b] app/controllers/api/connect/v4/repositories/installer_controller.rb:4:in `index'
rmt-cli p e sles/15/x86_64
-- works.rmt-cli p l
-- breaks with:Traceback (most recent call last):
16: from /usr/bin/rmt-cli:33:in `<main>'
15: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/base.rb:444:in `start'
14: from /usr/share/rmt/lib/rmt/cli/base.rb:29:in `dispatch'
13: from /usr/share/rmt/lib/rmt/cli/base.rb:40:in `handle_exceptions'
12: from /usr/share/rmt/lib/rmt/cli/base.rb:29:in `block in dispatch'
11: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor.rb:369:in `dispatch'
10: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/invocation.rb:126:in `invoke_command'
9: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/command.rb:27:in `run'
8: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor.rb:242:in `block in subcommand'
7: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/invocation.rb:115:in `invoke'
6: from /usr/share/rmt/lib/rmt/cli/base.rb:29:in `dispatch'
5: from /usr/share/rmt/lib/rmt/cli/base.rb:40:in `handle_exceptions'
4: from /usr/share/rmt/lib/rmt/cli/base.rb:29:in `block in dispatch'
3: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor.rb:338:in `dispatch'
2: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor.rb:433:in `normalize_command_name'
1: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor.rb:453:in `find_command_possibilities'
/usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor.rb:453:in `sort': comparison of String with :ls failed (ArgumentError)
This mapping probably needs to be a string instead of a symbol:
Line 36 in bb928ca
Same for other Thor command mappings.
Tumbleweed users might also want to mirror SUSE repos. Could you add the package to openSUSE:Factory please?
There's already a special format for binaries that log to journald
(enabled with LOG_TO_JOURNALD
env variable). We should adjust the format to specify message severity level: the message needs to be prepended with numeric severity level in diamond brackets (<severity_number>message
). This would enable filtering the logs (journalctl -p
).
The following levels are supported by journald
:
<7>This is a DEBUG level message
<6>This is an INFO level message
<5>This is a NOTICE level message
<4>This is a WARNING level message
<3>This is an ERR level message
<2>This is a CRIT level message
<1>This is an ALERT level message
<0>This is an EMERG level message
I was experimenting with a web interface for RMT ( ๐)
And then I found this logic in the CLI code classes:
Lines 64 to 69 in bca4800
While this is not a real problem while we only have one CLI and no other means to manage RMT, I think this logic should be moved into or closer to the Product
model code, since it is not possible to call this logic from a controller action, like I would need for a web interface.
We're running into an issue where rmt mirrors only PRODUCT-Pool
and PRODUCT-Updates
repos. It's possible to fetch source RPMs for PRODUCT-Updates
if mirror_src
is set but I'm struggling to find how to mirrorPRODUCT-Debuginfo-Pool
, PRODUCT-Debuginfo-Updates
and PRODUCT-Source-Pool
repos for each configured product.
FWIW, I can see the repos I'm looking for in repositories
table of rmt
database, so I think it's something within rmt
that's filtering the repos out and not making mirrors of them.
On an empy database (before the first sync
)
rmt-cli repos --all
currently outputs
No repositories enabled.
But it should behave the same way than products --all
which in this case already outputs
Run "rmt-cli sync" to synchronize with your SUSE Customer Center data first.
Feedback for others who may try this: For a 0.1 Release, the quality of rmt is impressive. On several test and production SLE12 systems that were having issues with SMT and/or broken SUSEConnect issues - RMT just works. Server setup is easy (had one self inflicted YAML issue), client connections are very reliable (I can't break anything yet - believe me I've tried). I have performed some simple zypper patch on SLE 12 SP3 systems that worked fine as well as some SLE 12 SP2 -> SP3 migrations on several systems with absolutely no issues. I simply followed the instructions in the GitHub readme and added a cron job for root on the rmt server to sync and mirror against the SCC daily. Will continue to test and provide feedback, but I really like the direction this is headed.
Not urgent or imperative, but I have a potential enhancement request. At the moment, if multiple targets are presented to rmt-cli, the client fails with an error;
server:/ # rmt-cli repos enable SLES/12.2/x86_64 SLES/12.3/x86_64 sle-sdk/12.2/x86_64 sle-sdk/12.3/x86_64 PackageHub/12.2/x86_64 PackageHub/12.3/x86_64 sle-module-legacy/12/x86_64 sle-module-adv-systems-management/12/x86_64 sle-module-web-scripting/12/x86_64
ERROR: "rmt-cli repos enable" was called with arguments ["SLES/12.2/x86_64", "SLES/12.3/x86_64", "sle-sdk/12.2/x86_64", "sle-sdk/12.3/x86_64", "PackageHub/12.2/x86_64", "PackageHub/12.3/x86_64", "sle-module-legacy/12/x86_64", "sle-module-adv-systems-management/12/x86_64", "sle-module-web-scripting/12/x86_64"]
Usage: "rmt-cli repos enable TARGET"
Allowing multiple targets could make autoyast / automated / scripted setups slightly easier for admins in the future. When I present one target, it works just fine.
In addition to the "rmt-cli repos disable" or rmt-cli products disable" commands, RMT could have a way of removing repos that are no longer needed from local storage.
Originally from: https://bugzilla.suse.com/show_bug.cgi?id=1087785
For the csv export, would be beneficial to export with headers to import into spreadsheet. "--headers"
Data that is sent by Yast:
{
"arch"=>"x86_64",
"identifier"=>"SLES",
"version"=>"12.3-0",
"release_type"=>"",
"product"=>{"release_type"=>"", "identifier"=>"SLES", "version"=>"12.3-0", "arch"=>"x86_64"}
}
The product version isn't being cleaned up, so RMT searches for 12.3-0
in the DB -- which results in Product not found
error.
This might be fixed in Yast for SLES15, but SLES12 versions of Yast/zypper might be affected by this.
Lines 43 to 55 in 8c3e0d2
repository_url
and auth_token
parameters, mirror
should take them insteaduri
and how is it different from repository_url
?Clients download the rmt-client-setup
script over http. We should provide a checksum file that they can verify the downloaded file and avoid a man in the middle.
Ideas:
Make
From the logs of rmt-server-mirror.service
:
Mar 21 02:01:47 rmt rmt-cli[3454]: Mirroring repository SLE-Module-Legacy12-Pool to /usr/share/rmt/public/repo
Mar 21 02:01:47 rmt rmt-cli[3454]: I, [2018-03-21T02:01:45.507740 #3454] INFO -- : โ directory.yast
Mar 21 02:01:47 rmt rmt-cli[3454]: I, [2018-03-21T02:01:45.692643 #3454] INFO -- : โ license.de.txt
Mar 21 02:01:47 rmt rmt-cli[3454]: I, [2018-03-21T02:01:46.285904 #3454] INFO -- : โ license.es.txt
Mar 21 02:01:47 rmt rmt-cli[3454]: I, [2018-03-21T02:01:46.745088 #3454] INFO -- : โ license.fr.txt
Mar 21 02:01:47 rmt rmt-cli[3454]: I, [2018-03-21T02:01:47.020948 #3454] INFO -- : โ license.it.txt
Mar 21 02:01:47 rmt rmt-cli[3454]: I, [2018-03-21T02:01:47.226396 #3454] INFO -- : โ license.ja.txt
Mar 21 02:01:47 rmt rmt-cli[3454]: I, [2018-03-21T02:01:47.412607 #3454] INFO -- : โ license.ko.txt
Mar 21 02:04:21 rmt rmt-cli[3454]: /usr/share/rmt/lib/rmt/mirror.rb:125:in `initialize': No such file or directory @ rb_sysopen - /tmp/d20180321-3454-3jy31p/directory.yast (Errno::ENOENT)
Mar 21 02:04:21 rmt rmt-cli[3454]: from /usr/share/rmt/lib/rmt/mirror.rb:125:in `open'
Mar 21 02:04:21 rmt rmt-cli[3454]: from /usr/share/rmt/lib/rmt/mirror.rb:125:in `mirror_license'
Mar 21 02:04:21 rmt rmt-cli[3454]: from /usr/share/rmt/lib/rmt/mirror.rb:31:in `mirror'
Mar 21 02:04:21 rmt rmt-cli[3454]: from /usr/share/rmt/lib/rmt/cli/base.rb:90:in `mirror!'
Mar 21 02:04:21 rmt rmt-cli[3454]: from /usr/share/rmt/lib/rmt/cli/main.rb:26:in `block in mirror'
Mar 21 02:04:21 rmt rmt-cli[3454]: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/activerecord-5.1.4/lib/active_record/relation/delegation.rb:39:in `each'
Mar 21 02:04:21 rmt rmt-cli[3454]: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/activerecord-5.1.4/lib/active_record/relation/delegation.rb:39:in `each'
Mar 21 02:04:21 rmt rmt-cli[3454]: from /usr/share/rmt/lib/rmt/cli/main.rb:26:in `mirror'
Mar 21 02:04:21 rmt rmt-cli[3454]: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/command.rb:27:in `run'
Mar 21 02:04:21 rmt rmt-cli[3454]: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/invocation.rb:126:in `invoke_command'
Mar 21 02:04:21 rmt rmt-cli[3454]: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor.rb:369:in `dispatch'
Mar 21 02:04:21 rmt rmt-cli[3454]: from /usr/share/rmt/lib/rmt/cli/base.rb:22:in `block in dispatch'
Mar 21 02:04:21 rmt rmt-cli[3454]: from /usr/share/rmt/lib/rmt/cli/base.rb:33:in `handle_exceptions'
Mar 21 02:04:21 rmt rmt-cli[3454]: from /usr/share/rmt/lib/rmt/cli/base.rb:22:in `dispatch'
Mar 21 02:04:21 rmt rmt-cli[3454]: from /usr/lib64/rmt/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/base.rb:444:in `start'
Mar 21 02:04:21 rmt rmt-cli[3454]: from /usr/share/rmt/bin/rmt-cli:40:in `<main>'
Mar 21 02:04:21 rmt rmt-cli[3454]: I, [2018-03-21T02:01:47.872110 #3454] INFO -- : โ license.pt_BR.txt
Mar 21 02:04:21 rmt rmt-cli[3454]: I, [2018-03-21T02:01:48.319427 #3454] INFO -- : โ license.ru.txt
Mar 21 02:04:21 rmt rmt-cli[3454]: I, [2018-03-21T02:01:48.508310 #3454] INFO -- : โ license.txt
Mar 21 02:04:21 rmt rmt-cli[3454]: I, [2018-03-21T02:01:48.694386 #3454] INFO -- : โ license.zh_CN.txt
Mar 21 02:04:21 rmt rmt-cli[3454]: I, [2018-03-21T02:01:49.148291 #3454] INFO -- : โ license.zh_TW.txt
Mar 21 02:04:21 rmt rmt-cli[3454]: I, [2018-03-21T02:01:49.337903 #3454] INFO -- : โ repomd.xml
Mar 21 02:04:21 rmt rmt-cli[3454]: I, [2018-03-21T02:01:49.787041 #3454] INFO -- : โ repomd.xml.key
Mar 21 02:04:21 rmt rmt-cli[3454]: I, [2018-03-21T02:01:49.971263 #3454] INFO -- : โ repomd.xml.asc
Mar 21 02:04:21 rmt rmt-cli[3454]: I, [2018-03-21T02:01:50.423472 #3454] INFO -- : โ a520dfc6a1381cf7a81153e26f980b7991c1204df404cc87aa0406b96ab17eed-filelists.xml.gz
Mar 21 02:04:21 rmt rmt-cli[3454]: I, [2018-03-21T02:01:50.870874 #3454] INFO -- : โ 79e8ba0768866d978e5f41fe1316b27f1bec76f62674ce4bc16b37880208d359-susedata.xml.gz
Mar 21 02:04:21 rmt rmt-cli[3454]: I, [2018-03-21T02:01:51.318416 #3454] INFO -- : โ ff2b95bddf7a2c57a0d00a631abc32f852dac0b5efd92733facab87a9a194a96-other.xml.gz
Mar 21 02:04:21 rmt rmt-cli[3454]: I, [2018-03-21T02:01:51.539702 #3454] INFO -- : โ 2800235d054959bc659bac62602237e6f2963a2510bf67f838985d0f64a8f638-primary.xml.gz
Mar 21 02:04:21 rmt rmt-cli[3454]: I, [2018-03-21T02:01:51.742719 #3454] INFO -- : โ a008fd6cfa7ad376aefe60eeeb77c27ace54f8fa37331e7d311b7762d6482ddb-license.tar.gz
We've got several CLI commands left that do something along the lines:
if (something_wrong)
warn "Error!"
return
end
That isn't ideal, because that doesn't set the exit code. It would be better to raise RMT::CLI::Error
instead, which is handled in RMT::CLI::Base
to set the exit code and hide the stack trace.
This discussion started here: #177
I'm not a fan of try
(nor ruby's safe operator .&
)..
It basically makes code insecure about itself... is the thing there? or is it not? ๐ฑ if it's not, you have to check for nil
s all over the place. => Sadness.
When working with the Settings
object, our code is littered with Settings.try(:foo)
to check if the settings are really OK.
A few things that can be done to improve this:
Settings
class directly (which is the low-level construct provided by the railsconfig/config gem), only from lib/rmt/config.rb
. We have some settings accessed through Settings
and yet others through RMT::Config
, which is inconsistent. We can encapsulate all the try
s and nil
s n RMT::Config
, leaving the rest of the code confident about itself.config
gem, which is included somewhere in the RMT sources, ie. it's not user-modifiable. By staring with a clean, default config, then the config
gem will override the settings that are user-provided.config
gem's validation feature, checking whether the loaded settings are correct.Currently it's possible to specify the user and group for rmt-cli
commands to run with in the config:
Lines 23 to 25 in 6e2867b
However, the service that starts puma with the permissions specified in the service file:
Line 10 in 6e2867b
This can lead to unexpected consequences: i.e., changing the config setting will not make puma
to run as the same user, which can lead to issues with file permissions, etc.
Options:
rmt.conf
and remove the hardcode from the serviceCurrently, if you try to mirror a repository with a :
in the url, a FAT drive will fail to make the directory structure.
Example:
Mirroring repository at http://***/ibs/Devel:/SCC/SLE_12_SP3/
Can not create a local repository directory: Invalid argument @ dir_s_mkdir - /run/media/***/rmt/ibs/Devel:
The warning should make it more clear that the reason why the command failed is because of their FAT drive.
First registration attempt:
# ./rmt-client-setup https://my-local-rmt.example.org
Error: SCC returned 'Not all mandatory repositories are mirrored for product SUSE Linux Enterprise Server 12 SP3 x86_64' (422)
Enabled the repos on RMT, second registration attempt:
# ./rmt-client-setup https://my-local-rmt.example.org
The system is already registered. Please de-register first by calling:
$> SUSEConnect --de-register
$> SUSEConnect --cleanup
At this point turns out that /etc/zypp/credentials.d/SCCcredentials
file was written, however /etc/SUSEConnect
with RMT server URL wasn't, so de-registration isn't possible:
# ./bin/SUSEConnect -d
Error: Invalid system credentials, probably because the registered system was deleted in SUSE Customer Center. Check https://scc.suse.com whether your system appears there. If it does not, please call SUSEConnect --cleanup and re-register this system.
We could check if the product is fully mirrored during announce
API call (the base product data should be present there) and raise the error. Alternatively, this could be fixed by forcing SUSEConnect to write /etc/SUSEConnect
even if an error occurs.
Currently when you, let's say, enable a product, RMT outputs something like
3 repositories enabled
but it does not tell you which ones. So if you by accident enter a wrong product id, you might think your action was successful, even though you just enabled wrong repos.
Plus, when you enable the same product another time, RMT tells you 0 repositories enabled
, just because they were already enabled.
Instead, it should report what exactly the situation is after your action, in a reproducable manner. Something like:
enabled repositories:
- foo
- bar
- baz
Currently rmt-cli mirror
continues to mirror repositories even if errors are encountered. However, it doesn't set exit code if errors were encountered, as the result systemd
unit doesn't enter failed state. It is hard to tell if mirroring was successful or not, there are a lot of info messages displayed and it's hard to find error messages.
SMT displayed error summary at the very end with the list of repositories that weren't fully mirrored.
โ rmt-server-mirror.service - Mirror RMT
Loaded: loaded (/usr/lib/systemd/system/rmt-server-mirror.service; disabled; vendor preset: disabled)
Active: inactive (dead) since Tue 2019-02-12 18:46:38 CET; 3s ago
Process: 454 ExecStart=/usr/share/rmt/bin/rmt-cli mirror (code=exited, status=0/SUCCESS)
Main PID: 454 (code=exited, status=0/SUCCESS)
Feb 12 18:46:37 e127 systemd[1]: Started Mirror RMT.
Feb 12 18:46:38 e127 rmt-cli[454]: INFO: RMT version 1.1.1
Feb 12 18:46:38 e127 rmt-cli[454]: INFO: Mirroring repository SLE-Module-Toolchain12-Pool to /usr/share/rmt/public/repo/SUSE/Products/SLE-Module-Toolchain/12/x86_64/product/
Feb 12 18:46:38 e127 rmt-cli[454]: WARN: Error while mirroring metadata: repodata/repomd.xml - return code ssl_cacert
Feb 12 18:46:38 e127 rmt-cli[454]: INFO: Mirroring repository SLE-Module-Toolchain12-Updates to /usr/share/rmt/public/repo/SUSE/Updates/SLE-Module-Toolchain/12/x86_64/update/
Feb 12 18:46:38 e127 rmt-cli[454]: WARN: Error while mirroring metadata: repodata/repomd.xml - return code ssl_cacert
test
Currently both lib/rmt/repos.rb
and lib/rmt/products.rb
have a list
method that looks rather similar in its logic. Possibly, this can be refactored and moved into some shared method.
We still don't catch RecordNotFound exceptions when a user calls rmt-cli repos enable 123
and that ID is not in the database. Same for products. The fix is simple: Just look ahead in the database with find_by(id: 123)
. But when I started adding this check, I found the method change_repository_mirroring
already too complicated and hesitated to add another if-else.
What makes it (relatively) complicated is that we, for some reason, support rmt-cli repos enable some/product/identifier
which I find very unnecessary. In case a user wants to enable a whole product, there is rmt-cli products enable
... why put this extra functionality in the repos, too?
So I suggest we drop that and make it clear that:
repos enable
only works with a repo id.products enable
takes either a product id or a product identifier string.Then the cognitive complexity of both method would be better.
> docker build -t rmt .
Sending build context to Docker daemon 204.1MB
Step 1/9 : FROM opensuse/amd64:42.3
---> e1c1588cea46
Step 2/9 : RUN zypper --non-interactive install --no-recommend timezone gcc libffi48-devel make git-core zlib-devel libxml2-devel libxslt-devel ruby2.4-rubygem-bundler ruby2.4-rubygem-mini_portile2 ruby2.4-rubygem-nio4r ruby2.4-rubygem-websocket-driver ruby2.4-devel ruby2.4-rubygem-pg cron libmariadb-devel
---> Using cache
---> 97b554f256fb
Step 3/9 : RUN bundle config build.nokogiri --use-system-libraries
---> Using cache
---> d2f75b287852
Step 4/9 : ENV RAILS_ENV production
---> Using cache
---> 5bd26488f9ad
Step 5/9 : COPY . /srv/www/rmt/
---> 401a4d262eea
Step 6/9 : WORKDIR /srv/www/rmt/
---> ca1585d311de
Removing intermediate container d8718d67946b
Step 7/9 : RUN bundler.ruby2.4
---> Running in 1e3604d83b61
Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
installing your bundle as root will break this application for all non-root
users on this machine.
Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.
Fetching gem metadata from https://rubygems.org/..........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Fetching https://github.com/vcr/vcr.git
Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.
Using rake 12.0.0
Installing concurrent-ruby 1.0.5
Installing i18n 0.8.6
Installing minitest 5.10.3
Installing thread_safe 0.3.6
Installing tzinfo 1.2.3
Installing activesupport 5.1.3
Installing builder 3.2.3
Installing erubi 1.6.1
Installing mini_portile2 2.3.0
Installing nokogiri 1.8.1 with native extensions
Installing rails-dom-testing 2.0.3
Installing loofah 2.0.3
Installing rails-html-sanitizer 1.0.3
Installing actionview 5.1.3
Installing rack 2.0.3
Installing rack-test 0.6.3
Installing actionpack 5.1.3
Installing activemodel 5.1.3
Installing case_transform 0.2
Installing jsonapi-renderer 0.1.2
Installing active_model_serializers 0.10.6
Installing arel 8.0.0
Installing activerecord 5.1.3
Installing public_suffix 2.0.5
Installing addressable 2.5.1
Installing ast 2.3.0
Installing awesome_print 1.8.0
Installing byebug 9.0.6 with native extensions
Installing deep_merge 1.1.1
Installing config 1.4.0
Installing multi_json 1.12.2
Installing unf_ext 0.0.7.4 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
current directory: /usr/lib64/ruby/gems/2.4.0/gems/unf_ext-0.0.7.4/ext/unf_ext
/usr/bin/ruby.ruby2.4 -r ./siteconf20180118-1-q85i57.rb extconf.rb
checking for -lstdc++... no
creating Makefile
current directory: /usr/lib64/ruby/gems/2.4.0/gems/unf_ext-0.0.7.4/ext/unf_ext
make "DESTDIR=" clean
current directory: /usr/lib64/ruby/gems/2.4.0/gems/unf_ext-0.0.7.4/ext/unf_ext
make "DESTDIR="
compiling unf.cc
make: g++: Command not found
Makefile:209: recipe for target 'unf.o' failed
make: *** [unf.o] Error 127
make failed, exit code 2
Gem files will remain installed in /usr/lib64/ruby/gems/2.4.0/gems/unf_ext-0.0.7.4 for inspection.
Results logged to /usr/lib64/ruby/gems/2.4.0/extensions/x86_64-linux/2.4.0/unf_ext-0.0.7.4/gem_make.out
An error occurred while installing unf_ext (0.0.7.4), and Bundler cannot
continue.
Make sure that `gem install unf_ext -v '0.0.7.4'` succeeds before bundling.
The command '/bin/sh -c bundler.ruby2.4' returned a non-zero code: 5
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.