Giter Club home page Giter Club logo

rpmpack's Introduction

rpmpack (tar2rpm) - package rpms the easy way

Disclaimer

This is not an official Google product, it is just code that happens to be owned by Google.

Overview

tar2rpm is a tool that takes a tar and outputs an rpm. rpmpack is a golang library to create rpms. Both are written in pure go, without using rpmbuild or spec files. API documentation for rpmpack can be found in GoDoc.

Installation

$ go get -u github.com/google/rpmpack/...

This will make the tar2rpm tool available in ${GOPATH}/bin, which by default means ~/go/bin.

Usage of the binary (tar2rpm)

tar2rpm takes a tar file (from stdin or a specified filename), and outputs an rpm.

Usage:
  tar2rpm [OPTION] [FILE]
Options:
  -file FILE
        write rpm to FILE instead of stdout
  -name string
        the package name
  -release string
        the rpm release
  -version string
        the package version

Usage of the library (rpmpack)

API documentation for rpmpack can be found in GoDoc.

import "github.com/google/rpmpack"
...
r, err := rpmpack.NewRPM(rpmpack.RPMMetaData{Name: "example", Version: "3"})
if err != nil {
  ...
}
r.AddFile(rpmpack.RPMFile{
    Name: "/usr/local/hello",
    Body: []byte("content of the file"),
})
if err := r.Write(w); err != nil {
  ...
}

Usage in the bazel build system (pkg_tar2rpm)

There is a working example inside example_bazel

In WORKSPACE

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
git_repository(
    name = "rpmpack",
    remote = "https://github.com/google/rpmpack.git",
   branch = "master",
)

# The following will load the requirements to build rpmpack
http_archive(
    name = "io_bazel_rules_go",
    sha256 = "69de5c704a05ff37862f7e0f5534d4f479418afc21806c887db544a316f3cb6b",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.27.0/rules_go-v0.27.0.tar.gz",
        "https://github.com/bazelbuild/rules_go/releases/download/v0.27.0/rules_go-v0.27.0.tar.gz",
    ],
)

http_archive(
    name = "bazel_gazelle",
    sha256 = "62ca106be173579c0a167deb23358fdfe71ffa1e4cfdddf5582af26520f1c66f",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/bazel-gazelle/releases/download/v0.23.0/bazel-gazelle-v0.23.0.tar.gz",
        "https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.23.0/bazel-gazelle-v0.23.0.tar.gz",
    ],
)

load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies")

go_rules_dependencies()

go_register_toolchains(version = "1.16")

gazelle_dependencies()

load("@com_github_google_rpmpack//:deps.bzl", "rpmpack_dependencies")

rpmpack_dependencies()

In BUILD or BUILD.bazel

load("@bazel_tools//tools/build_defs/pkg:pkg.bzl", "pkg_tar")
load("@com_github_google_rpmpack//:def.bzl", "pkg_tar2rpm")

pkg_tar(
    name = "rpmtest-tar",
    srcs = [":content1.txt"],
    mode = "0644",
    ownername = "root.root",
    package_dir = "var/lib/rpmpack",
)

pkg_tar2rpm(
    name = "rpmtest",
    data = ":rpmtest-tar",
    pkg_name = "rpmtest",
    release = "3.4",
    version = "1.2",
    prein = "echo \"This is preinst\" > /tmp/preinst.txt",
)

Features

  • You put files into the rpm, so that rpm/yum will install them on a host.
  • Simple.
  • No spec files.
  • Does not build anything.
  • Does not try to auto-detect dependencies.
  • Does not try to magically deduce on which computer architecture you run.
  • Does not require any rpm database or other state, and does not use the filesystem.

Downsides

  • Is not related to the team the builds rpmlib.
  • May easily wreak havoc on rpm based systems. It is surprisingly easy to cause rpm to segfault on corrupt rpm files.
  • Many features are missing.
  • All of the artifacts are stored in memory, sometimes more than once.
  • Less backwards compatible than rpmbuild.

Philosophy

Sometimes you just want files to make it to hosts, and be managed by the package manager. rpmbuild can use a spec file, together with a specific directory layout and local database, to build/install/package your files. But you don't need all that. You want something similar to tar.

As the project progresses, we must maintain the complexity/value ratio. This includes both code complexity and interface complexity.

Disclaimer

This is not an official Google product, it is just code that happens to be owned by Google.

rpmpack's People

Contributors

aedwa038 avatar caarlos0 avatar cristianciutea avatar dependabot[bot] avatar djgilcrease avatar erikgeiser avatar felixoid avatar jarondl avatar marcsauter avatar nickajacks1 avatar phlax avatar sluongng avatar tanyabouman avatar weirdbob avatar yakhatape avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rpmpack's Issues

tar2rpm: Build Date is far in the future

The rpm created by tar2rpm contains a "Build Date" that is far in the future.
Steps to reproduce:

kali@kali:~/go$ date
Wed 08 Jul 2020 04:02:49 PM EDT

kali@kali:~/go$ ./tar2rpm -file /tmp/example.rpm -name "Example" -version "1.0" /tmp/input.tar

kali@kali:~/go$ rpm -qip /tmp/example.rpm

Name        : Example
Epoch       : 0
Version     : 1.0
Release     : 
Architecture: noarch
Install Date: (not installed)
Group       : 
Size        : 727851
License     : 
Signature   : (none)
Source RPM  : Example-1.0.src.rpm
Build Date  : Sun 13 Jul 2042 09:04:00 PM EDT
Build Host  : 
Relocations : (not relocatable)
Packager    : 
Vendor      : 
URL         : 
Summary     : 
Description :

Expectation:
The "Build Date:" line of the rpm query should contain something like "Wed 08 Jul 2020 04:02:49 PM EDT", but it's always "Sun 13 Jul 2042 09:04:00 PM EDT" even on subsequent delayed rebuilds of the rpm.

Support zstd compression (and optional compression levels)

I'd like to submit a PR to add Zstandard compression support. I already added it on a local branch an tested it with https://github.com/goreleaser/nfpm and Fedora 34 (see https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression). Is this something you'd consider merging?

I'd use the zstd implementation from https://github.com/klauspost/compress/tree/master/zstd which is provided "using a Go standard license". While the algorithm stands for itself, the Go implementation is way faster than the Go implementation of xz which makes it a great choice going forward. If you want you can check out the performance benchmarks I made over here: https://github.com/erikgeiser/benchpress

Before submitting the PR, I'd like to ask if you would be fine with changing the Compressor setting to optionally include a level such as gzip:9, zstd:fastest or zstd:best. For gzip the level can be parsed with strconv.Atoi and zstd already provides an EncoderLevelFromString function. I would not support levels for xz and lzma as the settings are too complex. This way, users can better prioritize install speed vs. download speed or even build speed. If you do not like this, I'll submit a PR with a hard coded level for zstd (probably "best"/zstd.SpeedBestCompression as it's the closest to level 19 which is used by Fedora).

Subprocess failed. Error

tar2rpm -name "ccal" -version "0.0.1" -release "v0.6.9" -arch "x86_64" -licence "MIT" -prein "/usr/bin/ccal-single" -file "ccal.rpm" ccal.tar

sudo zypper in ccal.rpm

正在加载软件源数据...
正在读取已安装的软件包...
正在解决软件包依赖关系...

将安装以下 1 个新软件包:
  ccal

1 个软件包将新装.
总下载大小:0 B。已缓存:2.6 MiB。 操作完成后,将使用额外的 5.8 MiB。
继续吗? [y/n/v/...? 显示全部选项] (y): y
在缓存 ccal.rpm 中                                                                                (1/1),   2.6 MiB (解压后   5.8 MiB)

正在检查文件冲突: ..............................................................................................................[完毕]
(1/1) 正在安装:ccal-0.0.1-v0.6.9.x86_64 ........................................................................................[错误]
安装 ccal-0.0.1-v0.6.9.x86_64 失败:
错误: Subprocess failed. Error: RPM 失败: /var/tmp/rpm-tmp.UtwbaF: line 1: /usr/bin/ccal-single: No such file or directory
error: %prein(ccal-0:0.0.1-v0.6.9.x86_64) scriptlet failed, exit status 127
error: ccal-0:0.0.1-v0.6.9.x86_64: install failed

What should I do?

Ghost files are not properly supported in the RPM - file is still installed when installing the package

With reference to Maximum RPM:

The %ghost Directive

As we mentioned in the Section called The %files List, if a file is specified in the %files list, that file will automatically be included in the package. There are times when a file should be owned by the package but not installed - log files and state files are good examples of cases you might desire this to happen.

The way to achieve this, is to use the %ghost directive. By adding this directive to the line containing a file, RPM will know about the ghosted file, but will not add it to the package.

The examples there go on to show common RPM operations. Essentially, what needs to happen is that the file is owned by the RPM, but no file contents are added to the same.

The file should be:

  1. Shown in rpm -qlp <package.rpm
  2. Shown as provided-by the rpm rpm -q --whatprovides <my-ghost-file>
  3. Not present in the filesystem after installing the package.

1 & 2 are currently achieved, but 3 is not correct. We end up with an empty file, instead.

Support RPM filetypes

in https://github.com/goreleaser/nfpm there is the need to support at least the ConfigFile type https://github.com/sassoftware/go-rpmutils/blob/master/tags.go#L138 because when you uninstall a rpm by default a *.rpmsave file is created for all config files which nfpm verifies as part of it's testing.

Support for this was initially added as #16 but was asked to create an issue around this to determine if and how to support this for the tar2rpm tool

My initial idea was to not support it for the tool, and only for the case where rpmpack is being used as a library. Another option is to provide flags to point to a tar of each filetype. The final option I can think of is to have an option like -config-file=path/to/config/file that would have to match h.Name in https://github.com/google/rpmpack/blob/master/tar.go#L62 to set the file type

Of the 3 options I am personally leaning toward the latter so the command would look something like tar2rpm -config-file etc/bla.conf -config-file etc/bla/overide.conf -file bla.rpm bla.tar which makes it so you do not need to separately tar up each type of file. We would want to ensure wildcard support for this so you could do -config-file etc/*

Open Question: which file types do we want to support from the tool?

  • ConfigFile
  • DocFile
  • IconFile
  • MissingFile
  • NoReplaceFile
  • SpecFile
  • GhostFile
  • LicenceFile
  • ReadmeFile
  • ExcludeFile
  • UnpatchedFile
  • PubKeyFile
  • PolicyFile

I know ConfigFile for sure as the need for that prompted adding this feature. I would also say DocFile, LicenceFile, ReadmeFile

Owner and Group seem reversed

Hi,

is this inversion wanted or a typo (rpm.go l.375):

	r.fileowners = append(r.fileowners, f.Group)
	r.filegroups = append(r.filegroups, f.Owner)

Trying to create an RPM with rpmpack and then read it with go-rpmutils, the owner and group are switched

support writing %pre, %preun, %post, and %postun scriptlets

I would like to be able to do something like

package main

import (
	"log"
	"os"

	"github.com/google/rpmpack"
)

func main() {

	r, err := rpmpack.NewRPM(rpmpack.RPMMetaData{
		Name:    "rpmsample",
		Version: "0.1",
		Release: "A",
		Arch:    "noarch",
	})
	if err != nil {
		log.Fatal(err)
	}
	r.AddScriptlet(
		rpmpack.RPMFile{
			Name:  "pre",
                        Body: []byte(`
echo "-------------"
echo "This is pre"
echo "-------------"
`),
		})
	if err := r.Write(os.Stdout); err != nil {
		log.Fatalf("write failed: %v", err)
	}

}

support boolean dependencies

I am trying to add support for boolean dependencies
https://rpm-software-management.github.io/rpm/manual/boolean_dependencies.html

I see the dependency tags all defined @https://github.com/rpm-software-management/rpm/blob/ab01b5eacf9ec6a07a5d9e1991ef476a12d264fd/include/rpm/rpmds.h

And the rich parser defined
https://github.com/rpm-software-management/rpm/blob/9f62aba3bd26eea51d411f0d17ee4dadbf2fd873/lib/rpmds.c#L1480

based on
https://github.com/rpm-software-management/rpm/blob/9f62aba3bd26eea51d411f0d17ee4dadbf2fd873/lib/rpmds.c#L1126

I think I need to set add a dependency of rpmlib(DynamicBuildRequires)", "4.15.0-1" and set RPMSENSE_RPMLIB to all rpms that should support it, but not 100% sure how to go about that yet.

comparators ("senses") bitmasks in relations are not mapped correctly

It looks like the comparators (<, >, =) are not mapped correctly to their rpm bitmask representations:
Example input (through goreleaser/nfpm):

- "dep"
- "dep < 2.0.0"
- "dep > 1.0.0"
- "dep = 1.5.0" 
- "dep <= 1.6.0"
- "dep >= 1.4.0"

Result:

$ rpm -qpR dist/test.rpm
dep
dep  2.0.0
dep < 1.0.0
dep > 1.5.0
dep > 1.6.0
dep <> 1.4.0

According to the RPM source the correct bitmasks are (with my => decimal calculation):

RPMSENSE_ANY = 0, // => 0
RPMSENSE_LESS = (1 << 1), // => 2
RPMSENSE_GREATER = (1 << 2), // => 4
RPMSENSE_EQUAL = (1 << 3), // => 8

However the rpmpack bitmasks are:

SENSE_ANY = 1 << iota >> 1, // iota == 0 so => 0
SENSE_LESS // 1 << 1 >> 1 => 1
SENSE_GREATER // 1 << 2 >> 1 => 2
SENSE_EQUAL // 1 << 3 >> 1 => 4

it looks to me like overcomplicating the bitshift iota has caused the shifts to be off.


I have tested and verified that adjusting the "sense" bitshifts results in a correct rpm:

SenseAny  rpmSense = 0
SenseLess          = 1 << iota
SenseGreater
SenseEqua

Result:

dep 0
dep 2 2.0.0
dep 4 1.0.0
dep 8 1.5.0
dep 10 1.6.0
dep 12 1.4.0
...
$ rpm -qpR dist/test.rpm
dep 
dep < 2.0.0
dep > 1.0.0
dep = 1.5.0
dep <= 1.6.0
dep >= 1.4.0

No prefix support for generated rpm

Hello, team, this project looks not support prefix for the rpm package. So the rpm package is not relocatable. Guys, is there a plan to support this feature?

Unable to install the built rpm (without force)

Ive been trying to build an rpm using pkg_tar2rpm

when i go to install the rpm im seeing the following errors

Error: Transaction check error:
  file /opt from install of envoy-0:1.19.0-pre.noarch conflicts with file from package filesystem-3.8-2.el8.x86_64
  file /usr from install of envoy-0:1.19.0-pre.noarch conflicts with file from package filesystem-3.8-2.el8.x86_64
  file /usr/bin from install of envoy-0:1.19.0-pre.noarch conflicts with file from package filesystem-3.8-2.el8.x86_64

if i use rpm -i --force my.rpm it works - im wondering what im doing wrong (or not doing right) that is fooing the paths in the package

xz, lzma compression don't work

for xz, lzma but not gzip compression types, the output rpms are apparently corrupt:

rpmsample | rpm2cpio | cpio -t
cpio: premature end of archive

tested with rpm2cpio from rhel7 and upstream 4.16.1.3

Add support for `version_file` in `pkg_tar2rpm`

It would be very helpful if a version file was supported for setting the package version - as eg pkg_deb does

if a version file could be passed in it would allow pkg_tar2rpm to read version info directly from eg a VERSION file rather than having to duplicate/hardcode the current version in BUILD/bzl files in addition to the canonical setting

tar2rpm: rpm -ivh of generated .rpm fails with cpio: Digest mismatch

I have generated a valid, functioning TAR file with Apache Ant (file says: POSIX tar archive).
When I create an RPM file with tar2rpm there are neither errors or warnings reported but when the generated .rpm file should be installed with "rpm -ivh" the installation fails with:

error: unpacking of archive failed on file /redhat-release;5f2712a5: cpio: Digest mismatch
error: Example-0:1.0-.noarch: install failed

Steps to reprodude:

Use this Dockerfile:

FROM oraclelinux:7

RUN yum -y install oracle-golang-release-el7
RUN yum -y install golang git ant file

RUN go get -u github.com/google/rpmpack/...

RUN echo "<project name='test' default='main' basedir='.'>" >/tmp/build.xml && \
    echo "  <target name='main'>" >>/tmp/build.xml && \
    echo "    <tar destfile='/tmp/generated-by-ant.tar'>" >>/tmp/build.xml  && \
    echo "      <tarfileset dir='/etc' username='root' group='root'>" >>/tmp/build.xml && \
    echo "        <include name='redhat-release'/>" >>/tmp/build.xml && \
    echo "      </tarfileset>" >>/tmp/build.xml && \
    echo "    </tar>" >>/tmp/build.xml && \
    echo "  </target>" >>/tmp/build.xml&& \
    echo "</project>" >> /tmp/build.xml

RUN ant -f /tmp/build.xml
 
ENTRYPOINT ["bash"]

Build and run:

docker build -t rpmtest .
docker run --rm -it rpmtest

Run these commands in the Docker Container:

# Run file to determine the file type:
file /tmp/generated-by-ant.tar
# List the contents of the tar file (or extract it, it will work)
tar -tvf /tmp/generated-by-ant.tar
# Run the tar2rpm tool to generate the rpm
/root/go/bin/tar2rpm -file /tmp/example.rpm -name "Example" -version "1.0" --build_time "$(date +%s)" /tmp/generated-by-ant.tar

#Try to install the rpm
rpm -ivh /tmp/example.rpm

The last command will produce this error:

[root@e5a144d2b99f /]# rpm -ivh /tmp/example.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:Example-0:1.0-                   ################################# [100%]
error: unpacking of archive failed on file /redhat-release;5f2712a5: cpio: Digest mismatch
error: Example-0:1.0-.noarch: install failed
[root@e5a144d2b99f /]# 

[idea] API to allow libary users to add and support their own tags

Currently rpmpack must maintain a full list of tags and values itself so every time a library needs to support a tag that is not currently in rpmpack they must make a PR here and then update their use after the PR is accepted. So I am proposing reworking the API a bit to make it easier for those using rpmpack as a library to add support for the tags they need without rpmpack explicitly knowing about or officially supporting it.

Rough API Idea

func (r *RPM) AddTag(rpmTag int, value IndexEntry) {...}
func (r *RPM) AddSignatureTag(rpmTag int, value IndexEntry) {...}
func NewIndexEntry(value interface{}) IndexEntry {...}

Export fileType type and tag constants

In my opinion, the fileType type from file_types.go should be exported, so libraries can for example write wrappers around RPM.AddFiles() like this:

func addFileWrapper(rpm *rpmpack.RPM, ..., fileType rpmpack.FileType) error {
    ...
    rpm.AddFile(rpmpack.RPMFile{
        ...
        Type: fileType
    })
}

Similarly, the tag constants from tags.go should also be exported to allow libraries to use them when extracting and validating RPM properties in tests via these tags.

nfpm, which depends on rpmpack, currently has to employ an ugly workaround for fileType in such a wrapper and redefines many tag constants for tests.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.