cannot compile nokogiri about gem-compiler HOT 23 CLOSED

@wtnb75 seems the problem of nokogiri, as it removes files that were part of the gem specification after the compilation:

https://github.com/sparklemotion/nokogiri/blob/v1.6.6.2/ext/nokogiri/extconf.rb#L63-L68

And those files were part of the original gem specification but no longer exist after extconf.rb has executed:

$ gem spec nokogiri-1.6.6.2.gem files | grep ports
- ports/archives/libxml2-2.9.2.tar.gz
- ports/archives/libxslt-1.1.28.tar.gz
- ports/patches/libxml2/0001-Revert-Missing-initialization-for-the-catalog-module.patch
- ports/patches/libxml2/0002-Fix-missing-entities-after-CVE-2014-3660-fix.patch
- ports/patches/libxslt/0001-Adding-doc-update-related-to-1.1.28.patch
- ports/patches/libxslt/0002-Fix-a-couple-of-places-where-f-printf-parameters-wer.patch
- ports/patches/libxslt/0003-Initialize-pseudo-random-number-generator-with-curre.patch
- ports/patches/libxslt/0004-EXSLT-function-str-replace-is-broken-as-is.patch
- ports/patches/libxslt/0006-Fix-str-padding-to-work-with-UTF-8-strings.patch
- ports/patches/libxslt/0007-Separate-function-for-predicate-matching-in-patterns.patch
- ports/patches/libxslt/0008-Fix-direct-pattern-matching.patch
- ports/patches/libxslt/0009-Fix-certain-patterns-with-predicates.patch
- ports/patches/libxslt/0010-Fix-handling-of-UTF-8-strings-in-EXSLT-crypto-module.patch
- ports/patches/libxslt/0013-Memory-leak-in-xsltCompileIdKeyPattern-error-path.patch
- ports/patches/libxslt/0014-Fix-for-bug-436589.patch
- ports/patches/libxslt/0015-Fix-mkdir-for-mingw.patch
- ports/patches/sort-patches-by-date

gem-compiler cannot rebuild a gem when the original files defined in the gemspec are no longer available.

Please check nokogiri documentation and reach out nokogiri developers about the issue, which is not gem-compiler's issue.

Cheers.

from gem-compiler.

Hit the same issue.

from gem-compiler.

@jk-shah and have the same answer, this is a Nokogiri problem that should be reported upstream, not here.

Cheers.

from gem-compiler.

@jk-shah thank you!

from gem-compiler.

In the meanwhile can I break the step after it untars nokogiri it so that I can change their code before the process as a temporary hack?

from gem-compiler.

just monkey patch it?

from gem-compiler.

@jk-shah 😕

Why not introduce a flag to gem-compiler that tells it to purge/sanitize the gem specification during re-packaging?

Example UX:

$ gem compile <gemname.gem> --purge-package

(Or --clean-files or --clean-package, whatever fits better)

Then you introduce a check for that flag and removes the reference to files that no longer exist.

I like that approach better than hacks, which I'm not sure where those could be introduced.

Feel free to send a pull request for this change.

from gem-compiler.

@mpapis I'm not sure where the monkey patch fits into the equation as gem-compiler unpacks, compiles and re-pack the gem, so see above comment for an alternative, which I'm open to accept pull requests for.

from gem-compiler.

@luislavena It's what I meant, maybe with a temporary workaround in form of patching ;)

from gem-compiler.

Places where this can be performed are here, after collecting artifacts and removal of extensions:

https://github.com/luislavena/gem-compiler/blob/master/lib/rubygems/compiler.rb#L41

You will also need to introduce the extra flags that are passed to the compiler from the command:

https://github.com/luislavena/gem-compiler/blob/master/lib/rubygems/commands/compile_command.rb

from gem-compiler.

Actually I patched nokogiri code and built a custom gem and used gem-compiler on it and there were no complains and go a binary gem.

All good except when the app tries to install it it still goes for the source gem and not the binary gem.
I am not familiar with the gem world but does gem install need to be used differently for these binary gems?

UPDATE: Got to install it with full name.

from gem-compiler.

@jk-shah can you push the fix upstream?

from gem-compiler.

Done: sparklemotion/nokogiri#1275 though it will
require review as I did a quick fix to make my case work.

On Sat, Apr 11, 2015 at 3:27 PM, Michal Papis [email protected]
wrote:

@jk-shah https://github.com/jk-shah can you push the fix upstream?

—
Reply to this email directly or view it on GitHub
#12 (comment)
.

from gem-compiler.

Ref: sparklemotion/nokogiri#1275

from gem-compiler.

I opened a PR with @luislavena's suggestion and named the flag --purge-package.

from gem-compiler.

I think we need to add more behaviour. With this flag Nokogiri compiles but after re-packaging the lib directory won't be inside. Looking at their repository, they don't provide a gemspec. Thoughts?

from gem-compiler.

It doesn't matter if nokogiri provides a gemspec in their repository.

Gem-compiler uses the gemspec of the gem that just extracted.

I haven't looked at your code, but I think the test is not properly
checking what really is going on.

I will suggest you look at the code you added that remove files, perhaps
will be better reject spec files after the extensions have been compiled.

Sorry for top posting. Sent from mobile.
On Apr 26, 2015 1:19 AM, "Andrés N. Robalino" [email protected]
wrote:

I think we need to add more behaviour. With this flag Nokogiri compiles
but after re-packaging the lib directory won't be inside. Looking at their
repository, they don't provide a gemspec. Thoughts?

Reply to this email directly or view it on GitHub
#12 (comment)
.

from gem-compiler.

@luislavena Before adding the flag feature the test fails as expected (Indicates the files originally in specification are missing).

There was a bug in the logic that didn't remove the files appropriately during re-packaging, which is fixed now.

from gem-compiler.

Just an FYI that this commit upstream in Nokogiri should address the underlying issue:

sparklemotion/nokogiri@20d7591

from gem-compiler.

Until a new version of the gem with the fix is pushed out you can build it from source.

git clone https://github.com/sparklemotion/nokogiri.git
cd nokogiri
bundle install
rake native gem

Then the compiled gem is in the pkg/ directory

from gem-compiler.

Seeing the same :(

$ gem compile nokogiri-1.6.7.2.gem
Unpacking gem: 'nokogiri-1.6.7.2' in temporary directory...
Building native extensions.  This could take a while...
WARNING:  See http://guides.rubygems.org/specification-reference/ for help
ERROR:  While executing gem ... (Gem::InvalidSpecificationException)
    ["ports/archives/libxml2-2.9.2.tar.gz", "ports/archives/libxslt-1.1.28.tar.gz"] are not files

Will follow your advice drampelt, thank you.

from gem-compiler.

@xanview, thanks for your note.

I've opened a new bug with nokogiri: sparklemotion/nokogiri#1436

from gem-compiler.

@xanview @flavorjones

To deal with removal of files form the original package, --prune option was introduced.

$ gem list gem-compiler

*** LOCAL GEMS ***

gem-compiler (0.4.0)

$ gem fetch nokogiri
Downloaded nokogiri-1.6.7.2

$ gem compile nokogiri-1.6.7.2.gem --prune
Unpacking gem: 'nokogiri-1.6.7.2' in temporary directory...
Building native extensions.  This could take a while...
... (supressed RubyGems warnings) ....
  Successfully built RubyGem
  Name: nokogiri
  Version: 1.6.7.2
  File: nokogiri-1.6.7.2-x86_64-linux.gem

$ ls -l -h nokogiri*
-rw-rw-r-- 1 luis luis 8,9M feb 29 10:36 nokogiri-1.6.7.2.gem
-rw-rw-r-- 1 luis luis 1,5M feb 29 10:38 nokogiri-1.6.7.2-x86_64-linux.gem

from gem-compiler.