nodejs / gyp-next Goto Github PK
View Code? Open in Web Editor NEWA fork of the GYP build system for use in the Node.js projects
License: BSD 3-Clause "New" or "Revised" License
A fork of the GYP build system for use in the Node.js projects
License: BSD 3-Clause "New" or "Revised" License
As discussed in #54 McCabe code complexity as measured by flake8 is at 101 when it should be more like 10.
The task here is to run flake8 --max-complexity=100
and ensure that the most complex functions have great test coverage and then refactor them to get their complexity score lower. Repeat this process with other complex functions by lowering the flake8 --max-complexity=xx
value.
I believe this was caused by #121
A copy action like the following results in a forward slashed path for the copy
command, and the system complains it cannot find the file specified:
{
"target_name": "copy_binary",
"type":"none",
"dependencies" : [ "build_binary"],
'actions': [
{
'action_name': 'copy_binary',
'inputs': ['<(module_root_dir)/build/<(CONFIGURATION_NAME)/binary.exe'],
'outputs': ['<(module_root_dir)/../build_app/binary_$(PlatformShortName).exe'],
'action': ['SET COPYCMD=/Y && copy', '<@(_inputs)', '<@(_outputs)'],
}
]
}
In the vcproj we can see:
<CustomBuild Include="C:\Users\alexc\code\redacted\client\www\electron\native\build\$(ConfigurationName)\binary.exe">
<FileType>Document</FileType>
<Command>call call SET COPYCMD=\Y && copy "C:/Users/alexc/code/redacted/client/www/electron/native/build/$(Configuration)/binary.exe" "C:/Users/alexc/code/redacted/client/www/electron/build_app/binary_$(PlatformShortName).exe"
if %errorlevel% neq 0 exit /b %errorlevel%</Command>
<Message>copy_binary</Message>
<Outputs>C:\Users\alexc\code\redacted\client\www\electron\build_app\binary_$(PlatformShortName).exe</Outputs>
</CustomBuild>
To me, reading the comment here does not match the added code:
# If the argument starts with a slash or dash, it's probably a command line
# switch
# Return the path with forward slashes because the command using it might
# not support backslashes.
arguments = [i if (i[:1] in "/-") else _FixPath(i, "/") for i in cmd[1:]]
I believe it should be something like:
arguments = [_FixPath(i, "/") if (i[:1] in "/-") else _FixPath(i) for i in cmd[1:]]
OR don't call FixPath at all if it's not a command line switch and enforce backslashes be used directly in the inputs/outputs.
Can it provide completions for common shells? I guess https://docs.iterative.ai/shtab/ or https://docs.iterative.ai/shtab/#alternatives can be useful. TIA!
% ./build.sh --python=python3 --gyp -v
run_scanbuild gyp -f ninja --depth=/Users/dkeeler/src/nss/nss --generator-output=. -Dpython=python3 -Dtarget_arch=arm64 -Denable_sslkeylogfile=1 -Dnss_dist_dir=/Users/dkeeler/src/nss/dist -Dnss_dist_obj_dir=/Users/dkeeler/src/nss/dist/Debug -Dnspr_include_dir=/Users/dkeeler/src/nss/dist/Debug/include/nspr -Dnspr_lib_dir=/Users/dkeeler/src/nss/dist/Debug/lib /Users/dkeeler/src/nss/nss/nss.gyp
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/bin/gyp", line 8, in <module>
sys.exit(script_main())
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/gyp/__init__.py", line 686, in script_main
return main(sys.argv[1:])
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/gyp/__init__.py", line 678, in main
return gyp_main(args)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/gyp/__init__.py", line 645, in gyp_main
[generator, flat_list, targets, data] = Load(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/gyp/__init__.py", line 111, in Load
os.path.join(output_dir, "out", default_variables["build_type"]),
KeyError: 'build_type'
0.12.1 seems to work fine, but anything after that fails with this error.
The project is currently licensed to Google Inc. using a BSD 3-Clause license. Ideally, we should be using MIT instead? Or atleast relicense it to new contributors? Anyway, we could also do this one file at a time. Or use a dual-license. Let's use this thread to brainstorm this.
Hi, folks.
I had the idea to test on upcoming Python versions before they become stable.
Current results (Python 3.13): I'm happy to share that Python 3.13 Alpha 2 and master
branch of gyp-next
seem to be working okay together, at least at the time I'm writing this:
Suggestion: If folks here want, I'm sure it wouldn't be too hard to modify the test matrix of the "Python tests" workflow to run against the prerelease Python version on a schedule, perhaps once weekly, or once monthly?
Let me know if this is something you are interested in, and I can look at the cleanest way to implement it into the workflow .yaml
for CI. (Easiest thing would be to just add 3.13
to the test matrix now, and test every PR/push against it, I suppose.)
I'm working on a module that helps with build/OS environment checking (similar to autoconf) and would like to have a standardized placement of the file that describes the checking to be done. Ideally the user would just stick the file in the same directory as the relevant gyp that utilizes it, but I'm not sure if the path to the current gyp/gypi (with or without filename included) is available in one way or another?
I've looked through the documentation but don't see anything obvious standing out.
We could use this GH action: https://github.com/pypa/gh-action-pypi-publish
Hey there!
I belong to an open source security research community, and a member (@dwisiswant0) has found an issue, but doesn’t know the best way to disclose it.
If not a hassle, might you kindly add a SECURITY.md
file with an email, or another contact method? GitHub recommends this best practice to ensure security issues are responsibly disclosed, and it would serve as a simple instruction for security researchers in the future.
Thank you for your consideration, and I look forward to hearing from you!
(cc @huntr-helper)
Linking to another issue where gyp-next
is used: nodejs/node-gyp#2485
But the problem is somewhere in the file path generation inside this repo.
{
'sources': [
'!(echo filename with space.cc)',
],
'libraries': [
'!@(pkg-config --libs-only-l apr-1)',
],
}
Miss <
before !
.
Our automated tests currently run ruff --max-complexity=101
when the recommended setting is 10
, not 101
. We do not need to reduce complexity all the way down to 10 but we should examine our worst offending functions and try to reduce their cyclomatic complexity.
Strong tests should be put in place before modifying these functions because at 101, we should expect severe spaghetti code with substantial corner cases and side effects. GenerateOutput()
is one function that is 739 lines long!
Please modify only one file per pull request given the difficulty of reviewing changes to older yet crucial code.
% flake8 --ignore=E203,W503 --max-line-length=88 --statistics --max-complexity=20
./gyp/pylib/gyp/generator/xcode.py:641:1: C901 'GenerateOutput' is too complex (101)
./gyp/pylib/gyp/generator/cmake.py:624:1: C901 'WriteTarget' is too complex (78)
./gyp/pylib/gyp/generator/ninja.py:2207:1: C901 'GenerateOutputForConfig' is too complex (55)
./gyp/pylib/gyp/input.py:759:1: C901 'ExpandVariables' is too complex (48)
./gyp/pylib/gyp/__init__.py:316:1: C901 'gyp_main' is too complex (45)
./gyp/pylib/gyp/generator/make.py:1571:5: C901 'MakefileWriter.WriteTarget' is too complex (40)
./gyp/pylib/gyp/generator/ninja.py:1349:5: C901 'NinjaWriter.WriteLinkForArch' is too complex (35)
./gyp/pylib/gyp/generator/make.py:2230:1: C901 'GenerateOutput' is too complex (34)
./gyp/pylib/gyp/xcode_emulation.py:571:5: C901 'XcodeSettings.GetCflags' is too complex (32)
./gyp/pylib/gyp/input.py:2529:1: C901 'ProcessListFiltersInDict' is too complex (27)
./gyp/pylib/gyp/generator/ninja.py:1057:5: C901 'NinjaWriter.WriteSourcesForArch' is too complex (25)
./gyp/pylib/gyp/generator/xcode.py:138:5: C901 'XcodeProject.Finalize1' is too complex (25)
./gyp/pylib/gyp/generator/ninja.py:378:5: C901 'NinjaWriter.WriteSpec' is too complex (24)
./gyp/pylib/gyp/MSVSNew.py:230:5: C901 'MSVSSolution.Write' is too complex (21)
./gyp/pylib/gyp/input.py:2974:1: C901 'Load' is too complex (21)
./gyp/pylib/gyp/generator/ninja.py:686:5: C901 'NinjaWriter.WriteRules' is too complex (21)
16 C901 'GenerateOutput' is too complex (101)
Coincidence: Our most complex function has a McCabe complexity of 101 and this is issue #101.
Blocked on nodejs/node-gyp#2645
It's permanently red, so I disabled it in https://github.com/nodejs/gyp-next/actions/workflows/node-gyp.yml
I think we should make separate workflow files so they can be restarted independently.
For Node.js:
tools/gyp
with the code from the PR/branch being testedFor node-gyp:
gyp
with the code from the PR/branch being testedI think release-please can do this.
The build tooling of this build tooling (ironically) is broken. A lot of files in the project exist as if gclient
and other Chromium tooling (like the commit queue) were being used in the project which is definitely not true.
I guess we should cleanup these and also set up new build tooling (Travis? Circle? AppVeyor?). @cclauss wdyt?
Getting this with node-gyp 8.3.0, which presumably has the patch from PR #124 to fix this.
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | win32 | x64
gyp info find Python using Python version 3.10.0 found at "C:\Code\Tools\Python3\python.exe"
gyp ERR! find VS
gyp ERR! find VS msvs_version not set from command line or npm config
gyp ERR! find VS VCINSTALLDIR not set, not running in VS Command Prompt
gyp ERR! find VS unknown version "undefined" found at "C:\Program Files\Microsoft Visual Studio\2022\Community"
gyp ERR! find VS could not find a version of Visual Studio 2017 or newer to use
https://pypi.org/project/gyp-next/ only has gyp-next 0.4.0, which is about 2 years old at this point. It would be great to have the more recent releases available there.
Take a look at the commit history for upstream (Google) gyp
: https://chromium.googlesource.com/external/gyp/+log/refs/heads/master
As I am writing this, upstream gyp
has updates as recently as the 12th of May 2020. These might fix bugs that we are still affected by. (I don't think the NodeJS or node-gyp copies of gyp
have been synced with upstream (Google) gyp
in quite some time, right? Maybe as long ago as 2015?)
Unexpectedly to me, there are many Python 3-related patches landed there in 2018, 2019 and 2020. About half of the commits in those years are Python 3 related. But there are other fixes as well.
Perhaps these are redundant to changes here, but perhaps not. I think it would be worthwhile seeing what has changed there, and if we could benefit from those things here/in the NodeJS world.
When installing a native module using node-gyp
, commands fail to run if there is a space in the path anywhere of the CWD.
A CWD like /home/abc/My Awesome Repo/project
will fail.
This has been discussed here: kelektiv/node.bcrypt.js#961
and here: nodejs/node-gyp#65
And has been found to be an underlying issue with gyp itself
https://github.com/nodejs/gyp-next/actions/runs/6560318675/job/17817683531
Run ruff --format=github .
ruff --format=github .
shell: /bin/bash -e {0}
env:
pythonLocation: /Users/runner/hostedtoolcache/Python/3.8.18/x64
PKG_CONFIG_PATH: /Users/runner/hostedtoolcache/Python/3.8.18/x64/lib/pkgconfig
Python_ROOT_DIR: /Users/runner/hostedtoolcache/Python/3.8.18/x64
Python[2](https://github.com/nodejs/gyp-next/actions/runs/6560318675/job/17817683531#step:6:2)_ROOT_DIR: /Users/runner/hostedtoolcache/Python/[3](https://github.com/nodejs/gyp-next/actions/runs/6560318675/job/17817683531#step:6:3).8.18/x6[4](https://github.com/nodejs/gyp-next/actions/runs/6560318675/job/17817683531#step:6:4)
Python3_ROOT_DIR: /Users/runner/hostedtoolcache/Python/3.8.18/x[6](https://github.com/nodejs/gyp-next/actions/runs/6560318675/job/17817683531#step:6:6)4
error: unexpected argument '--format' found
tip: to pass '--format' as a value, use '-- --format'
Usage: ruff check [OPTIONS] [FILES]...
For more information, try '--help'.
OK, so I think I've found a way of doing this without patching gyp-next (apologies I can't push a commit because I don't own this PR), but this seems to compile OK:
index aebb528..f75147f 100644
--- a/tools/v8_gypfiles/v8.gyp
+++ b/tools/v8_gypfiles/v8.gyp
@@ -796,6 +796,13 @@
],
}],
['OS=="win"', {
+ 'conditions': [
+ ['v8_target_arch=="arm64"', {
+ # Windows on Arm requires explicitly enabling the Microsoft
+ # Arm Assembler (marmasm) for this component.
+ 'msvs_enable_marmasm': 1
+ }]
+ ]
'sources': [
'<!@pymod_do_main(GN-scraper "<(V8_ROOT)/BUILD.gn" "\\"v8_base_without_compiler.*?is_win.*?sources \+= ")',
],
Originally posted by @richard-townsend-arm in nodejs/node#34337 (comment)
macOS 12.3.1
Python 3.8.9
node 14.19.3
gyp info using node@14.19.3 | darwin | x64
gyp verb command configure [ '-f', 'xcode' ]
gyp verb download using dist-url https://electronjs.org/headers
gyp verb find Python Python is not set from command line or npm configuration
gyp verb find Python Python is not set from environment variable PYTHON
gyp verb find Python checking if "python3" can be used
gyp verb find Python - executing "python3" to get executable path
gyp verb find Python - executable path is "/Applications/Xcode.app/Contents/Developer/usr/bin/python3"
gyp verb find Python - executing "/Applications/Xcode.app/Contents/Developer/usr/bin/python3" to get version
gyp verb find Python - version is "3.8.9"
gyp info find Python using Python version 3.8.9 found at "/Applications/Xcode.app/Contents/Developer/usr/bin/python3"
gyp verb get node dir compiling against --target node version: 17.1.1
gyp verb command install [ '17.1.1' ]
gyp verb download using dist-url https://electronjs.org/headers
gyp verb install input version string "17.1.1"
gyp verb install installing version: 17.1.1
gyp verb install --ensure was passed, so won't reinstall if already installed
gyp verb install version is already installed, need to check "installVersion"
gyp verb got "installVersion" 9
gyp verb needs "installVersion" 9
gyp verb install version is good
gyp verb get node dir target node version installed: 17.1.1
gyp verb build dir attempting to create "build" dir: /Users/lim/Projects/dvc-monorepo/tools/build
gyp verb build dir "build" dir needed to be created? Yes
gyp verb build/config.gypi creating config file
gyp verb build/config.gypi writing out config file: /Users/lim/Projects/dvc-monorepo/tools/build/config.gypi
gyp verb config.gypi checking for gypi file: /Users/lim/Projects/dvc-monorepo/tools/config.gypi
gyp verb common.gypi checking for gypi file: /Users/lim/Projects/dvc-monorepo/tools/common.gypi
gyp info spawn /Applications/Xcode.app/Contents/Developer/usr/bin/python3
gyp info spawn args [
gyp info spawn args '/Users/lim/.nvm/versions/node/v14.19.3/lib/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args 'binding.gyp',
gyp info spawn args '-f',
gyp info spawn args 'xcode',
gyp info spawn args '-I',
gyp info spawn args '/Users/lim/Projects/dvc-monorepo/tools/build/config.gypi',
gyp info spawn args '-I',
gyp info spawn args '/Users/lim/.nvm/versions/node/v14.19.3/lib/node_modules/node-gyp/addon.gypi',
gyp info spawn args '-I',
gyp info spawn args '/Users/lim/Library/Caches/node-gyp/17.1.1/include/node/common.gypi',
gyp info spawn args '-Dlibrary=shared_library',
gyp info spawn args '-Dvisibility=default',
gyp info spawn args '-Dnode_root_dir=/Users/lim/Library/Caches/node-gyp/17.1.1',
gyp info spawn args '-Dnode_gyp_dir=/Users/lim/.nvm/versions/node/v14.19.3/lib/node_modules/node-gyp',
gyp info spawn args '-Dnode_lib_file=/Users/lim/Library/Caches/node-gyp/17.1.1/<(target_arch)/node.lib',
gyp info spawn args '-Dmodule_root_dir=/Users/lim/Projects/dvc-monorepo/tools',
gyp info spawn args '-Dnode_engine=v8',
gyp info spawn args '--depth=.',
gyp info spawn args '--no-parallel',
gyp info spawn args '--generator-output',
gyp info spawn args 'build',
gyp info spawn args '-Goutput_dir=.'
gyp info spawn args ]
Traceback (most recent call last):
File "/Users/lim/.nvm/versions/node/v14.19.3/lib/node_modules/node-gyp/gyp/gyp_main.py", line 45, in <module>
sys.exit(gyp.script_main())
File "/Users/lim/.nvm/versions/node/v14.19.3/lib/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 662, in script_main
return main(sys.argv[1:])
File "/Users/lim/.nvm/versions/node/v14.19.3/lib/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 654, in main
return gyp_main(args)
File "/Users/lim/.nvm/versions/node/v14.19.3/lib/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 639, in gyp_main
generator.GenerateOutput(flat_list, targets, data, params)
File "/Users/lim/.nvm/versions/node/v14.19.3/lib/node_modules/node-gyp/gyp/pylib/gyp/generator/xcode.py", line 1336, in GenerateOutput
xct.AddDependency(xcode_targets[dependency])
File "/Users/lim/.nvm/versions/node/v14.19.3/lib/node_modules/node-gyp/gyp/pylib/gyp/xcodeproj_file.py", line 2690, in AddDependency
XCTarget.AddDependency(self, other)
File "/Users/lim/.nvm/versions/node/v14.19.3/lib/node_modules/node-gyp/gyp/pylib/gyp/xcodeproj_file.py", line 2389, in AddDependency
other_project_ref = pbxproject.AddOrGetProjectReference(other_pbxproject)[1]
File "/Users/lim/.nvm/versions/node/v14.19.3/lib/node_modules/node-gyp/gyp/pylib/gyp/xcodeproj_file.py", line 2991, in AddOrGetProjectReference
self._properties["projectReferences"] = sorted(
TypeError: '<' not supported between instances of 'builtin_function_or_method' and 'builtin_function_or_method'
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack at ChildProcess.onCpExit (/Users/lim/.nvm/versions/node/v14.19.3/lib/node_modules/node-gyp/lib/configure.js:259:16)
gyp ERR! stack at ChildProcess.emit (events.js:400:28)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:285:12)
gyp ERR! System Darwin 21.4.0
Related: nodejs/TSC#648.
If so, I want to add check to enhance this.
https://pypi.org/project/gyp-next/
Closed by #234
As part of the transfer we should asked that the following be added:
CODE_OF_CONDUCT.md
CONTRIBUTING.md
We need to add testing to gyp. Perhaps it'll be nice to start with unit tests?
@cclauss assigning this to you, feel free to unassign.
TL;DR this commit broke compilation of nodejs-mobile for iOS from an x64 macBook. See PR nodejs-mobile/nodejs-mobile#9
(I'm not sure if it's good practice to comment on old PRs, so feel free to tell me to move this discussion elsewhere.)
I maintain nodejs-mobile lately, and I'm updating our fork from Node 12.x to Node 16.x. Since we're compiling node.js for iOS, we want the arch
to be arm64
, but the host clang has flags like __amd64__
and this mixed with the iPhoneOS SDK headers ends up causing an Unsupported architecture
compilation error.
Maybe a solution would be to patch this gyp-next script to detect whether we're running in arm64 darwin (such checks are common throughout the codebase, I've noticed) and then use the new logic. Else, use the old logic.
What do you think?
Originally posted by @staltz in #78 (comment)
As discussed in #54, flake8 --max-line-length=88
currently find 60+ lines in Python files that are longer than 88 characters. psf/black will not auto-fix most of these issues so it is a manual job to wrap these lines at 88 chars max without breaking any functionality or tests.
The cp
option -a
in copy_cmd fails on Solaris. Currently npm install
of a binary add-on gives:
cp: illegal option -- a
Usage: cp [-f] [-i] [-p] [-@] [-/] f1 f2
cp [-f] [-i] [-p] [-@] [-/] f1 ... fn d1
cp -r|-R [-H|-L|-P] [-f] [-i] [-p] [-@] [-/] d1 ... dn-1 dn
A work around is to use node-gyp configure
, edit build/Makefile and change the cp
syntax to use -pPR
. Then run node-gyp build
:
bash-4.4$ diff build/Makefile build/Makefile.bak
134c134
< cmd_copy = rm -rf "$@" && cp -pPR "$<" "$@"
---
> cmd_copy = rm -rf "$@" && cp -af "$<" "$@"
Also see nodejs/node-gyp#562 (comment)
npm rebuild
in Android/termux meet this bug. However I cannot found android_ndk_path
in gyp's code repo. How to solve it? TIA!
"gyp" is taken on PyPI (https://pypi.org/project/gyp/). I guess it would make sense to contact the uploader/Chromium folks/Google OSPO about this.
/cc @cclauss
Add a helper script to git-node
in node-core-utils
for updating gyp in-tree exactly like V8.
We're getting this pretty regularly in node-gyp CI: https://travis-ci.com/github/nodejs/node-gyp/jobs/339883554
The command "npm test" exited with 0.
10714.02s$ GYP_MSVS_VERSION=2015 GYP_MSVS_OVERRIDE_PATH="C:\\Dummy" python -m pytest
1072============================= test session starts ==============================
1073platform linux -- Python 3.5.6, pytest-4.6.6, py-1.7.0, pluggy-0.13.1
1074rootdir: /home/travis/build/nodejs/node-gyp
1075collected 27 items
1076
1077gyp/pylib/gyp/MSVSSettings_test.py F.F..... [ 29%]
...
I don't really know how to interpret the log output that follows, but it's been consistent. I can restart it in Travis and it should pass again, this shows up in maybe 50% of Travis runs and it seems to be most common on Node.js 12 + Python 3.5, but I don't have numbers to back that up.
Hi!
Just a quick question, but node-gyp is really good at finding the right location of the msbuild.exe to use when compiling. For some very specific compilation stuff we do with libsodium we also need to find it, and have so far been duplication the logic in sodium-native for that.
This made me wonder, is there a CLI command or something equivalent to print the location of msbuild.exe so we can just re-use the great logic already in this tool?
I noticed if I try any kind of expansion (e.g. command or variable) in 'includes'
list items, they don't get expanded or replaced but gyp instead tries to use them literally.
Am I doing something wrong, is this intentional, or is it a bug?
Mainly to fix nodejs/node-v8#161 (comment)
In short: Windows .obj files generated in flattened directory causing linker errors
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.