Comments (11)
CentOS7, which is the base distro for manylinux2014
, is the reason pytorch cannot use CXX11_ABI. Moving to manylinux2014
is easy and makes sense since the older distros are EOL, but I am not sure what the tangible benefits are for pytorch.
For a nice summary of glibc versions vs. distros, along with EOL dates, there is https://github.com/mayeut/pep600_compliance
from builder.
@malfet's feedback was: let's upgrade to manylinux2014
for now - CentOS 7 is still around, so let's do that first. manylinux1
to manylinux2014
upgrade is valuable.
from builder.
Copying selected comments from pypa/manylinux#1012, which discusses the fact that manylinux_2_24
is based on Ubuntu, which does not recompile newer gcc versions for the libstdc++
needed for manylinux_2_24
compliance:
The GCC version in
manylinux_2_24
is a dismal 6.3, much worse than both the year based manylinuxes and not even new enough for C++17 language support (mostly added in GCC 7)! This is a huge step backward for an image with a higher GLIBC version.
Once you install gcc-9, it will update at least libstdc++ which is a no go. RHEL dev toolset took care of that and also other libraries like libgcc_s so that binaries produced with devtoolset would still be compatible with the base image with no action.
Bottom line: in order to use the manylinux_2_24
standard for wheels (which would enable using CXX11_ABI
), someone needs to put in the effort to build a gcc-9 or gcc-10 for manylinux_2_24
.
from builder.
To add to this:
- currently we still have
manylinux1
wheels - that meansglibc
2.5 (2006) - upgrading to
manylinux2014
is much easier - that would beglibc
2.17 (2012) manylinux_2_24
givesglibc
2.24 (2016)
@mattip you said 2.5 -> 2.17 was much less interesting than 2.17 -> 2.24. Can you explain why that is? Newer is better, but getting rid of manylinux1
would also be interesting I'd think?
from builder.
CentOS 7 is old but is not EOL. It will be supported till June 30th, 2024. It's just it will not have new features and will not add support for new hardware. But it is not a problem for build pipelines. Usually we run it in docker so it doesn't matter if its kernel can support the latest CPU. And for release managers, we should always target the oldest distro as possible as we can.
But if PyTorch team is willing to add a new set of packages for manylinux_2_24, it is absolutely great.
from builder.
I have one additional comment: if pytorch decided to build manylinux2014, then you need to replace the GCC from 9.x to 8.x , as long as you still need to support CUDA 10.2.
from builder.
let's upgrade to manylinux2014 for now
@malfet: is that "stop creating linux86_64 and create only manylinux2014 instead" or "add manylinux2014 wheels in addition to the linux86_64 wheels"
from builder.
let's upgrade to manylinux2014 for now
@malfet: is that "stop creating linux86_64 and create only manylinux2014 instead" or "add manylinux2014 wheels in addition to the linux86_64 wheels"
From looking at build_all_docker.sh, one would think that manylinux2014
is now the default, but the other one appears to be being built due to a possible MANYLINUX_VERSION
vs MANY_LINUX_VERSION
variable name variation, per update 2 in this other issue
from builder.
Yes, that does look like a typo. Are the wheels being built? xref #979
from builder.
Yes, that does look like a typo. Are the wheels being built? xref #979
Honestly, I'm having trouble building the wheels inside either container 'the right way' because I'm not sure which scripts meta uses to build them inside the container. The documentation only points to how to build the docker container image.
Running manywheel/build.sh
results in an error complaining the gcc version in the original Docker image is too old.
Running said script with the 2014 docker image results in an openssl so file not being found.
Inside the container, used the following command in the folder where pytorch builder is checked out where /mnt/pytorch-1.11.0-cuda-11.3-src
is the path to the pytorch source checkout:
GPU_ARCH_TYPE=cuda OVERRIDE_TORCH_CUDA_ARCH_LIST=3.5 DESIRED_CUDA=11.3 DESIRED_PYTHON=3.9 PYTORCH_ROOT=/mnt/pytorch-1.11.0-cuda-11.3-src manywheel/build.sh
from builder.
@malfet: is that "stop creating linux86_64 and create only manylinux2014 instead" or "add manylinux2014 wheels in addition to the linux86_64 wheels"
This was definitely "drop manylinux1
, upgrade existing jobs to manylinux2014
".
from builder.
Related Issues (20)
- `numpy v2.0.0rc1` on pytorch/ conda channel problem HOT 6
- Deprecate LEGACY_DOCKER_IMAGE
- Docker Images Validate. Fix arm64 docker builds to not contain cuda versions HOT 1
- Build cuda:12.4.0-cudnn8-devel-ubuntu22.04 docker image and host it in pytorch AWS HOT 3
- OOM error in building ARM wheel
- Triton build is failing from time to time with error on python 3.12 with Exit code 137
- Move binary Docker manywheel and conda build workflows and scripts to pytorch/pytorch repo
- Unable to build docker image from Builder Repo HOT 4
- Are cuda aarch64 build available anywhere? HOT 3
- Move release and analytics workflows scripts to test-infra
- Move magma builds from builder to test-infra
- Move all validation workflows and scripts from builder to test-infra
- [BE] More code-reuse across tests HOT 1
- CentOS Docker failures: mirrorlist.centos.org no longer online HOT 1
- Unable to build Docker image with release/2.4 branch HOT 1
- Packaging for libcufile
- Extend validation framework with additional tests HOT 1
- Cannot install scikit-learn from conda-forge channel alongside pytorch from the pytorch channel
- Run builder/test/smoke_test.py on nightly builds
- Support aarch64 and aarch64 conda builds
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from builder.