hadrieng2 / hwlocality Goto Github PK
View Code? Open in Web Editor NEWRust bindings to Open MPI Portable Hardware Locality "hwloc" library, covering version 2.0 and above.
License: MIT License
Rust bindings to Open MPI Portable Hardware Locality "hwloc" library, covering version 2.0 and above.
License: MIT License
hwloc provides interoperability with a bunch of GPU/heterogeneous system APIs. The amount of support code needed for each is small, but each will need to get a dedicated feature and suitable cfg+doc(cfg)+cfg(doc) annotations.
Also, testing is going to be problematic since CI nodes don't have GPUs and I don't even have local setups for some of these APIs.
I am trying to get GitHub actions and container image builds to succeed and looks like documentation right now is not quite accurate in terms of dependencies necessary.
About Linux it current says:
In addition to a valid C build environment, this requires autotools on Unices and CMake on Windows
While I noticed that for Ubuntu automake
,libtool
and pkg-config
are all necessary for compilation.
Didn't get to macOS yet, but I expect similar surprises there.
Would be great to make documentation more exhaustive regarding dependencies.
UPD: libtool
and pkg-config
are also needed on macOS.
On Alpine Linux, pthread_t
type is mismatched between libc
and std
.
ThreadId
= libc::pthread_t
's type is *mut libc::c_void
!std::os::unix::thread::JoinHandleExt::as_pthread_t
's return type is RawPthread
= u64
.docker run --rm -it alpine:3.19 # Digest: x86_64 => sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0
# In the docker container, we'll do the rest
apk add cargo git hwloc-dev eudev-dev
git clone "https://github.com/HadrienG2/hwlocality.git"
cd hwlocality
cargo test --all # Error!
Compiling hwlocality-sys v0.4.0 (/hwlocality/hwlocality-sys)
Compiling hwlocality v1.0.0-alpha.1 (/hwlocality)
error[E0277]: `*mut c_void` doesn't implement `std::fmt::Display`
--> src/cpu/binding.rs:805:62
|
805 | Self::Thread(id) => format!("the thread with TID {id}"),
| ^^^^ `*mut c_void` cannot be formatted with the default formatter
|
= help: the trait `std::fmt::Display` is not implemented for `*mut c_void`
= note: in format strings you may be able to use `{:?}` (or {:#?} for pretty-print) instead
= note: this error originates in the macro `$crate::__export::format_args` which comes from the expansion of the macro `format` (in Nightly builds, run with -Z macro-backtrace for more info)
error[E0308]: mismatched types
--> src/lib.rs:288:9
|
286 | fn as_thread_id(&self) -> ThreadId {
| -------- expected `*mut c_void` because of return type
287 | use std::os::unix::thread::JoinHandleExt;
288 | self.as_pthread_t()
| ^^^^^^^^^^^^^^^^^^^ expected `*mut c_void`, found `u64`
|
= note: expected raw pointer `*mut c_void`
found type `u64`
Some errors have detailed explanations: E0277, E0308.
For more information about an error, try `rustc --explain E0277`.
error: could not compile `hwlocality` (lib) due to 2 previous errors
warning: build failed, waiting for other jobs to finish...
error[E0277]: `*mut libc::c_void` doesn't implement `std::fmt::Display`
--> src/cpu/binding.rs:805:62
|
805 | Self::Thread(id) => format!("the thread with TID {id}"),
| ^^^^ `*mut libc::c_void` cannot be formatted with the default formatter
|
= help: the trait `std::fmt::Display` is not implemented for `*mut libc::c_void`
= note: in format strings you may be able to use `{:?}` (or {:#?} for pretty-print) instead
= note: this error originates in the macro `$crate::__export::format_args` which comes from the expansion of the macro `format` (in Nightly builds, run with -Z macro-backtrace for more info)
error[E0308]: mismatched types
--> src/lib.rs:288:9
|
286 | fn as_thread_id(&self) -> ThreadId {
| -------- expected `*mut libc::c_void` because of return type
287 | use std::os::unix::thread::JoinHandleExt;
288 | self.as_pthread_t()
| ^^^^^^^^^^^^^^^^^^^ expected `*mut c_void`, found `u64`
|
= note: expected raw pointer `*mut libc::c_void`
found type `u64`
error: could not compile `hwlocality` (lib test) due to 2 previous errors
If we cast the type ThreadId
to *mut libc::c_void
, then all examples would be failed because ThreadId
is not thread-safe.
So I think the plausible option is: type ThreadId = u64
.
Expose hwloc's CPU core kinds, used to handle heterogeneous systems like ARM big.LITTLE and Intel Adler Lake:
In principle, it is possible to extend hwloc by writing plugins for it. However, the compile-time constant used to evaluate ABI compatibility is a header define, which means I can't check it from Rust, so I'm not sure if exposing this from the Rust side is a good idea. It will probably require at least a little build.rs hackery to get back the define value into Rust.
In any case, here's the docs:
I have a function that looks like this:
pub fn example(mut tp: hwlocality::topology::Topology) {
let cpuset = tp.cpuset();
for index in cpuset.iter_set().map(usize::from).collect::<Vec<_>>() {
println!("cpu {}", index);
}
}
I call it like this:
example(tp.clone());
Here is a backtrace:
#0 0x00005555556bc6e5 in hwloc_bitmap_next (set=0x0, prev_cpu=0) at /home/thomas/hwlocality/hwloc/hwloc/bitmap.c:1402
#1 0x00005555556edbe3 in hwlocality::bitmaps::{impl#0}::next_set::{closure#0} (bitmap=0x0, prev=0) at src/bitmaps/mod.rs:868
#2 0x00005555556edaed in hwlocality::bitmaps::Bitmap::next<hwlocality::bitmaps::{impl#0}::next_set::{closure_env#0}> (self=0x7fffffffce18, index=..., next_fn=...)
at src/bitmaps/mod.rs:854
#3 0x00005555556a0d46 in hwlocality::bitmaps::Bitmap::next_set (self=0x7fffffffce18, index=...) at src/bitmaps/mod.rs:867
#4 0x00005555556678d2 in hwlocality::bitmaps::{impl#32}::next<&hwlocality::bitmaps::Bitmap> (self=0x7fffffffcd90)
at /home/thomas/hwlocality/src/bitmaps/mod.rs:1189
#5 0x0000555555615fe3 in core::iter::adapters::map::{impl#2}::next<usize, hwlocality::bitmaps::BitmapIterator<&hwlocality::bitmaps::Bitmap>, fn(hwlocality::bitmaps::indices::BitmapIndex) -> usize> (self=0x7fffffffcd90) at /rustc/8c74a5d27c644a0f7a22bb2fa8dd3ff8257bc220/library/core/src/iter/adapters/map.rs:103
#6 0x0000555555686c19 in alloc::vec::Vec<usize, alloc::alloc::Global>::extend_desugared<usize, alloc::alloc::Global, core::iter::adapters::map::Map<hwlocality::bitmaps::BitmapIterator<&hwlocality::bitmaps::Bitmap>, fn(hwlocality::bitmaps::indices::BitmapIndex) -> usize>> (self=0x7fffffffcd30, iterator=...)
at /rustc/8c74a5d27c644a0f7a22bb2fa8dd3ff8257bc220/library/alloc/src/vec/mod.rs:2795
#7 0x00005555556895da in alloc::vec::spec_extend::{impl#0}::spec_extend<usize, core::iter::adapters::map::Map<hwlocality::bitmaps::BitmapIterator<&hwlocality::bitmaps::Bitmap>, fn(hwlocality::bitmaps::indices::BitmapIndex) -> usize>, alloc::alloc::Global> (self=0x7fffffffcd30,
iter=<error reading variable: Cannot access memory at address 0x0>) at /rustc/8c74a5d27c644a0f7a22bb2fa8dd3ff8257bc220/library/alloc/src/vec/spec_extend.rs:17
#8 0x0000555555685ddd in alloc::vec::spec_from_iter_nested::{impl#0}::from_iter<usize, core::iter::adapters::map::Map<hwlocality::bitmaps::BitmapIterator<&hwlocality::bitmaps::Bitmap>, fn(hwlocality::bitmaps::indices::BitmapIndex) -> usize>> (iterator=...)
at /rustc/8c74a5d27c644a0f7a22bb2fa8dd3ff8257bc220/library/alloc/src/vec/spec_from_iter_nested.rs:43
#9 0x000055555568962d in alloc::vec::spec_from_iter::{impl#0}::from_iter<usize, core::iter::adapters::map::Map<hwlocality::bitmaps::BitmapIterator<&hwlocality::bitmaps::Bitmap>, fn(hwlocality::bitmaps::indices::BitmapIndex) -> usize>> (iterator=<error reading variable: Cannot access memory at address 0x0>)
at /rustc/8c74a5d27c644a0f7a22bb2fa8dd3ff8257bc220/library/alloc/src/vec/spec_from_iter.rs:33
#10 0x0000555555689575 in alloc::vec::{impl#14}::from_iter<usize, core::iter::adapters::map::Map<hwlocality::bitmaps::BitmapIterator<&hwlocality::bitmaps::Bitmap>, fn(hwlocality::bitmaps::indices::BitmapIndex) -> usize>> (iter=...) at /rustc/8c74a5d27c644a0f7a22bb2fa8dd3ff8257bc220/library/alloc/src/vec/mod.rs:2695
#11 0x000055555561626d in core::iter::traits::iterator::Iterator::collect<core::iter::adapters::map::Map<hwlocality::bitmaps::BitmapIterator<&hwlocality::bitmaps::Bitmap>, fn(hwlocality::bitmaps::indices::BitmapIndex) -> usize>, alloc::vec::Vec<usize, alloc::alloc::Global>> (
self=<error reading variable: Cannot access memory at address 0x0>)
at /rustc/8c74a5d27c644a0f7a22bb2fa8dd3ff8257bc220/library/core/src/iter/traits/iterator.rs:1895
#12 0x000055555562a9dd in benchmark::cpuload::example (tp=...) at src/cpuload.rs:88
#13 0x000055555568a7ce in benchmark::main () at src/main.rs:45
Expose hwloc's memory attributes, used to discriminate fast memory from slow memory on platforms with multiple memory tiers (HBM, DDR, NVRAM...).
Expose windows specific APIs behind suitable cfg flags with associated doc(cfg)/cfg(doc): https://hwloc.readthedocs.io/en/v2.9/group__hwlocality__windows.html
Hi,
I'm trying to bind and migrate all of my thread's memory to a numanode, but I'm getting an unsupported feature error despite my system supporting it.
assertion failed: support::MemoryBindingSupport::default().set_current_thread() == true
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
$ pacman -Q | grep hwloc
hwloc 2.10.0-1
$ hwloc-info --support | grep thisthread_membind
membind:set_thisthread_membind = 1
membind:get_thisthread_membind = 1
How should I go about binding and migrating my thread's memory to the desired numa node?
I'm using main in the above test with the latest feature gate.
hwlocality = { git="https://github.com/HadrienG2/hwlocality", branch="main", features=["hwloc-latest"]}
P.S. - this library is awesome, makes setting up locality a breeze. Thanks for your hard work :>
Expose hwloc interoperability with the OpenFabrics networking library (infiniband etc), add suitable feature+cfg+doc(cfg)+cfg(doc): https://hwloc.readthedocs.io/en/v2.9/group__hwlocality__openfabrics.html
Supply chain attacks are real, it is always better to download a known good commit hash rather than moving target
Expose the hwloc functionality for telling how "close" or "far away" two topology objects are:
To avoid issues like #114 in the future, I should add a musl build to CI. Hopefully replicating the dockerfile listed there, but with clippy lints and maybe tests on latest hwloc + rustc, should be enough to catch the obvious stuff.
The bundled
feature is currently hardcoded to error out on macOS (and this combination is disabled in CI) because if I let such a build run through, it will bomb at binary linking time due to the fact that for some unknown reason, the resulting internal hwloc build does not correctly set the -framework
flags that it needs in its pkgconfig file.
I do not have a Mac around, so I cannot investigate this on my side. Patches from Mac users are welcome!
While I have commented on several other issues about what's needed before a final v1 release, I just realized that a single issue that centralizes this knowledge would make this info more visible, so here goes.
Before releasing v1.0.0 of hwlocality and expending more energy on advertising this project to the world, I want to...
Result
....and I think that's it. Will expand if anything else comes to mind.
Expose libc::sched_(get|set)affinity
interoperability behind suitable cfg flags with associated doc(cfg)/cfg(doc): https://hwloc.readthedocs.io/en/v2.9/group__hwlocality__glibc__sched.html
The way configuration of hwloc works by default is that it looks at some of the system libraries (that is my assumption anyway) and based on what it find, requires extra dependencies.
For example in some cases it needs cudart
and in others it doesn't.
My use case is to only know CPU topology and to be able to pin threads to a set of CPU cores, so all of the extra stuff just makes it harder to build.
At the same time configuration of hwloc has some options to disable CUDA and other things, which I think might have been helpful to expose. For example build minimal version by default and require vendored-full
or something for default build with complete feature-set (just because features in Cargo are additive).
WDYT?
I don't see a hwlocality crate on crates.io.
hwloc tagged their 2.10 release a couple days ago.
The changelog is pretty short, and not super explicit about the new heterogeneous memory stuff, but a quick diff of the include directory suggests that the affected API surface is minimal and will be easy to wrap. The big change is removal of netloc, which hwlocality never supported to begin with.
I'm waiting for openSUSE Tumbleweed to package this release before getting started.
Latest version of hwlocality-sys
pulls in reqwest
, which by default pulls OpenSSL, which means OpenSSL is now an installation requirement, which is a huge dependency with very bad implication in terms of developer experience for Rust developers.
It is possible to opt-in into vendored version of OpenSSL, but reqwest
also supports rustls-tls
, as an alternative.
Consider disabling reqwest
's default features and using rustls-tls
feature instead.
Expose Linux-specific API functions and libnuma interoperability behind suitable cfg flags with associated doc(cfg)/cfg(doc):
libnuma support should also be gated behind a suitable feature, and will depend on the availability of a minimally working binding.
There is some kind of test test that in my project (when building for a specific target with -Z build-std
) exceeds path limit:
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(382,5): error MSB3491: Could not write lines to file "hwloc_get_largest_objs_inside_cpuset.dir\Release\hwloc_ge.C408B399.tlog\hwloc_get_largest_objs_inside_cpuset.lastbuildstate". Path: hwloc_get_largest_objs_inside_cpuset.dir\Release\hwloc_ge.C408B399.tlog\hwloc_get_largest_objs_inside_cpuset.lastbuildstate exceeds the OS max path limit. The fully qualified file name must be less than 260 characters. [C:\actions-runner_work\subspace\subspace\target\x86_64-pc-windows-msvc\production\build\hwlocality-sys-aa1805acd0c31c09\out\build\tests\hwloc_get_largest_objs_inside_cpuset.vcxproj]
Do you think it is possible to shorten it?
Also I'm wondering why tests are built when I just use it as a library dependency.
UPD: I have a fix, will submit a bit later.
Expose hwloc's support for sharing topologies between processes: https://hwloc.readthedocs.io/en/v2.9/group__hwlocality__shmem.html
Expose the hwloc functionality for exporting topologies to two textual formats:
Will enable me to drop the once_cell dependency.
Without hacking, windows CI is broken by the fact that pkgconfiglite binary releases are hosted on SourceForge and that host is constantly blocking downloads from github CI nodes.
As a stopgap solution, I'm putting a savage mirror of the binaries as an attachment to this issue: pkg-config-lite-0.28-1_bin-win32.zip.
If someone managed to figure out a way to get around SourceForge's download blocus, suggestions are most welcome!
I noticed that right now type ThreadId = hwloc_thread_t
type alias is used, while there is already ThreadId in standard library that would ideally be used instead, at least as an option.
I'm working on a software that can't expect user to install hwlock library and the fact that hwlocality has bundled (it is typically called vendored though) version is a big win.
However, I noticed it much later because the first line in prerequisites is:
You will need a system with hwloc >=2.0.0 and associated development packages installed.
I think it should mention right there something like "or optionally vendored version can be used".
Similarly https://crates.io/crates/hwlocality-sys mentions bindings, but not the fact that it has vendored option too.
Without reading cargo features and/or source code it is not possible to discover that, while for me the fact that I can statically link to a recent version of the library is a major benefit of the library comparing to the alternatives.
Tried to cross-compile for aarch64 Linux and got this:
#22 108.6 Compiling hwlocality v1.0.0 (https://github.com/HadrienG2/hwlocality?rev=3141847b0a463f38adcf623a2d720931757a38ae#3141847b)
#22 109.0 error[E0308]: mismatched types
#22 109.0 --> /root/.cargo/git/checkouts/hwlocality-e92c49001d785139/3141847/src/ffi/mod.rs:107:22
#22 109.0 |
#22 109.0 107 | snprintf(buf.as_mut_ptr(), buf.len()),
#22 109.0 | -------- ^^^^^^^^^^^^^^^^ expected `*mut u8`, found `*mut i8`
#22 109.0 | |
#22 109.0 | arguments to this function are incorrect
#22 109.0 |
#22 109.0 = note: expected raw pointer `*mut u8`
#22 109.0 found raw pointer `*mut i8`
#22 109.0
#22 109.0 error[E0277]: the trait bound `Box<[u8]>: From<Vec<i8>>` is not satisfied
#22 109.0 --> /root/.cargo/git/checkouts/hwlocality-e92c49001d785139/3141847/src/ffi/mod.rs:116:13
#22 109.0 |
#22 109.0 116 | buf.into()
#22 109.0 | ^^^^ the trait `From<Vec<i8>>` is not implemented for `Box<[u8]>`
#22 109.0 |
#22 109.0 = help: the following other types implement trait `From<T>`:
#22 109.0 <Box<OsStr> as From<Cow<'_, OsStr>>>
#22 109.0 <Box<OsStr> as From<OsString>>
#22 109.0 <Box<OsStr> as From<&OsStr>>
#22 109.0 <Box<Path> as From<Cow<'_, Path>>>
#22 109.0 <Box<Path> as From<PathBuf>>
#22 109.0 <Box<Path> as From<&Path>>
#22 109.0 <Box<CStr> as From<Cow<'_, CStr>>>
#22 109.0 <Box<CStr> as From<CString>>
#22 109.0 and 18 others
#22 109.0 = note: required for `Vec<i8>` to implement `Into<Box<[u8]>>`
#22 109.0
#22 109.0 Some errors have detailed explanations: E0277, E0308.
#22 109.0 For more information about an error, try `rustc --explain E0277`.
#22 109.1 error: could not compile `hwlocality` (lib) due to 2 previous errors
#22 109.1 warning: build failed, waiting for other jobs to finish...
#22 ERROR: process "/bin/sh -c /root/.cargo/bin/cargo build --locked -Z build-std --profile $PROFILE --bin subspace-farmer --target aarch64-unknown-linux-gnu && mv target/*/*/subspace-farmer subspace-farmer && rm -rf target" did not complete successfully: exit code: 101
------
> [stage-0 16/16] RUN /root/.cargo/bin/cargo build --locked -Z build-std --profile production --bin subspace-farmer --target aarch64-unknown-linux-gnu && mv target/*/*/subspace-farmer subspace-farmer && rm -rf target:
109.0 <Box<Path> as From<&Path>>
109.0 <Box<CStr> as From<Cow<'_, CStr>>>
109.0 <Box<CStr> as From<CString>>
109.0 and 18 others
109.0 = note: required for `Vec<i8>` to implement `Into<Box<[u8]>>`
109.0
109.0 Some errors have detailed explanations: E0277, E0308.
109.0 For more information about an error, try `rustc --explain E0277`.
109.1 error: could not compile `hwlocality` (lib) due to 2 previous errors
109.1 warning: build failed, waiting for other jobs to finish...
Looks like there are some platform-dependent type?
Expose hwloc's ability to compute diffs of topologies that are similar but not quite identical: https://hwloc.readthedocs.io/en/v2.9/group__hwlocality__diff.html
build.rs
is built for the host system, and the cfg
variables are set accordingly, when cross compiling the same information is presented at build script runtime via env vars CARGO_CFG_*
.
Cases that may trip:
Tried to compile on x86-64 macOS for aarch64 and it failed like this:
configure: error: in `/Users/runner/work/subspace/subspace/target/aarch64-apple-darwin/production/build/hwlocality-sys-5187208e9a266943/out/build':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
thread 'main' panicked at /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/autotools-0.2.6/src/lib.rs:781:5:
Not sure if this is hwlocality's fault yet, but this is the first package that failed in the project.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.