joaomlneto / unstickymem Goto Github PK

Currently we only have a README.md in the root directory.

Not only this is kind of limiting, but right now it's completely outdated.

Before making things public (after the publication), we should document it properly.

Sections:

• How to install
• How to compile
• How to run (LD_PRELOAD or linking in the source code)
• Library modes
• Library options (configuration file, environment variables)

There is a bug that does not uniformly distribute the memory among the nodes.

The decision-making algorithm does the following:

Initialization steps

1. Interleave all memory objects (must use numactl --interleave=all for now)
2. Wait a few seconds before starting the tuning process.

Tuning process

1. Take N measurements of the memory stall rate, T microseconds apart.
2. Discard k measurements (the k/2 smallest and k/2 greatest).
3. Compute the average
4. If the average is lower, then keep going (shift a few more pages into the local node). Otherwise, stop.

Calling place_pages with the same arguments results in extra pages being moved into the local node.

We wanted to improve it anyways -- rewrite!!

Currently we assume there is 1 local/worker node.

The function takes one parameter (%local) that places local% of the segment in the local node, and the remainder is interleaved between the three other nodes.

Provide another version that can have different weights for each node.

Algorithm should treat all the contiguous mapped memory as blocks of N pages.

Should place x% of each block in the local node and (1-x)% interleaved in the remaining nodes.

Measured stall rate and decisions taken by the algorithm

Measurements at 2018-10-10 on intel14cores-v2

Currently we only support checking the memory map once and acting upon it.

If we update the memory mapping, the place_pages is not idempotent. (see #9 for explanation and example).

@gureya which version of Streamcluster are you using?

I recall comments about multiple versions existing during the last Skype call.

Currently we will just crash if we're running on a machine with a single NUMA node.

We should simply show a warning and do nothing.

Algorithm should treat all the regions as one and place x% in the local node and (1-x)% interleaved in the other Nodes.

Write some basic documentation for users and future contributors

The mechanism to avoid stopping due to noise is simply not working.

If we get a higher stall rate we trigger this confirmation mechanism. However, we will stop the adaptive placement independently of the stall rate being lower.

This needs to be fixed in all adaptive mechanisms (weighted and uniform)

We need to find a way of checking what data segments exist, in order to pass them to mbind.

• How does the performance of mmap/malloc change depending on the number of mappings?
• What are the limitations?
• What is the data structure where these mappings are stored? Can we use a better one?

We should try and do some functional tests to parts of the library.

Not an easy task, given the nature of the library.

Old bug, but ideally we want to avoid running setup-counters-$(arch) all the time.

Probably a good time to learn about packaging and stuff :-)

Currently it will work on Broadwell/Haswell (which is the case of intel14cores).

Intel Performance Counters:

• 1 (countreg 0x40000001) - Haswell - Core Clock Cycles
• 111 (event 0xa2, mask 0x01) - Haswell - Any Resource Stall

AMD Performance Counters:

• ??????????

Changes are being tracked on the perf branch

As memory goes from an interleaved state to a local state; it would be nice to offer functionality to make it go in the opposite way.

A new user with a brand new NUMA machine with default e.g. Ubuntu -- how should I install/run your application? Should cover package dependencies and a comprehensive list of commands.

If we're dealing with an application that makes many small allocations, the algorithm in WeightedAdaptiveMode won't work.

@gureya has a fix ready in his branch :-P

I'm just writing it here not to forget we had a problem, and to (try and) make a unit test later.

The fix is basically:

1. Set the default memory policy to MPOL_INTERLEAVE on all nodes in the library constructor
2. Ignoring any new segments sent from MemoryMap until the WeightedAdaptive mode starts
3. Upon WeightedAdaptive mode starting, apply initial weights
4. Proceed normally

The side effects are that the segments will be uniformly interleaved until the node starts, although we don't expect this to be of significance.

We want to be able to know the set of all existing segments to apply mbind on.

To this end, we'll intercept standard C/linux functions that deal with memory mappings/allocations:

• mmap
• munmap
• sbrk

This mechanism has significant advantages over reading /proc/self/maps (#11):

• Deal with user semantics rather than the kernel -- we won't observe a segment getting fragmented as when reading the /proc/self/maps file.
• Up-to-date information -- we are able to have an always up-to-date set of segments
• Thread-safety -- we won't be plagued by read-free-mbind crashes, as we are able to stop the process from making changes during vulnerability periods.

• What license to include in the repository?
• What copyright notice to include in the source files?

Actually, they can stay -- but the application should be able to effortlessly generate them.

There should be a background thread that does things automatically (sleep, measure, decide new weights, apply mbind).

• The thread should be bound to the same core in the same NUMA node.

Change library name to match the one on the paper.

The way we are dealing with algorithms won't scale and quickly becomes a mess.

Instead of spawning a thread that (currently) has a scan mode and an adaptive mode, refactor this into a Factory/Strategy pattern.

Evaluation of the Scalar Penta-diagonal solver of NAS Parallel Benchmarks.