Giter Club home page Giter Club logo

Comments (23)

dmanno avatar dmanno commented on August 10, 2024

Hi John,

We are actively looking into this. We will update progress here. If you could please share a bit of the details around the FS being indexed, that would be great. We are expecting that there are directories with very high amounts of sub-dirs, causing the queues to grow quickly. If you'd like to discuss via email, feel free to reach out to [email protected]

from gufi.

tacc-desantis avatar tacc-desantis commented on August 10, 2024

Hello Dominic!

Thanks for the reply! Just for public visibility, the file system I'm trying to index is BeeGFS. It is 8.1 PB usable and currently has 1,034,075,745 inodes in use. I'll email you directly with additional details.

John DeSantis

from gufi.

calccrypto avatar calccrypto commented on August 10, 2024

Hi. Please try out these two branches:

insitu
gufi_dir2index -C <count> ..., where count is the maximum number of subdirectories in each directory that are enqueued for parallel processing. Any remaining subdirectories are processed in the current directory's thread, reducing the number of active allocations. The value that should be used will depend on the distribution of branching factors of the specific tree you are using.

deferred-queue
gufi_dir2index -M <bytes> ..., where bytes is a soft target memory size for the queues to attempt to reduce the overall memory footprint to. bytes is divided by the number of threads. Work is put on a separate per-thread queue that is not processed until the normal per-thread queue is emptied.

from gufi.

tacc-desantis avatar tacc-desantis commented on August 10, 2024

Hello,

Thank you! I am running the deferred-queue build now. I'll update this issue with the results once it is done.

John DeSantis

from gufi.

tacc-desantis avatar tacc-desantis commented on August 10, 2024

Hello,

I'm pleased to state that I've noticed stability in the memory footprint so far! I'll update again once the indexing finishes. Feel free to review the screen shot I've included to show that the deferred-queue branch has had a positive effect, including a small decrease in the amount of memory being used

Screenshot from 2023-03-21 16-43-39

And for what it's worth, the directory in question has 38,574,219 sub directories.

from gufi.

tacc-desantis avatar tacc-desantis commented on August 10, 2024

Hello,

I wanted to provide an update. Unfortunately, indexing on two specific directories still continues to fail due to memory usage, despite either branch and various combinations of arguments, so we've reached out to the users in question to tidy up their directories.

Once we hear back from them I'll get a count of their current directory structures and verify items have been culled. I'll then re-run the indexing process (both methods) and report back.

Thanks again!
John DeSantis

from gufi.

calccrypto avatar calccrypto commented on August 10, 2024

Please give the reduce-work-size branch a try. The data structure that is enqueued has been reduced in size by over 40%. Additionally, if CMake detects the zlib library and -C (with no value this time) is passed to gufi_dir2index, the work items will be compressed.

from gufi.

tacc-desantis avatar tacc-desantis commented on August 10, 2024

Hello,

The reduce-work-size branch is extremely promising! So far, not only has the indexing process run longer (4x) for this specific directory, it's memory footprint has been reduced by ~83%!!!! Please see the screen shot below to see the fruits of your labor.

Screenshot from 2023-03-30 09-56-19

I'll update again once the process completes or is terminated.

from gufi.

tacc-desantis avatar tacc-desantis commented on August 10, 2024

Hello,

Both indexes completed without any logged errors. CPU and memory statistics for both processes is listed below:

56,591,355 directories & 80% reduction in memory footprint:

23962.18user 9703.66system 35:46:25elapsed 26%CPU (0avgtext+0avgdata 41947132maxresident)k 104162440inputs+3706836056outputs (24major+44473160minor)pagefaults 0swaps

38,574,219 directories & 91.5% reduction in memory footprint:

9857.33user 3419.66system 14:54:09elapsed 24%CPU (0avgtext+0avgdata 18088904maxresident)k 30993064inputs+1370928848outputs (2major+25345753minor)pagefaults 0swaps

We will need to provide more storage for the out of tree index though, as these two directories consumed all 3.7 TB allocated for the indexes.

Thank you!
John DeSantis

from gufi.

garygrider avatar garygrider commented on August 10, 2024

from gufi.

tacc-desantis avatar tacc-desantis commented on August 10, 2024

Hello,

Thanks so much for the challenging use case to help make the software better.

Certainly! I thoroughly appreciate the time and fixes the development team produced.

Potentially these things could help you and of course it would be nice to know if these tools
can handle such a wide directory structure that your site has, something we could help ensure works.

Agreed. Currently, I've only been using gufi_find for querying the tree, so I'll need to read-up on available documentation (including recent slides) and look at some of the commit logs to take advantage of these options. Despite this, gufi_find is still a gem, as our current production need is to speed up file system purging. Once we have developed a proper workflow for this process, we'll most likely apply it to several other production file systems, and then begin to utilize GUFI's other features for richer file system accounting.

If there are any other features the development team would like to test against large, production file systems, let me know.

Thank you again!
John DeSantis

from gufi.

garygrider avatar garygrider commented on August 10, 2024

from gufi.

calccrypto avatar calccrypto commented on August 10, 2024

The changes made in the 3 branches have been merged into one branch and then merged with master via #125. Compression with zlib has been changed to use -e since -C was used by the insitu branch.

Please let us know if the combining of all of the changes broke something.

from gufi.

tacc-desantis avatar tacc-desantis commented on August 10, 2024

Hello,

The changes made in the 3 branches have been merged into one branch and then merged with master via #125. Compression with zlib has been changed to use -e since -C was used by the insitu branch.

Excellent!

Please let us know if the combining of all of the changes broke something.

Will do! I am running an index now. Also, I've deliberately have skipped the indexing of the problematic directories now since users are actively removing content; one of the logs had 1k+ lines stating that directories couldn't be opened due to the active file/directory removals.

Thanks again for such amazing responses from the development team!

John DeSantis

from gufi.

tacc-desantis avatar tacc-desantis commented on August 10, 2024

Gary,

Glad it is helping you. Yes guf_find is about 1/50th of the capability gufi has.

Where's that "mind blown" gif???

Very complex things are pretty easy like make me a sorted histogram of all the file types in the entire system to simple things like find me the top 3 largest directories. Hope you find use for some of it…

Thanks for these suggestions, a seed has been planted...

John DeSantis

from gufi.

tacc-desantis avatar tacc-desantis commented on August 10, 2024

Hello,

Marking this as resolved! I am no longer seeing memory issues that I had seen previously. Thanks again for all of the development team's hard work.

John DeSantis

from gufi.

tacc-desantis avatar tacc-desantis commented on August 10, 2024

from gufi.

garygrider avatar garygrider commented on August 10, 2024

from gufi.

tacc-desantis avatar tacc-desantis commented on August 10, 2024

Gary,

Try
./gufi_query -n 1 -E "select path(uid,uid),* from entries;" /gufi_index/lo_scemo/ | head -10

And of course
./gufi_query -n 1 -E ‘select path(uid,uid) || “\” || name, size, uid, uidtouser(uid,20) from entries;" /gufi_index/lo_scemo/ | head -10

These work perfectly, thank you for the insight!

We need to produce a “sql guide go gufi”
I imagine we will because we have a tutorial coming up at MSST in May

A guide would be most beneficial, especially since (as you already said) there are a lot more data points available via gufi_query vs. the wrapper tools gufi_find, gufi_ls, etc. I'd love to attend MSST, but it's too short of a notice and I'm in the middle of a cross country move :(

You can look at the entire schema in the code or just go to a dir and do
Sqlite3 dbname where dbname is the name of the database in that dir
Then something like .schema or something

Agreed, that's what I did since we all love to abuse and exploit SQL!

There is a LOT there and happy to describe if it helps

You may want to rescind that offer, hahaha!

Thanks again,
John DeSantis

from gufi.

calccrypto avatar calccrypto commented on August 10, 2024

I pushed some commits that restore the original path(), epath(), and fpath() sqlite functions. path() no longer takes in arguments. I have renamed the function with 2 arguments (summary.name and summary.rollupscore) to rpath(), which is meant for handling path names in rolled up indices. Its intended use in the -E query is:

SELECT rpath(summary.name, summary.rollupscore) || "/" || pentries.name
FROM summary, pentries
WHERE summary.inode == pentries.pinode;

However, because we are not in the business of forcing user queries to be in a certain format, it is easy to misuse rpath().

uidtouser, gidtogroup, and several other functions had an extra unexpected argument. They have been removed. The extra argument was to help with alignment of output, but they were always unused and set to 0.

from gufi.

garygrider avatar garygrider commented on August 10, 2024

from gufi.

garygrider avatar garygrider commented on August 10, 2024

from gufi.

calccrypto avatar calccrypto commented on August 10, 2024

@tacc-desantis I just pushed GUFI-SQL.docx, Gary's guide on using GUFI's SQL schema and functions, to the docs directory. Please pull the latest version of GUFI to get all of the features mentioned in the guide.

from gufi.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.