totsubdirs in summary tables,about mar-file-system/gufi

Comments (6)

garygrider commented on August 10, 2024

Totsubdirs is in tree summary but not in summary We don’t store subdirs in gufi because we want to make moving subtrees possible, so the directories are derived from readdir, not from an query. We could add them to summary though, no reason not to Thx From: John Bent ***@***.***> Sent: Wednesday, October 4, 2023 12:24 PM To: mar-file-system/GUFI ***@***.***> Cc: Subscribed ***@***.***> Subject: [EXTERNAL] [mar-file-system/GUFI] totsubdirs in summary tables (Issue #143) I'm wondering whether it might be useful to include totsubdirs in the summary tables to complement totfiles and totlinks. — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/mar-file-system/GUFI/issues/143__;!!Bt8fGhp8LhKGRg!CmpD2H3od8AZCyiQIRGgClm6-AsPPs-74rProfrHdBkqMihpZJgRIfd91v8qusyKboWM_4GBZGA5m_1wpK2XzJl3$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ACZXDGMPIOYWNEZRQW6GQW3X5WSTXAVCNFSM6AAAAAA5TBRDS2VHI2DSMVQWIX3LMV43ASLTON2WKOZRHEZDMNZVGI4TKNQ__;!!Bt8fGhp8LhKGRg!CmpD2H3od8AZCyiQIRGgClm6-AsPPs-74rProfrHdBkqMihpZJgRIfd91v8qusyKboWM_4GBZGA5m_1wpCo1csVN$>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.******@***.***>>

from gufi.

johnbent commented on August 10, 2024

I think it makes sense to add. @calccrypto added a super cool arg to gufi_stats where it shows you the filesize bins. It's basically that CASE statement that we discussed but using sql log() instead of using a case for each bin. Also, has the nice feature that we don't have to query max file size first.

So now I want to use that same logic to show direntries but we currently only have filecount and linkcount. I think that most people consider subdirs when they want to know how many items are in a directory. For example, people do things like 'ls -l /some/dir | wc -l' and that includes files, and links, and subdirs.

from gufi.

garygrider commented on August 10, 2024

Brad, Dominic and I had this discussion at length one night, I remember because I was standing outside a hotel freezing waiting for uber. Anyway, he wanted to put dir names in entries (with a type d). there is a type f and type l (I think there might have been a type M once for mkdev like things). That way someone could do ls -l. I said, that’s a really bad idea because one of the really cool attributes of gufi is that you can unhook the index at any directory and move it and hook it up in another (move a directory), you could tar directory tree up and move it and give it to people and …. We could detect moves of directories when reindexing (something I worked on and Jeff inman and I figured out a crazy method involving parking lots to figure what all has moved in a tree after the fact (non trivial). We never implemented any of that in incremental update. We did convince cray to add a thing to lustre log so that it just logs that a top level dir has changed somewhere (instead of logging all million files under it that changed), so we could do incremental that way, we know what “parts” of the tree changed and just reindex them from the top of where it logged…. I understand people want to think of gufi as a file system and do an ls -l just from sql, but if you do that you have killed the ability to pick up gufi trees and move them and graft them and distribute them…. So anyway I am against it because it limits future improvements in incremental indexing that someone could do based on the earlier work. 1. I am not against putting in number of subdirs in a summary record though. It’s a bit of a lie if you pick up the subdir index and move it but maybe its not so important. I still dont think we want to put dir records into the entries table because it makes move dir ugly, but the counts I think are ok. 2. I think we should put in a histogram for everything we can think of into the summary record (or make a new table for just distribution info and put it all in there, they could join on that directories inode and a view would be trivial to make to see the entire summary plus distro info. I think the we should do same to treesummary as well (put it in the record or make a separate table) and put in all the histograms we can think of. So number of subdirs., num files, num links, mtime, ctime, num xattrs, file sizes, blocks …(basically anything that is a count or a size I thnk we need to then modify the real code (not python 😊 ) and have the process that makes summary (essentially a subroutine in the loader) to calculate these and store them when summary is created (in the summary record or in an adjacent table), could be an option in the index create/loader or not. So the process creates rich distro info when it creates the index summary records to begin with so its just there. Not just subdirs but all these fields should have distro info created at index time and stored. Then I think we need to modify the rollup/treesummary creation code to populate either a distro enhanced treesummary record or another adjacent table with treesummary distro info and think about what distros we want to keep at the tree summary level. It would be distros of distros of course which is largely just adding bins together. The bins we need to decide on for each variable, something that makes sense. So if we do that, we end up with a summary record (or view) in each dir that has rich distro for every variable that is a count/sum/size.. And we have a rich tree summary record (or view) in each dir which is rich distro of the tree below 1. Then the longitudinal table should look a lot like a summary+treesummary with all the distro info in it too, one record per top level dir (like /search/home/ggrider). OR. One for every level of the index tree for every top level dir. If we use the level thing we need to summarize the summary and treesummary records horizontally at the level and think about what that record should contain. 2. I mean its nice that we are adding verbiage to help people understand and debug their queries with some python, and we are learning how to use sql a little more, but if we want to really make this rich we gotta go into the real code and add histogram to summary and treesummary or make a summaryhist and treesummaryhist table in each dir and populate them when we are building, so the info is kept for the longitudinal sweep every period. We don’t want to be walking a bunch of times to create the longitudinal info. 3. Maybe it would be best to make the 2 new tables with the new view so we mess with less code? Although adding fields in sql is not disruptive either. Anyway, I wanted to mention that discussion we had long ago about why we don’t keep subdir info in entries in the directory’s index. Which by osmosis is why we don’t have subdir counts in summary. (but we could put counts easy enough when building the index) A nice feature would be to have a function that will tell you number of subdirs. Subdircount((path() and that user func would do a readdir+ and count. That of course will get you some immediate satisfaction that you can do a histogram of the subdirs now, but it really wont solve the longitudinal load/save unless you want to be doing millions of readdirs when you create the longitudinal records. I mean it would take 10 minutes to write the subdircount() thing and we should but it seems like we are just pecking around the edges of all this. So I think its time for surgery and not just messing around with real code in addition to adding handy functions. Change summary schema or add summaryhist table. (all fields we can think of with bins we want) Change treesummary schema or add treesummaryhist table (all fields we can think of with bins from summary rolled up) Mod load process to populate summary/summaryhist Mod the treesummary create process/rollup to populate treesummary/treesummaryhist Then think about longitudinal records one per top level dir or one per top level dir * level Another thing we could do to help solve the problem of ls -l | wc -l is this – which I considered doing a few times and just didn’t Make a virtual table (its not too hard but its not a little python either), that has a virtual entries record for every subdir. Then you can have your cake and eat it too. we could mod vrpentries view to concatentate the records from the virtualsubdir table to the view and presto you can list the dirs., files, links in vrpentries but you don’t have to actually have real recorded subdir records. So you can count them in a query if you really want to but if you move that index to some other spot, you can still do that. The directory summary summaryhist record might be wrong on subdir count if you do that. That might be another reason to put his histogram info in another table, and put dir count in there because if you are moving stuff around in the index you need to redo some summary info and we could say you have to blow away your summaryhist table just like we say blow away your treesummary table when you move stuff around in your index, but you don’t have to blow away your summary record. Is anyone following what I am saying. I said a bunch of this stuff a week ago and it seemed to go into the ether? Do we need to have a white board or something? From: John Bent ***@***.***> Reply-To: mar-file-system/GUFI ***@***.***> Date: Friday, October 6, 2023 at 12:50 AM To: mar-file-system/GUFI ***@***.***> Cc: Gary Grider ***@***.***>, Comment ***@***.***> Subject: [EXTERNAL] Re: [mar-file-system/GUFI] totsubdirs in summary tables (Issue #143) I think it makes sense to add. @calccrypto<https://urldefense.com/v3/__https:/github.com/calccrypto__;!!Bt8fGhp8LhKGRg!AyZKGJZH75sGMVQC63T1cqhBnCrtFxyzrsc8onVFL5DLOpwO7b0DGrvX-4pHMsInnXYtsCWY3_vgafQhMF6AJhZR$> added a super cool arg to gufi_stats where it shows you the filesize bins. It's basically that CASE statement that we discussed but using sql log() instead of using a case for each bin. Also, has the nice feature that we don't have to query max file size first. So now I want to use that same logic to show direntries but we currently only have filecount and linkcount. I think that most people consider subdirs when they want to know how many items are in a directory. For example, people do things like 'ls -l /some/dir | wc -l' and that includes files, and links, and subdirs. — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/mar-file-system/GUFI/issues/143*issuecomment-1750072178__;Iw!!Bt8fGhp8LhKGRg!AyZKGJZH75sGMVQC63T1cqhBnCrtFxyzrsc8onVFL5DLOpwO7b0DGrvX-4pHMsInnXYtsCWY3_vgafQhMMcgR1Q0$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ACZXDGPPIOOLIMAAQCCYARLX56S3XAVCNFSM6AAAAAA5TBRDS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJQGA3TEMJXHA__;!!Bt8fGhp8LhKGRg!AyZKGJZH75sGMVQC63T1cqhBnCrtFxyzrsc8onVFL5DLOpwO7b0DGrvX-4pHMsInnXYtsCWY3_vgafQhMGJUx9yx$>. You are receiving this because you commented.Message ID: ***@***.***>

from gufi.

johnbent commented on August 10, 2024

“Is anyone following what I am saying. I said a bunch of this stuff a week ago and it seemed to go into the ether? Do we need to have a white board or something?” :) I think I’m mostly following. I definitely understand your point about putting this into the actual indexing code instead of putting it into post-processing code. I don’t understand the thing about how moving a directory could create problems. White board sessions are always good. Maybe we can find a white board at the Panasas office and have a gufi deep dive there. They might want to attend probably. John

…

On Fri, Oct 6, 2023 at 11:30 PM garygrider ***@***.***> wrote: Brad, Dominic and I had this discussion at length one night, I remember because I was standing outside a hotel freezing waiting for uber. Anyway, he wanted to put dir names in entries (with a type d). there is a type f and type l (I think there might have been a type M once for mkdev like things). That way someone could do ls -l. I said, that’s a really bad idea because one of the really cool attributes of gufi is that you can unhook the index at any directory and move it and hook it up in another (move a directory), you could tar directory tree up and move it and give it to people and …. We could detect moves of directories when reindexing (something I worked on and Jeff inman and I figured out a crazy method involving parking lots to figure what all has moved in a tree after the fact (non trivial). We never implemented any of that in incremental update. We did convince cray to add a thing to lustre log so that it just logs that a top level dir has changed somewhere (instead of logging all million files under it that changed), so we could do incremental that way, we know what “parts” of the tree changed and just reindex them from the top of where it logged…. I understand people want to think of gufi as a file system and do an ls -l just from sql, but if you do that you have killed the ability to pick up gufi trees and move them and graft them and distribute them…. So anyway I am against it because it limits future improvements in incremental indexing that someone could do based on the earlier work. 1. I am not against putting in number of subdirs in a summary record though. It’s a bit of a lie if you pick up the subdir index and move it but maybe its not so important. I still dont think we want to put dir records into the entries table because it makes move dir ugly, but the counts I think are ok. 2. I think we should put in a histogram for everything we can think of into the summary record (or make a new table for just distribution info and put it all in there, they could join on that directories inode and a view would be trivial to make to see the entire summary plus distro info. I think the we should do same to treesummary as well (put it in the record or make a separate table) and put in all the histograms we can think of. So number of subdirs., num files, num links, mtime, ctime, num xattrs, file sizes, blocks …(basically anything that is a count or a size I thnk we need to then modify the real code (not python 😊 ) and have the process that makes summary (essentially a subroutine in the loader) to calculate these and store them when summary is created (in the summary record or in an adjacent table), could be an option in the index create/loader or not. So the process creates rich distro info when it creates the index summary records to begin with so its just there. Not just subdirs but all these fields should have distro info created at index time and stored. Then I think we need to modify the rollup/treesummary creation code to populate either a distro enhanced treesummary record or another adjacent table with treesummary distro info and think about what distros we want to keep at the tree summary level. It would be distros of distros of course which is largely just adding bins together. The bins we need to decide on for each variable, something that makes sense. So if we do that, we end up with a summary record (or view) in each dir that has rich distro for every variable that is a count/sum/size.. And we have a rich tree summary record (or view) in each dir which is rich distro of the tree below 1. Then the longitudinal table should look a lot like a summary+treesummary with all the distro info in it too, one record per top level dir (like /search/home/ggrider). OR. One for every level of the index tree for every top level dir. If we use the level thing we need to summarize the summary and treesummary records horizontally at the level and think about what that record should contain. 2. I mean its nice that we are adding verbiage to help people understand and debug their queries with some python, and we are learning how to use sql a little more, but if we want to really make this rich we gotta go into the real code and add histogram to summary and treesummary or make a summaryhist and treesummaryhist table in each dir and populate them when we are building, so the info is kept for the longitudinal sweep every period. We don’t want to be walking a bunch of times to create the longitudinal info. 3. Maybe it would be best to make the 2 new tables with the new view so we mess with less code? Although adding fields in sql is not disruptive either. Anyway, I wanted to mention that discussion we had long ago about why we don’t keep subdir info in entries in the directory’s index. Which by osmosis is why we don’t have subdir counts in summary. (but we could put counts easy enough when building the index) A nice feature would be to have a function that will tell you number of subdirs. Subdircount((path() and that user func would do a readdir+ and count. That of course will get you some immediate satisfaction that you can do a histogram of the subdirs now, but it really wont solve the longitudinal load/save unless you want to be doing millions of readdirs when you create the longitudinal records. I mean it would take 10 minutes to write the subdircount() thing and we should but it seems like we are just pecking around the edges of all this. So I think its time for surgery and not just messing around with real code in addition to adding handy functions. Change summary schema or add summaryhist table. (all fields we can think of with bins we want) Change treesummary schema or add treesummaryhist table (all fields we can think of with bins from summary rolled up) Mod load process to populate summary/summaryhist Mod the treesummary create process/rollup to populate treesummary/treesummaryhist Then think about longitudinal records one per top level dir or one per top level dir * level Another thing we could do to help solve the problem of ls -l | wc -l is this – which I considered doing a few times and just didn’t Make a virtual table (its not too hard but its not a little python either), that has a virtual entries record for every subdir. Then you can have your cake and eat it too. we could mod vrpentries view to concatentate the records from the virtualsubdir table to the view and presto you can list the dirs., files, links in vrpentries but you don’t have to actually have real recorded subdir records. So you can count them in a query if you really want to but if you move that index to some other spot, you can still do that. The directory summary summaryhist record might be wrong on subdir count if you do that. That might be another reason to put his histogram info in another table, and put dir count in there because if you are moving stuff around in the index you need to redo some summary info and we could say you have to blow away your summaryhist table just like we say blow away your treesummary table when you move stuff around in your index, but you don’t have to blow away your summary record. Is anyone following what I am saying. I said a bunch of this stuff a week ago and it seemed to go into the ether? Do we need to have a white board or something? From: John Bent ***@***.***> Reply-To: mar-file-system/GUFI ***@***.***> Date: Friday, October 6, 2023 at 12:50 AM To: mar-file-system/GUFI ***@***.***> Cc: Gary Grider ***@***.***>, Comment ***@***.***> Subject: [EXTERNAL] Re: [mar-file-system/GUFI] totsubdirs in summary tables (Issue #143) I think it makes sense to add. @calccrypto< https://urldefense.com/v3/__https:/github.com/calccrypto__;!!Bt8fGhp8LhKGRg!AyZKGJZH75sGMVQC63T1cqhBnCrtFxyzrsc8onVFL5DLOpwO7b0DGrvX-4pHMsInnXYtsCWY3_vgafQhMF6AJhZR$> added a super cool arg to gufi_stats where it shows you the filesize bins. It's basically that CASE statement that we discussed but using sql log() instead of using a case for each bin. Also, has the nice feature that we don't have to query max file size first. So now I want to use that same logic to show direntries but we currently only have filecount and linkcount. I think that most people consider subdirs when they want to know how many items are in a directory. For example, people do things like 'ls -l /some/dir | wc -l' and that includes files, and links, and subdirs. — Reply to this email directly, view it on GitHub< https://urldefense.com/v3/__https:/github.com/mar-file-system/GUFI/issues/143*issuecomment-1750072178__;Iw!!Bt8fGhp8LhKGRg!AyZKGJZH75sGMVQC63T1cqhBnCrtFxyzrsc8onVFL5DLOpwO7b0DGrvX-4pHMsInnXYtsCWY3_vgafQhMMcgR1Q0$>, or unsubscribe< https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ACZXDGPPIOOLIMAAQCCYARLX56S3XAVCNFSM6AAAAAA5TBRDS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJQGA3TEMJXHA__;!!Bt8fGhp8LhKGRg!AyZKGJZH75sGMVQC63T1cqhBnCrtFxyzrsc8onVFL5DLOpwO7b0DGrvX-4pHMsInnXYtsCWY3_vgafQhMGJUx9yx$>. You are receiving this because you commented.Message ID: ***@***.***> — Reply to this email directly, view it on GitHub <#143 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPT2PRMYPUC6EYW5YBNXBTX6AIYJAVCNFSM6AAAAAA5TBRDS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJQG43TSOBSGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

from gufi.

garygrider commented on August 10, 2024

That sounds good We wanted to avoid if we wanted to move an index tree because we could detect that it was just moved to be able to just move it, assuming we ever get sophisticated enough to do cool incremental update concepts and logging that tells us that something has changed in this 1/1000 of the tree or… If you put information in the parent db about a child dir (subdir) and then you move that child subdir, you have to go update the parent db and move the subdir. If you have no information about the subdir (or very contained information) then you can just move the subdir and you are done. So that is how it is in gufi now “except” for treesummaries but if you are moving subtrees you obviously need to redo rollups and treesummaries as you changed the tree but you have nothing else to do. If you start putting info about the subdirs in the parents db (in summary record) or as a record in entries (type=d) then you have created a mess to clean up in that db when you move a dir out of it, or into it frankly. So its weird, you cant do an ls -l function in gufi – it wasn’t designed for that, but you can do an ls -lR or find like the wind. To do an ls -l you would do the normal thing which is select something from vrpentries or really just entries is the underlying table but you wouldn’t get the subdirs, so you would have to go through and do a readdir of all the subdirectories int that dir, go open their db’s and pull their summary record and list all those too to do the equivalent of an ls -l. I had this same discussion with Dominic and Brad. If you make ls -l easy, you make ls -lR hard (store the subdir only (name or inode) in the parent but not the summary info or ….) and make tree rearrangement hard and we don’t want to make ls -lR hard. I do think a udf countsubdirs() would be good to have, that udf could just readdir+ and give an answer back, that way you could get your post processing number of dirs histogram with your sql log function stuff. It’s a trivial udf. You could copy any number of the udf’s and have it working in 10 minutes and you get what you want for dirs. But Should we put anything at all about the subdirs in the parent db? We already have in treesummary and rollup because those are tree operations. So I am thinking maybe instead of putting anything about the subdirs in the parent summary or entries tables, lets make a new summary table or a new entries table, summaryhist and maybe subdirentries (although I really don’t think this subdirentries is needed except if we just want to replicate ls -l). We put all the cool histogram stuff you want about that directory in that record (so you can use this table or not depending on if you want cool histograms) and we put the subdir stats/histo stuff in there. So now if we do a move/remove/add a subdir to an parent somewhere, we have to 1. Blow away the treesmmary db 2. Blow away the summaryhist and subdirentries (if we choose to do this) 3. Unrollup from that parent down 4. Move the subdirtree in or out 5. Rerun the rollup which populates he treesummary and could also populate the summary hist pretty easily Easy to keep the ability to move trees around and you aren’t cleaning out fields in records or anything hard. Then I said to myself, ok that’s pretty cool we have a summaryhist, maybe we need a treesummaryhist to put the tree rollup version of all those cool histograms to capture distributions of that dir in summaryhist and distributions of all the tree below in treesummaryhist And as for the subdirentries table – you could populate that with the rollup too Or you could make it a virtual table that is slow because it has to readdir then open up the subdir records (if allowed to by perms) and emulate a subdirentries table for those people that just absolutely want an ls -l function But I think this is low priority if we do it at all Rollup already has to examine all the perms of all the subdirs below and the parent you are processing to decide if something can be rolled up securely, so rollup is already processing all subdirs to decide if it can rollup or not. it even has to copy the summary and entries records to the parent from all the subdirs if rollup is allowed (placing them properly so we can unrollup if necessary), so it is almost already doing what is necessary for the summaryhist and treesummary hist population and it could populate a subdirentries table if we chose to do that. I just don’t know if its worth it. It adds nothing to the longitudinal study as long as we do a summaryhist and treesummary hist that have directory disto info in them. If you want to find the largest directory by number of files you could do that now. If you wanted to find the largest directories by the number of subdirs -that is missing but if you had the countsubdirs() function you could easily. (select against summary countsubdirs() sort by countsubdirs() limit 1 and then aggregate/sort, but if you wanted to bin dirs. by how many subdirs, its possible I suppose but not trivial. If you have a summaryhist that already have the histo for num subdirs in it, then its crazy simple and fast and the same process that creates all summary and treesummary records is already touching all the metadata required to populate summaryhist and treesummaryhist. This raving seems to let us keep the “moving trees is easy” and have rich distribution info for everything including subdirs, and even give us a way to support ls -l if we just want to do that, either through a virtual table or a disposable one like treesummary and other rollup related tables. Of course you might say – gary why are you trying so hard to preserve this ability to move index subtrees around easily. It’s a good question. I hold out hope that the elegant mechanisms Inman and I put together might be useful if/when trees get to be really massive and incrementals with simple (what part of the tree changed hints) are no longer economical. From: John Bent ***@***.***> Reply-To: mar-file-system/GUFI ***@***.***> Date: Friday, October 6, 2023 at 1:56 PM To: mar-file-system/GUFI ***@***.***> Cc: Gary Grider ***@***.***>, Comment ***@***.***> Subject: [EXTERNAL] Re: [mar-file-system/GUFI] totsubdirs in summary tables (Issue #143) “Is anyone following what I am saying. I said a bunch of this stuff a week ago and it seemed to go into the ether? Do we need to have a white board or something?” :) I think I’m mostly following. I definitely understand your point about putting this into the actual indexing code instead of putting it into post-processing code. I don’t understand the thing about how moving a directory could create problems. White board sessions are always good. Maybe we can find a white board at the Panasas office and have a gufi deep dive there. They might want to attend probably. John

On Fri, Oct 6, 2023 at 11:30 PM garygrider ***@***.***> wrote: Brad, Dominic and I had this discussion at length one night, I remember because I was standing outside a hotel freezing waiting for uber. Anyway, he wanted to put dir names in entries (with a type d). there is a type f and type l (I think there might have been a type M once for mkdev like things). That way someone could do ls -l. I said, that’s a really bad idea because one of the really cool attributes of gufi is that you can unhook the index at any directory and move it and hook it up in another (move a directory), you could tar directory tree up and move it and give it to people and …. We could detect moves of directories when reindexing (something I worked on and Jeff inman and I figured out a crazy method involving parking lots to figure what all has moved in a tree after the fact (non trivial). We never implemented any of that in incremental update. We did convince cray to add a thing to lustre log so that it just logs that a top level dir has changed somewhere (instead of logging all million files under it that changed), so we could do incremental that way, we know what “parts” of the tree changed and just reindex them from the top of where it logged…. I understand people want to think of gufi as a file system and do an ls -l just from sql, but if you do that you have killed the ability to pick up gufi trees and move them and graft them and distribute them…. So anyway I am against it because it limits future improvements in incremental indexing that someone could do based on the earlier work. 1. I am not against putting in number of subdirs in a summary record though. It’s a bit of a lie if you pick up the subdir index and move it but maybe its not so important. I still dont think we want to put dir records into the entries table because it makes move dir ugly, but the counts I think are ok. 2. I think we should put in a histogram for everything we can think of into the summary record (or make a new table for just distribution info and put it all in there, they could join on that directories inode and a view would be trivial to make to see the entire summary plus distro info. I think the we should do same to treesummary as well (put it in the record or make a separate table) and put in all the histograms we can think of. So number of subdirs., num files, num links, mtime, ctime, num xattrs, file sizes, blocks …(basically anything that is a count or a size I thnk we need to then modify the real code (not python 😊 ) and have the process that makes summary (essentially a subroutine in the loader) to calculate these and store them when summary is created (in the summary record or in an adjacent table), could be an option in the index create/loader or not. So the process creates rich distro info when it creates the index summary records to begin with so its just there. Not just subdirs but all these fields should have distro info created at index time and stored. Then I think we need to modify the rollup/treesummary creation code to populate either a distro enhanced treesummary record or another adjacent table with treesummary distro info and think about what distros we want to keep at the tree summary level. It would be distros of distros of course which is largely just adding bins together. The bins we need to decide on for each variable, something that makes sense. So if we do that, we end up with a summary record (or view) in each dir that has rich distro for every variable that is a count/sum/size.. And we have a rich tree summary record (or view) in each dir which is rich distro of the tree below 1. Then the longitudinal table should look a lot like a summary+treesummary with all the distro info in it too, one record per top level dir (like /search/home/ggrider). OR. One for every level of the index tree for every top level dir. If we use the level thing we need to summarize the summary and treesummary records horizontally at the level and think about what that record should contain. 2. I mean its nice that we are adding verbiage to help people understand and debug their queries with some python, and we are learning how to use sql a little more, but if we want to really make this rich we gotta go into the real code and add histogram to summary and treesummary or make a summaryhist and treesummaryhist table in each dir and populate them when we are building, so the info is kept for the longitudinal sweep every period. We don’t want to be walking a bunch of times to create the longitudinal info. 3. Maybe it would be best to make the 2 new tables with the new view so we mess with less code? Although adding fields in sql is not disruptive either. Anyway, I wanted to mention that discussion we had long ago about why we don’t keep subdir info in entries in the directory’s index. Which by osmosis is why we don’t have subdir counts in summary. (but we could put counts easy enough when building the index) A nice feature would be to have a function that will tell you number of subdirs. Subdircount((path() and that user func would do a readdir+ and count. That of course will get you some immediate satisfaction that you can do a histogram of the subdirs now, but it really wont solve the longitudinal load/save unless you want to be doing millions of readdirs when you create the longitudinal records. I mean it would take 10 minutes to write the subdircount() thing and we should but it seems like we are just pecking around the edges of all this. So I think its time for surgery and not just messing around with real code in addition to adding handy functions. Change summary schema or add summaryhist table. (all fields we can think of with bins we want) Change treesummary schema or add treesummaryhist table (all fields we can think of with bins from summary rolled up) Mod load process to populate summary/summaryhist Mod the treesummary create process/rollup to populate treesummary/treesummaryhist Then think about longitudinal records one per top level dir or one per top level dir * level Another thing we could do to help solve the problem of ls -l | wc -l is this – which I considered doing a few times and just didn’t Make a virtual table (its not too hard but its not a little python either), that has a virtual entries record for every subdir. Then you can have your cake and eat it too. we could mod vrpentries view to concatentate the records from the virtualsubdir table to the view and presto you can list the dirs., files, links in vrpentries but you don’t have to actually have real recorded subdir records. So you can count them in a query if you really want to but if you move that index to some other spot, you can still do that. The directory summary summaryhist record might be wrong on subdir count if you do that. That might be another reason to put his histogram info in another table, and put dir count in there because if you are moving stuff around in the index you need to redo some summary info and we could say you have to blow away your summaryhist table just like we say blow away your treesummary table when you move stuff around in your index, but you don’t have to blow away your summary record. Is anyone following what I am saying. I said a bunch of this stuff a week ago and it seemed to go into the ether? Do we need to have a white board or something? From: John Bent ***@***.***> Reply-To: mar-file-system/GUFI ***@***.***> Date: Friday, October 6, 2023 at 12:50 AM To: mar-file-system/GUFI ***@***.***> Cc: Gary Grider ***@***.***>, Comment ***@***.***> Subject: [EXTERNAL] Re: [mar-file-system/GUFI] totsubdirs in summary tables (Issue #143) I think it makes sense to add. @calccrypto< https://urldefense.com/v3/__https:/github.com/calccrypto__;!!Bt8fGhp8LhKGRg!AyZKGJZH75sGMVQC63T1cqhBnCrtFxyzrsc8onVFL5DLOpwO7b0DGrvX-4pHMsInnXYtsCWY3_vgafQhMF6AJhZR$%3E added a super cool arg to gufi_stats where it shows you the filesize bins. It's basically that CASE statement that we discussed but using sql log() instead of using a case for each bin. Also, has the nice feature that we don't have to query max file size first. So now I want to use that same logic to show direntries but we currently only have filecount and linkcount. I think that most people consider subdirs when they want to know how many items are in a directory. For example, people do things like 'ls -l /some/dir | wc -l' and that includes files, and links, and subdirs. — Reply to this email directly, view it on GitHub< https://urldefense.com/v3/__https:/github.com/mar-file-system/GUFI/issues/143*issuecomment-1750072178__;Iw!!Bt8fGhp8LhKGRg!AyZKGJZH75sGMVQC63T1cqhBnCrtFxyzrsc8onVFL5DLOpwO7b0DGrvX-4pHMsInnXYtsCWY3_vgafQhMMcgR1Q0$%3E, or unsubscribe< https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ACZXDGPPIOOLIMAAQCCYARLX56S3XAVCNFSM6AAAAAA5TBRDS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJQGA3TEMJXHA__;!!Bt8fGhp8LhKGRg!AyZKGJZH75sGMVQC63T1cqhBnCrtFxyzrsc8onVFL5DLOpwO7b0DGrvX-4pHMsInnXYtsCWY3_vgafQhMGJUx9yx$%3E. You are receiving this because you commented.Message ID: ***@***.***> — Reply to this email directly, view it on GitHub <#143 (comment)><https://urldefense.com/v3/__https:/github.com/mar-file-system/GUFI/issues/143*issuecomment-1750779821*3E__;IyU!!Bt8fGhp8LhKGRg!CD9Dev3xgNvHSMuANHjYhHvGOPSIA0y2rhFGAHtTvcLoQMCdG6SmvDfLszIGYTo7nEm4Ii5fVhYmLGH4mlI5-EvP$>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPT2PRMYPUC6EYW5YBNXBTX6AIYJAVCNFSM6AAAAAA5TBRDS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJQG43TSOBSGE><https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AAPT2PRMYPUC6EYW5YBNXBTX6AIYJAVCNFSM6AAAAAA5TBRDS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJQG43TSOBSGE*3E__;JQ!!Bt8fGhp8LhKGRg!CD9Dev3xgNvHSMuANHjYhHvGOPSIA0y2rhFGAHtTvcLoQMCdG6SmvDfLszIGYTo7nEm4Ii5fVhYmLGH4mo9_nc0s$> . You are receiving this because you authored the thread.Message ID: ***@***.***>

— Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/mar-file-system/GUFI/issues/143*issuecomment-1751336322__;Iw!!Bt8fGhp8LhKGRg!CD9Dev3xgNvHSMuANHjYhHvGOPSIA0y2rhFGAHtTvcLoQMCdG6SmvDfLszIGYTo7nEm4Ii5fVhYmLGH4mjlqB3Z2$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ACZXDGNPRJNPK7QXHPYPOW3X6BO6NAVCNFSM6AAAAAA5TBRDS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJRGMZTMMZSGI__;!!Bt8fGhp8LhKGRg!CD9Dev3xgNvHSMuANHjYhHvGOPSIA0y2rhFGAHtTvcLoQMCdG6SmvDfLszIGYTo7nEm4Ii5fVhYmLGH4mmi3RBXt$>. You are receiving this because you commented.Message ID: ***@***.***>

from gufi.

calccrypto commented on August 10, 2024

4c28be4 adds the sqlite function subdirs(srollsubdirs, sroll) using a new column in vrsummary to get the correct number of immediate subdirs of each directory whether or not the index was rolled up. After moving subtrees and redoing rollups, the subdir count is automatically computed when querying.

from gufi.

totsubdirs in summary tables about gufi HOT 6 CLOSED

Comments (6)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent