When you issue r.ownertree in ipython, you acually get r.ownertree.__repr__(). This prints {} because

• the tree object is a subclass of dict,
• the (apparent) default dict objects is not filled in, but instead the .tree attribute is
• repr does not exist
  When you issue print(r.ownertree), you invoke the r.ownertree.__lst__() and this pretty-prints the tree
  If you need the tree dict object, documentation states you should use r.ownertree.tree

No args in function?

Also this part in ownertree

return self._ownertree if self._ownertree else self.tree("GPBD_OWNER_ID")

else part yields

>>> r.tree("GBPD_OWNER_ID")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/henri/pyracf/lib/python3.8/site-packages/pyracf/__init__.py", line 1016, in tree
    higher_ups = self._groups.groupby(linkup_field)
  File "/home/henri/pyracf/lib/python3.8/site-packages/pandas/core/frame.py", line 8252, in groupby
    return DataFrameGroupBy(
  File "/home/henri/pyracf/lib/python3.8/site-packages/pandas/core/groupby/groupby.py", line 931, in __init__
    grouper, exclusions, obj = get_grouper(
  File "/home/henri/pyracf/lib/python3.8/site-packages/pandas/core/groupby/grouper.py", line 985, in get_grouper
    raise KeyError(gpr)
KeyError: 'GBPD_OWNER_ID'

when loading from pickles?

Using this code:

import time
from pyracf import IRRDBU
mysys = IRRDBU('/x/IRRDBU00.txt')
mysys.parse()
while mysys.status['status'] != 'Ready':
    time.sleep(5)
mysys.xls('/x/y/my.xlsx')

I'm getting the below error:

Traceback (most recent call last):
  File "/x/x/test.py", line 7, in <module>
    mysys.xls('/x/y/my.xlsx')
  File "/x/y/opt/miniconda3/lib/python3.9/site-packages/pyracf/__init__.py", line 548, in xls
    if self._records[DSACC_RECORDTYPE]['parsed'] + self._records[GRACC_RECORDTYPE]['parsed'] == 0:
NameError: name 'DSACC_RECORDTYPE' is not defined

yaml aggressively converts str values to int, float or bool, so you have to quote words like YES and NO, or they'll be booled.
also, yaml quietly allows duplicate keys (and keeps the last) in dict definitions.

StrictYAML does not do automatic (re-)typing, and has a built-in syntax validator.
You have to jump through some hoops to allow flow style specification ( [DSBD,GRBD] or [NONE,READ] instead of their multi-line equivalent).

pros and cons pypi

So far, this applies to RuleVerifier, but could have more use.

When using the .acl(...) suffix method sometimes the 'column names' get stripped of their prefix.
It's workable, but would be more consitent is (for instance) this

>>> r.datasetAccess.gfilter('SYS1.S*.**').acl()
                  NAME VOL  USER_ID  AUTH_ID  ACCESS
0   SYS1.SECRET.*.**        USER1  USER1     ALTER

Would be

>>> r.datasetAccess.gfilter('SYS1.S*.**').acl()
            USBD_NAME VOL  USER_ID  AUTH_ID  ACCESS
0   SYS1.SECRET.*.**        USER1  USER1   ALTER
```

ProfileFrames show all columns in the Frame, plus the index fields. This may result in useful information being invisible (caught in the ellipsis), such as DSBD_UACC, and not so informative fields, like the recordtype taking prominent place.
The default view of a Frame should be useful, we should not claim that "the user can rearrange things if they want."

unload file has x'001a' (CTRL-Z) at the end after file transfer, and parse_t() tries to find the layout of the record for record type x'001a'. This results in KeyError, obviously:

File "/pyracf-dev-0.9/src/pyracf/__init__.py", line 324, in parse_t
    offsets = RACF._recordtype_info[r]["offsets"]
KeyError: '\x1a'

We need to check if r in RACF._recordtype_info, or add a try:

Currently, everything is shipped in package pyracf.

Core irrdbu00 reader is in pyracf/__init__.py, which defines the RACF class. The RACF class has methods:

• coded as functions in __init__.py
• defined through property() as class properties, this includes most of the publisher names
• inherited from ProfilePublisher, XlsWriter

RACF instances create dfs:

• dataframes (starting with _) are defined as ProfileFrame() and then linked using setattr

It is easy to delete XlsWriter from the RACF class, and force users to import the module containing XlsWriter.
The same applies to RuleVerifier, i.e. create verifier instances explitly using v = RuleVerifier(r), but where should we store the rule_verifier.py module?

Also, RuleVerifier needs some of the same classes that RACF uses. That implies creating a separate directory with class definitions, that modules in both pyracf and verifier directories import from.

Same could be done with the ProfilePublisher class (optional), and the FrameFilter class (also shared between pyracf and verifier).

Do we want multiple packages?

.acl( ) provides support for normal ACL and conditional ACL, and even for combined output.

Many customers will object if CATYPE, CANAME, and their 2 less known siblings are shown on all reports: "what is this, I have not asked for it." So currently, if there is no record in the output with CATYPE filled in, the 4 columns are dropped.

Programmatic users that integrate output of .acl( ) with other data now have to include trivial logic to deal with the conditional conditional columns:

# combine r.datasets, r.datasetAccess and r.datasetConditionalAccess
access = r.datasets.acl()
if 'CATYPE' in access.columns:
  # CATYPE is there
else:
  # only normal ACL

Is there a need to always add the 4 columns, for these automated data slurpers? Say, add an option "all_columns=True" (with default = False)?

Commits that change a release level in setup.py or in PKG-INFO, or create binary blobs, should not make code changes. This combo of changes makes it impossible to cherry-pick those commits into development branches. Example and example.

The unload file has UID values stored as fixed length, leading zero, numbers:

_NAME
ADCDA      0000990008
ADCDB      0000990009
ADCDMST    0000000000
ASH        0000000000
BILLA      0000000100
              ...    
ZOSCAGL    0000009081
ZOSCSRV    0000009082
ZOSMFAD    0000990007
ZOSUGST    0000009080
9735       0000000100

This makes it harder to get a list of all users with UID 0, because you have to be careful count the number of characters.

r.userOMVS.loc[r.userOMVS.USOMVS_UID=='0000000000']
vs
r.userOMVS.loc[r.userOMVS.USOMVS_UID=='0']

We could strip the leading zeros in correlate(). Objections?

Currently we have .gfilter() and .rfilter(), they implement (positive) selection. We originally decided on gfilter (generic filter) as the functionality we needed is similar to the SEARCH FILTER( ) command in TSO, and .select .match and .filter were already taken by Pandas.
Looking more closely at the built-in .filter method, it does something similar although it does not directly allow generic filters:

>>> r.users.filter(['ADCDA','IBMUSER'],axis=0)
 	USBD_RECORD_TYPE 	USBD_NAME 	USBD_CREATE_DATE 	USBD_OWNER_ID 	USBD_ADSP 	USBD_SPECIAL 	USBD_OPER 	USBD_REVOKE 	USBD_GRPACC 	USBD_PWD_INTERVAL 	... 	USBD_PPHENV_EXISTS 	USBD_PWD_ALG 	USBD_LEG_PWDHIST_CT 	USBD_XPW_PWDHIST_CT 	USBD_PHR_ALG 	USBD_LEG_PHRHIST_CT 	USBD_XPW_PHRHIST_CT 	USBD_ROAUDIT 	USBD_MFA_FALLBACK 	USBD_PHR_INTERVAL
_NAME 																					
ADCDA 	0200 	ADCDA 	2007-06-01 	IBMUSER 	NO 	NO 	NO 	YES 	NO 	180 	... 	NO 	LEGACY 	000 	000 	NOPHRASE 	000 	000 	NO 	NO 	00000
IBMUSER 	0200 	IBMUSER 	1995-06-06 	IBMUSER 	NO 	YES 	YES 	YES 	NO 	000 	... 	NO 	LEGACY 	000 	000 	NOPHRASE 	000 	000 	NO 	NO 	00000

2 rows × 52 columns

>>> t.users.filter(regex='IBM.*',axis=0)
 	USBD_RECORD_TYPE 	USBD_NAME 	USBD_CREATE_DATE 	USBD_OWNER_ID 	USBD_ADSP 	USBD_SPECIAL 	USBD_OPER 	USBD_REVOKE 	USBD_GRPACC 	USBD_PWD_INTERVAL 	... 	USBD_PPHENV_EXISTS 	USBD_PWD_ALG 	USBD_LEG_PWDHIST_CT 	USBD_XPW_PWDHIST_CT 	USBD_PHR_ALG 	USBD_LEG_PHRHIST_CT 	USBD_XPW_PHRHIST_CT 	USBD_ROAUDIT 	USBD_MFA_FALLBACK 	USBD_PHR_INTERVAL
_NAME 																					
IBMUSCN 	0200 	IBMUSCN 	2019-11-11 	IBMUSER 	NO 	NO 	NO 	NO 	NO 	180 	... 	NO 	NOPASSWORD 	000 	000 	NOPHRASE 	000 	000 	NO 	NO 	00000
IBMUSECN 	0200 	IBMUSECN 	2019-11-11 	IBMUSER 	NO 	NO 	NO 	NO 	NO 	180 	... 	NO 	NOPASSWORD 	000 	000 	NOPHRASE 	000 	000 	NO 	NO 	00000
IBMUSER 	0200 	IBMUSER 	1995-06-06 	IBMUSER 	NO 	YES 	YES 	YES 	NO 	000 	... 	NO 	LEGACY 	000 	000 	NOPHRASE 	000 	000 	NO 	NO 	00000

3 rows × 52 columns

There are 3 issues I would like to address:

1. we now have gfilter() and rfilter(), after 1 week not using pyracf, I have to think hard what the method name is, again. r.users.select('IBM*')..... nope. We chose gfilter because the name was available, well, a strange looking function name is not always the right choice
2. someone mentioned rfilter reminded him of "RACF filters" and gfilter didn't remind him of generic patterns.
3. it would be great to be able to SELECT and also EXCLUDE records, like so:
   r.users.gfilter('IBM*').gfilter('NOT', 'IBMUSER')
   or
   r.users.gfilter('IBM*').gfilter('IBMUSER' ,exclude=True)
   so we can show all users starting with IBM, except IBMUSER.

I wonder, should be drop gfilter and rfilter, and take a different verb?

a. take it or leave it:
r.users.take('IBM*').leave('IBMUSER')
b. leave feels like "leave it in" and "skip" is the opposite of include:
r.users.take('IBM*').skip('IBMUSER')
but also
r.users.skip('IBMUSER')

The straight-forward use of take and skip should focus on RACF idiom, so it uses generic patterns. More advanced users might write regex filters, but they would be familiar with Python's r-string type (to take the backslash at face value).
But there is no way for the function to check if the caller entered an r prefix for the string. And we do not want to force the more advanced user to always start their regex pattern with a ^.
So what it we add an 'r' keyword in these methods that activates regex, like so:

r.users.take('r', 'IBM.*').skip('IBMUSER')

or more intuitively for regex connoisseurs force them to enter the program object or package name (basically anything that is not a str:

r.datasets.take(re, r'SYS\d\..*').skip(re, r'SYS1\.\*\*')

r.generals.take(re, 'FACILITY', r'BPX\..*')

There are too many deficiencies with gollum (github's version of Wiki), in combination with sphinx.

• sphinx does not generate a _Sidebar.md, you have to rename the index.md into _Sidebar.md
• gollum doesn't accept .md filetypes in URIs, you have to add a sphinx global option to kill those in the generated md flies
• gollum does not respect directories (folders) in the URI of links, so the source directory with autodoc files results in 404 errors from the _Sidebar, meaning, you would have to edit _Sidebar.md anyway
• there is only 1 Wiki link that is branch independent
• you have to maintain a separate git and do a separate push to update the Wiki
• a summary

Advantage of Wiki? Ehrm, it displays all of the docs contents, where github's interpretation of rst files is lacking (doesn't support table from file, doesn't interpret the autodoc directives).

So, lets

1. rename the _build directory to build,
2. build md files after any change using make markdown,
3. git add docs/build/markdown/* (including ./source)
4. put a link to docs/build/markdown/index.md into README.md
5. deactivate Wiki
6. Bob's your uncle

This works, except...

1. Make sure the .md filetype is present in the URI of all links
2. selecting the link from README, you are dumped into Preview mode, with the navigator as a sidebar, and no fancy sidebar.
3. somehow, the one single screenshot in Installation results in 404. That's because Installation.md looks for _static/file.png, whereas the file really is in ../_static/file.png

Still, it does provide a manual for the working branches.

When requesting a record type that's NOT in the unload, parse_fancycli fails.... as the 'key' is not in the stats..._records thinggy

I should use

# Autofit the worksheet.
worksheet.autofit()

Here :)

Looks like we cannot always 'just' activate the _correlate.. as there can be required other types. For instance, running a parse on users and groups only yields the following:

r2.parse(recordtypes=['0100','0200'])
True
>>> Exception in thread Thread-2 (parse_t):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/henri/repos/pyracf/venv/lib/python3.10/site-packages/pyracf/__init__.py", line 339, in parse_t
    self._correlate()
  File "/home/henri/repos/pyracf/venv/lib/python3.10/site-packages/pyracf/__init__.py", line 379, in _correlate
    raise StoopidException("Need to parse GPMEM and USCON first...")
pyracf.StoopidException: Need to parse GPMEM and USCON first...

Maybe not run _correlate if recordtypes is specified?
Maybe parse required records regardless of what user specified?
Maybe even say goodbye to the selection of what records to parse, as parsing is lightning-fast now anyway?

Need to add to offsets and in parsing et al...