Emphasis on simple- this is not going to be an optimal preprocessing setup either way, we just want to make it a bit better through simple means that don't require any additional installs, configs, etc.

@netj Chris mentioned doing this (I think you were there?)

Facilitates applying gold standard labels (among other things)
@jason-fries

statspadding

Should we focus on the hold out set? What are the important metrics to users wanting to refine their DSRs?

Apparently there are some problems here with the parser interface (@chrismre)

Useful for NER systems, and unary Relations aren't friendly

Fix getting pushed with #44

E.g. have one for getting to an Extractions object, then dump this, and have a separate notebook for writing rules & testing model?

Fix getting pushed with new example notebook

1. User gets random subsample of candidates in Mindtagger, and labels them
2. User gets statistics over the labeling functions, some w.r.t. to this label set (e.g. empirical acc., etc.)
3. Learn model
4. Get precision stats
5. Log (?) and repeat

Stats to show for label function development:

• Coverage
• Overlap
• Conflict
• Empirical accuracy
• Show labeling functions and/or candidates that are conflict heavy (+ low emp. accuracy lfs)

Questions:

• Should we be proscriptive, and automatically (opaquely) split their label set into a "label fn. validation set" and a test set (as default option which can be turned off)?
• How to integrate ground truth that they bring in externally?

Let the user evaluate how well do their rules work if:

1. We pick the first rule that triggers as the labeler, based on the ordering of rules passed in
2. Majority vote
3. Logistic regression model...

E.g. for an even quicker-and-dirtier option / baselines!

For regex matching, it would be very helpful to have access to the text of a single sentence without any tokenization. When tagging chemical names, for example, we frequently get these tokenization type artifacts:
Li ( 3 ) PS ( 4 ) vs. Li(3)PS(4)
Some of these can be fixed with modified regexes, but it would be nice to operate on the original text itself. As far as mapping back to tokens for entity tags, we could just consider a match as anything that overlaps with the original span.

Desired initial functionalities:

• Take as input a directory filepath, a single filename, a filename pattern
• Strip XML, HTML (i.e. strip tags without corrupting basic sentence structures)

Ideally there would be some simple way to extend so that users could write basic XML/HTML parser modules (e.g. to grab metadata, preserve section structure, etc.) via some python library (e.g. lxml, beautifulsoup). This kind of solution would not be performant, but could potentially be very simple...

Just need to wait one sec...

Each parse_sents call requires a new Java process and a 10sec+ delay to load the models.

Need to be able to set initialization params and "train unsupervised" so scikit-learn version would need modifications... could also use scipy.opt

One simple way to have db connectivity in the notebook is our favorite extension, ipython-sql. We could initially just build some helper functions around this (or any other psql connector).

However, in DDL we pass around an object containing the entire dataset (Relations)- this would allow us to connect to the database in a way that is opaque to the user, turning this Relations object into essentially a cache for the DeepDive db...

What else?

1. Some users' envs may not allow sudo. Even if they do, we'd still want to use virtualenv.
2. We could use a requirements.txt to list all py dependencies.

How do we illustrate best practices for debugging vs. enhancement vs. revision to new users?

When we reopen MindTagger, we can keep the same sample as before, but not reload the tags. @netj is there a nice way to do this with the API like how we retrieve the tags, or should we form and dump tags.json to the instance directory?

See example in ddlite.py

Also, don't pair the same token to itself for a relation

Should give each extraction a unique id

This breaks one cell in the extraction example

DeepDive can do l1 logistic regression, so ddlite should too

We need to output the basic deepdive calibration plots (notebooks are perfect for this!), as well as potentially some other histograms which guide users towards correct error analysis / debugging procedures.

We also need to output the marginals, which is a minor sub-function to add in.

Example notebooks are the de facto documentation. Example notebooks are not actually documentation.

E.g. user should be able to access a path_between attribute of a Relation object, etc. This can / is currently being done with treedlib, however I am trying to decouple these two repos... however can bring this back in under the covers in a more limited form (e.g. Relation objects initialize with several dependency path attributes like path_between, but don't expose the direct XPath mechanisms to the user)

Refine saving and loading annotation dumps from MindTagger during DSR refinement. I realize "items.csv" is dumped in the MindTagger directory under some unique folder id and tags can be fetched using get_mindtagger_tags() on the MindTagger instance, but the metrics associated with these values should be wrapped up in some sort of "classification_report" type function.

This will be almost identical to the existing DictionaryMatch operator.

This RegexMatch operator, if essentially copied from DictionaryMatch, will be able to trivially do e.g. POS tag sequence matches (using match_attrib=poses); however this could be wrapped / presented as a separate operator...?

Support sparse matrices as optional input

Either Extractions should be able to handle a list of list of sentences, or the parsers should output a flat list of sentences

Not a big deal, but shows up as -LRB- and -RRB- in MindTagger which looks like a lot like a gene to the underinformed

Rather than a fully random sample, do we want

• Some all-abstained candidates?
• Some high conflict candidates?
• Some low conflict candidates?
• Some candidates with probability close to 0.5?

Helpful for people interested in Relations-style applications

Would be good to separate the concepts of Extractions as a data container/operator and the learning algorithms it implements? Should also probably implement Relation and Entity using a proxy pattern.

@ajratner let's talk before deciding either way on this?

edit: Adding question marks so it sounds like a question?

Could always run the good ole essential-feature-is-behind-a-paywall con