s-e-d-a / seda-ml-summarization Goto Github PK

Currently, we know that attempting to parse data from 3,890 documents, with 115,000 distinct words, takes about 24.2 GB of RAM. We should find a way to parse the data using a more reasonable amount of memory.

Potential ideas include streaming the data during parsing with an efficient module like pandas, or using memmap to prevent keeping all of the data in RAM.

Need to be able to read in the formats of the datasets for training and the text generated by the text detection

Right now the DBN requires some nonempty validation/test sets to run without error (because of the construction of the theano code). Since these are unnecessary in unsupervised learning they should be removed.

Query-Oriented Multi-Document Summarization via Unsupervised Deep Learning Paper

Currently downloading, installing, and running Jetty Solr takes a considerable amount of time.

Need to set a variable of some sort to indicate it has already been done so re-provisioning is not bogged down every time.

Summary Generation Sub-Tasks in-order

Solve the optimization problem. This is the fourth and final step in summary generation.

Classical knapsack problem in dynamic programming

f_k (lambda_k) = max{mu_k In_k + f_{k-1} (lambda_{k-1}}
lambda_k = lambda_{k+1} - mu_{k+1} l_{k+1}
K = t, 1 <= t <= T
lambda_0 = 0
lambda_T = N_s
f_0 (lambda_0) = 0

f_k (lambda_k) = max of summary importance in stage k
K = stage variable to describe current sentence
lambda_k = remaining length before K starts
mu_k = decision variable for sentence

Need to formulate a method to check whether a core does not exist, that way if it is the first time indexing to a new core, the core should automatically be created without having to specify -n for new.

Basic building block for the hidden layer.

RBM is a two-layer recurrent neural network in which stochastic binary inputs and outputs are connected using symmetrically weighted connections.

The initial parameters of the first RBM are also determined by the query words.

numpy, scipy, pandas, scipy, vincent, ipython, ipython-notebook for using web browser http://localhost:18888 for development

Summary Generation Sub-Tasks in-order

Word Extraction from Importance Matrix AF. This is the second step in summary generation.

AF_i,n = importance of ith word in vocabulary to the nth node of the hidden layer H3 (extraction layer)

Extract 10 words with the largest AF_i,n value in every nth node of the hidden layer.
This set of words is defined as UN

In order to run the Deep Belief Network on the term frequency data, the dataset field in DBN.py needs to reflect the expected input.

The format appears to be

[ [train_x, train_y], [valid_x, valid_y], [test_x, test_y]]

where train_x, train_y, valid_x, valid_y, test_x and test_y are numpy arrays representing the corresponding array.

We need to take the term frequency data and wrap it up in these arrays and then run

cPickle.dump(your_own_dataset, open("file.pkl", "wb"), protocol=-1)

Making the Theano_DBN features to dict script component generalizable to generic input

Generate Documents into an uploadable format and upload them to SOLR server.

Decide on the best format to receive and how to receive it.

First, generate a vocabulary with length V based on the words appearing in document topic set D. (V is the length of the vocabulary of D.)

Then, calculate the feature vectors

of the document set D and

of the single document .

is the tf (term frequency) value of the
word in the vocabulary of D calculated in all documents.

is the tf value of word in the vocabular of D calculated in .

Need to figure out the correct combination of parameters in order to make sure Solr loads the cores on restart

For example:

python DBN_Symmetric_RBM.py "../data/australianLegalDataset_term_frequencies"

python DBN_Symmetric_RBM.py "../data/nist_dataset"

A script is needed to generate the summary label from the document (catchphrases I think) and then evaluate the generated summary with the label.

Summary Generation Sub-Tasks in-order

The importance of every sentence In_t must be calculated. This is the third step in summary generation.

In_t = \sum_i w_i

where

w_i = lambda if mu_i is contained in UN AND mu_i is contained in q
w_i = 1 if mu_i is contained in UN
w_i = 0 otherwise

lambda = query word importance factor
mu_i = the word in sentence s_t
Importance of Summary: In = \sum_t In_t such that Le <= N_s
Le = l_1 + l_2 + ... + l_t + ... + l_T <= N_s

Write a script/commands in order to extract features from data on SOLR server.

If evaluation summaries available for documents, use ROUGE toolkit as part of the trainTopic.sh epilogue to determine score of trained dataset.

Reference: http://kavita-ganesan.com/rouge-howto

We need to run extensive tests on this component to ensure it is complete. The base testing confirms that the class runs as expected, but I have not seen it against large amounts of data and there could be an unknown bug I haven't seen yet... But who knows

Summary Generation Sub-Tasks in-order

Importance Matrix AF must be calculated. This is the first step in summary generation.

AF_i,n = importance of ith word in vocabulary to the nth node of the hidden layer H3 (extraction layer)

Format data and make sure data gets into Python correctly.

One of a pair of scripts, trainTopic.sh and runQuery.sh

trainTopic.sh takes arguments for topic and dataset

Installing Solr using wget and running it with Jetty through java -jar start.jar