Automatic Text Summarization with Neural Networks

Automatic Text Summarization is one of the more interesting fields in text processing and text analysis. I had previous experience with artificial neural networks (ANNs) programmed in MATLAB and decided to implement the model by Khosrow Kaikhah, published in June 2004 under the same title as this blog entry. I did not follow the paper exactly, especially in regard to the ANN architecture.

The basic premise is relatively simple: One seperates texts (six simple texts from www.newsinlevels.com) into paragraphs and single sentences. Each sentence is then analyzed in regard to the following seven features:

1. Sentence is in Paragraph which follows the Title [0/1]
2. Location of the Paragraph containing the Sentence within the Text [0-1]
3. Location of the Sentence within the Paragraph [0-1]
4. First Sentence in Paragraph [0/1]
5. Length of the Sentence [0+]
6. Number of the Text's most important (frequent) Words in the Sentence [0+]
7. Number of Title Words in the Sentence [0+]

All these values are retrieved with different MATLAB functions and constructs. In regard to feature number (1), this value is "1" for all sentences which are in the leading paragraph and "0" otherwise. For (2), the position within the text is given in percentages. Again, this value is the same for all sentences in each paragraph. A text with 3 paragraphs would result in giving all sentences in the first paragraph the value 0.333, since one is "a third" through the paragraph after reading this paragraph. The last paragraph (or more precisely, each sentence in it) would have the value "1". Feature number [3] is exactly the same, only in regard to the position of the sentence within the paragraph, giving every sentence a unique location, with the last sentence having the position "1". [4] is easy, either "0" or "1" depending on whether or not the sentence is the first in the paragraph. Feature [5] simply shows how many symbols are contained in the sentence. The last two features are slightly more complex. [6] basically creates a list of the most used words in the text. I decided to use the three most frequent words for each text and counted for every sentence the respective occurrences. The last feature [7] counts how often a word contained in the title is occurring in the sentence. For the last two features, I only used words with a minimum length of 4/3, so I could erase common words like "the" or "and" from the text analysis and very short words common in titles like "a" or "is" from the title analysis.

Basically, the selection of these exact features is dependent on some studies and general assumptions that a sentence should be more important if its "scoring" high on several of those measures. For a more detailed intuition, please refer to the original paper. The picture above shows the resulting array for the first few texts (every row is a sentence). The columns represent the features from [1] to [7]. This array is then z-score normalized, which is good practice for ANN inputs.

Obviously, before we can train the ANN, we need to provide it with targets. So a test person summarizes (selects summary sentences) six test texts. This is only a "0"/"1" vector, implying whether or not the respective sentence is a summary sentence or not.

I used a simple feed-forward ANN with three layers, four hidden neurons in the hidden layer and the Levenberg-Marquardt training algorithm. The input layers are obviously the seven input feautres, and the output layer consists of a single node. The neural network learns to map the sentences in the training sample to the target vector (sentence is summary sentence or not) by adapting internal structures, processing weights or other methods. After sufficient rounds of training, MATLAB terminates the process.

Now, it is really simple. We have the trained network, and "show" it another text, one the test person has never seen before (see picture above), but should be reasonably similar to the other texts. As a result, it produces a value from "0" to "1" for each sentence in the new text. If this value is reasonably high (i chose a value of over 0.8), I would assume that the ANN recognizes it as a summary sentence. Obviously, this threshold is not objective, maybe a sentence limit would be appropriate, e.g. choose a threshold so that a summary of 3, 4 or 10 sentences is produced.

If we now only "count" the sentences which are classified as being summary sentences, the following summary is produced (all lower case due to the processing; keep in mind that this is an article written in very simple english)

"a man from slovakia is happy. he is working on a flying car. the car has an original design. when you want to fly, the car must change. people want to make the car which can fly next year for others who will buy this car."

This is by no means perfect, but for a very low amount of training texts (six), relatively short texts (three to five paragraphs with one to six sentences per paragraph) and a very quick choice of the ANN structure, the result is pretty good.

The best part is that it includes the test reader's own personal reading/summarization style. With further work, this model could be a very legitimate tool for automatic text summarization.