The segmentation model is split into two smaller models. The first model segments the amendment text from the surrounding text like introductory and closing words. The second model segments individual articles and the amendments within them.
The most notable extracted features are:
Whether the text is inside a heading;
Whether the text starts with a list marker, e.g.: 1., a., 1), etc.;
Whether the sentence matches a Regular Expression testing for article titles.
The tags used are:
preamble
heading
paragraph
amendments
article
clause (clause of an article, can be nested)
postamble
The first model is evaluated on the entire text of the amendments documents. The result of the first segmentation is then fed into the evaluation of the second model which segments the individual articles and amendments within them.
Both models are trained using Conditional Random Fields on a training. The corpus size of the text segmentation model is 35 documents. The corpus size of the amendments segmentation model is 40 documents. Both corpora are drawn from published Dutch amendment documents and tagged by hand. Both models were trained with a maximum iteration count of 1000 using L2 regularization running on 8 threads in parallel. The training of the text segmentation model took 4 minutes while training the article model takes around 6 minutes on a laptop with an Intel i7-4702HQ processor and 16GB of RAM. Training was clearly CPU bound; the 8 logical processors were 100% utilized. Memory usage was around 1.5 GB. Training speed can possibly be improved using the GPU rather than the CPU. However, this was not explored.
The trained models are evaluated against previously unseen examples. Both models are scored on the overall performance of all their labels. Both models are scored using the accuracy and F1 metrics which are common scoring metrics.
Table 1. Evaluation results
|
Model name |
Accuracy |
F1 |
|---|---|---|
|
Text segmentation |
99,53% |
99,72% |
|
Amendment segmentation |
95,36% |
95,31% |
As seen in Table 1, “Evaluation results”, the models are quite accurate. This is mostly due to the predictable and precise nature of the amendment documents.
The result of the evaluation of both models is fed back to the UI where the user sees a suggestion to apply markup an entire article by pressing one button. This UI allows the user to make corrections in case the model did not predict the correct structure.