Modeling policy texts with AI in SysML
One of the goals of model-based systems engineering (MBSE) is to replace documents with models. However, in addition to technical documents such as specifications, there are many other documents that would benefit from modeling. These include contracts, guidelines, manuals, data sheets and even legal texts.
In March 2022, Max Chen and Shamsnaz Virani Bhada published a paper on the topic entitled "Converting natural language policy article into MBSE model" at an INCOSE conference. They used SysML for this.
The following is the content and assessment of this work.
Little SysML to be seen
To get straight to the point: The paper (download) contains only one SysML diagram, namely a package diagram. Although the authors have oriented themselves to SysML, the relevant subset is very small. Nevertheless, the approach is interesting and is similar to Semiantthe quality assistant for specifications that I co-developed.
The authors presented their paper at the Conference on Systems Engineering Research | CSER 2022 before. The conference is organized by the Systems Engineering Research Center (SERC) together with INCOSE aligned.
Policy texts include complex and sometimes complicated documents that describe structures, definitions, roles and responsibilities. However, even if they are available digitally, they are almost always unstructured (PDF, Wiki, MS Office, etc.). Such documents can quickly show inconsistencies, redundancies, gaps, contradictions and other problems. In practice, reviews are usually intended to find these problems. The quality varies drastically.
The overall quality of the documents (policy texts) varies greatly in terms of structure and language behavior
Max Chen and Shamsnaz Virani Bhada in "Converting natural language policy article into MBSE model"
Some companies use languages such as BPMNbut this is the exception rather than the rule. Therefore, the use of Natural Language Processing (NLP) is attractive for this task.
For this work, the researchers used the standard NLP tool SpaCy (official website) is used. It is designed to be easily integrated and customized. SpaCy is able to utilize common ML features including Part-of-Speech (POS) tagging, dependency parsing, Named-Entity Recognition (NER), Entity Linking (EL), rule-based matching, and much more.
Procedure
Tasks like this typically go through a "pipeline" consisting of several steps. The researchers' system consisted of the following steps:
- Pre-NLP text cleanup: Extract text and section structure from PDF documents
- Structural analysis: Checking the headings of chapters and subsections
- NLP pipeline: Tokenizing, normalizing, parsing, tagging, merging punctuation, merging clusters
- Named Entity Recognition (NER): Examine the text paragraphs for persons, organizations, laws, etc.
- Modeling actions: Searching the text paragraphs for semantic rule matches (actions) and their responsible actors
- Compilation & Integration: Create an integrated SysML model based on the scanned document
Steps 4 and 5 are particularly interesting. NER (Named Entity Recognition) is about identifying the entities of the system. As this involves policy texts, the actuators are the most important entities.
In step 5 (modeling), the aim is to find relationships between actor and activity. For this purpose, the relations identified by spaCy between the parts of the sentence are examined, especially with regard to relations that indicate a dependency between subject and verb.
The final step, the creation of the model, was carried out manually by the authors.
Result
The authors tested the system with two versions of the "Veteran Health Affair Handbook" for which manually created SysML models already existed. They created guidelines for comparing the models, which enabled an objective comparison of the models.
Not surprisingly, good document structures can significantly improve the quality of the result. For example, one of the manuals had explicit headings that were automatically used by the system for the package structure. The other manual had no headings, which led to a much poorer result.
Use existing structures! In systems engineering, we almost always have good structures in requirements management tools that can be used for activities of this kind.
Named Entity Recognition (NER)
In this important step, the authors' results were rather disappointing. Two statistical indicators are often quoted in NLP, Precision and recall. Both were rather mediocre. Even worse, the authors tried to improve the quality of the result by adjusting the extraction rules. Although this improved the result for one of the manuals, it worsened it for the other.
Modeling actions
The system has not covered itself in glory with this step either. Pronouns were a major challenge. In paragraphs, the role was only referenced after the first sentence and not written out again. Good for readability, bad for automation.
Conclusion
The work shown here is a good example of what can be achieved with modern NLP libraries. It shows the possibilities but also the limitations. The fatal thing is that it is unclear how we can overcome the weaknesses. In particular, "generic fine-tuning" seems almost impossible, as the authors at NER had to realize.
There are two concrete solutions to these challenges. The first would be to develop a company- or team-specific system. There is a risk here that the effort required for maintenance and further development will be drastically underestimated. However, such an approach can be worthwhile for larger companies, especially if the initiative is strategically planned and receives appropriate resources.
The second approach is interactivity. Users have a higher tolerance for incorrect results if they can correct them once and then they never appear again. The authors are planning to develop a user interface for further work. I would like this to be geared towards interactivity.
