MBSE with ChatGPT: A (scientific) field report
There has actually been far too much written about hyped technologies such as generative AI recently (much of it hot air). But the publication in a journal (Software and Systems Modeling at Springer) gave me hope that the authors would provide some substance for the discourse. And they did. The authors examine the use of ChatGPT in software modeling to see how useful and mature the technology already is. The authors come to a similar conclusion to mine: there is still a lot to do!
The article
Specifically, it is about the article On the assessment of generative AI in modeling tasks: an experience report with ChatGPT and UML. The authors are researchers at the University of Malaga. One of the authors has in the MoLa blog a summary in English written.
The authors address a total of 8 questions. Similar to In my work, the authors rely on PlantUML for the visualization.
The authors conducted experiments in two phases. First, they familiarized themselves with ChatGPT and explored its possibilities and limitations. One interesting finding was that ChatGPT is not particularly good at dealing with abstract entities. Creating a model for a vehicle, its wheels and engine is much easier for ChatGPT than an abstract description of a system consisting of the elements A, B and C.
The aim of the second phase was to find answers to the specific research questions. To this end, the authors created a total of ten exercise examples with sample solutions. The aim was then to get ChatGPT to produce the sample solution as a result.
1. does ChatGPT generate syntactically correct UML models?
Yes - at least quite often. However, the quality of the result seems to depend on the language selected. Besides PlantUML the authors USE one. Generated PlantUML had significantly fewer syntactical errors than generated USE.
2. does ChatGPT generate semantically correct UML models?
Before we can answer this question, we need to consider what semantically correct actually means. The authors mean that the model reflects the intention of the author. The result is sobering: ChatGPT often produces models that make little sense from a modeling or domain perspective. I also found this to be the case in my experiments.
The authors also struggled with unrepeatable results. However, this is due to the fact that they used the chatbot and not an API to GPT. The API allows much more precise settings to be made, how strongly chance may influence the answer ("Temperature"). For ChatGPT, the value is 0.7. For my experiments, I used the API and set the value to 0 in order to have as much determinism as possible.
The determinism of GPT can be significantly increased via the API, in contrast to ChatGPT
How sensitive is ChatGPT with regard to domain and context?
Both points have a significant influence on the result. As already mentioned, ChatGPT generates good models for known domains and poor models for abstract or unknown domains. Furthermore, the models built in a longer chat context are usually better. The authors had good experience with generating a comparatively small model in the first step and then expanding it in dialog with ChatGPT.
How big can the models be?
ChatGPT has serious problems with models consisting of more than 10-12 classes. Scaling is therefore a significant problem.
Which modeling concepts work well, which don't?
ChatGPT recognizes associations quite well. Aggregation, composition and inheritance also work reasonably well. As the models are very small, this result should be treated with caution.
Other concepts require explicit specifications in the prompt. This applies, for example, to enumerations, multiple inheritance and constraints. ChatGPT was even worse at dealing with abstraction.
ChatGPT has major problems with abstraction
How strongly does the formulation of the input influence the result?
As mentioned, the authors had to contend with variability due to the rather high default temperature (0.7). Accordingly, ChatGPT often produced different results even with the same input. The authors also occasionally ran into endless loops. In such situations, the authors ended the dialog and simply started a new one.
How do formulation strategies influence the result?
As expected, the formulation of the query to an AI system is of great importance. In this respect, the question here is not "whether", but "how extremely" the formulation affects the result. The authors have developed a number of strategies. These include the iterative construction of a model, dealing with variability and deciding at which point it is better to start a new dialog.
The formulation of input for AI systems is so important that it now has its own job title: Prompt Engineer
How sensitive is ChatGPT with regard to the selected notation?
As already mentioned, the system works better with PlantUML than with USE. The authors also experimented with OCLfor which the results are often incorrect.
This result is actually not particularly surprising: After all, GPT was trained with the information on the Internet, and we find considerably more PlantUML models there than USE models or OCL fragments.
SysML v2 was specified with its own text notation, which would of course be perfect for the output. However, as the Internet provides virtually no examples of this, we can forget about this approach, at least today.
Conclusion
The authors see potential for ChatGPT to support activities in MBSE. However, due to the scaling problem alone, the authors speak of "model fragments" and not models. In this role, the system can support modelers in finding solutions for specific situations.
The authors also point out the possibility of generating test cases from models, even if they did not explicitly evaluate this approach. Personally, however, I am skeptical here: the moment we have a model, we can also generate test cases without AI. There are already a lot of tools for this task.
The authors rate the use in education as the most promising, at least today. An AI system can support students in learning, provide feedback, evaluate models, etc.
One thing is clear: Generative AI, such as ChatGPT, can support MBSE activities. There is also sure to be a "killer app" that will turn the market upside down in the next few years. Only time will tell when this happens and which company will be the first to achieve success here.
