Survey results 2024: How users see the use of AI in requirements engineering
Some of you may have noticed that I've been running a survey on AI in requirements management with Gunnar Harde on LinkedIn for the last few weeks. The survey is now closed.
In order to better understand the results, we have transferred the use cases examined to five action priority matrices. This allows us to quickly decide how we should take action. Specifically, we answer the following questions for each of the 27 use cases: (1) Is it available and useful? Then use it immediately; (2) If it is not available but useful: Then service providers and tool manufacturers should implement it. (3) Is it available but not very useful? Then it is worth looking at it opportunistically. (4) Or is it not available and not very useful? Then we should simply ignore the use case.
We have posted the results on LinkedIn today: Take part in the discussion there!
Do you prefer English?
The survey
We asked the participants a total of five double questions, all following the same pattern:
- How valuable are the following use cases?
- How usable are the following use cases?
The participants were then each given lists of use cases, which they had to put in order according to the question.
With this approach, we wanted to find out where AI is already well established today and where users see an urgent need for AI solutions.
Example model generation: The participants saw the AI model generation use case in first placebut settled on today's solutions in terms of usability to the last place to.
Question about Requirements Explanation
The subject areas
The survey dealt with five main topic areas relating to AI-based use cases in the Requirements managementExploration, Specification, Allocation, Explanation and Estimation. Respondents were asked to rate the value and usefulness of these use cases.
Exploration
In the first question, on the subject of ExplorationThe topic was methods that enable a requirements engineer to familiarize themselves efficiently with a specific subject area. This includes, for example, answering domain-specific questions based on large data sets or automatically transcribing interviews and whiteboard notes. The aim is to facilitate access to relevant information and automate documentation.
Today's AI co-interviewers are crap!
The result of the survey on the topic Exploration shows a clear dissatisfaction with the current status of AI-supported co-interviewers. Respondents feel that the current solutions are inadequate and unhelpful. Instead of making work easier, AI co-interviewers currently seem more likely to cause frustration. Pure note-taking, on the other hand, was described as a mature, functioning technology.
Specification
The second question, on the subject of SpecificationThe first project, "Requirements Management", related to support for the specification of requirements. This includes the automatic identification of requirements from stakeholder input, the creation of personas and the generation of acceptance criteria and test cases. The focus here is on automation and increasing efficiency in the creation of precise and testable requirements.
AI does not create requirements - it merely checks them
The results of the survey show that respondents see the generation of requirements by AI more as a gimmick. It looks nice in demos, but if the result is a zero-eight-fifteen drone from the internet, then these requirements don't help anyone. Rather than actually creating requirements, the strength of AI lies in the verification and quality assurance of existing requirements. However, this already existed before ChatGPT. Not only that: ML-based quality checkers deliver deterministic results, generative AI does not.
Allocation
The third question dealt with Allocation and the automatic assignment of requirements to specific components, teams or sources. This also includes the traceability of requirements and the detection of dependencies or duplicates. The aim of these use cases is to make the management and distribution of requirements more efficient and to improve collaboration within teams.
Finding dependencies is the way to unleash the potential of AI
The result shows that the automatic identification of dependencies has great potential. This does not just mean classic traceability, but also relationships to customer information, support requests and, of course, to design or tests. However, the participants' assessment that AI is already able to handle this task reasonably well today is only in the midfield. It is clear that any approach to making traceability more efficient and reliable has enormous potential.
Explanation
In the area Explanation the fourth question dealt with the simplification and translation of requirements. The use cases in this area include optimizing the formulation of requirements for better readability, translating them into simple language or other languages and converting models into text and vice versa. The aim is to improve the comprehensibility and accessibility of requirements for different stakeholders.
Deriving models from text is a challenge for AI
The respondents find the idea of deriving models from text very attractive. However, they also see a lot of potential for improvement in the performance of AI. I can understand that, after all I had carried out some experiments on this myself and also examined the literature.
This question was actually a little too narrow: It was about modeling as an aid to explanation. However, the potential for deriving models from requirements goes far beyond the use of explanation.
Estimation
The fifth question dealt with the topic Estimation and focused on automatically estimating the value, complexity and risk of requirements. These use cases aim to support the prioritization of requirements by providing valuable insights into their potential impact and challenges.
AI can only estimate moderately well, there is still room for improvement
Respondents were ambivalent about the performance of AI in this area. However, risk assessment emerged as the most important use case.
It would certainly be interesting to find out what kind of risks we are talking about here: a risk relating to functional safety has a different significance than the risk of not being able to meet production targets due to bottlenecks at the supplier of a microchip.
Conclusion
The survey on AI in requirements engineering shows that users recognize the potential of the technology, but still see considerable room for improvement. In view of the growing economic pressure, the results offer clear recommendations for action: Users should specifically match the available AI solutions with their own needs, while Tool manufacturer and Consulting company become more active in areas where there is a need but a lack of mature solutions.
The road to optimization is still long, but with targeted use and continuous development, AI can change requirements engineering in the long term and make it more efficient.
