Some of you may have noticed that I've been running a survey on AI in requirements management with Gunnar Harde on LinkedIn for the last few weeks. The survey is now closed, here are the results!
In order to better understand the results, we have transferred the use cases examined to five action priority matrices. This allows us to quickly decide how we should take action. Specifically, we answer for each of the 27 use cases: (1) Is it available and useful? Then use it immediately; (2) If it is not available but useful: Then service providers and tool manufacturers should implement it. (3) Is it available but not very useful? Then it is worth looking at it opportunistically. (4) Or is it not available and not very useful? Then we should simply ignore the use case.
We published the results on LinkedIn: Join the discussion!
Prefer German?
The Survey
We asked the participants a total of five double questions, all following the same pattern:
- How valuable are the following use cases?
- How useful are the following use cases?
The participants were then each given lists of use cases, which they were asked to rank in order according to the question.
With this approach, we wanted to find out where AI is already well established today and where users see an urgent need for AI solutions.
Example of model generationThe participants saw the AI model generation use case in first place, but ranked today's solutions last in terms of usability.
Question about Requirements Explanation
The Subject Areas
The survey focused on five main areas of AI-based use cases in requirements management: Exploration, Specification, Allocation, Explanation and Estimation. Respondents were asked to rate the value and usefulness of these use cases.
Exploration
The first question, on the topic of exploration, dealt with methods that enable a requirements engineer to familiarize themselves efficiently with a specific subject area. This includes, for example, answering domain-specific questions based on large data sets or automatically transcribing interviews and whiteboard notes. The aim is to facilitate access to relevant information and automate documentation.
AI Co-Interviewers Suck Today!
Interestingly, the results of this survey show a clear dissatisfaction with the current status of AI-supported co-interviewers. The respondents feel that the current solutions are inadequate and unhelpful. Instead of making work easier, the AI co-interviewers currently seem more likely to cause frustration. Pure note-taking, on the other hand, was described as a mature, functioning technology.
Specification
The second question, on the topic of specification, related to support for the specification of requirements. This includes the automatic identification of requirements from stakeholder input, the creation of personas and the generation of acceptance criteria and test cases. The focus here is on automation and increasing efficiency in the creation of precise and testable requirements.
AI doesn't create requirements - it checks them
The results of this survey show that respondents tend to view the generation of requirements using AI as a gimmick. It looks nice in demos, but if the result is a zero-eight-fifteen drone from the internet, then these requirements don't help anyone. Rather than actually creating requirements, the strength of AI lies in the verification and quality assurance of existing requirements. However, this already existed before ChatGPT. Not only that: ML-based quality checkers deliver deterministic results, generative AI does not.
Allocation
The third question dealt with allocation and the automatic assignment of requirements to specific components, teams or sources. This also includes the traceability of requirements and the detection of dependencies or duplicates. The aim of these use cases is to make the management and distribution of requirements more efficient and to improve collaboration within teams.
Finding dependencies is key to unleashing AI potential
The result shows that the automatic identification of dependencies has great potential. This does not just mean classic traceability, but also relationships to customer information, support requests and, of course, to design or tests. However, the participants' assessment that AI is already able to handle this task reasonably well today is only in the midfield. It is clear that any approach to making traceability more efficient and reliable has enormous potential.
Explanation
In the area of explanation, the fourth question dealt with the simplification and translation of requirements. The use cases in this area include optimizing the formulation of requirements for better readability, translating them into simple language or other languages and converting models into text and vice versa. Here, the aim is to improve the comprehensibility and accessibility of requirements for different stakeholders.
Modeling from text is key - and challenging for AI
The respondents find the idea of deriving models from text very attractive. However, they also see a lot of potential for improvement in the performance of AI. I can understand that; after all, I had conducted some experiments on this myself and also examined the literature.
This question was actually a bit too narrow: It was about modeling as an aid to explanation. However, the potential for deriving models from requirements goes far beyond the use of explanation.
Estimation
The fifth question addressed the topic of estimation and focused on automatically estimating the value, complexity and risk of requirements. These use cases aim to support the prioritization of requirements by providing valuable insights into their potential impact and challenges.
AI is only moderately capable at estimation, there is room for improvement
Respondents were ambivalent about the performance of AI in this area. However, risk assessment emerged as the most important use case.
It would certainly be interesting to find out what kind of risks are involved here: a risk relating to functional safety has a different significance than the risk of not being able to achieve production targets due to bottlenecks at the supplier of a microchip.
Bottom Line
The survey on AI in requirements engineering shows that users recognize the potential of the technology, but still see considerable room for improvement. In view of the growing economic pressure, the results offer clear recommendations for action: Users should specifically match the available AI solutions with their own needs, while tool vendors and consulting firms can become more active in areas where there is a need but a lack of mature solutions.
The road to optimization is still long, but with targeted use and continuous development, AI can change requirements engineering in the long term and make it more efficient.