Measures of... everything measurable in systems engineering
Predictability is one of the many reasons why we Systems Engineering practise. But we can only calculate if we also measure. ISO 15288 defines a separate process for collecting assessments (measurement process) as well as several assessment categories (e.g. Measures of Effectiveness, MOE).
As we have more options for processing than ever before ("big data"), we should make sure that we make the best possible use of the options for evaluation. Below is a brief overview of the available assessments/measures, what the processes look like and what we can do with them.
Systematically collect valuations
The ISO 15288 defines its own "Measurement Process" in Clause 6.3.7. I am referring here to the ISO/IEC/IEEE 15288:2015 available to me, which was revised last year. However, it is the same clause in the 2023 revision.
The INCOSE Systems Engineering Manual (Review) describes in somewhat more detail than the standard how the assessment process is structured.
The purpose of the evaluation process is to collect, analyze and document objective data regarding our development. This enables us to ensure that our product meets the quality requirements of our stakeholders. This is used by the Project management.
Ratings as part of the feedback loop
Appropriate valuations and indicators are an essential part of trade-off analysis to balance cost, schedule and technical objectives.
Assessments must be relevant. The first step is therefore to analyze which assessments we need and why ("measurement needs"). We can then derive the required data from this. We need to store this data. Among other things, this helps us to recognize trends. We need a strategy to be able to handle the assessments with confidence.
Ratings should be easy to collect and measure: Ideally, they should even be automatically generated by our development system so that they are always up to date.
Evaluations must be evaluated regularly and quickly so that we can react to the results.
What should I measure?
The most important ratings are:
Measurement of effectiveness (MOE)They are decisive for how well the system fulfills the intended purpose from the customer's point of view.
Examples: Customer satisfaction, mission success rate or operational availability.
Measures of Performance (MOP): These focus on the technical performance of the system and often relate to the design.
Examples: Speed, accuracy or throughput.
Assessment of suitability (Measures of Suitability, MOS)These evaluate the suitability of the system for use in certain operating environments or conditions.
Examples: Reliability, maintainability or user-friendliness.
Evaluation of technical performance (Technical Performance Measures TPM): They track critical technical attributes of the system during development to assess progress and compliance with requirements.
Examples: Weight, power consumption or signal integrity.
Further information in the SeBoK
Many organizations already have their own evaluation criteria that are independent of development, but which can and should still be included in systems engineering. For example, key performance indicators (KPIs), which typically evaluate the economic success of the system, are widely used.
The Systems Engineering Book of Knowledge has a own page on the subject of measurement. The concepts presented here are discussed in much more detail there. You will also find examples, best practices and much more.
Conclusion
We live in the age of "big data": this is nothing new for us systems engineers. Nevertheless, or perhaps precisely because of this, we should regularly address the topic of evaluation: Because modern approaches to dealing with evaluations have enormous potential to accelerate development and make it predictable.
Photo by Ekaterina Grosheva on Unsplash
