The reward function: the linchpin of AI systems
When we develop systems for people, we have to make important decisions: Who are the users? What is important to the users? What is valuable to them? What problem will the system solve for these users, and how do we solve it? How do we know if we are ready? We need to answer all these questions, regardless of whether the target system uses AI or not. However, if it is an AI-based system, then the reward function plays an important role. Why this is the case and how we use it is the subject of the following section.
An AI system uses the reward function to define successes and failures. To function properly, this should be consciously defined and optimized with a team of key stakeholders.
Before we turn to the reward function, we need to clarify two other questions: First, do customer value and AI even go together, and if so, how? Secondly, do we use AI for the automation or augmentation (support) of activities? With this knowledge, we can then design the appropriate reward function.
The one described here is based on the People + AI Guidebook from Google. Fittingly also here the two-piece RE4AI.
Intersection of user needs and AI strengths
As with any development, identifying the right problem is one of the most important tasks. This step has nothing to do with AI to begin with and is well understood.
We typically use AI for the processes required for the Automation or Support of activities. We should already be thinking about this at this stage. A good technique is to examine existing workflows. If we "look over the shoulder" of the users, we can identify bottlenecks. This does not have to happen at work, workshops or interviews are also techniques we can use.
Does AI really make the product better?
It is important to question whether AI really makes the product better. Often a rule-based or heuristic solution works just as well, if not better. A simpler solution has the added benefit of being easier to create, explain, debug and maintain. AI can even make the product worse in some situations.
AI is probably better
- Recommendation of different content for different users
- Prediction of future events.
- Personalization
- Understanding natural language
- Recognition of a class of objects or elements
- Detection of low-frequency events that change over time
- A domain-specific agent or bot function
- Dynamic content
AI is probably worse
- Predictability has a high priority.
- Task with static or very limited data.
- Minimization of expensive errors.
- Complete transparency is required
- Optimization for high speed and low costs
- Tasks with high value
Automation or augmentation?
There are tasks that people would like to have done by AI. But people also like to be in control and want to do many tasks themselves. In these cases, AI can help to complete tasks faster, more efficiently and sometimes even more creatively. Ideally, automation and augmentation work together to simplify and improve the outcome of a long, complicated process.
The following lists help to decide whether (or in which areas of the task) we should automate or support:
Automation
Success is measured with:
- Increased efficiency
- Greater security
- Reduction of tedious tasks
- Enabling new features that would not have been possible without automation
Criteria for the use of automation:
- Users lack the knowledge for the activity
- Tasks are boring, repetitive, cumbersome or dangerous
Augmentation
Success is measured with:
- Users have more fun with a task
- Users need more control than would be possible with automation
- Responsibility remains with the user
- Users can scale their services
- Creativity increases
Criteria for the use of automation:
- Users enjoy the activity
- Responsibility for the result lies with people
- The stakes are high
- Clear preferences are difficult to communicate
The Reward Function
Every AI model is controlled by a "reward function". Other terms are reward function, target function or loss function. This is a mathematical formula that the AI model uses to distinguish "correct" from "incorrect" predictions. It determines the action or behaviour that our system tries to optimize for and is an important factor in the final user experience.
Many AI models predict whether a certain element belongs to a certain category or not. Such a binary classification is a good way of explaining how the reward function is used.
An AI model can be right or wrong, and there are two types of "right" and "wrong", as the following matrix illustrates:
We use the reward function to weight the results. In the simple example of a binary evaluation, we can therefore assign different weightings to the four results. Weighing up the costs of false positives and false negatives is an important decision that will shape the experience of our users.
It is tempting to weight everything equally. However, this is not what users want. For example, if an AI is supposed to recognize a dangerous situation, then a false negative could lead to great harm. However, if there are too many false positives, this could annoy users to such an extent that they switch off or ignore the system.
Precision and recall
Precision and recall are the terms that describe the breadth and depth of results that AI delivers to users and the types of errors that users see.
The higher the precision, the more certain we can be that the results of the model are correct. The disadvantage, however, is that the number of false negatives increases, as potentially relevant results are excluded.
Recall refers to the proportion of true positive results that were correctly categorized out of all true positive and false negative results. The higher the recall rate, the more certain we can be that all relevant results are included somewhere in the output. The downside, however, is that the number of false positives increases, as irrelevant results may have been included.
The following picture makes the whole thing a little clearer:
We have to take these compromises into account during development - there is no way around it. Each product is individual and it depends on what our users expect and what gives them the feeling of completeness of the task. Sometimes it helps users to see some lower certainty results alongside the 100 % results so they can be sure the system hasn't missed anything. In other cases, displaying lower confidence results might make users trust the system less.
Evaluation of the reward function
When developing the reward function, it is essential that we involve the stakeholders. How many users there will be is also relevant here. Here are some criteria for the evaluation:
- Assessment of inclusivity - Inclusivity means ensuring that all product users are taken into account. This includes people with different backgrounds, different perspectives and dimensions such as origin, gender, age or body shape, to name but a few.
- Evaluation over time - The reward function should take into account the effects over time, e.g. how long individual users have been using the system.
- Creation of sample scenarios - Our system could have indirect effects on users. These are notoriously difficult to predict, but it is worth taking them into account when designing your reward function.
Be prepared for negative effects
The more use cases AI covers, the more important it becomes to monitor the negative effects and respond to them appropriately. Even with the best planning, it is almost certain that situations will arise that we have not anticipated. We should therefore regularly review the relevant key figures and triggers. Incidentally, this is where it is important, MLOps so that we can continuously adapt the system without any problems.
Conclusion
The first step for a successful AI product is to tailor your product to the needs of the users. Once we have found this, we need to check whether AI really is the best approach. If this is the case, then the reward function plays a central role in ensuring that our product provides a great experience for all your users in the long term.
Image source: Source: PAIR Guidebook, BY-NC-SA
