Collect AI data correctly in 6 steps

By now, word has gotten around that training data plays a central role for AI models. But how do we get good AI data? And since everyone is currently talking about "Artificial General Intelligence", the question arises as to whether we still need training data at all. The short answer is: Of course! Read on to hear the long answer.

The one described here is based on the People + AI Guidebook from Google. Three weeks ago I already wrote a summary of the first chapter, which deals with needs and success criteria went. Today, we will focus on the content of the second chapter, the collection of data and its evaluation.

Why AI data is so important

In order for AI-based products to make predictions, we need a learning model so that the product can recognize patterns and correlations in data. This data is called training data and can be collections of images, videos, text, audio and more.

We can either use existing data sources or explicitly collect new data for training purposes. We may already have data, but often we would use open or commercially available data sources. For example, we could use an existing catalog of birds to develop a bird detection system.

However, the data must be labeled. For example, not only should the image be labeled with the bird shown on it, ideally the area in the image where the bird is located should also be marked.

The quality of the AI data directly determines the performance of the system and therefore user satisfaction. At this point, it is worth thinking again about whether an AI system really produces better results than another approach.

If we are sure that machine learning should be used, then we can achieve this in six steps:

① Work with high-quality data right from the start

If we do not pay attention to the quality of the data from the outset, then there is a risk of Data cascade. This means that negative effects accumulate that are difficult to correct afterwards.

But what is high data quality anyway? High quality AI data...

• Precisely represent a real fact or a real entity
• Were collected, stored and used responsibly
• Are reproducible
• Can be maintained over time
• Are reusable in all relevant applications
• Have empirical and explanatory power

② Translate user needs into data properties

We have already established that training data consists of examples and labels. Furthermore, data has features that are of interest to us. For example, a feature of a bird could be its color or wingspan. Which features are important for the problem to be solved and what do we need to consider?

These relationships are not simple, so we should definitely create a specification for our data set that collects the relevant requirements regarding the AI data in a verifiable way.

The creation of a data set specification is strongly recommended

We have a lot to consider here: Is the data balanced? How was it collected? How was it transformed? The chapter in the PAIR book contains helpful checklists. The most important points are

• Are the data representative and complete: If, for example, an app is to be used for the detection of birds worldwide, then a database of European birds is not enough.
• Are all the required features available? Since male and female birds sometimes look extremely different, the sex should be included as a feature in the database.
• Are the labels well selected and consistently applied? This also includes ensuring that labels from different data sets are consistent. An example would be the label "child", which could include different age groups in different data sets.
• Systematically collect data records: Based on the specification, possible data sets must be checked for suitability. This also includes topics such as data protection or systematic processing, as well as a strategy for updating the data.
• Record the need for data: Quality is not the same as quantity. We should think about how much data we really need. This may mean using only some of the existing AI data and supplementing the data with newly collected data in a targeted manner.
• Balancing over-adjustment and under-adjustment: Overfitting means that the ML model is tailored too closely to the training data. An overfitted model can make very good predictions on the training data, but perform worse on new data. Underfitting is the opposite, where a model has not correctly captured the complexity of the relationships between the features of the training dataset. It therefore cannot make good predictions with new data.
• Maintaining fairness: At each stage of development, human biases can be incorporated into the ML model. The data is collected from people in the real world and reflects their personal experiences and prejudices. If we are not careful or do not actively counteract them, these patterns can be implicitly recognized and reinforced by the ML model.
• Use data from as many different groups as possible: The training data should reflect the diversity and cultural context of the people who will use it. To check this, there are tools such as Facets orWIT.
• Managing data protection and security: In order to cover these topics, we may need to consult experts such as lawyers in addition to appropriate data protection agreements and audits.
• Regulate user consent to the use of data: Different countries have different rules that we must observe. We should also ensure that our behavior is perceived as transparent by users.
• Minimize the risk of user identification: Is there a risk of inadvertently disclosing user data? What would be the consequences?

③ Responsible data collection

Now we can start collecting the data. The preparations just described should ensure that we collect the correct data in a legally clean manner.

The easiest and cheapest way is to use data that has already been collected. As described above, not all data is suitable and may still need to be processed. If we need to collect new data, a dataset specification helps to get this right from the start.

New data can be collected implicitly or explicitly. For example, the telemetry of an app usage is an implicit data collection, while a user survey is an explicit data collection.

The classic question: "Was this result useful?" is a form of explicit data collection.

To ensure the quality of the data in the long term, it must be maintained. We should also document this. Maintenance falls into three areas: Preventive maintenance ensures that certain potential problems do not arise in the first place. Adaptive maintenance adapts existing data to a changing reality, such as changing map material. And finally, corrective maintenance eliminates identified problems.

④ Prepare and document data

How do we know that our model works? We need not only training data but also test data. To do this, we divide our AI data into these two groups. After we have trained the data with one part, we can test the model with the second part. A typical split is 60% (training) to 40% (testing).

The AI data is usually processed in a pipeline, whereby the term MLOps established, analogous to DevOps. We also need to document the data, likewise analogous to the documentation of software code.

The following have proven to be suitable for documenting AI data Data Cards established.

Data cards are structured summaries of key facts about various aspects of ML datasets that are required by stakeholders throughout the lifecycle of a project for responsible AI development.

The Data Cards Playbook

⑤ Preparation for labeler

We have already learned that correct labeling increases the quality of the model. Labelers can be humans, but also automatic processes. However labeling is done, we need to take it into account in our development process.

If we use people for labelling, diversity is also helpful here to neutralize unconscious bias. Furthermore, we should provide them with a guideline to ensure consistent labeling.

Labeling should be supported by suitable tools and support the labelers' workflows. It can also be useful to take the labelers' self-confidence into account and, in case of doubt, to use the results of several people.

⑥ Tuning the model

Once we have trained our model with the training data, we evaluate the results. This allows us to determine whether the model meets the needs of the target users according to our success metrics. If not, we need to adjust it accordingly.

There are tools for this phase, but ultimately we have to test, test, test. So that we can track our changes, we should version the changes made to the model and the data. To get really good results, we need to do a lot of rounds here, which is why a properly established MLOps can avoid a lot of frustration.

Conclusion

Data is the foundation of every ML system. Responsibly sourced data helps us to develop better systems and thus better meet the needs of users.

This is not the end of the journey to a successful AI system, but we have covered one of the most important aspects.

Photo from Mika Baumeister on Unsplash