Analyzing German texts with spaCy - step by step

One of the best-known open source libraries for Natural Language Processing (NLP) is spaCy. Reason enough to roll up our sleeves and carry out a few experiments with spaCy.

In this practical tutorial, I show how we can analyze user stories written in German with just four lines of Python code. And what a few more lines can do.

I have already written a lot in this blog about NLP written. Due to the current interest in ChatGPT interest has certainly increased. This mini-tutorial shows how we can produce our first results with spaCy in half an hour.

Installation

If you would like to try out what is shown here yourself, you must first Python 3 install. Python has now established itself as the most important programming language for NLP.

We still need a package manager for Python. I use pip, but there are also others.

We don't even need a code editor (IDE) for this small example. However, if you want to develop the code further, I recommend the following IDE (code editor) Visual Studio Code to install. It works on all common operating systems and has good support for programming in Python.

After installing Python and pip, we install spaCy:

pip install -U pip setuptools wheel
pip install -U spacy

Finally, we need a language model. Here we can select a language model on the spaCy website. For this tutorial I use a small German model and install it as follows:

python3 -m spacy download en_core_news_sm

That's it already!

A first test

Now we can write our first NLP program:

import spacy
nlp = spacy.load("de_core_news_sm")
doc = nlp("As the host, I would like to invite my friends so that we can spend a nice evening together, for example.")
print([(w.text, w.pos_) for w in doc])

In the first two lines, we import spaCy and configure it with our German language model. In the third line, we let spaCy loose on a user story written in German. In the fourth and last line, we print the result on the console:

[
  ('As', 'ADP'),
  ('Host', 'NOUN'),
  ('would like', 'AUX'),
  ('me', 'PRON'),
  ('my', 'DET'),
  ('friends', 'NOUN'),
  ('invite', 'VERB'),
  (',', 'PUNCT'),
  ('so that', 'SCONJ'),
  ('we', 'PRON'),
  ('e.g.', 'NOUN')
  ('one', 'DET'),
  ('beautiful', 'ADJ'),
  ('evening', 'NOUN'),
  ('together', 'ADV'),
  ('spend', 'VERB'),
  ('can', 'AUX'),
  ('.', 'PUNCT')
]

We can see that spaCy has broken down the sentence into tokens and classified them. Part of speech tags (POS tags) are used for classification. In the Model description we can look up which POS tags are available for our model ("Label Scheme"). Of course, we can easily guess the meaning of many identifiers (e.g. AUX → auxiliary verb). We can also use spacy.explain() to print out a short English description:

print(spacy.explain("SCONJ"))
→ subordinating conjunction

Interesting is the token "e.g.", which was not misinterpreted as a punctuation mark. This shows nicely that NLP does more than just recognize patterns.

In addition to .pos (Part of Speech), the tokens contain further information, specifically:

• Lemma: The neutral form of the token, e.g. beautiful → beautiful
• Day: Basically like POS, only with more details, e.g. I → PPER (personal pronoun, instead of PRON pronoun)
• Shape: Upper/lower case, where each letter is replaced with an "X", e.g. 'e.g.' → 'x.X.'
• is alpha: If the token consists of letters, e.g. 'e.g.' → False
• is stop: If the token is a "stop word", i.e. an extremely common word in the selected language, e.g. 'so that' → True
• Dep: Syntactic dependency between the tokens - more on this in a moment.

Syntactic dependencies

The syntactic dependencies are the most powerful result of the code written so far. To understand the syntactic dependencies, it helps to visualize them first. To do this, we only need to add the following two lines to our code:

from spacy import displacy
displacy.serve(doc, style="dep")

When we run the code, Python starts a web server whose page we can call up in the web browser. For this example, the result looks like this:

This is a tree structure. The root of the tree is the word "would like", which returns the value "ROOT".

Outlook

It's exciting what we can do with just a few lines of code. What I have shown here is just the tip of the iceberg. spaCy can recognize terms (named entities), evaluate similarities and much more.

The code shown so far has no practical use yet, but can be quickly expanded accordingly. For example, we could use the dependencies to analyze the sentence structure and decide whether a sentence is a user story or not.

Or we could find the subject, object and predicate to create a Ontology build up.

To be truly practical, the quality must be sufficient to help users and not frustrate them. In practice, this means that we need a robust framework for development and systematically improve the system with appropriate training data. I already mentioned this in the HOOD blog wrote an article about MLOps.

Have fun experimenting with spaCy!

Photo by Kari Shea on Unsplash