TL;DR:

This post discusses Cohere's multilingual embedding model for cross-lingual text classification in 100+ languages—excelling in sentiment analysis, content moderation, and intent recognition, all while outperforming alternatives.

Can companies and developers build systems that serve a global audience from day one? When we announced Cohere’s multilingual model, we highlighted the search capabilities of Cohere’s multilingual embedding model. In this article, we’ll demonstrate how it can also be used to easily build a text classifier across languages.

Some key use cases for this model include:

Sentiment Analysis: Analyze customer sentiment from online sources, such as weets, in any language.
Content Moderation: Fight spam and hate-speech in international communities like online gaming.
Intent Recognition: Classify the user's intent based on a set of predefined intents (e.g., booking a flight, ordering food, etc.).

In many of these use cases, it was necessary to collect training data in each language individually, which is time consuming and hard to get. This was especially the case for smaller teams as they don’t have access to a global pool of data annotators.

This is no longer needed with the Cohere multilingual embedding model: it is sufficient to create a training dataset in a single language, and the model will work automatically across 100+ languages. Further, in many cases, just a handful of training examples per class are needed to produce a strong text classifier.

What are Embedding Models?

Embedding models translate text into numeric representations (called vectors or embeddings). These numeric representations enable the computer to understand the content of the text and to perform respective actions, in this case, classify the content into categories.

The Cohere multilingual embedding model was trained in such a way as to only focus on the content of the text and ignore the language used. Hence, texts in different languages are mapped to roughly equal points in the vector space. As we see in the following visualization, the English sentence, “Are you open on weekends?” and the Japanese translation, “週末は営業していますか?” are mapped to roughly the same place in the vector space.

This allows us to easily build a text classifier that works across languages. We start with annotating some data. For example, in the following picture, we collect comments from a social network and mark them as “wanted” or “unwanted.” The language in which we collect the data doesn’t matter. All training examples are then projected to the before-mentioned vector space, and we learn a decision boundary that tells what is wanted or unwanted on the platform.

When a new comment is posted on the platform, we project the text to the vector space, and depending on its position to the decision boundary, we then decide if the content is wanted or not.

Starting with Cohere Classify

You can use the Cohere Classify endpoint to quickly create such a cross-lingual classifier: Here, you provide your training examples and the new text as inputs to an API call, and Cohere handles everything in the background for you.

In the above example, we provide some examples for wanted and unwanted content in English, and then classify comments in various languages.

Building a Cross-Lingual Intent Recognition System

In the following sections, we show step by step how to use an embedding model (TODO: add a link to the embedding blog post) to build a system that can recognize user intents across languages.

As a basis, we use the Amazon MASSIVE dataset, which contains transcribed voice commands for 60 different intents in 51 typologically-diverse languages. We will train the model on the English dataset with 10 training examples per intent, and then apply it to the 50 other languages. Performance is measured by accuracy.

Once the model is trained, we have three approaches to building our cross-lingual intent recognition system.

Approach 1) Nearest Neighbor Classification

A popular approach for intent recognition is to use nearest neighbor classification. Here, we embed all training examples using Cohere’s embedding model into a vector space.

In the above image, we visualize the vector space with just two dimensions. Here, we see that commands for the alarm are in the upper-left corner, while commands for the music are in the right corner.

When a new command is sent to our assistant, such as the command `play a song`,, we project it to the same vector space and find the nearest neighbor. In the above example, it is `play music`, and from the training data we know that this example is associated with the label `play_music`.

We have different options for measuring distances between points in vector spaces. Common options are cosine similarity, dot product, and euclidean distance. Which distance to use depends on the model and on the respective task. Hence we are going to test all three.

Implementing the code is quite easy,. See Colab - Cross-Lingual Intent Recognition with Nearest Neighbor Classifier for an example

Cohere multilingual-22-12 model	Accuracy English (US)	Avg. accuracy on 51 non-English languages
Nearest Neighbor - Cosine	51.3	42.9
Nearest Neighbor - Dot product	58.1	51.3
Nearest Neighbor - Euclidean distance	50.3	41.1

From the above table, we see that it doesn’t perform too badly: with just 10 training examples per intent, we get an accuracy of 58 for an English and 51 for a non-English command. The distance metric used makes a big difference—here, dot product performed the best.

The advantage of this approach is its simplicity: classification of new examples is extremely quick, and new training data can easily be added. However, it is not necessarily the best approach and subjective to noise in the training data.

Approach 2) Nearest Centroid Classification

We can improve the accuracy further by moving from nearest neighbor to nearest centroid classification. Here, we first compute the average of all embeddings in the same class.

Now, every class is represented by a single vector, which is simply the mean of all training examples of that label. We then perform the nearest neighbor search as before. See Colab - Cross-Lingual Intent Recognition with Nearest Centroid Classifier for a code example

Cohere multilingual-22-12 model	Accuracy English (US)	Avg. accuracy on 51 non-English languages
Nearest Neighbor - Dot product	58.1	51.3
Nearest Centroid - Cosine	72.9	63.6
Nearest Centroid - Dot product	57.8	49.9
Nearest Centroid- Euclidean distance	73.1	63.8

First, we observe a significant gain in accuray: performance for English increases by 15 points and for non-English by 12.5 points. Furthermore, although dot product was the best distance metric for nearest neighbor search, it does not perform well in centroid search. Instead, Euclidean distance performs the best.

The advantage of the nearest centroid classifier is even higher classification speed, especially with more training examples per class. Adding new training examples is still not too complicated, we just need to compute the new centroid.

Approach 3) Logistic Regression Classifier

Finally, we test adding a logistic regression classifier. This classifier learns a decision boundary between the examples. When we want to classify a new command, we project it to the vector space and measure its position relative to the decision boundary. Everything in the top-left of the image below would be classified as `alarm_set`.

See Colab - Cross-Lingual Intent Recognition with Logistic Regression to access the code and learn how to train this classifier.

In terms of classification performance, we see a small increase for English and a slight decrease for other languages.

Cohere multilingual-22-12 model	Accuracy English (US)	Avg. accuracy on 51 non-English languages
Nearest Neighbor - Dot product	58.1	51.3
Nearest Centroid - Euclidean distance	73.1	63.8
Logistic Regression Classifier	73.5	62.7

The more complex logistic regression classifier is especially beneficial when you have more training examples. In the table below, you see the performance for training with up to 50 examples per class. Here, we see clear outsized performance for the logistic regression classifier.

Cohere multilingual-22-12 model (50 examples/class)	Accuracy English (US)	Avg. accuracy on 51 non-English languages
Nearest Neighbor - Dot product	67.8	59.9
Nearest Centroid - Euclidean distance	77.4	68.2
Logistic Regression Classifier	80.7	69.7

Benchmark:How Well Does the Cohere Multilingual Model Perform?

In the following, we compare the Cohere multilingual-22-12 model with the most popular open source model for multilingual text embeddings: the paraphrase-multilingual-mpnet-base-v2 model. We observe with the Cohere model, especially for non English languages, a nice boost performance of up to 10 points. You can find the evaluation results for all languages in this spreadsheet.

Model	Accuracy English (US)	Avg. accuracy on 51 non-English languages
Training with 10 examples/class
Cohere multilingual-22-12	73.5	62.7
sentence-transformers	72.0	54.4
Training with 50 examples/class
Cohere multilingual-22-12	80.7	69.7
sentence-transformers	79.7	59.5

Using Cross-Lingual Classification in Your Application

You can start with multilingual classification in the Cohere Playground. Select the `multilingual-22-12` model from the Parameters section on the right, and you’re ready to go!

You can add this capability to your applications by using the Cohere SDK. The Text Classification (Classify) page walks you through that process.

In the Python SDK, the call to Classify looks like this:

The call has these parameters:

Model: here we specify the multilingual model we want to use for this task. Check the changelog to see the latest available models. At the time of writing, `multilingual-22-12` is the latest version.
`examples_list`: is a list of labeled examples that demonstrate the classification task to the model. If we’re doing sentiment classification, for example, this list would contain positive sentences indicated as `positive` and negative sentences indicated as `negative`.
Inputs: This is the list of texts we want to classify.

We can provide the examples and inputs like this:

Notice how the examples can be in one language and the inputs in another language. That’s what the term cross-lingual refers to and is a capability of this model. The more examples we provide the model, and the more of them for the target language, the better we can expect the results to be.

You can copy this code in a notebook (just make sure to plug in your API key):

import cohere
from cohere.responses.classify import Example

co = cohere.Client('API_KEY')


classifications = co.classify(
 model='embed-multilingual-v2.0',
 examples=[Example("The order came 5 days early", "positive"),
           Example("The item exceeded my expectations", "positive"),
           Example("I ordered more for my friends", "positive"),
           Example("I would buy this again", "positive"),
           Example("I would recommend this to others", "positive"),
           Example("The package was damaged", "negative"),
           Example("The order is 5 days late", "negative"),
           Example("The order was incorrect", "negative"),
           Example("I want to return my item", "negative"),
           Example("The item\'s material feels low quality", "negative")],
          


 inputs=["This item was broken when it arrived",
         "Qualité raisonnable pour le prix",
         "هناك مشكلة في الطلب"])


# View the results
print('The confidence levels of the labels are: {}'.format(
      classifications.classifications))

Final Thoughts

As the world continues to become more connected, the ability to understand and interpret text-based data in multiple languages has become a critical skill for companies. By using cross-lingual classification models, organizations can gain a deeper understanding of customer sentiment and behavior, enabling them to improve customer engagement and marketing efforts.

However, it's important to note that both cross-lingual and monolingual classification have their own unique strengths and weaknesses, and companies should choose the right approach based on the specific needs and goals of their organization. Whether it's to analyze customer feedback, predict consumer trends, or understand market dynamics, the use of text classification models will continue to play a pivotal role in shaping the future of business and commerce.

Are you ready to build text classifiers for 100+ languages using just one language's training data? Look no further! Cohere's multilingual embedding model is here to supercharge your sentiment analysis, content moderation, and intent recognition projects. Unleash the Power of Cross-Lingual Classification in NLP with Cohere. Want to learn more? Get in touch with sales.