Topic identification in nutshell

How much knowledge is lost between messages every day?

Conversations and pages are the pillars of Tamarind.

Pages allow users to curate and persist knowledge in a more structured way, while conversations serve as a mere channel of communication: often ephemeral, filled mostly with shallow content and noise.

Or at least that's what it seems at first glance.

When I started building Tamarind I had no doubt that messages were actually the prime source of knowledge and value creation. It was basically the sole idea that sparked this project.

All applications have their own search engines nowadays, allowing you to search something written in the past, if you know what you are looking for. And that's a big If. We tend to search for accurate names and tickets, but not for ideas.

We can retrieve the what, not the how. Nor the why.

It is always us asking the question first. But what if the tool could prompt us the answer before we even finish writing the question? This is where Tamarind comes in.

In order to achieve such degree of semantic retrieval, the tool must understand first what we are talking about in our conversations. Furthermore, it needs to filter out the noise, read between the lines and capture the essence of our messages.

But conversations are, by definition, an infinite stream of messages. How do we know what the theme is about? Does it have one topic or many? When does a topic stop being relevant? These are only a few questions that we had to answer in order to build one of the critical components of Tamarind's Semantic Engine: the ability of identifying what conversations are about.

The challenge was not only how to identify what was the underlying theme of a given group of messages, but to tell which messages were actually relevant enough to determine the theme of a conversation. A flawed sample of data can compromise the whole pipeline of information. This is where we introduced embeddings into the equation.

Numbers as the common language

Embeddings are a bit like the Rosetta Stone of the application. They give us a way to represent meaning through numbers, making it easier to compare concepts and measure their likeness. Here they are used to compare new written messages against existing conversation topics, within a spectrum of similarity. We still cannot tell if the message is semantically related to the current topic, only that it shares some correlation with it.

Once these messages were identified (using multiple kinds of heuristics, scoring strategies and embeddings), the next step was to tell if a message represented a new topic, or it belonged to an existing one.

If we already identified the topic of a conversation, what happens if someone now writes a new message that has nothing to do with it? Do we just replace one theme for another? How long do we need to wait until it is confirmed the conversation has changed its tone and focus?

Tamarind takes some of those elements into consideration and creates a set of topics for each conversation. These themes have scores that decay with time. If a topic that we identified in the past hasn't been mentioned in a while, it loses momentum, thus reducing the chances of being "the current theme" of the conversation.

Topic drifting

The key here is to understand that these messages are being written by humans, and humans tend to ramble. It is quite common to see a conversation momentarily drift from work to football, or movies, only going back to the main topic of discussion after a few "useless" messages. The system has to be opened to new "candidates" at any point, but it also needs to be resilient enough to tell the difference between banter and value.

The theme of a conversation is not defined by a single message, but by a mixture of them. And we not only need their embeddings, we also need a literal description of what the conversation is about, because it will help us later comparing potential new topics with the existing ones. This is where LLMs come into play. An interface capable of distilling the essence of a series of messages into a refined semantic concept that is worth persisting. A more reliable interpretation of meaning that what embeddings can offer, invoked when we needed to tell whether a topic is repeated or not, and what is it about.

Topic identification was only one key element that was needed to achieve more complex tasks later on. Once you know what's being discussed, you can proactively surface relevant data, or suggest ideas to your users whenever they ask questions. The raw material is there: users write thousands of words per day on chats with other users, or directly as documentation pages. The challenge is to separate the signal from the noise, and bring it back to the users exactly when they need it.