How to split a topic

[ITAYC0HEN]

Hey,
After running Bertopic, I sometimes find myself unhappy with certain specific topics. For example, where a topic is grouping together documents I believe that can further be split to two or more topics. Sometimes, tuning different parameters or specifying seed words work, but often not.

In such cases, where I am happy with the overall topics created, except one or two that I wish to split, what can I do?

What I imagine is something like this pseudo code:

topic_to_split = [3, 8, 23]
topic_model.split_topics(docs, topic_to_split)

# < This will somehow execute a clustering algorithm on the documents belonging to these topics and
# create the newly created topics, thus increasing the number of topics >

If you can and this functionality exists, please suggest a code example for an automatic approach like shown above, as well as a manual approach that looks like this:

1. Get the mapping of documents to topics
2. manually create new topic (add empty topic)
3. manually move certain documents from certain topics to existing or newly created topics.

[MaartenGr]

Thank you for the extensive description! Although not currently implemented, a .split_topics would definitely be possible I think as it would indeed be running a clustering algorithm on the subset of topics based on an input clustering algorithm.

You could do this yourself by simply clustering the topics that you want to have more fine-grained topics from and simply adding them using Manual BERTopic.

[ITAYC0HEN]

Thank you! Would you mind sharing a quick code as an example? :)

Also, would Manual modeling cause any future issues when working with topic_model or should it work smoothly?

[MaartenGr]

I do not have the code by hand but it would simply be selecting the topic you want to split and run BERTopic something like this:

# For instance, topic 12
split_model = BERTopic().fit(documents_related_to_one_topic)

and then make sure that you replace the original topic with those of the split model:

#Replace all instances of topic 12
original_model.topics_

# with the assignment of the split model:
split_model.topics_

You could also simply merge the original_model and split_model. The only thing to keep in mind is that you would have to remove the original topic in original_model.

[ITAYC0HEN]

Say I take the manual approach of manually updating the topic_model.topics_ list. I will need to add few more topics, say if I splitted a topic into 5 more topics I will need to increase the number of topics from the existing X topics to X+4.

It is possible to do that, however, it does not reflect in topic_model.get_topic_info(), how can I update it to reflect the change?

I noticed that ".merge_topics()` executes the following in order to apply all the needed changes. I guess having to execute all these is a bit too much for the manual approach.

[...]
        # Update topics
        documents.Topic = documents.Topic.map(mapping)
        self.topic_mapper_.add_mappings(mapping)
        documents = self._sort_mappings_by_frequency(documents)
        self._extract_topics(documents, mappings=mappings)
        self._update_topic_size(documents)
        self._save_representative_docs(documents)
        self.probabilities_ = self._map_probabilities(self.probabilities_)

What approach would you recommend to take?

Edit:

Okay I performed something like this, which does nothing except forcing BERTopic to update all the required information based on my updated topic_model.topics_ :D

topic_model.merge_topics(messages_list, [6,6])

The problem is, that this won't work if I will add a new topics index. Say there were 12 topics and now I include in topic_model.topics_ values of 13, 14, 15, this will fail in

BERTopic/bertopic/_bertopic.py

Line 3914 in 424cefc

embds = np.array(self.topic_embeddings_)[np.array(topic_ids) + self._outliers]

[MaartenGr]

Ah, you're making this more complex than needed! When I say the manual approach, I mean the link I shared before: https://maartengr.github.io/BERTopic/getting_started/manual/manual.html

from bertopic import BERTopic
from bertopic.backend import BaseEmbedder
from bertopic.cluster import BaseCluster
from bertopic.vectorizers import ClassTfidfTransformer
from bertopic.dimensionality import BaseDimensionalityReduction

# Prepare our empty sub-models and reduce frequent words while we are at it.
empty_embedding_model = BaseEmbedder()
empty_dimensionality_model = BaseDimensionalityReduction()
empty_cluster_model = BaseCluster()
ctfidf_model = ClassTfidfTransformer(reduce_frequent_words=True)

# Fit BERTopic without actually performing any clustering
topic_model= BERTopic(
        embedding_model=empty_embedding_model,
        umap_model=empty_dimensionality_model,
        hdbscan_model=empty_cluster_model,
        ctfidf_model=ctfidf_model
)
topics, probs = topic_model.fit_transform(docs, y=y)

The y that you pass here is simply a list of topic IDs, one for each document. So the only thing that you need to do is to populate y with original_model.topics_ and replace the topics you split with those in split_model.topics_.

For instance:

# If `original_model.topics_` would look like this:
[1, 2, 3, 3, 4, 4]

# And you would split topic 3 into subtopics, then `split_model.topics_` might look something like this
[0, 1]

# As a result, your `y` variable could look something like this:
y = [1, 2, 5, 6, 4, 4]

# Notice that the documents that were initially topic 3 are now split into topics 5 and 6 which represent topics 0 and 1 respectively
# as shown in `split_model.topics_`. Then, you only need to pass `y` to BERTopic