[Question] No similarity beetween WSI Reports Embeddings and WSI Slide Image

[vicky-gupta]

---

What's your question?

Edit: I’m updating my question to make it more detailed for better understanding.

---

Hey Trident Team,

I’ve been using the TITAN model within the Trident framework to encode my WSI slides and reports.
Here’s the command I used to encode the WSI slides:

python TRIDENT/run_batch_of_slides.py \
  --task all \
  --wsi_dir "./Train_700" \
  --job_dir "./TRIDENT/new_tri_pro_train_part1" \
  --slide_encoder titan \
  --patch_size 512 \
  --mag 20 \
  --custom_list_of_wsis train_mpp_part1.csv

For the text reports, I used TITAN’s text encoder as follows:

# Encode the report text using TITAN (following the correct method)
with torch.no_grad():
    texts = [report_text]
    tokenized_text = model.text_encoder.tokenizer(texts).to(device)
    text_embedding = model.encode_text(tokenized_text, normalize=True)

    # Convert to NumPy
    if isinstance(text_embedding, torch.Tensor):
        text_embedding = text_embedding.cpu().numpy()

    # Remove batch dimension if present
    if len(text_embedding.shape) > 1 and text_embedding.shape[0] == 1:
        text_embedding = text_embedding.squeeze(0)

I’ve carefully reviewed the TITAN paper and tried to replicate the pretraining setup as closely as possible.

---

Task

Goal: Find the best-matching report for each WSI image.

Embeddings generation pipeline:

• For WSI slides:

  WSI Image → Vision Encoder → Embeddings (not L2-normalized)
  → Apply L2 normalization along dim=-1 → Embeddings (L2-normalized)
  

• For WSI reports:

  Report Text → TITAN Text Encoder → Embeddings (L2-normalized)
  

Then, computed cosine similarity between the WSI and report embeddings.

---

Problem

I’m getting almost zero similarity scores — suggesting the embeddings might not be aligned in the same shared space.

I suspect I might be missing a step or detail from the paper. Could you please advise which section or part of the TITAN paper I should revisit to understand how to properly align or normalize these embeddings?

Also, any feedback on my pipeline or assumptions would be greatly appreciated.

---

[guillaumejaume]

@andrewsong90 @sophiajw @HHenryD, can you take a look at that when you have time. thx

[HHenryD]

Hi @vicky-gupta, Thanks for your interest in TITAN.

Trident extracted slide embedding is before the projector layer, which projects slide embedding into the image-text latent space. slide_embedding = slide_embedding @ model.vision_encoder.proj will do the projection. Alternatively, you can also use model.zero_shot to calculate the cosine similarity.

We also provided a zero shot demo using TITAN, which can found here.

Hope this helps.

[vicky-gupta]

Hey @HHenryD — thanks a lot for your response, it was really helpful!

Cc: @guillaumejaume @andrewsong90 @sophiajw

I tried both approaches:

• Zero-shot inference
• Projecting embeddings into the same latent space

However, the cosine similarity (or probability) between my WSI slide embeddings and corresponding report embeddings remains very low — almost negligible.

Do you think I should try fine-tuning TITAN on my own dataset of paired WSI Slides and Report Texts?

Here’s a sample of what my report data looks like (applied manipulation later to use it with TITAN) :

[
  {
    "id": "PIT_01_00002_01.tiff",
    "report": "Breast, core-needle biopsy;\n  Invasive carcinoma of no special type, grade I (Tubule formation: 2, Nuclear grade: 2, Mitoses: 1)"
  },
  {
    "id": "PIT_01_00003_01.tiff",
    "report": "Breast, core-needle biopsy;\n  Invasive carcinoma of no special type, grade II (Tubule formation: 3, Nuclear grade: 2, Mitoses: 1)"
  },
  {
    "id": "PIT_01_00004_01.tiff",
    "report": "Breast, core-needle biopsy;\n  Invasive carcinoma of no special type, grade II (Tubule formation: 3, Nuclear grade: 2, Mitoses: 1)"
  }
]

Any insights on whether fine-tuning might help improve the alignment between image and text embeddings would be greatly appreciated.

Thanks again for your guidance!

[HHenryD]

Hi @vicky-gupta,

Do you mind sharing your cosine similarity calculation code? Or could you try your code on a paired TCGA slide and its corresponding diagnosis and see how's the similarity? Thank you.