The mapping between category number and category type

[gusye1234]

Great work on the memory benchmark!

But I noticed LOCOMO repo/paper didn't state the mapping relation between category number and type.
I notice you're using single-hop, multi-hop, temporal, open-domain, adversial, but what are number of those type?

I looked into the dataset, and I think category 2 is temporal, but I can't assert the rest.

[zcg-joker]

I'm also very interested in understanding this issue.

To investigate, I started by reviewing the code in the repository's task_eval/evaluation.py file, specifically within the eval_question_answering function. I observed that:

# single-hop, temporal, open-domain eval without splitting for sub-answers 
if line['category'] in [2, 3, 4]:
    all_ems.append(f1_score(output, answer))

# multi-hop eval by splitting entire phrase into sub-answers and computing partial F1 for each
elif line['category'] in [1]:
    all_ems.append(f1(output, answer))

# adversarial eval --> check for selection of correct option
elif line['category'] in [5]:
    if 'no information available' in output.lower() or 'not mentioned' in output.lower():
        all_ems.append(1)
    else:
        all_ems.append(0)
else:
    print(line)
    raise ValueError

However, upon carefully examining the data itself (particularly the locomo10.json file), it appears the defined categories might not align perfectly with the actual content. For instance, my review suggests that category 2 consistently contains time-related questions (many seem to start with "When").

Based on my examination of the data, I believe the intended category mapping might be as follows:

• Category 1: Multi-hop
• Category 2: Temporal
• Category 3: Open-domain
• Category 4: Single-hop
• Category 5: Adversarial

This is just my personal interpretation based on the current data and code. I would greatly appreciate clarification from the author(s) if possible.

Thank you again for your work on this project!

[dial481]

This mismatch was also confirmed by:

1. MemMachine (September 2025) - Documented in Appendix A of their benchmark analysis, noting: "Upon closer inspection of the LoCoMo source code and the locomo10.json dataset, it becomes clear that the numerical IDs...do not directly align with the sequential order in which they are presented in the paper's text."

2. Goal-Directed Search paper (arXiv:2511.21726v1, November 2025) - Appendix E documents the same finding: "We found a mismatch between the human-readable descriptions in the LoCoMo white paper and the actual category IDs used in the released code."

Issue

Section 4.1 of the LoCoMo paper presents categories in a numbered list:

"(1) Single-hop questions require answers based on a single session; (2) Multi-hop questions require synthesizing information from multiple different sessions; (3) Temporal reasoning questions...; (4) Open-domain knowledge questions...; (5) Adversarial questions..."

However, the integer category field in locomo10.json does not follow this ordering. The actual mapping in the source code (task_eval/evaluation.py) is:

Category ID	Actual Type
1	Multi-hop
2	Temporal
3	Open-domain
4	Single-hop
5	Adversarial

This can be verified from the evaluation code comments:

# single-hop, temporal, open-domain eval without splitting for sub-answers 
if line['category'] in [2, 3, 4]:
    all_ems.append(f1_score(output, answer))

# multi-hop eval by splitting entire phrase into sub-answers and computing partial F1 for each
elif line['category'] in [1]:
    all_ems.append(f1(output, answer))

# adversarial eval --> check for selection of correct option
elif line['category'] in [5]:

Impact

Without explicit documentation, researchers may:

• Report category-level results with incorrect labels
• Make invalid cross-paper comparisons
• Spend time debugging non-existent issues in their implementations

Suggested Fix

Add a mapping table to the README in the QA annotation section:

### Category Mapping

| Category ID | Type | Description |
|-------------|------|-------------|
| 1 | Multi-hop | Requires synthesizing information from multiple sessions |
| 2 | Temporal | Requires temporal reasoning and time-related cues |
| 3 | Open-domain | Requires integrating with external/commonsense knowledge |
| 4 | Single-hop | Requires answer from a single session |
| 5 | Adversarial | Designed to test recognition of unanswerable questions |