Correct label errors in locomo10.json QA

[dial481]

I'm running the benchmark on a new project I've been working on and it identified three benchmark label errors in locomo10.json that penalize correct model reasoning.

Question	Line	Issue	Answer Key Says	Transcript Says	Error Type
Q1	21	Wrong field	"Psychology, counseling certification"	"counseling and mental health" (no psychology, no certification)	Hallucination
Q2	46	Wrong day	"Sunday before 25 May 2023"	"last Saturday"	Temporal Mismatch
Q3	1155	Wrong person	Caroline shared "abstract painting with blue streaks"	Caroline shared drawing; Melanie shared paintings	Speaker Misattribution

Line 21:

{
"question": "What fields would Caroline be likely to pursue in her educaton?",
"answer": "Psychology, counseling certification",
"evidence": [
"D1:9",
"D1:11"
],
"category": 3
},

Issue: Psychology is not mentioned in that conversation at all, although counseling and mental health is explicitly mentioned. "Psychology" is mentioned in another conversation between Tim and John (line 28587). "Certificate" and "certification" are also mentioned in other conversations but not between Caroline and Melanie.

Line 4410:

[
"Caroline is planning to continue her education and explore career options in counseling or mental health to support those with similar issues.",
"D1:9"
]

Line 1708:

{
"speaker": "Caroline",
"dia_id": "D1:11",
"text": "I'm keen on counseling or working in mental health - I'd love to support those with similar issues."
},

Line 46:

{
"question": "When did Melanie run a charity race?",
"answer": "The sunday before 25 May 2023",
"evidence": [
"D2:1"
],

Issue: The transcript clearly indicates "last Saturday" not "last Sunday".

Line 1754:

"session_2_date_time": "1:14 pm on 25 May, 2023",
"session_2": [
{
"speaker": "Melanie",
"dia_id": "D2:1",
"text": "Hey Caroline, since we last chatted, I've had a lot of things happening to me. I ran a charity race for mental health last Saturday \u2013 it was really rewarding. Really made me think about taking care of our minds."
},

Line 1155:

{
"question": "What kind of painting did Caroline share with Melanie on October 13, 2023?",
"answer": "An abstract painting with blue streaks on a wall.",
"evidence": [
"D17:14"
],
"category": 4
},

Issue: The answer key attributes Melanie's painting to Caroline. According to the transcript, Caroline shared a drawing of a woman in a dress (D17:14, BLIP caption: "a photo of a drawing of a woman in a dress"). Melanie shared the paintings, including the one with "blue streaks on a wall." This creates an unintentionally adversarial question.

Line 3977:

"speaker": "Caroline",
"img_url": [
"https://i.redd.it/50qvgfuva33b1.jpg"
],
"blip_caption": "a photo of a drawing of a woman in a dress",

[dial481]

Additional benchmark label errors in conv-26:

Line 200 (conv-26)

• Question: What books has Melanie read?
• Truth: "Nothing is Impossible", "Charlotte's Web"
• Actual Text: "This book I read last year reminds me to always pursue my dreams..." (D7:8)
• Reason: GROUND_TRUTH_ERROR. The title "Nothing is Impossible" is completely absent from the source text. The text only refers to "This book". This is a hallucination in the benchmark's ground truth. Global search confirmed absence. D7:11 Referrs to "Becoming Nicole".

Line 226 (conv-26)

• Question: When did Melanie read the book "nothing is impossible"?
• Truth: 2022
• Actual Text: "Caroline, so glad you got the support! Your experience really brought you to where you need to be. You're gonna make a huge difference! This book I read last year reminds me to always pursue my dreams, just like you are doing!" (D7:D8)
• Reason: GROUND_TRUTH_ERROR. The question itself contains a hallucination ("nothing is impossible") which is not in the source text.

Line 317 (conv-26)

• Question: What did Melanie paint recently?
• Truth: sunset
• Actual Text: "Here's one I did last week. It's inspired by the sunsets." (D17:12)
• Reason: INACCURATE_EVIDENCE_REF. The answer "sunset" is correct per global context search (D17:12), but the cited evidence (D9:17) states "Wow, Caroline! It really conveys unity and strength - such a gorgeous piece! My kids and I just finished another painting like our last one." D8:6 (correctly cited) mentions: "blip_caption": "a photo of a painting of a sunset with a palm tree", "query": "painting vibrant flowers sunset sky",

Line 375 (conv-26)

• Question: What kind of art does Caroline make?
• Truth: abstract art
• Actual Text: "I've been trying out abstract stuff recently." (D17:13)
• Reason: INACCURATE_EVIDENCE_REF. (D17:13) most directly references "abstract art" per global context search , but the cited evidence (D11:12, D11:8, D9:14) does not mention "abstract", although D9:14 has a URL with 'abstract' in it.

Line 486 (conv-26)

• Question: What symbols are important to Caroline?
• Truth: Rainbow flag, transgender symbol
• Actual Text: "The rainbow flag mural is important... The eagle symbolizes freedom..." (D14:15), "blip_caption": "a photo of a person holding a necklace with a cross and a heart", "query": "pendant transgender symbol", (D4:1)
• Reason: GROUND_TRUTH_ERROR. The text explicitly lists "rainbow flag," "eagle," "a person holding a necklace with a cross and a heart," and "pendant transgender symbol".

Line 571 (conv-26)

• Question: What does Melanie do with her family on hikes?
• Truth: Roast marshmallows, tell stories
• Actual Text: "Thanks, Caroline! It's awesome seeing the kids get excited learning something new about nature. Those moments make being a parent worth it. We roasted marshmallows and shared stories around the campfire. Those simple moments make the best memories. What inspires you with your volunteering?" (D16:4), "We always look forward to our family camping trip. We roast marshmallows, tell stories around the campfire and just enjoy each other's company. It's the highlight of our summer!" (D10:12)
• Reason: GROUND_TRUTH_ERROR. The question specifically asks about hikes. The activities (marshmallows, stories) are described as happening around the campfire while camping. Hiking is referenced in D14:1, D8:34, D3:14, D16:2, D18:5 but without camping, roasting marshmallows or telling stories. Summary D4:8 combines hiking with camping and roasting marshmallows but not telling stories: "Melanie went camping with her family in the mountains last week and had a great time exploring nature, roasting marshmallows, and hiking." (D4:8, summary)

[dial481]

Additional benchmark label errors in conv-30:

Line 5545 (conv-30)

• Question: How long did it take for Jon to open his studio?
• Truth: six months
• Actual Text: Lost job Jan 19, 2023 (D1:2). Opened studio June 20, 2023 (D15:5).
• Reason: GROUND_TRUTH_ERROR & INACCURATE_EVIDENCE_REF. Jan 19 to June 20 is exactly 5 months and one day. Benchmark claims "six months". This is a factual calculation error. The cited evidence is incorrect: (D1:2),(D15:13)

Line 5756 (conv-30)

• Question: What advice does Gina give to Jon about running a successful business?
• Truth: build relationships with customers, create a strong brand image, stay positive
• Actual Text: "Got any advice or tips on running a successful biz?" (Gina asking Jon, D7:4). "Yeah, brand identity is key... build relationships... stay positive" (Jon answering Gina, D7:5).
• Reason: GROUND_TRUTH_ERROR / ATTRIBUTION. The question asks what advice Gina gives Jon. The text clearly shows Jon giving this advice to Gina. This is a standard question (Category 4), not marked as Adversarial/Trap, indicating a genuine attribution error.

[dial481]

Additional benchmark label errors in conv-41, conv-42, conv-43

Line 9310 (conv-41)

• Question: When did Maria meet Jean?
• Truth: February 24, 2023
• Actual Text: "Hey John, how's it going? Just wanted to give you the heads up on what's been happening lately- I took a creative writing class recently, and it was super enlightening!" (D7:1)
• Reason: INACCURATE_EVIDENCE_REF. No mention of Jean. The cited evidence does not contain the answer or the name "Jean". "While volunteering yesterday, I met this amazing woman, Jean, who had been through a lot, yet stayed optimistic and resilient. She showed me the importance of gratitude and connection." (D7:5)

Line 16583 (conv-42)

• Question: What pets wouldn't cause any discomfort to Joanna?
• Truth: Hairless cats or pigs
• Actual Text: "I'm allergic to most reptiles and animals with fur. It can be a bit of a drag, but I find other ways to be happy." (D2:23).
• Reason: HALLUCINATION_EXTERNAL: The text doesn't refer to pigs or hairless anything. (conv-26 mentions a "guinea pig")

Line 17127 (conv-42)

• Question: What alternative career might Nate consider after gaming?
• Truth: an animal keeper at a local zoo...
• Actual Text: Nate expresses joy in caring for his turtles (D5:8, D19:3), but never mentions a career change or "zoo".
• Reason: HALLUCINATION. The intent for a career change or become a zookeeper is fabricated/hallucinated from his hobby of keeping turtles.

Line 24045 (conv-43)

• Question: In which month's game did John achieve a career-high score in points?
• Truth: June 2023
• Actual Text: "So much has happened in the last month - on and off the court. Last week I scored 40 points, my highest ever, and it feels like all my hard work's paying off." (D3:1, July 16).
• Reason: GROUND_TRUTH_ERROR. "Last week" from July 16 is July 9-15. The event happened in July, not June.

Line 24139 (conv-43)

• Question: After how many weeks did Tim reconnect with the fellow Harry Potter fan from California?
• Truth: three weeks
• Actual Text: "Last week [July 9-15] I had a nice chat" (D3:2). " Guess what? Just skyped with that Harry Potter fan I met in CA [Aug 9]" (D5:1).
• Reason: GROUND_TRUTH_ERROR. The gap between July 9-15 and Aug 9 is 25-31 days. "Three weeks" is inaccurate.

Line 24530 (conv-43)

• Question: When did John achieve a career-high assist performance?
• Truth: December 11, 2023
• Actual Text: "I had a career-high in assists last Friday" (D23:3, Dec 11).
• Reason: GROUND_TRUTH_ERROR, INACCURATE_EVIDENCE_REF. Dec 11 is Monday. Last Friday was Dec 8. The event date is Dec 8. (D3:1) only appears as an "observation": "John scored 40 points in a game last week, his highest ever." requiring reference back to session 3 to find the date ("4:21 pm on 16 July, 2023") (D23:2) does not reference the basketball game at all.

[dial481]

Line 31790 (conv-44)

• Question: When did Audrey make muffins for herself?
• Truth: The week of April 3rd to 9th
• Actual Text: "I made some of my favorite treats last week" (D3:18, Sunday, April 16). "Last week" from April 16 is April 9-15.
• Reason: GROUND_TRUTH_ERROR. The dates are off by one week.

Line 31798 (conv-44)

• Question: When did Audrey see a hummingbird?
• Truth: first week of May 2023
• Actual Text: "Last week I finally went on a hike" (D4:1, May 3). "Last week" from May 3 is April 24-30.
• Reason: GROUND_TRUTH_ERROR. The event happened in late April, not the first week of May.

Line 32095 (conv-44)

• Question: When did Audrey get into an accident in the park?
• Truth: between October 19 and 24, 2023
• Actual Text: "I had an unexpected adventure last week" (D25:2, Oct 24). "Last week" is Oct 15-21.
• Reason: GROUND_TRUTH_ERROR. While "last week" overlaps with the 19th-24th, the benchmark's range (19-24) includes dates (22, 23, 24) that are definitely not "last week" and excludes days that are in "last week"

Line 39214 （conv-47）

• Question: How many days did James plan to spend on his trip in Canada?
• Truth: 19 days
• Actual Text: Leaving "day after tomorrow" (July 11, based on D16:9 July 9 date). "I plan to return on July 20" (D16:13)
• Reason: GROUND_TRUTH_ERROR. 19 vs 9 is likely a typo or calculation error. Accurate duration is 9 days

Line 54615 (conv-49)

• Question: When did Evan's son fall off his bike?
• Truth: Thursday before December 17, 2023
• Actual Text: "My son had an accident last Tuesday" (D20:3, Dec 17).
• Reason: GROUND_TRUTH_ERROR. Tuesday vs Thursday. Last Tuesday from Dec 17 is Dec 12.

Line 54671 (conv-49)

• Question: When did Evan have a drunken night with his friends?
• Truth: January 9, 2023
• Actual Text: "Yesterday I went out..." (D24:3, Jan 10, 2024).
• Reason: GROUND_TRUTH_ERROR. Year mismatch (2023 vs 2024). Yesterday is Jan 9, 2024.

Line 61235 (conv-50)

• Question: When did Calvin first get interested in cars?
• Truth: at an early age
• Actual Text: ""Thanks, Calvin! It's always been my thing. My dad took me to my first car show when I was 10, and I was wow-ed!" (D26:6). This is Dave speaking.
• Reason: ATTRIBUTION_ERROR. The question asks about Calvin, but the text describes Dave's history. This question is not labeled as adversarial.

[dial481]

As already noted in papers such as [Goal-Directed Search (ArXiv:2511.21726)](https://arxiv.org/html/2511.21726v1), the designated "Adversarial" category (Category 5) is widely excluded from evaluation because it lacks standardized ground truth (as confirmed by [Mem0](https://arxiv.org/abs/2407.07818) and [Zep](https://blog.getzep.com/lies-damn-lies-statistics-is-mem0-really-sota-in-agent-memory/)).

However, the errors identified in this issue (#27) highlight that "Adversarial" issue also affects some of the standard questions.. Specifically, we are seeing three distinct failure modes:

1. Intentionally False Premise (Category 5 - "The Trap")
The question contains a verifiable error designed to trick the model.

• Example (Line 5957): "What is Jon's favorite painting style?" (Jon is a dancer who likes contemporary dance).
• Outcome: A hallucinating model grabs the keyword "contemporary." A dumb model refuses. A clever model corrects the user: "Jon doesn't paint; he saids favorite style of dance is contemporary." The current evaluation logic may penalize the clever model.

2. Unintentionally False Premise (Standard Category - This Issue)
Standard questions that accidentally function as Type 1 traps due to annotation errors.

• Example (Line 1155): "What painting did Caroline share?" (Melanie shared the paintings, Caroline shared a drawing).
• Outcome: Just like the intentional trap, a model that corrects the attribution error ("Actually, Melanie shared paintings and Caroline shared a drawing.") is penalized for not matching the erroneous ground truth.

3. Intentionally Missing (Category 5)
The information genuinely does not exist. The correct answer is "I don't know," but these questions currently lack standardized ground truth to score against.

The Core Issue:
The benchmark currently penalizes Cooperative Correction. Whether the false premise is intentional (Jon's dance) or accidental (Caroline's painting), the evaluation rewards either "dumb" retrieval (matching the hallucinations) or strict refusal. It fails to account for models capable of identifying and correcting the premise.

Secondary Issue: LLM Judge Semantic Drift and False Positives

Adding to the issues above, the benchmark's partial reliance on LLMs as judges introduces a compounding evaluation error: semantic attribute conflation.

Example from our testing (Line 1155):

• Our Model's Answer: "Caroline shared a drawing of a woman in a blue dress"
• Original Answer Key: "An abstract painting with blue streaks on a wall."
• Actual Truth: "Caroline shared a drawing of a woman in a dress" (no color specified in source; the "blue painting" was shared by Melanie).
• Judge Outcome: GPT-4.1 marked this CORRECT

The judge conflated the "blue" from Caroline's drawing with Melanie's "abstract painting with blue streaks". The hallucinated detail ("blue dress") was accepted as equivalent to the dataset's ground truth (which was itself erroneous).

Recent research confirms (see [Evaluating Alignment and Vulnerabilities in LLMs-as-Judges](https://arxiv.org/abs/2406.12624) and [Mitigating Agreeableness Bias](https://arxiv.org/abs/2510.11822)) that LLM judges exhibit a systematic positive bias:

• High True Positive Rate (>96%): Good at identifying correct answers.
• Low True Negative Rate (30-40%): Poor at identifying incorrect ones.

This means LLM judges may penalize correct models for refusing hallucinations while rewarding hallucinating models that the LLM judges deem "close enough" semantically.

The Compounding Problems:
When combined with the three types of adversarial failures identified above, this creates a situation where:

1. Models are penalized for cooperative correction (Type 1 & 2 above).
2. Models are rewarded for hallucinations the judge deems semantically plausible (False Positive).
3. Category 5 questions are evaluated against inconsistent standards, making benchmark comparisons unreliable.
4. A significant number of the standard Q&A pairs contain significant errors.

[dial481]

Line 1203 (conv-26)

• Question: How did Melanie's son handle the accident?
• Truth: He was scared but reassured by his family
• Actual Text: "The kids look so cute, Mel! I bet they bring lots of joy. How did they handle the accident?" (D18:6), "Thanks! They were scared but we reassured them and explained their brother would be OK. They're tough kids." (D18:7)
• Reason: GROUND_TRUTH_ERROR. In the cited evidence Caroline's question refers to Melanie's son, but Melanie's answer "scared but reassured" refers to her son's siblings, not her son. Earlier in the conversation it was explicitly stated that the son was "OK"/"okay": "We were all freaked when my son got into an accident. We were so lucky he was okay." (D18:1) "Thank goodness your son's okay." (D18:2) "Thankfully, my son's ok..." (D18:3)

[dial481]

Line 1132 (conv-26)

• Question: What setback did Melanie face in October 2023?
• Truth: She got hurt and had to take a break from pottery.
• Actual Text: "BTW, recently I had a setback. Last month I got hurt and had to take a break from pottery, which I use for self-expression and peace." (D17:8 "10:31 am on 13 October, 2023")
• Reason: GROUND_TRUTH_ERROR. "Last month" would be September 2023 not October 2023. However, in October Melanie experienced another setback, her son got into an "accident", implied to be a accident by the attached media. The question could be correct by either changing the time frame to September 2023 in the question or changing the answer and the evidence to match her son's accident (D18:1).

[dial481]

Line 803 (conv-26)

• Question: What is Melanie's hand-painted bowl a reminder of?
• Truth: art and self-expression
• Actual Text: "Yep, Melanie! I've got some other stuff with sentimental value, like my hand-painted bowl. A friend made it for my 18th birthday ten years ago. The pattern and colors are awesome-- it reminds me of art and self-expression." (D4:5)
• Reason: ATTRIBUTION_ERROR. The bowl belongs to Caroline not Melanie. This is an unintentionally adversarial question due to misattribution and the fact that Melanie and her children also like pottery, bowls and cups.

Line 1147 (conv-26)

• Question: What painting did Melanie show to Caroline on October 13, 2023?
• Truth: A painting inspired by sunsets with a pink sky.
• Actual Text: Yeah, Here's one I did last week. It's inspired by the sunsets. The colors make me feel calm. What have you been up to lately, artistically? (D17:12)
• Reason: GROUND_TURTH_ERROR. Melanie showed two paintings to Caroline on October 13, 2023. "Thanks, Caroline! I painted it because it was calming. I've done an abstract painting too, take a look! I love how art lets us get our emotions out." "blip_caption": "a photo of a painting on a wall with a blue background" (D17:14)

[Timing04]

hello，i also find an error
line 16696
{
"question": "What movie did Joanna watch on 1 May, 2022?",
"answer": "Lord of the Rings",
"evidence": [
"D10:1"
],
"category": 2
}
it seems like category 2 is about temproal, and th prompt in the code is added "Use DATE of CONVERSATION to answer with an approximate date". But the ground-truth is not an date.

[dial481]

Update: Complete Audit

Since filing this issue, I completed a comprehensive audit of all 1,540 non-adversarial questions (categories 1–4) across all 10 conversations. Because benchmark is still seeing active use in published evaluations as of February 2026, it seemed worthwhile to open source the data so that anyone can use it to adjust their scores.

Findings: 99 score-corrupting errors across 1,540 questions (6.4%)

Category	Questions	Errors	Error Rate	Adjusted Ceiling
Cat 1 (multi-hop)	282	28	9.9%	90.1%
Cat 2 (temporal)	321	26	8.1%	91.9%
Cat 3 (open-domain)	96	9	9.4%	90.6%
Cat 4 (single-hop)	841	36	4.3%	95.7%
Total	1,540	99	6.4%	93.6%

An additional 57 citation-only errors (wrong evidence dialog IDs, correct answers) were also documented. These don't affect scoring but indicate broader annotation quality issues.

Error types include HALLUCINATION (facts not present anywhere in the transcript), TEMPORAL_ERROR (incorrect date calculations), ATTRIBUTION_ERROR (statements attributed to the wrong speaker), and INCOMPLETE (answers that omit facts explicitly stated in the source text).

Methodology: Two-pass audit, first verifying answers against cited evidence dialog IDs, then searching the full conversation transcript when evidence checks failed. The original dataset is included unmodified with SHA256 verification.

Everything is published with full reproducibility data at dial481/locomo-audit — per-conversation audit packages, the consolidated errors.json, and the complete audit report.

The objective is to give the community a way to interpret existing LoCoMo scores with appropriate context, and that anyone reporting results on this benchmark can reference adjusted ceilings. Happy to discuss any of the specific findings.