You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
NOTE:The content of the "Millbrook City RAG Challenge" is fictional and generated using AI techniques for synthetic scenarios generation with the purpose of evaluating real-life challenges for RAG systems.
The scenario is realistic and complex enough where naive RAG may cause a lot of hallucinations.
In questions for which the answers requires understanding relationships, approaches like GraphRAG or chunk conextualization should do well
In areas where transcripts involves multiple languages, the system will do better with embedding model with multilingual capabilities
A system handling all types of challenges present in the documents will require advanced techniques like multiple type of store options and Agentic RAG, to handle the complexity while maintaining the simplicity of each challenge.
The scenario is short enough that it will fit in the context window of a modern LLM. In that case, the challenge becomes more about how well the LLM can work with ambiguous reference, temporal complexity, domain confusion, etc.
The scenario should allow for discussion of many approaches and understand when they work, when they fail, and when to use one approach vs another, or when to combine many.
Background
The Millbrook City RAG Challenge.md is a comprehensive ~3,200-word RAG testing document about Millbrook, a diverse small city. The document is structured as 10 interconnected documents that will challenge RAG systems across multiple dimensions.
Domain confusion: PE license (Professional Engineer vs. Physical Education)
Relationship complexity: Multi-degree professional and personal connections
The document tests domain-specific language across medical, engineering, educational, legal, and municipal contexts while maintaining narrative coherence through the Bridge 7 infrastructure crisis that connects all characters and institutions.
TESTING CHALLENGES EMBEDDED IN THIS DOCUMENT
Ambiguous References
Multiple "Romano" family members (Isabella Romano vs. Rosa Romano)
Two doctors named "Dr. Okafor" (Elena Vasquez's colleague vs. Sofia's husband)
Bridge construction dates: 1963 vs. 1892 city incorporation
Temporal Complexity
Events span from 1892 (city founding) to November 2024
Rosa Romano's bakery established 1963, same year as bridge construction
Cross-referencing career timelines and family relationships
Contradictory Information
Bridge load capacity: 25 tons (current) vs. 40 tons (original) vs. 15-ton restrictions
Economic impact: $47,000/day vs. 25-35% business impact estimates
Response time increase: +4.7 minutes vs. "minimal" in newspaper
Missing Context Scenarios
What happened to Isabella Romano's parents?
Why did Sofia Okafor move from Texas to Millbrook?
Details about Miguel Vasquez's (Elena's husband) architectural projects
Domain Confusion
"PE License" (Professional Engineer) vs. physical education context
"Code-switching" (linguistic) vs. programming context
Medical "attending" (physician) vs. meeting attendance
Relationship Complexity
Carmen Vasquez (daughter of Dr. Elena) friends with Lily Chen (daughter of Marcus)
Emmanuel Okafor (son of Dr. Kwame) friends with Diego Vasquez (son of Dr. Elena)
Professional relationships intersecting with personal connections across multiple institutions
Millbrook Municipal Planning Department Publication Date: March 15, 2024 Report ID: MPD-2024-003
Executive Summary
Millbrook, incorporated in 1892, serves as a diverse mid-sized city of 47,832 residents spanning 23.4 square miles. The municipality encompasses four distinct neighborhoods: Historic Downtown, Riverside District, University Heights, and New Millbrook. Population density averages 2,043 residents per square mile, with significant demographic variation across districts.
Demographic Breakdown by Neighborhood
Historic Downtown (Population: 12,847)
Median Age: 34.2 years
Hispanic/Latino: 35.2%
African American: 23.1%
White: 28.7%
Asian: 8.9%
Other: 4.1%
Riverside District (Population: 15,234)
Median Age: 41.6 years
White: 45.3%
Hispanic/Latino: 26.8%
African American: 15.2%
Asian: 9.1%
Other: 3.6%
University Heights (Population: 11,982)
Median Age: 28.9 years
White: 42.1%
Asian: 24.3%
Hispanic/Latino: 18.7%
African American: 11.2%
Other: 3.7%
New Millbrook (Population: 7,769)
Median Age: 36.8 years
Hispanic/Latino: 38.9%
White: 31.2%
African American: 18.4%
Asian: 7.8%
Other: 3.7%
DOCUMENT 2: BIOGRAPHICAL DOSSIERS
PROFILE: Dr. Elena Vasquez
Position: Chief of Cardiology, Hospital Director Institution: Millbrook General Hospital Address: 1247 Medical Plaza Dr., Riverside District DOB: July 23, 1975 Education:
Professional History:
Dr. Vasquez joined Millbrook General in 2006 as an attending cardiologist. She assumed directorial responsibilities in January 2020, overseeing a 247-bed facility with 340 staff members. Vasquez pioneered the hospital's telemedicine cardiac monitoring program, serving rural communities within a 50-mile radius.
Family: Married to Miguel Vasquez (architect), two children: Carmen (16) and Diego (13)
Languages: Spanish (native), English (fluent), Portuguese (conversational)
PROFILE: Marcus Chen
Position: Physics Teacher (Lincoln High School), City Councilman (District 2) Address: 892 Elm Street, University Heights DOB: November 11, 1981 Education:
BS Physics, UC Berkeley (2003)
MEd Secondary Education, State University (2005)
Public Administration Certificate, Community College (2019)
Professional History:
Chen has taught at Lincoln High School since 2006, developing the Advanced Placement Physics program that achieved 89% pass rates. Elected to city council in 2020, representing University Heights and portions of Historic Downtown. Champions education funding and sustainable development initiatives.
Family: Single parent to daughter Lily Chen (14), student at Lincoln High School
Languages: English (native), Mandarin (fluent), Spanish (basic)
PROFILE: Sofia Okafor
Position: Principal Civil Engineer Company: Okafor Engineering Solutions Address: 156 Commerce Way, Historic Downtown DOB: March 8, 1988 Education:
BS Civil Engineering, Howard University (2010)
MS Structural Engineering, Stanford (2012)
PE License: Texas (#89472), California (#67821)
Professional History:
Founded Okafor Engineering Solutions in 2018 after six years with municipal engineering departments. Current projects include Millbrook Bridge rehabilitation, University Heights stormwater management, and Historic Downtown pedestrian safety improvements. Firm employs 12 engineers and 6 support staff.
Family: Married to Dr. Kwame Okafor (pediatrician), one son: Emmanuel (8)
Languages: English (native), Igbo (fluent), French (conversational)
Business Management Certificate, Community College (1995)
Professional History:
Immigrated to Millbrook in 1989, initially working construction before opening the bakery in 1994. The establishment serves as an informal community center, hosting cultural events and serving as a meeting place for the Islamic Community Center board. Al-Rashid mentors young entrepreneurs through the Small Business Association.
Family: Married to Fatima Al-Rashid, three children: Omar (28, attorney), Layla (25, nurse), Samir (22, college student)
Languages: Arabic (native), English (fluent), Spanish (basic)
PROFILE: Isabella Romano
Position: Nursing Student (Senior), Part-time CNA at Sunset Manor Address: 567 Cypress Avenue, New Millbrook DOB: January 15, 2002 Education:
BSN Program, Millbrook Community College (Expected May 2025)
CNA Certification (2022)
Professional History:
Isabella works 20 hours weekly at Sunset Manor Assisted Living while completing her nursing degree. She specializes in geriatric care and plans to pursue psychiatric nursing after graduation. Her grandmother, Rosa Romano, founded Romano's Family Bakery in 1963.
Family: Lives with grandmother Rosa Romano (89), parents deceased
Languages: English (native), Italian (fluent), Spanish (conversational)
PROFILE: Dr. James Patterson
Position: Principal, Lincoln High School Address: 445 Academic Circle, University Heights DOB: August 19, 1968 Education:
BA History, Millbrook State (1990)
MEd Educational Leadership, State University (1995)
EdD Educational Administration, Regional University (2008)
PROFILE: Rebecca Martinez
Position: Emergency Department Nurse Manager Institution: Millbrook General Hospital Address: 778 Riverside Park Dr., Riverside District DOB: December 3, 1985
PROFILE: David Kim
Position: Software Engineer, Municipal IT Department Address: 334 Tech Lane, University Heights DOB: May 22, 1992
PROFILE: Maria Santos
Position: Social Worker, Family Services Division Address: 223 Community Blvd, Historic Downtown DOB: April 17, 1979
PROFILE: Robert Thompson
Position: Fire Chief, Millbrook Fire Department Address: 445 Safety Street, Riverside District DOB: October 8, 1973
DOCUMENT 3: MILLBROOK CITY COUNCIL MEETING MINUTES
Date: October 14, 2024 Time: 7:00 PM - 9:45 PM Location: Council Chambers, City Hall Attendees: Mayor Jennifer Walsh, Councilman Marcus Chen (District 2), Councilwoman Sarah Mitchell (District 1), Councilman Roberto Silva (District 3), Councilwoman Patricia Johnson (District 4)
AGENDA ITEM 3.2: INFRASTRUCTURE ASSESSMENT REPORT
Councilman Chen: The engineering assessment from Okafor Engineering Solutions indicates that Bridge 7 over Millbrook Creek requires immediate attention. Sofia Okafor's team identified structural deficiencies in the eastern support pylons.
Mayor Walsh: What's the projected timeline for repairs, and how will this impact the University Heights commuter route?
Councilman Chen: According to the preliminary report, repairs could take 4-6 months. Sofia recommends a phased approach to maintain partial traffic flow. The estimated cost is $2.3 million.
Councilwoman Johnson: Has Dr. Vasquez been consulted about emergency vehicle access to the hospital during construction?
Councilman Silva: That's a valid concern. The hospital serves as our primary trauma center, and any delays could be critical.
Motion: Councilwoman Mitchell moves to approve $50,000 for detailed engineering assessment by Okafor Engineering Solutions. Seconded by Councilman Chen. PASSED 5-0
AGENDA ITEM 4.1: SMALL BUSINESS SUPPORT PROGRAM
Mayor Walsh: Ahmed Al-Rashid submitted a proposal for a microloan program supporting immigrant entrepreneurs. The Islamic Community Center has offered to provide financial literacy workshops.
Councilman Chen: This aligns with our economic development goals. The Historic Downtown district would benefit significantly from increased small business activity.
Councilwoman Johnson: What oversight mechanisms are proposed?
[Meeting continues with budget discussions, zoning changes, and public comments...]
DOCUMENT 4: MEDICAL RECORD EXCERPT
MILLBROOK GENERAL HOSPITAL Patient ID: MG-2024-7841 Patient: Emmanuel Okafor DOB: 06/12/2016 Guardian: Sofia Okafor (mother), Dr. Kwame Okafor (father)
Date of Service: September 22, 2024 Attending Physician: Dr. Kwame Okafor, Pediatrics Chief Complaint: Acute onset abdominal pain
History of Present Illness:
8-year-old male presented to ED with sudden onset severe RLQ abdominal pain, duration 6 hours. Patient reports nausea, one episode of vomiting. No fever documented at home. Mother (Sofia) reports normal bowel movements, no recent illness. Father (Dr. Okafor) concerned about appendicitis given symptom presentation.
Physical Examination:
Vitals: T 100.2°F, HR 110, BP 95/60, RR 22, O2 Sat 98% RA
General: Alert, moderately distressed
Abd: Tender RLQ with guarding, positive McBurney's point tenderness
Nursing Notes (Rebecca Martinez, RN):
Patient's aunt, Isabella Romano, arrived to provide additional support. Family demonstrates strong community connections. Sofia expressed concerns about missing work commitments at engineering firm but prioritizes child's care appropriately.
DOCUMENT 5: ACADEMIC TRANSCRIPT
LINCOLN HIGH SCHOOL Student: Carmen Vasquez Student ID: LHS-2025-0234 DOB: 03/14/2008 Parent/Guardian: Dr. Elena Vasquez, Miguel Vasquez
Academic Year 2023-2024 - Junior Year
Course
Teacher
Grade
Credits
AP Physics C
Marcus Chen
A+
1.0
AP Spanish Literature
Ms. Rodriguez
A
1.0
Honors Chemistry
Dr. Williams
A-
1.0
AP Calculus BC
Ms. Thompson
A
1.0
English III Honors
Mr. Davis
B+
1.0
World History
Ms. Carter
A-
1.0
GPA: 3.92 (Weighted: 4.31) Class Rank: 12 of 342
Extracurricular Activities:
Science Olympiad Team Captain
Volunteer, Millbrook General Hospital (50+ hours)
Math Tutoring Program (organized by Lily Chen)
Teacher Comments:Mr. Chen (AP Physics): "Carmen demonstrates exceptional analytical skills and natural leadership. Her bilingual abilities have been valuable in peer tutoring sessions with Spanish-speaking students."
Principal Patterson's Note: "Carmen shows strong potential for pre-med programs. Her volunteer work at her mother's hospital demonstrates commitment to healthcare service."
DOCUMENT 6: PERSONAL DIARY ENTRY
From the diary of Lily Chen Date: November 1, 2024
Dad came home from the city council meeting tonight looking tired again. I know he's stressed about the bridge project - he keeps talking about how Ms. Okafor's engineering report shows it's worse than anyone thought. He said something about Dr. Vasquez being worried about ambulance routes to the hospital.
Carmen and I studied together today for Mr. Chen's (my dad's) physics test. It's so weird having my dad as a teacher! Carmen is super smart though - she's already thinking about medical school like her mom. She told me that her little brother Diego has been hanging out with Emmanuel Okafor after school. Their families are getting close since Dr. Okafor (Emmanuel's dad) and Dr. Vasquez work together at the hospital.
I helped Isabella Romano with her CNA homework today at the library. She's so good with elderly people - she told me about how she takes care of her grandmother and works at the nursing home. She speaks Italian with her grandmother, which sounds so beautiful. She said her family's bakery used to compete with Mr. Al-Rashid's place, but now they're friends and sometimes share suppliers.
Dad says I need to start thinking about colleges soon. Carmen's already planning to apply to pre-med programs. Maybe I'll study engineering like Ms. Okafor - she's so cool and smart. Dad says she's doing amazing work on the city infrastructure.
Following our October 14 council discussion, my team completed detailed analysis of Bridge 7's structural integrity. The situation is more critical than initially assessed.
Key Findings:
Eastern pylon foundation shows 15% displacement from original position
Load-bearing capacity reduced to 65% of design specifications
Projected timeline for catastrophic failure: 8-14 months without intervention
Recommended Action Plan:
Phase 1 (Immediate): Install temporary support structures ($180,000)
Phase 3 (Month 4-6): Deck rehabilitation and safety upgrades ($950,000)
Impact Assessment:
Dr. Vasquez at Millbrook General Hospital expressed concerns about emergency vehicle access. I recommend establishing alternative routes through University Heights, adding approximately 4.7 minutes to response times. This route passes Lincoln High School, where Principal Patterson has agreed to coordinate traffic management during school hours.
Ahmed Al-Rashid and other Historic Downtown business owners have requested economic impact mitigation strategies. Reduced foot traffic could affect businesses by 25-35% during construction.
Timeline Coordination:
I've coordinated with David Kim from Municipal IT to implement smart traffic routing systems. Rebecca Martinez from the hospital emergency department will participate in weekly progress reviews to ensure medical service continuity.
Professional Engineering Seal:
Sofia Okafor, PE #89472
Principal Engineer, Okafor Engineering Solutions
DOCUMENT 8: CONVERSATION TRANSCRIPT
Date: October 25, 2024 Location: Al-Rashid Mediterranean Bakery Participants: Ahmed Al-Rashid (Owner), Dr. Elena Vasquez, Marcus Chen, Isabella Romano
[TRANSCRIPT BEGINS - 2:15 PM]
Ahmed:[في العربية] أهلاً وسهلاً، دكتورة إيلينا! Welcome, Dr. Elena! Coffee is fresh.
Dr. Vasquez: Gracias, Ahmed. Marcus, I'm glad you could meet here. We need to discuss the bridge situation and hospital access.
Marcus: Sofia's latest report is concerning. The timeline is tighter than we hoped.
Isabella:[entering] Scusi, Mr. Al-Rashid. Is this the meeting about the bridge? My nursing instructor said the hospital might need alternative emergency plans.
Ahmed: Yes, Isabella. Please, sit. Dr. Vasquez was just explaining.
Dr. Vasquez: The issue is that our emergency response times could increase significantly. Para los pacientes cardíacos, four additional minutes could be critical.
Marcus:[switching to basic Mandarin with Isabella] 你明白吗？Do you understand the medical concerns?
Isabella: Sí, yes. At Sunset Manor, we sometimes need quick hospital transfers. I can translate for the Italian residents who might not understand the construction delays.
Ahmed: My bakery sees many community members daily. I can help spread information about the alternative routes. [في العربية] المجتمع يجب أن يعرف - The community must know.
Dr. Vasquez: That's helpful. Isabella, how is your grandmother Rosa handling the changes in the neighborhood?
Isabella: Nonna Rosa remembers when they built the original bridge in 1963 - the same year she opened Romano's Bakery. She says "questi ponti vengono e vanno" - these bridges come and go.
Marcus: We need to coordinate with the schools too. Lily mentioned that students use Bridge 7 to get to their after-school jobs.
Ahmed: Omar, my eldest son - the attorney - he says the city should consider legal protections for businesses affected by the construction.
Dr. Vasquez: Sofia's engineering assessment mentions that Dr. Okafor - Kwame, mi colega en pediatría - has concerns about ambulance routing to the children's ward.
Isabella: Speaking of Dr. Okafor, his son Emmanuel was asking about the construction when I saw him at the library with Diego Vasquez.
Marcus: That's right - Carmen mentioned that Diego and Emmanuel have become friends. Small world in Millbrook.
[TRANSCRIPT CONTINUES...]
DOCUMENT 9: NEWSPAPER ARTICLE
THE MILLBROOK HERALD November 2, 2024
"Community Rallies Around Bridge Crisis"
By Staff Reporter Jennifer Liu
As Millbrook's Bridge 7 faces its most significant structural challenge since construction in 1963, community leaders are demonstrating the collaborative spirit that defines this diverse city.
Sofia Okafor, whose engineering firm identified the critical infrastructure failure, has been working around the clock with city officials to minimize disruption. "This isn't just about engineering," Okafor explained during yesterday's public forum. "It's about maintaining the connections that make Millbrook function as a community."
The bridge crisis has united unlikely allies. Ahmed Al-Rashid, whose Mediterranean bakery has served Historic Downtown for three decades, has partnered with Rosa Romano's family bakery to establish a "Bridge Support Network" providing discounted meals for construction workers.
Dr. Elena Vasquez, director of Millbrook General Hospital, praised the community response. "When Emmanuel Okafor - Dr. Kwame Okafor's son - was treated in our emergency department last month, we saw how this community comes together. Now we're seeing it again with the bridge situation."
City Councilman Marcus Chen, who represents the affected University Heights district, notes that the crisis has strengthened inter-neighborhood cooperation. "My daughter Lily and her friend Carmen Vasquez have organized study groups to help students affected by the longer commute times," Chen reported.
The construction timeline remains challenging, but early coordination efforts show promise. Rebecca Martinez, who manages emergency services at the hospital, confirmed that response time increases have been minimal thanks to proactive routing strategies.
"This is what makes Millbrook special," reflected Isabella Romano, a nursing student whose family has operated in Historic Downtown since the 1960s. "We don't just live in the same city - we take care of each other."
DOCUMENT 10: TECHNICAL ENGINEERING REPORT
OKAFOR ENGINEERING SOLUTIONS Project: Millbrook Bridge 7 Structural Rehabilitation Report ID: OES-2024-MB7-FINAL Principal Engineer: Sofia Okafor, PE Date: November 1, 2024
EXECUTIVE SUMMARY
Comprehensive structural analysis of Bridge 7 spanning Millbrook Creek reveals critical deficiencies requiring immediate intervention. Load-bearing capacity has degraded to 65% of original design specifications due to foundation settlement and material fatigue.
DETAILED FINDINGS
Foundation Analysis:
Eastern pylon shows 15.3cm displacement from vertical axis
Steel reinforcement: 12% cross-sectional loss (maximum observed)
REMEDIATION STRATEGY
Phase 1: Emergency Stabilization (30 days)
Install temporary shoring system beneath eastern span
Implement load restrictions: 15-ton maximum vehicle weight
Establish continuous monitoring with strain gauges and tiltmeters
Estimated cost: $180,000
Phase 2: Foundation Reconstruction (60 days)
Excavate and replace eastern pylon foundation
Install micropile support system extending to bedrock
Implement dewatering system for construction zone
Estimated cost: $1,200,000
Phase 3: Superstructure Rehabilitation (45 days)
Deck replacement with post-tensioned concrete system
Upgrade expansion joint assemblies
Install cathodic protection system for rebar
Estimated cost: $950,000
RISK ASSESSMENT
Public Safety Factors:
Current load rating: 25 tons (reduced from 40 tons)
Probability of catastrophic failure: 15% within 12 months without intervention
Emergency vehicle routing impact: +4.7 minutes average response time via alternative routes
Economic Impact Analysis:
Daily traffic volume: 12,400 vehicles
Commercial vehicle percentage: 18%
Economic loss per day of closure: $47,000 (based on detour costs and business impact)
RECOMMENDATIONS
Immediate Action Required: Implement Phase 1 stabilization within 14 days
Traffic Management: Coordinate with Millbrook PD and Emergency Services for alternative routing
Public Communication: Establish community liaison protocols with local business leaders
Long-term Planning: Consider seismic upgrade compliance for future resilience
Quality Assurance:
All analysis conducted in accordance with AASHTO LRFD Bridge Design Specifications (9th Edition) and ACI 318-19 Building Code Requirements for Structural Concrete.
Professional Engineer Certification:
This report prepared under direct supervision of Sofia Okafor, PE License #89472, Texas Board of Professional Engineers.
Question: "Who owns Romano's Bakery, and how is this person related to Isabella Romano?"
Challenge: Tests ability to distinguish between Isabella Romano (nursing student) and Rosa Romano (grandmother, bakery founder)
Expected Answer: Rosa Romano owns/founded Romano's Family Bakery in 1963. Isabella Romano is her granddaughter who lives with her.
Wrong Answers a RAG might give:
Isabella Romano owns the bakery
They are the same person
No relationship established
QUESTION 1B (Difficulty: Hard)
Question: "Dr. Okafor treated a patient in the emergency department. What was the patient's condition, and what is Dr. Okafor's relationship to the patient?"
Challenge: Tests disambiguation between Dr. Kwame Okafor (pediatrician) and Dr. Elena Vasquez's colleague also named "Dr. Okafor"
Expected Answer: Dr. Kwame Okafor (pediatrician) treated Emmanuel Okafor for acute appendicitis. Emmanuel is Dr. Okafor's son.
Wrong Answers a RAG might give:
Confusing which Dr. Okafor
Incorrect patient-doctor relationship
Missing the family connection
QUESTION 1C (Difficulty: Medium)
Question: "There are two bakeries mentioned in the documents. What are their names and who runs them?"
Challenge: Tests ability to identify and distinguish between Al-Rashid Mediterranean Bakery and Romano's Family Bakery
Expected Answer: Al-Rashid Mediterranean Bakery (run by Ahmed Al-Rashid) and Romano's Family Bakery (founded by Rosa Romano in 1963)
Wrong Answers a RAG might give:
Only identifying one bakery
Confusing the owners
Missing the historical founding information
CHALLENGE CATEGORY 2: TEMPORAL COMPLEXITY
QUESTION 2A (Difficulty: Hard)
Question: "What significant infrastructure event happened in Millbrook in 1963, and how does it relate to current events in 2024?"
Challenge: Tests ability to connect historical events (bridge construction, bakery founding) with current crisis
Expected Answer: Bridge 7 was constructed in 1963 (same year Rosa Romano founded her bakery). In 2024, this bridge requires major structural repairs due to deterioration.
Wrong Answers a RAG might give:
Missing the temporal connection
Confusing construction vs. repair dates
Not linking historical and current events
QUESTION 2B (Difficulty: Medium)
Question: "How long has Ahmed Al-Rashid been operating his bakery, and what was his career before opening it?"
Challenge: Tests timeline calculation and career progression tracking
Expected Answer: Al-Rashid has been operating his bakery for 30 years (opened 1994, document dated 2024). Before that, he worked in construction after immigrating in 1989.
Wrong Answers a RAG might give:
Incorrect timeline calculations
Missing the immigration date
Confusing pre-bakery career
QUESTION 2C (Difficulty: Medium)
Question: "When did Dr. Elena Vasquez assume her current role as Hospital Director, and how long had she been at the hospital before that promotion?"
Challenge: Tests career progression timeline within the same institution
Expected Answer: She became Hospital Director in January 2020. She had been at Millbrook General since 2006, so 14 years before her promotion.
Wrong Answers a RAG might give:
Confusing start date vs. promotion date
Incorrect timeline calculations
Missing the career progression details
CHALLENGE CATEGORY 3: CONTRADICTORY INFORMATION
QUESTION 3A (Difficulty: Hard)
Question: "What is the current load capacity of Bridge 7, and are there any restrictions in place?"
Challenge: Tests ability to identify and resolve conflicting load capacity information
Expected Answer: The current load rating is 25 tons (reduced from original 40 tons), but there are temporary 15-ton restrictions due to safety concerns.
Wrong Answers a RAG might give:
Providing only one number without context
Missing the temporary restrictions
Not explaining the capacity reduction
QUESTION 3B (Difficulty: Medium)
Question: "How much time will be added to emergency response times due to the bridge construction, and how has this actually affected hospital operations?"
Challenge: Tests ability to reconcile specific projections (+4.7 minutes) with general reports ("minimal impact")
Expected Answer: The engineering report projects +4.7 minutes average response time, but the newspaper reports actual increases have been "minimal" due to proactive routing strategies.
Wrong Answers a RAG might give:
Only citing one source
Not recognizing the apparent contradiction
Missing the distinction between projected vs. actual impact
QUESTION 3C (Difficulty: Hard)
Question: "What is the projected economic impact of the bridge construction on local businesses?"
Challenge: Tests handling of different economic impact measurements
Expected Answer: Two different measures are given: $47,000 per day economic loss (from detour costs), and 25-35% reduction in foot traffic for Historic Downtown businesses.
Wrong Answers a RAG might give:
Only providing one metric
Confusing the different types of economic impact
Missing the distinction between general economic loss and specific business impact
CHALLENGE CATEGORY 4: MISSING CONTEXT SCENARIOS
QUESTION 4A (Difficulty: Medium)
Question: "What happened to Isabella Romano's parents?"
Challenge: Tests ability to recognize when information is not provided in the documents
Expected Answer: The documents indicate Isabella's parents are deceased but do not provide details about what happened to them.
Wrong Answers a RAG might give:
Making up details not in the documents
Claiming no information about parents is provided
Confusing with other family information
QUESTION 4B (Difficulty: Hard)
Question: "Why did Sofia Okafor move from Texas to Millbrook to start her engineering firm?"
Challenge: Tests recognition of missing contextual information about motivation
Expected Answer: The documents indicate Sofia has a Texas PE license and founded her firm in Millbrook in 2018, but do not explain the reason for relocating.
Wrong Answers a RAG might give:
Fabricating reasons not in the text
Not recognizing the Texas connection
Missing the timeline of firm founding
QUESTION 4C (Difficulty: Medium)
Question: "What architectural projects has Miguel Vasquez (Elena's husband) worked on in Millbrook?"
Challenge: Tests recognition when a person is mentioned but their professional work is not detailed
Expected Answer: Miguel Vasquez is identified as an architect and Elena's husband, but no specific projects in Millbrook are mentioned in the documents.
Wrong Answers a RAG might give:
Inventing projects not mentioned
Confusing with other construction/engineering work
Missing that he's an architect
CHALLENGE CATEGORY 5: DOMAIN CONFUSION
QUESTION 5A (Difficulty: Medium)
Question: "What does Sofia Okafor's PE license enable her to do professionally?"
Challenge: Tests correct interpretation of "PE" as Professional Engineer vs. Physical Education
Expected Answer: Sofia's PE (Professional Engineer) license #89472 enables her to practice engineering and stamp engineering documents in Texas (she also has a California PE license #67821).
Wrong Answers a RAG might give:
Confusing with Physical Education
Not understanding professional licensing
Missing the multi-state licensing aspect
QUESTION 5B (Difficulty: Hard)
Question: "In the medical record, what does 'attending physician' refer to, and how does this differ from meeting attendance mentioned elsewhere?"
Expected Answer: "Attending physician" refers to Dr. Kwame Okafor's role as the supervising doctor responsible for patient care, not meeting attendance. This is standard medical terminology.
Wrong Answers a RAG might give:
Confusing with meeting attendance
Not understanding medical hierarchy
Missing the professional role distinction
QUESTION 5C (Difficulty: Medium)
Question: "What type of 'code-switching' is mentioned in the documents?"
Challenge: Tests linguistic vs. programming context understanding
Expected Answer: Linguistic code-switching - the practice of alternating between languages within conversations, as shown in the bakery conversation transcript.
Wrong Answers a RAG might give:
Relating to computer programming
Missing the multilingual context
Not understanding sociolinguistic terminology
CHALLENGE CATEGORY 6: RELATIONSHIP COMPLEXITY
QUESTION 6A (Difficulty: Hard)
Question: "How are Carmen Vasquez and Emmanuel Okafor connected, and trace the path of their relationship through their families' professional connections?"
Challenge: Tests multi-hop relationship reasoning across professional and personal domains
Expected Answer: Carmen Vasquez (daughter of Dr. Elena Vasquez) is friends with Emmanuel Okafor (son of Dr. Kwame Okafor). Their mothers work together at Millbrook General Hospital - Elena as Hospital Director and Kwame as a pediatrician.
Wrong Answers a RAG might give:
Missing the family connections
Not linking professional and personal relationships
Confusing the hospital hierarchy
QUESTION 6B (Difficulty: Hard)
Question: "What connections exist between the Chen family and the bridge project, considering both professional and personal relationships?"
Challenge: Tests ability to trace complex relationship networks across multiple contexts
Expected Answer: Marcus Chen (city councilman) works professionally with Sofia Okafor on the bridge project. His daughter Lily Chen is friends with Carmen Vasquez, whose mother (Dr. Elena Vasquez) has concerns about bridge impact on hospital access. Lily also helps Isabella Romano with studies, connecting to another affected family.
Wrong Answers a RAG might give:
Only identifying direct professional connections
Missing the student friendship networks
Not connecting family relationships to professional issues
QUESTION 6C (Difficulty: Hard)
Question: "How many degrees of separation exist between Ahmed Al-Rashid and Dr. Kwame Okafor, and what is the path?"
Challenge: Tests complex network analysis across multiple document types
Expected Answer: Multiple paths exist:
Direct: Al-Rashid → community meetings → hospital discussions → Dr. Okafor
Through children: Al-Rashid → community connections → Emmanuel Okafor → Dr. Okafor (father)
Through Isabella: Al-Rashid → Isabella Romano (who knows about Emmanuel's hospital visit) → Dr. Okafor
Wrong Answers a RAG might give:
Only finding one connection path
Missing community network connections
Not recognizing multiple relationship types
CHALLENGE CATEGORY 7: MULTILINGUAL AND CULTURAL COMPLEXITY
QUESTION 7A (Difficulty: Medium)
Question: "What languages does Dr. Elena Vasquez speak, and how do her language skills impact her role in the community?"
Challenge: Tests information synthesis across multiple documents and contexts
Expected Answer: Dr. Vasquez speaks Spanish (native), English (fluent), and Portuguese (conversational). Her bilingual abilities help in medical care and community meetings, as noted in Carmen's academic comments about tutoring Spanish-speaking students.
Wrong Answers a RAG might give:
Missing some languages
Not connecting language skills to professional/community impact
Confusing with other multilingual characters
QUESTION 7B (Difficulty: Hard)
Question: "In the bakery conversation transcript, identify all instances of code-switching and explain what languages are being mixed."
Challenge: Tests detailed linguistic analysis and cultural understanding
Expected Answer: Multiple instances:
Ahmed: Arabic ("أهلاً وسهلاً" - welcome) to English
Dr. Vasquez: Spanish ("Gracias," "Para los pacientes cardíacos") mixed with English
Marcus: English to Mandarin ("你明白吗？")
Isabella: Italian ("Scusi," "questi ponti vengono e vanno") to Spanish/English
Ahmed: Arabic ("المجتمع يجب أن يعرف") translated to English
Question: "According to the engineering report, what are the specific technical reasons Bridge 7 needs immediate attention, and how do these relate to the timeline concerns expressed in the city council meeting?"
Challenge: Tests integration of technical and policy information across document types
Expected Answer: Technical reasons include 15.3cm pylon displacement, 65% load capacity, micro-cracking, and 18% natural frequency reduction. These support the council's concern about the 8-14 month failure timeline and need for immediate $2.3M intervention.
Wrong Answers a RAG might give:
Only citing general concerns without technical details
Not connecting engineering data to policy discussions
Missing the timeline correlation
QUESTION 8B (Difficulty: Medium)
Question: "What medical considerations are mentioned regarding the bridge construction, and which healthcare professionals are involved in planning?"
Challenge: Tests cross-domain professional coordination
Expected Answer: Main concerns are increased emergency response times (+4.7 minutes) affecting cardiac patients. Dr. Elena Vasquez (Hospital Director) and Rebecca Martinez (Emergency Nurse Manager) are involved in planning alternative routes and weekly progress reviews.
Wrong Answers a RAG might give:
Missing specific medical professionals
Not identifying the cardiac patient concern
Confusing the response time impact
SCORING GUIDE FOR RAG SYSTEMS:
EXCELLENT (90-100%): Correctly answers questions requiring multi-hop reasoning, resolves ambiguities, identifies contradictions, and recognizes missing information
GOOD (75-89%): Handles most complex relationships and domain-specific terminology but may miss some subtle contradictions or temporal connections
FAIR (60-74%): Retrieves basic information accurately but struggles with multi-document synthesis and complex reasoning
POOR (Below 60%): Frequent errors in disambiguation, temporal reasoning, and relationship mapping
CRITICAL FAILURES:
Fabricating information not in documents
Confusing domain-specific terminology
Missing obvious contradictions
Failing to recognize missing information scenarios
