Josue Torres-Fonseca (he/him)
Hi! I am a PhD candidate at the University of Michigan, Department of Computer Science and Engineering. I work with Dr. Joyce Chai as a member of the Situated Language and Embodied Dialogue (SLED) Lab. My research focuses on language understanding, cognitive development, and embodied agents, specifically, how robots can take advantage of their ability to interact with the world around them to learn language during spoken interaction with others. My recent work evaluates whether multimodal LLM agents can recognize and mitigate safety hazards while completing household tasks (SafetyALFRED, Findings of ACL 2026), and builds conversational robots that help people living with cognitive decline reflect on their resilience (under review at HRI 2027). Prior to joining the University of Michigan, I graduated from Boise State University with a B.S. in Computer Science (Magna Cum Laude, Minor in Applied Mathematics), where during my last two years I performed research advised by Dr. Casey Kennington as a member of the Speech Language and Interactive Machines (SLIM) Lab.
-
Doctor of Philosophy in Computer Science and Engineering
Expected May 2028, University of Michigan
GPA: 4.0
Advisor: Dr. Joyce Chai -
Master of Science in Computer Science and Engineering
May 2025, University of Michigan
GPA: 4.0
Advisor: Dr. Joyce Chai -
Bachelor of Science in Computer Science, Minor in Applied Mathematics
May 2023, Boise State University
GPA: 3.889, Magna Cum Laude, Dean's List
Advisor: Dr. Casey Kennington
Research publications and preprints.
2027
- Robot to Elicit Resilience Reflection for Individuals Living with Cognitive DeclineUnder review at the ACM/IEEE International Conference on Human-Robot Interaction (HRI) 2027 -- *Equal contribution, co-first authors
2026
2025
2022
Conference presentations and other talks.
2022
Courses I've assisted in teaching.
Tutor, Operating Systems
Jan 2023 – May 2023Boise State University
- Taught students how concepts in class apply to their programming assignments
- Educated students on how concepts taught in class may be used in industry settings
- Assisted students in planning out and debugging their programming assignments
Tutor, Introduction to Theory of Computation
Jan 2021 – May 2021Boise State University
- Assisted students by guiding them through their assignments
- Facilitated individual and group tutoring sessions
Tutor, Data Structures
Aug 2020 – Dec 2020Boise State University
- Worked with students to enhance their understanding of topics discussed in class
- Guided students through debugging their code and understanding how their code worked
- Communicated and effectively explained the purpose of data structures and how they worked
Lab Assistant, Microprocessors
Aug 2020 – Dec 2020Boise State University
- Guided students through lab work and debugging hardware and software issues
- Helped students understand concepts related to the labs and how the labs demonstrated these concepts
- Assisted professor in grading papers
History of academic appointments.
Computer Science and Engineering Division, University of Michigan
August 2023 - PresentPhD Candidate, Situated Language and Embodied Dialogue Group
Supervised by Dr. Joyce Chai.
Project 2: Robot to Elicit Resilience Reflection (under review at HRI 2027, co-first author)
- Designed and built an embodied conversational robot that encourages individuals living with cognitive decline to reflect on their resilience, co-designed with 10 dementia experts and 14 people with cognitive impairment
- Developed a structured dialogue manager that steps through rapport-building, challenges, and resilience reflection phases; blinded raters scored its pacing above end-to-end LLM prompting in 35 of 40 simulated conversations
- Fine-tuned Qwen3-VL-32B on 15K dialogue turns distilled from Gemini 2.5 Pro, plus a resilience topic classifier reaching 84% accuracy; served both on-premise with vLLM to protect participant privacy
- Deployed in 7 lab and 3 in-home studies with individuals living with cognitive decline
Project 1: SafetyALFRED (Findings of ACL 2026)
- Created a safety-centric dataset using PDDL to generate trajectories that complete household tasks while mitigating six categories of kitchen hazards across 30 AI2-THOR scenes
- Evaluated 11 state-of-the-art VLMs on hazard recognition and safe embodied planning, finding models correctly recognized hazards up to 92% of the time but fixed them less than 60% of the time
- Developed a decoupled multi-agent framework separating hazard recognition from task execution, raising the rate at which Qwen3-VL-32B fixed appliance-misuse hazards from 0.7% to 71.1%
- Open-sourced the code and dataset; presented a poster at ACL 2026
Department of Electrical Engineering and Computer Science, MIT
May 2022 - November 2022Undergraduate Researcher, Language & Intelligence Group
Supervised by Dr. Jacob Andreas.
- Designed and built an interpretable ML model providing neuron annotations using no data
- Presented a poster at the MIT Summer Research Program poster session and delivered a lightning talk on the MIT Grad Diversity YouTube channel
Department of Computer Science, Boise State University
August 2021 - August 2023Undergraduate Researcher, Speech Language and Interactive Machines Group
Supervised by Dr. Casey Kennington.
Project 2: Symbol and Communicative Grounding through Object Permanence (SIGdial 2022)
- Designed and built an incremental spoken dialogue system capable of transcribing and understanding speech, detecting and gaining a concrete understanding of objects, and mapping objects in its environment using SLAM and the Anki Cozmo robot
- Approximated object permanence by extending Cozmo’s SLAM map to remember each object it has seen and its location; in an A/B study with 24 participants, this raised the robot’s accuracy answering what/where questions about learned objects from 70.0% to 82.7% and nearly doubled the questions participants asked
Project 1: HADREB (LREC 2022)
- Collected and analyzed HADREB, a dataset of over 1,000 robot emotional behaviors described and rated by 32 participants on the Misty II and Cozmo robots
- Showed a zero-shot language model predicts perceived emotion from the descriptions with up to 79.3% accuracy
Department of Computer Science, Texas State University
June 2020 - August 2020Undergraduate Researcher, Intelligent Security Group
Supervised by Dr. Mina Guirguis; funded by NSF as a Smart & Connected Communities REU site.
- Designed and built a game and reward system to secure a simulated water tank using game theory
- Simulated and secured a water tank with check blocks acting as both an intrusion detection and prevention system
- Prepared presentations and a report on research findings for REU participants and organizers
Other appointments I've held in industry.
Paylocity
May 2021 - August 2021Software Engineer Intern
- Contributed to software development across the full lifecycle as part of a 13-member team
- Worked with product owners and leads to translate use cases into features
- Designed, developed, tested, and deployed new features in project releases
- Applied Agile and Scrum methodologies; gained full-stack experience with JavaScript, React/Redux, C#, .NET, SQL, and Git
- Gemini Academic Program Award ($10,000) — 2026 - 2027
- NSF Graduate Research Fellowship ($159,000) — 2023 - 2028
- Dean's List — 2018 - 2023
- Boise State University Top Ten Scholar — 2023
- Barry M. Goldwater Scholarship ($7,500) — 2022
- Federal Employee Education & Assistance Fund Scholarship ($1,000) — 2018