Josue Torres-Fonseca (he/him)
Hello! I’m a Ph.D. candidate in Computer Science and Engineering at the University of Michigan, advised by Dr. Joyce Chai in the SLED Lab. My research studies how embodied AI agents, from robots to multimodal language models, understand language and act safely in the physical world. Some directions I’m particularly interested in include:
-
Safety in embodied agents. When multimodal agents recognize a household hazard, do they act to mitigate it, from high-level planning down to low-level control, and where inside these models is safety information lost? I study this with benchmarks and mechanistic interpretability. (see ACL 2026)
-
Grounding language. How can robots learn what words mean by interacting with the world, and how does grounding emerge mechanistically inside language models? (see ICML 2026, SIGdial 2022, SIGdial 2025, LREC 2022)
-
Conversational robots for well-being. How can embodied conversational agents help people living with cognitive decline reflect on and strengthen their resilience? (HRI 2027 under review)
My work is supported by the NSF Graduate Research Fellowship. I enjoy working with and mentoring students, so if you’re interested in any of these directions, feel free to reach out!
-
Doctor of Philosophy in Computer Science and Engineering
Expected May 2028, University of Michigan
GPA: 4.0
Advisor: Dr. Joyce Chai -
Master of Science in Computer Science and Engineering
May 2025, University of Michigan
GPA: 4.0
Advisor: Dr. Joyce Chai -
Bachelor of Science in Computer Science, Minor in Applied Mathematics
May 2023, Boise State University
GPA: 3.889, Magna Cum Laude, Dean's List
Advisor: Dr. Casey Kennington
Research publications and preprints.
* Equal contribution, co-first authors
2027
- Robot to Elicit Resilience Reflection for Individuals Living with Cognitive DeclineUnder review at the ACM/IEEE International Conference on Human-Robot Interaction (HRI) 2027
2026
2025
2022
Conference presentations and other talks.
2026
2022
Courses I've assisted in teaching.
Tutor, Operating Systems
Jan 2023 – May 2023Boise State University
- Taught students how concepts in class apply to their programming assignments
- Educated students on how concepts taught in class may be used in industry settings
- Assisted students in planning out and debugging their programming assignments
Tutor, Introduction to Theory of Computation
Jan 2021 – May 2021Boise State University
- Assisted students by guiding them through their assignments
- Facilitated individual and group tutoring sessions
Tutor, Data Structures
Aug 2020 – Dec 2020Boise State University
- Worked with students to enhance their understanding of topics discussed in class
- Guided students through debugging their code and understanding how their code worked
- Communicated and effectively explained the purpose of data structures and how they worked
Lab Assistant, Microprocessors
Aug 2020 – Dec 2020Boise State University
- Guided students through lab work and debugging hardware and software issues
- Helped students understand concepts related to the labs and how the labs demonstrated these concepts
- Assisted professor in grading papers
History of academic appointments.
Computer Science and Engineering Division, University of Michigan
August 2023 - PresentPhD Candidate, Situated Language and Embodied Dialogue Group
Supervised by Dr. Joyce Chai.
ReMI: A Conversational Robot to Elicit Resilience Reflection (under review at HRI 2027, co-first author)
- Designed and built an embodied conversational robot that encourages individuals living with cognitive decline to reflect on their resilience, co-designed with 10 dementia experts and 14 people with cognitive impairment
- Developed a structured dialogue manager that steps through rapport-building, challenges, and resilience reflection phases; blinded raters scored its pacing above end-to-end LLM prompting in 35 of 40 simulated conversations
- Fine-tuned Qwen3-VL-32B on 15K dialogue turns distilled from Gemini 2.5 Pro, plus a resilience topic classifier reaching 84% accuracy; served both on-premise with vLLM to protect participant privacy
- Deployed in 7 lab and 3 in-home studies with individuals living with cognitive decline
SafetyALFRED: Evaluating Safety-Conscious Planning of Multimodal LLMs (Findings of ACL 2026)
- Created a safety-centric dataset using PDDL to generate trajectories that complete household tasks while mitigating six categories of kitchen hazards across 30 AI2-THOR scenes
- Evaluated 11 state-of-the-art VLMs and found a gap between recognizing and acting: models identified hazards in question answering up to 92% of the time, but fixed them less than 60% of the time as embodied agents
- Developed a decoupled multi-agent framework separating hazard recognition from task execution, raising the rate at which Qwen3-VL-32B fixed appliance-misuse hazards from 0.7% to 71.1%
- Open-sourced the code and dataset; presented a poster at ACL 2026
Department of Electrical Engineering and Computer Science, MIT
May 2022 - November 2022Undergraduate Researcher, Language & Intelligence Group
Supervised by Dr. Jacob Andreas through the MIT Summer Research Program (MSRP).
- Designed and built an interpretable ML model providing neuron annotations using no data
- Presented a poster at the MIT Summer Research Program poster session and delivered a lightning talk on the MIT Grad Diversity YouTube channel
Department of Computer Science, Boise State University
August 2021 - August 2023Undergraduate Researcher, Speech Language and Interactive Machines Group
Supervised by Dr. Casey Kennington.
Symbol and Communicative Grounding through Object Permanence on a Mobile Robot (SIGdial 2022)
- Studied how object permanence helps a robot learn language, building an incremental spoken dialogue system that transcribes and understands speech, detects and grounds objects, and maps them with SLAM on the Anki Cozmo robot
- Extended Cozmo’s SLAM map to remember each object it has seen and its location; in an A/B study with 24 participants, this raised the robot’s accuracy answering what/where questions about learned objects from 70.0% to 82.7% and nearly doubled the questions participants asked
HADREB: A Dataset of Human Appraisals of Robot Emotional Behaviors (LREC 2022)
- Collected and analyzed HADREB, a dataset of over 1,000 robot emotional behaviors described and rated by 32 participants on the Misty II and Cozmo robots
- Showed a zero-shot language model predicts perceived emotion from the descriptions with up to 79.3% accuracy
Department of Computer Science, Texas State University
June 2020 - August 2020Undergraduate Researcher, Intelligent Security Group
Supervised by Dr. Mina Guirguis through an NSF Research Experiences for Undergraduates (REU) site in Smart & Connected Communities.
- Designed and built a game and reward system to secure a simulated water tank using game theory
- Simulated and secured a water tank with check blocks acting as both an intrusion detection and prevention system
- Prepared presentations and a report on research findings for REU participants and organizers
Other appointments I've held in industry.
Paylocity
May 2021 - August 2021Software Engineer Intern
- Contributed to software development across the full lifecycle as part of a 13-member team
- Worked with product owners and leads to translate use cases into features
- Designed, developed, tested, and deployed new features in project releases
- Applied Agile and Scrum methodologies; gained full-stack experience with JavaScript, React/Redux, C#, .NET, SQL, and Git
- Gemini Academic Program Award ($10,000) — Spring 2026 - Spring 2027
- NSF Graduate Research Fellowship ($159,000) — Fall 2023 - Spring 2027
- Dean's List — Fall 2018 - Spring 2023
- Boise State University Top Ten Scholar — Spring 2023
- Barry M. Goldwater Scholarship ($7,500) — Fall 2022 - Spring 2023
- Federal Employee Education & Assistance Fund Scholarship ($1,000) — Fall 2018
Students I've collaborated with and advised on research.
Current
- Yuzhang Mei: Undergraduate, University of Michigan (2026 - Present)
- Jun Wang: Master's, University of Michigan (2026 - Present)
Past
- Avi Bhattacharya: Undergraduate, University of Michigan (2026)
→ Software / ML Engineer, ServiceNow
If you're a University of Michigan student interested in my research and would like to work with me, please send me an email.