feat: use json-repair for LLM output parsing by ProgrammerPlus1998 · Pull Request #954 · MemMachine/MemMachine

ProgrammerPlus1998 · 2026-01-16T07:21:49Z

Purpose of the change

Improve JSON parsing robustness for LLM-generated outputs. Large language models sometimes produce non-standard JSON format (trailing commas, single quotes, missing quotes, etc.), causing parse failures with standard json.loads(). This PR uses json-repair library to automatically fix and parse malformed JSON from LLM outputs while maintaining efficiency by only applying it where needed.

Description

This PR replaces json.loads() with repair_json() from the json-repair library specifically for parsing direct LLM outputs. The change improves error tolerance when models produce non-standard JSON format without impacting performance of standard JSON parsing operations.

Changes:

Add json-repair>=0.54 dependency to pyproject.toml
Update LLM judge evaluation files to use repair_json for parsing model responses
Update Bedrock language model to use repair_json for parsing structured outputs from text
Update OpenAI language models to use repair_json for function call arguments parsing

Scope: This change only affects parsing of direct LLM text outputs. API responses and file I/O operations continue to use standard json module to maintain efficiency.

Fixes/Closes

N/A - This is a preventive improvement rather than fixing a specific reported issue.

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Refactor (does not change functionality, e.g., code style improvements, linting)
Documentation update
Project Maintenance (updates to build scripts, CI, etc., that do not affect the main project)
Security (improves security without changing functionality)

How Has This Been Tested?

Manual verification (list step-by-step instructions)

Test Steps:

Reviewed all modified files to ensure repair_json is only used for LLM output parsing
Verified standard json.load/loads/dump/dumps remain for API responses and file operations
Checked that all modified locations handle potential malformed JSON from LLM outputs

Test Results:

All modifications correctly target LLM output parsing locations
Standard JSON operations remain unchanged for efficiency
Import statements properly added to all affected files

Checklist

Maintainer Checklist

Confirmed all checks passed
Contributor has signed the commit(s)
Reviewed the code
Run, Tested, and Verified the change(s) work as expected

Screenshots/Gifs

N/A

Further comments

Performance Note: The json-repair library is slightly slower than standard json.loads() but provides much better error tolerance. By limiting its use to only LLM output parsing (where malformed JSON is more likely), we balance robustness and efficiency.

Modified Files:

evaluation/locomo/episodic_agent/llm_judge.py - LLM judge response parsing
evaluation/locomo/episodic_memory/llm_judge.py - LLM judge response parsing
src/memmachine/common/language_model/amazon_bedrock_language_model.py - Bedrock structured output parsing
src/memmachine/common/language_model/openai_responses_language_model.py - OpenAI function call arguments
src/memmachine/common/language_model/openai_chat_completions_language_model.py - OpenAI function call arguments

Copilot

Pull request overview

This PR enhances JSON parsing robustness for LLM-generated outputs by integrating the json-repair library. LLMs occasionally produce malformed JSON (e.g., trailing commas, single quotes), which causes standard json.loads() to fail. The change targets only direct LLM output parsing locations while preserving standard JSON operations elsewhere for efficiency.

Changes:

Added json-repair>=0.54 dependency
Updated LLM output parsing in evaluation judge files and language model implementations to use repair_json
Broadened exception handling from JSONDecodeError to generic Exception in OpenAI language models

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pyproject.toml	Adds json-repair library dependency
evaluation/locomo/episodic_memory/llm_judge.py	Uses repair_json for parsing LLM judge responses
evaluation/locomo/episodic_agent/llm_judge.py	Uses repair_json for parsing LLM judge responses
src/memmachine/common/language_model/amazon_bedrock_language_model.py	Uses repair_json for parsing Bedrock structured outputs
src/memmachine/common/language_model/openai_responses_language_model.py	Uses repair_json for parsing function call arguments and broadens exception handling
src/memmachine/common/language_model/openai_chat_completions_language_model.py	Uses repair_json for parsing function call arguments and broadens exception handling

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

jealous · 2026-01-16T07:50:54Z

@malatewang @o-love I think this would be helpful when LLM doesn't return a correct json.

ProgrammerPlus1998 · 2026-01-16T08:15:37Z

I've updated the code to properly catch TypeError and ValueError exceptions that repair_json() can throw. Specifically:

TypeError occurs when passing None or non-string types to repair_json()
ValueError can occur when JSON format is beyond repair

jealous · 2026-01-22T07:09:32Z

Hi @ProgrammerPlus1998, thanks for the updates. I triggered the CI, and some unit tests failed due to the changes. Those tests appear to be validating invalid JSON inputs. You may want to remove them or adjust them to use inputs that even json-repair cannot handle.

There are also many lint errors in the uv.lock file, along with a very large number of changed lines. Normally we shouldn’t see lint errors or such a large diff in that file, so it would be good to check what happened there.

Additionally, I noticed that some of your commits are signed, but not all of them. We require all commits to be signed before merging. The easiest way to address this would be to squash the commits and sign the resulting commit.

Let me know if you have any questions.

- Add json-repair library to handle common LLM JSON formatting errors - Update language model implementations (OpenAI, Bedrock) to use json-repair - Fix unit tests to validate JSON repair instead of expecting errors - Use real MemMachine examples in tests (project creation, memory search) - Remove unused json imports after json-repair integration This change improves robustness when handling LLM outputs with malformed JSON (unquoted keys, single quotes, trailing commas, etc.)

ProgrammerPlus1998 · 2026-01-22T09:23:49Z

Hi @jealous ,
Thank you for the feedback! I've addressed all the issues:

Unit Tests Fixed ✅ I've updated the failing tests. The original tests expected ValueError to be raised for invalid JSON, but since we're now using the json-repair library to auto-fix malformed JSON from LLM outputs, I've changed the tests to:
Verify that invalid JSON is successfully repaired (not raising errors)
Confirm the repaired JSON is correctly parsed into Python objects
Use real MemMachine scenarios (project creation, memory search) with multiple common LLM JSON formatting errors (unquoted keys, single quotes, trailing commas)
The test functions have been renamed from test_json_error to test_json_repair to accurately reflect their behavior.
All Commits Signed ✅ I've squashed all commits into a single signed commit.
uv.lock File Regarding the large diff in uv.lock: I'm using uv 0.6.14 (a4cec56dc 2025-04-09). I've regenerated the lock file, and it shows no additional changes, which suggests the current lock file is already in the correct format for this uv version. The original changes were likely due to:
The json-repair dependency addition
Lock file format differences between uv versions
If you'd like me to use a specific uv version to minimize the diff, please let me know and I'll be happy to regenerate it.
Let me know if there's anything else that needs adjustment. Thanks for your patience!

jealous · 2026-01-22T18:58:15Z

I triggered the CI and looks like there are still some issues. Could you please run following command locally in the project's root folder to check:

check and fix the format with ruff format .
static code analysis and possible fix with ruff check --fix .
type check with ty check src.
You may need to install ruff or ty with pip.
And you may also want to resolve the conflicts in uv.lock. Maybe just pull the latest change and add the new dependency.

- Fix import sorting (I001) in 4 files - Sort imports alphabetically and place json_repair correctly - evaluation/locomo/episodic_agent/llm_judge.py - evaluation/locomo/episodic_memory/llm_judge.py - src/memmachine/common/language_model/openai_chat_completions_language_model.py - src/memmachine/common/language_model/openai_responses_language_model.py - Fix code style issues in test files - Remove whitespace from blank lines (W293) - Replace single quotes with double quotes in multiline strings (Q001) - Rename unused unpacked variables to _content (RUF059) - tests/memmachine/common/language_model/test_openai_chat_completions_language_model.py - tests/memmachine/common/language_model/test_openai_responses_language_model.py All ruff checks now pass successfully.

# Conflicts: # uv.lock

Add json-repair v0.55.0 to lock file for LLM output parsing.

ProgrammerPlus1998 · 2026-01-23T02:27:51Z

Hi! Great catch - I've fixed everything! 🎉

✅ Ruff checks: All passing now!

Fixed import sorting (I001) across 4 files
Cleaned up test file formatting (W293, Q001, RUF059)
Both ruff format . and ruff check . are happy now ✨

✅ UV lock conflicts: Completely resolved!

Upgraded uv to v0.9.26 to maintain format consistency
Super clean change: only +13 lines for json-repair dependency
Zero conflicts, zero upgrades to existing packages - minimal impact! 🚀

✅ Type checking: Will verify with ty check src right away

Everything's pushed and ready for review. The CI should be all green now! 🟢

Thanks for the thorough feedback - really appreciate it! Let me know if anything else needs attention.

ProgrammerPlus1998 · 2026-01-23T04:19:50Z

@jealous I've reviewed the error logs but I'm having trouble understanding the root cause and whether it's related to my recent changes. Could you help investigate this issue? Thanks!

jealous · 2026-01-23T05:25:58Z

I don't think it's related to your change. I just triggered a re-run for that job. I think everything should be find. Really appreciate your contribution.

This reverts commit 1d7866f.

* feat: use json-repair for malformed LLM JSON output recovery - Add json-repair library to handle common LLM JSON formatting errors - Update language model implementations (OpenAI, Bedrock) to use json-repair - Fix unit tests to validate JSON repair instead of expecting errors - Use real MemMachine examples in tests (project creation, memory search) - Remove unused json imports after json-repair integration This change improves robustness when handling LLM outputs with malformed JSON (unquoted keys, single quotes, trailing commas, etc.) * fix: resolve ruff linting issues - Fix import sorting (I001) in 4 files - Sort imports alphabetically and place json_repair correctly - evaluation/locomo/episodic_agent/llm_judge.py - evaluation/locomo/episodic_memory/llm_judge.py - src/memmachine/common/language_model/openai_chat_completions_language_model.py - src/memmachine/common/language_model/openai_responses_language_model.py - Fix code style issues in test files - Remove whitespace from blank lines (W293) - Replace single quotes with double quotes in multiline strings (Q001) - Rename unused unpacked variables to _content (RUF059) - tests/memmachine/common/language_model/test_openai_chat_completions_language_model.py - tests/memmachine/common/language_model/test_openai_responses_language_model.py All ruff checks now pass successfully. * chore: add json-repair dependency to uv.lock Add json-repair v0.55.0 to lock file for LLM output parsing.

jealous requested review from Copilot and sscargal January 16, 2026 07:46

Copilot AI reviewed Jan 16, 2026

View reviewed changes

Comment thread src/memmachine/common/language_model/openai_responses_language_model.py Outdated

Comment thread src/memmachine/common/language_model/openai_chat_completions_language_model.py Outdated

jealous requested a review from a team January 16, 2026 19:39

ProgrammerPlus1998 force-pushed the feat/use-json-repair-for-llm-output branch from 38c3997 to 5cc565c Compare January 22, 2026 09:15

Merge branch 'main' into feat/use-json-repair-for-llm-output

fdf7c2e

ProgrammerPlus1998 added 3 commits January 23, 2026 09:45

Merge branch 'main' into feat/use-json-repair-for-llm-output

ea35324

# Conflicts: # uv.lock

chore: add json-repair dependency to uv.lock

67a7275

Add json-repair v0.55.0 to lock file for LLM output parsing.

jealous approved these changes Jan 23, 2026

View reviewed changes

tianAndrew approved these changes Jan 23, 2026

View reviewed changes

tianAndrew merged commit 1d7866f into MemMachine:main Jan 23, 2026
52 of 53 checks passed

ProgrammerPlus1998 added a commit to ProgrammerPlus1998/MemMachine that referenced this pull request Jan 23, 2026

Revert "feat: use json-repair for LLM output parsing (MemMachine#954)"

93b69b6

This reverts commit 1d7866f.

Navigation Menu

Uh oh!

feat: use json-repair for LLM output parsing - #954

feat: use json-repair for LLM output parsing#954
tianAndrew merged 5 commits into
MemMachine:mainfrom
ProgrammerPlus1998:feat/use-json-repair-for-llm-output

ProgrammerPlus1998 commented Jan 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

jealous commented Jan 16, 2026

Uh oh!

ProgrammerPlus1998 commented Jan 16, 2026

Uh oh!

jealous commented Jan 22, 2026

Uh oh!

ProgrammerPlus1998 commented Jan 22, 2026

Uh oh!

jealous commented Jan 22, 2026

Uh oh!

ProgrammerPlus1998 commented Jan 23, 2026

Uh oh!

ProgrammerPlus1998 commented Jan 23, 2026 •

edited

Loading

Uh oh!

jealous commented Jan 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

ProgrammerPlus1998 commented Jan 16, 2026

Purpose of the change

Description

Fixes/Closes

Type of change

How Has This Been Tested?

Checklist

Maintainer Checklist

Screenshots/Gifs

Further comments

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

jealous commented Jan 16, 2026

Uh oh!

ProgrammerPlus1998 commented Jan 16, 2026

Uh oh!

jealous commented Jan 22, 2026

Uh oh!

ProgrammerPlus1998 commented Jan 22, 2026

Uh oh!

jealous commented Jan 22, 2026

Uh oh!

ProgrammerPlus1998 commented Jan 23, 2026

Uh oh!

ProgrammerPlus1998 commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jealous commented Jan 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ProgrammerPlus1998 commented Jan 23, 2026 •

edited

Loading