Skip to content

Feature: JSON.FILTER #1465

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
devBaunz wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Feature: JSON.FILTER #1465

devBaunz wants to merge 1 commit into from

Conversation

@devBaunz
Copy link

@devBaunz devBaunz commented Nov 25, 2025 •
edited
Loading

Suggestion for an API - JSON.FILTER

Syntax

JSON.FILTER key1 [key2 ...] < path > < filter-expression >

Description

The command evaluates the filter-expression against each key's JSON document. If the filter returns non-empty results, the path is fetched from that document; otherwise, null is returned for that key position.

Complexity:

O(M*N) where M is the number of keys and N is the size of the document. Both filter-expression and path are evaluated against each document, with filter evaluation occurring first. When filter or path evaluate to multiple values: O(N1+N2+...+Nm) where m is the number of keys and Ni is the size of the i-th key

Basically a lightweight way of filtering and accessing the JSON tree at different points while adhering to JSONPath.
This allows user to have more complex operations without suffering a big performance impact, so that we can do things such as:

  1. Filter data at ROOT such as $.[?($.type == "metadata")] - to fetch any element that is a "metadata"
  2. Allow implementing a arr.filter((item) => predicate(item)) without having to first fetch the entire arr
  3. Allow detaching the filtering aspect of JSONPath from the actual data-fetch, so we can check something along the tree but retrieve a different element

Implementation details:

  • Follows JSON.MGET pattern for consistency
  • Validates filter syntax upfront using compile()
  • Uses calc_once() for filter evaluation
  • Supports both legacy (.) and modern ($) path formats
  • Returns array with results or null for non-matching keys

NOTE: This is an initial draft, it lacks tests and a more in-depth SPEC file, but this is mainly to discuss the concept, idea and reasoning of why this should/not be added

@CLAassistant
Copy link

CLAassistant commented Nov 25, 2025 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@devBaunz devBaunz mentioned this pull request Nov 25, 2025
Open
@devBaunz devBaunz marked this pull request as ready for review November 27, 2025 14:21
@ephraimfeldblum
Copy link
Contributor

ephraimfeldblum commented Nov 30, 2025

thank you for the contribution.

there's a small typo in https://github.com/RedisJSON/RedisJSON/pull/1465/files#diff-fe8692f5492557c8fe9633a9784c4369d58ae070dd14d9e63696f57256a4efe9R1047 (missing a :).

can you please add tests verifying correctness of both the happy path as well as any error paths. eg something like

def testFilterCommandErrors(env):
    env.expect('JSON.FILTER', '{doc}:1', '$').raiseError()
    env.expect('JSON.FILTER', '{doc}:1', '$', '$[?(@.a>0)]').raiseError()
    env.cmd('JSON.SET', '{doc}:1', '$', '{"a":1}')
    env.expect('JSON.FILTER', '{doc}:1', '$', '$[?(@.a>').raiseError()
    env.expect('JSON.FILTER', '{doc}:1', '$..[', '$[?(@.a>0)]').raiseError()
def testFilterCommandBasic(env):
    env.cmd('JSON.SET', '{doc}:1', '$', '{"name":"Alice","age":30,"city":"NYC","active":true}')
    env.cmd('JSON.SET', '{doc}:2', '$', '{"name":"Bob","age":25,"city":"LA","active":false}')
    env.cmd('JSON.SET', '{doc}:3', '$', '{"name":"Charlie","age":35,"city":"NYC","active":true}')
    env.cmd('JSON.SET', '{doc}:4', '$', '{"name":"Diana","age":28,"city":"SF","active":true}')

    res = env.cmd('JSON.FILTER', '{doc}:1', '{doc}:2', '{doc}:3', '{doc}:4', '$', '$[?(@.active==true)]')
    env.assertEqual(len(res), 4)
    env.assertNotEqual(res[0], None)
    env.assertEqual(res[1], None)
    env.assertNotEqual(res[2], None)
    env.assertNotEqual(res[3], None)

    res = env.cmd('JSON.FILTER', '{doc}:1', '{doc}:2', '{doc}:3', '{doc}:4', '$.name', '$[?(@.age>28)]')
    env.assertEqual(len(res), 4)
    env.assertEqual(json.loads(res[0]), ["Alice"])
    env.assertEqual(res[1], None)
    env.assertEqual(json.loads(res[2]), ["Charlie"])
    env.assertEqual(res[3], None)

    res = env.cmd('JSON.FILTER', '{doc}:1', '{doc}:2', '{doc}:3', '{doc}:4', '$', '$[?(@.city=="NYC")]')
    env.assertEqual(len(res), 4)
    env.assertNotEqual(res[0], None)
    env.assertEqual(res[1], None)
    env.assertNotEqual(res[2], None)
    env.assertEqual(res[3], None)

    res = env.cmd('JSON.FILTER', '{doc}:1', '{doc}:missing', '{doc}:3', '$', '$[?(@.active==true)]')
    env.assertEqual(len(res), 3)
    env.assertNotEqual(res[0], None)
    env.assertEqual(res[1], None)
    env.assertNotEqual(res[2], None)

    env.cmd('SET', '{doc}:wrong_type', 'not a json')
    res = env.cmd('JSON.FILTER', '{doc}:1', '{doc}:wrong_type', '{doc}:3', '$', '$[?(@.active==true)]')
    env.assertEqual(len(res), 3)
    env.assertNotEqual(res[0], None)
    env.assertEqual(res[1], None)
    env.assertNotEqual(res[2], None)

    env.cmd('JSON.SET', '{doc}:nested1', '$', '{"user":{"name":"Eve","score":85},"status":"active"}')
    env.cmd('JSON.SET', '{doc}:nested2', '$', '{"user":{"name":"Frank","score":92},"status":"inactive"}')
    env.cmd('JSON.SET', '{doc}:nested3', '$', '{"user":{"name":"Grace","score":78},"status":"active"}')

    res = env.cmd('JSON.FILTER', '{doc}:nested1', '{doc}:nested2', '{doc}:nested3', '$.user.name', '$[?(@.user.score>80)]')
    env.assertEqual(len(res), 3)
    env.assertEqual(json.loads(res[0]), ["Eve"])
    env.assertEqual(json.loads(res[1]), ["Frank"])
    env.assertEqual(res[2], None)

    env.cmd('JSON.SET', '{doc}:arr1', '$', '{"items":[1,2,3],"count":3}')
    env.cmd('JSON.SET', '{doc}:arr2', '$', '{"items":[4,5],"count":2}')
    env.cmd('JSON.SET', '{doc}:arr3', '$', '{"items":[6,7,8,9],"count":4}')

    res = env.cmd('JSON.FILTER', '{doc}:arr1', '{doc}:arr2', '{doc}:arr3', '$.count', '$[?(@.count>2)]')
    env.assertEqual(len(res), 3)
    env.assertEqual(json.loads(res[0]), [3])
    env.assertEqual(res[1], None)
    env.assertEqual(json.loads(res[2]), [4])

    res = env.cmd('JSON.FILTER', '{doc}:1', '{doc}:2', '{doc}:3', '.name', '.[?(@.age>28)]')
    env.assertEqual(len(res), 3)
    env.assertEqual(json.loads(res[0]), "Alice")
    env.assertEqual(res[1], None)
    env.assertEqual(json.loads(res[2]), "Charlie")

    res = env.cmd('JSON.FILTER', '{doc}:1', '{doc}:2', '{doc}:3', '$', '$[?(@.age>100)]')
    env.assertEqual(len(res), 3)
    env.assertEqual(res[0], None)
    env.assertEqual(res[1], None)
    env.assertEqual(res[2], None)

    env.cmd('JSON.SET', '{doc}:multi1', '$', '{"a":1,"nested":{"a":2,"b":3}}')
    env.cmd('JSON.SET', '{doc}:multi2', '$', '{"a":4,"nested":{"a":5,"b":6}}')

    res = env.cmd('JSON.FILTER', '{doc}:multi1', '{doc}:multi2', '$..a', '$[?(@.a>0)]')
    env.assertEqual(len(res), 2)
    env.assertEqual(json.loads(res[0]), [1, 2])
    env.assertEqual(json.loads(res[1]), [4, 5])
def testFilterCommandComplex(env):
    env.cmd('JSON.SET', '{doc}:store1', '$', '{"store":{"book":[{"category":"reference","price":8.95},{"category":"fiction","price":12.99}]}}')
    env.cmd('JSON.SET', '{doc}:store2', '$', '{"store":{"book":[{"category":"fiction","price":8.99},{"category":"fiction","price":22.99}]}}')
    env.cmd('JSON.SET', '{doc}:store3', '$', '{"store":{"book":[{"category":"reference","price":15.00}]}}')

    res = env.cmd('JSON.FILTER', '{doc}:store1', '{doc}:store2', '{doc}:store3', '$.store.book[*].price', '$.store.book[?(@.category=="fiction")]')
    env.assertEqual(len(res), 3)
    env.assertNotEqual(res[0], None)
    env.assertNotEqual(res[1], None)
    env.assertEqual(res[2], None)

    env.cmd('JSON.SET', '{doc}:deep1', '$', '{"level1":{"level2":{"value":10}}}')
    env.cmd('JSON.SET', '{doc}:deep2', '$', '{"level1":{"level2":{"value":5}}}')

    res = env.cmd('JSON.FILTER', '{doc}:deep1', '{doc}:deep2', '$..value', '$..level2[?(@.value>7)]')
    env.assertEqual(len(res), 2)
    env.assertEqual(json.loads(res[0]), [10])
    env.assertEqual(res[1], None)

@devBaunz
Copy link
Author

devBaunz commented Nov 30, 2025

@ephraimfeldblum Sure thing, sorry about it - was actually copy-pasting from my local fork (which pre-dates the macro usage) so didnt check it in-depth.

I'll get to it today - will also differentiate JSON.FILTER (new impl) and JSON.MFILTER (current impl)
as per discussion at #1466

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ephraimfeldblum ephraimfeldblum Awaiting requested review from ephraimfeldblum

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@devBaunz @CLAassistant @ephraimfeldblum