Initial Pass at Lambda Filters - Seeking Feedback / Help

[bcameron1231]

This is the initial pass at building Lambda Expression for OData Filters.

Overall feelings - Not loving it, it feels a bit kludgy. It tried to implement the operations where there was a clear separation of AND/OR conditionals but that proved difficult and will need some thoughts on. Additionally, constructing without lambda using the FilterBuilder object, is also a bit kludgy and probably needs to be re-worked (as well as re-named).

I have another branch where I'm working on improving this, but haven't invested a ton yet to show. Would love to see if this effort can be improved.

1. Allows for the ability pass in Standard Odata filter (for backwards compatibility).
2. Allows for agility to use Lambda expressions with Intellisense
3. Supports passing in date objects that can be automatically handled by the library
4. Supports creating chained and/or conditionals
5. Supports for re-using the underlying filter builder objects.

Sample usage

  //standard odata
  const r = await sp.web.lists.getByTitle("SPFx List").items.filter("Title eq 'Beau'")();    
  
  //lambda
  const r2 = await sp.web.lists.getByTitle("SPFx List").items.filter(i=> i.field("Title").equals("Beau").or(i=>i.field("SPFxCostCenter").equals("Information")))();
  const r3 = await sp.web.lists.getByTitle("SPFx List").items.filter(i=> i.field("Title").equals("Test").and(i=>i.field("SPFxCostCenter").equals("Information")))();
  const r4 = await sp.web.lists.getByTitle("SPFx List").items.filter(i=>(i.field("Title").equals("Beau")).or(i=>i.field("Title").equals("Cameron")))();
  
  //filter builder - very kludgy, needs to be re-worked
 let builder = new FilterBuilder();
 builder.field("Title").equals("Test");
 const r5 = await sp.web.lists.getByTitle("SPFx List").items.filter(builder.build())();

[Tanddant]

Hi @bcameron1231

I've been working on a similar concept as well - I went for less of a Lambda, and more of a CAMLjs style

OData.Where<ITask>()
    .NumberField("taskProductId").EqualTo(ProjectId)
    .And()
    .DateField("Created").LessThanOrEqualTo(new Date())
    .ToString();

Where The fields passed into DateField or NumberField (and so on) are required to be keys of the ITask interface

Output

taskProductId eq 58 and Created le '2023-11-20T14:18:45.455Z'

I've also implemented support for nested queries

I.e

OData.Where().Some([
    OData.Where<IProject>().NumberField("EditorId").EqualTo(60),
    OData.Where<IProject>().NumberField("EditorId").EqualTo(6),
    OData.Where<IProject>().NumberField("EditorId").EqualTo(85),
]).ToString()

( EditorId eq 60 or EditorId eq 6 or EditorId eq 85 )

These can also be nested (my main gripe with lambda expressions, is how unreadable they get at scale)

OData.Where().Some([
    OData.Where().All([
        OData.Where<any>().TextField("ProjectStatus").NotEqualTo("Done"),
        OData.Where<any>().DateField("Deadline").LessThan(new Date()),
    ]),
    OData.Where().All([
        OData.Where<any>().TextField("ProjectStatus").EqualTo("Critical"),
    ]),
]).ToString()

"( ( ProjectStatus neq 'Done' and Deadline lt '2023-11-20T14:29:02.608Z' ) or ( ProjectStatus eq 'Critical' ) )

[bcameron1231]

Thanks @Tanddant. Some great ideas here. Do you think you could create a branch with some of these changes implemented, or do you need some help with that?

We think there there is definitely a lot to like to about your implementation, but there may be a middle ground between the approach that I have and what you've implemented that could take this to the next level. I would love to see the inner workings of it.

Let me know how you'd like to best collaborate on this.

[Tanddant]

Hi @bcameron1231

Would love to help out - bit busy this week (need to finalize a bunch of stuff ahead of ESPC) - I spent 30 minutes trying to get the repo to work locally, but failed 😅

I've added my code at the end of the spqueryable.ts in my own fork, let me know if you have any questions - I'm at that phase of the solution where I'm like "I like this, it could work, and scale, but there's probably a reason no one has done this before if it's this easy, so I'm missing something obvious"

Also the naming could use a rework or two

Tanddant@4c43d58#diff-bdc963fcde94659fffe052eb963737c9440c80ed8435a039055a7892e3d8a7bb

[Tanddant]

There are also some filters I'm unaware of how works (never used) day, month, year, hour, minute, second, that maybe we should also think of a way to incorporate

docs

[Tanddant]

@bcameron1231 - I've moved my code here for now, as I keep iterating on it a bit, maybe we can have a chat post ESPC about what makes sense, I like the ideas you're working with using lambda expressions, but can certainly see the benefits of having type validation, maybe there's a golden middle ground

[bcameron1231]

@bcameron1231 - I've moved my code here for now, as I keep iterating on it a bit, maybe we can have a chat post ESPC about what makes sense, I like the ideas you're working with using lambda expressions, but can certainly see the benefits of having type validation, maybe there's a golden middle ground

That sounds good to me. I have some ideas we can start playing with. Let's connect in December.

[Tanddant]

Hey @bcameron1231, My calendar (and post-conference flu) is starting to clear up a bit!

Before I left i rewrote it to be based upon passing around a single class rather then just a string array (Rewrite branch), other then that I haven't had much time to look at it - I'm a bit stuck on getting PnPjs to run on my machine - so I can't really try it out in the full context, but let me know if there's anything you need me to do 😊

[bcameron1231]

@Tanddant Sent you a message on discord. We can try and get it resolved :)

[patrick-rodgers]

@Tanddant - what's the latest status here? Do we want to merge this into v4 so other can contribute? Are you still actively developing? No pressure on timeline, just not sure where we are with this work. Thanks!

[Tanddant]

@patrick-rodgers we have a well functioning solution, but for some odd reason I can't build it without commenting out some of it, and then "uncommenting" it while the code is running, @bcameron1231 wanted to take a look, but to my knowledge has been super busy 😊

[Tanddant]

@bcameron1231 Have you had a chance to look at this? 😊

[bcameron1231]

@Tanddant Not yet. :( I've been extremely busy recently.

[Tanddant]

@bcameron1231 All good, no worries - take care of yourself and relax when there's a bit of down time 😊

[Tanddant]

@bcameron1231 Are you down to see if we can get this working? 😊

[Tanddant]

No pressure, but I think I have this working like below, but would like either some "challenging queries" to test with, or someone else to review what I've got

Feel free to check my branch https://github.com/Tanddant/pnpjs/tree/v4-ODataFilterQuery

My Model (and lists) look like this

export interface IEmployee {
    Firstname: string;
    Lastname: string;
    Age: number;
    Department: IDepartment;
    Employed: boolean;
    Manager: IEmployee;
    SecondaryDepartment: IDepartment[];
}

export interface IDepartment {
    Title: string;
    DepartmentNumber: number;
    Alias: string;
    Manager: IEmployee;
    HasSharedMailbox: boolean;
}

Basic usage, there are three different ways to use these functions, all demonstrated here, my personal preference is number 1, but all three work, so I'm not planning on actively blocking it, all three have full on intellisense including validation that you can't use a numberfield to query a choice field and so on, so it'll only suggest the relevant fields for you, even knowing which fields are queryable on lookups.

// Get all active employees who either:
// 1. Work in Consulting, are managed by Dan, and are over 20
// 2. Work in Marketing and are over 30

// Method 1
const r = await sp.web.lists.getByTitle("Employees").items.filter<IEmployee>(f => f.All(
    f.BooleanField("Employed").IsTrue(),
    f.Some(
        f.All(
            f.LookupField("Department").TextField("Alias").EqualTo("Consulting"),
            f.LookupField("Manager").TextField("Firstname").EqualTo("Dan"),
            f.NumberField("Age").GreaterThan(20)
        ),
        f.All(
            f.LookupField("Department").TextField("Alias").EqualTo("Marketing"),
            f.NumberField("Age").GreaterThan(30)
        )
    )
))();
// (Employed eq 1 and ((Department/Alias eq 'Consulting' and Manager/Firstname eq 'Dan' and Age gt 20) or (Department/Alias eq 'Marketing' and Age gt 30)))

// Method 2
const r1 = await sp.web.lists.getByTitle("Employees").items.filter<IEmployee>(f => f.All(
    f => f.BooleanField("Employed").IsTrue(),
    f => f.Some(
        f => f.All(
            f => f.LookupField("Department").TextField("Alias").EqualTo("Consulting"),
            f => f.LookupField("Manager").TextField("Firstname").EqualTo("Dan"),
            f => f.NumberField("Age").GreaterThan(20)
        ),
        f => f.All(
            f => f.LookupField("Department").TextField("Alias").EqualTo("Marketing"),
            f => f.NumberField("Age").GreaterThan(30)
        )
    )
))();
// (Employed eq 1 and ((Department/Alias eq 'Consulting' and Manager/Firstname eq 'Dan' and Age gt 20) or (Department/Alias eq 'Marketing' and Age gt 30)))

// Method 3
const r2 = await sp.web.lists.getByTitle("Employees").items.filter<IEmployee>(SPOData.Where<IEmployee>().All(
    SPOData.Where<IEmployee>().BooleanField("Employed").IsTrue(),
    SPOData.Where<IEmployee>().Some(
        SPOData.Where<IEmployee>().All(
            SPOData.Where<IEmployee>().LookupField("Department").TextField("Alias").EqualTo("Consulting"),
            SPOData.Where<IEmployee>().LookupField("Manager").TextField("Firstname").EqualTo("Beau"),
            SPOData.Where<IEmployee>().NumberField("Age").GreaterThan(20)
        ),
        SPOData.Where<IEmployee>().All(
            SPOData.Where<IEmployee>().LookupField("Department").TextField("Alias").EqualTo("Marketing"),
            SPOData.Where<IEmployee>().NumberField("Age").GreaterThan(30)
        )
    )
))();
// (Employed eq 1 and ((Department/Alias eq 'Consulting' and Manager/Firstname eq 'Beau' and Age gt 20) or (Department/Alias eq 'Marketing' and Age gt 30)))

Some other random examples I could come up with

let list: IItems = sp.web.lists.getByTitle("Employees").items;

// Get all employees who work in Consulting and were hired before 2020
await list.filter<IEmployee>(f => f.All(
    f.LookupField("Department").TextField("Alias").EqualTo("Consulting"),
    f.DateField("HireDate").LessThan(new Date(2020, 1, 1))
))();
// (Department/Alias eq 'Consulting' and HireDate lt '2020-01-31T23:00:00.000Z')

// Get all employees who work in Marketing, were hired today, and have an 'a' in their first name
await list.filter<IEmployee>(f => f.All(
    f.DateField("HireDate").IsToday(),
    f.LookupField("Department").TextField("Alias").EqualTo("Marketing"),
    f.TextField("Firstname").Contains("a")
))();
// ((HireDate gt '2024-10-18T22:00:00.000Z' and HireDate lt '2024-10-19T21:59:59.999Z') and Department/Alias eq 'Marketing' and substringof('a', Firstname))

// Get all employees who work in Consulting, are not employed, and have a birthday in the next 30 days
await list.filter<IEmployee>(f => f.All(
    f.LookupField("Department").TextField("Alias").EqualTo("Consulting"),
    f.BooleanField("Employed").IsFalse(),
    f.DateField("Birthday").IsBetween(new Date(), new Date(new Date().getTime() + 30 * 24 * 60 * 60 * 1000))
))();
// (Department/Alias eq 'Consulting' and Employed eq 0 and (Birthday gt '2024-10-19T15:05:15.410Z' and Birthday lt '2024-11-18T15:05:15.410Z'))

// Get all employees named Dan, Jeppe, or Beau
await list.filter<IEmployee>(f => f.TextField("Firstname").In(["Dan", "Jeppe", "Beau"]))();
// (Firstname eq 'Dan' or Firstname eq 'Jeppe' or Firstname eq 'Beau')

// Get all employees who work in Consulting and are managed by Beau
await list.filter<IEmployee>(f => f.All(
    f.LookupField("Department").TextField("Alias").EqualTo("Consulting"),
    f.LookupField("Manager").TextField("Firstname").EqualTo("Beau")
))();
// (Department/Alias eq 'Consulting' and Manager/Firstname eq 'Beau')

// Get all employees who's fist name is Dan or last name is Toft
await list.filter<IEmployee>(f => f.TextField("Firstname").EqualTo("Dan").Or().TextField("Lastname").EqualTo("Toft"))();
// Firstname eq 'Dan' or Lastname eq 'Toft'

// Get alle active employees in consulting under the age of 30 who's name has a 'd' in it
// Or any employees who still works with us but are over the age of 60
await list.filter<IEmployee>(f => f.All(
    f.BooleanField("Employed").IsTrue(),
    f.Some(
        f.All(
            f.LookupField("Department").TextField("Alias").EqualTo("Consulting"),
            f.NumberField("Age").LessThan(30),
            f.TextField("Firstname").Contains("d")
        ),
        f.NumberField("Age").GreaterThan(60)
    )
))();
// (Employed eq 1 and ((Department/Alias eq 'Consulting' and Age lt 30 and substringof('d', Firstname)) or Age gt 60))


// Get everyone who is 30 or above and works in a department with an alias containing 'd' or 't'
await list.filter<IEmployee>(f => f.NumberField("Age").GreaterThanOrEqualTo(30).And().Some(f.LookupField("Department").TextField("Alias").Contains("d"), f.LookupField("Department").TextField("Alias").Contains("t")))();
// Age ge 30 and (substringof('d', Department/Alias) or substringof('t', Department/Alias))


// Get all employees who are employed and over 20
await list.filter<IEmployee>(f => f.All(f.BooleanField("Employed").IsTrue(), f.NumberField("Age").GreaterThan(20)))();
// (Employed eq 1 and Age gt 20)

[Tanddant]

I also need some opinions on what folks like more for the "grouping" operators

Pass in an array

await list.filter<IEmployee>(f => f.All([
    f => f.BooleanField("Employed").IsTrue(),
    f => f.NumberField("Age").GreaterThan(20)
]))();

Or just pass in multiple arguments?

await list.filter<IEmployee>(f => f.All(
    f => f.BooleanField("Employed").IsTrue(),
    f => f.NumberField("Age").GreaterThan(20)
))();

(The only difference is the [...])