new tests: RemesPath vec fun w/ all args funcs

1. need to test that vectorized arg functions work correctly for both arrays and objects when the first argument is an iterable function of input and a non-first argument is also a function of input 2. fix typos in RemesPath docs 3. add UDL file for RemesPath, to make it slightly easier to write RemesPath queries
molsonkiko · Jan 5, 2024 · 3f7800f · 3f7800f
1 parent 889628d
commit 3f7800f
Show file tree

Hide file tree

Showing 6 changed files with 154 additions and 50 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -21,7 +21,21 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 }
 ```
 3. Add option for users to choose newline in JSON Lines.
-
+4. Add new arguments to `stringify` RemesPath function to allow pretty-printing or customization of other formatting settings
+5. Make it so find/replace form can emit Python code that is equivalent to the generated RemesPath query
+6. Make RemesPath error messages less confusing
+7. Implement a less thread-unsafe way to have ArgFunctions use context:
+    * Add `uses_context` field to ArgFunction instances, so that they have JQueryContext appended to their arguments, and they can reference fields of that JQueryContext.
+	* This way we don't have to have these methods mutating and referencing a global static variable.
+	* Additionally, the presence of a function with `uses_context=true` would serve as a flag that the query cannot be executed in parallel, because doing so would cause race conditions associated with the shared JQueryContext fields.
+8. Add `conditional_execution` as a field for ArgFunctions.
+    * The first function with this set to true would be `ifelse`, but we could also add `and` and `or` non-vectorized functions that work the same as the corresponding functions in Python.
+	* This would be implemented by adding a vanilla CurJson (with identity function) after the final argument. This CurJson would always be evaluated before the function was evaluated, and would thus provide a reference to input, to allow for conditional evaluation of other arguments.
+9. Make it so the regex search form makes a very basic effort to determine the quote character, delimiter, and number of columns in CSV files.
+    * maybe only try to do this for files with the `.csv` and `.tsv` extensions
+	* only test the `,` and `\t` delimiters, and only the `"` or `'` quote characters
+	* test only the first 10KB of the file, or first 25 lines, whichever comes first.
+
 ### To Be Changed
 
 - If there's a validation error inside of an `anyOf` list of schemas (i.e. JSON doesn't validate under *any* of the schemas), the error message is rather uninformative, and says only "the JSON didn't validate under any of the schemas", but not *why* it didn't validate.
@@ -42,6 +56,10 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 
 ## [6.2.0] - (UNRELEASED) YYYY-MM-DD
 
+### Added
+
+1. A [RemesPath user-defined language (UDL) file](/RemesPath%20UDL.xml), providing some very basic syntax highlighting. It is buggy, but that is because the UDL system is inherently buggy, not because I did anything wrong (as far as I know).
+
 ### Changed
 
 1. The [`ifelse` vectorized function in RemesPath](/docs/RemesPath.md#vectorized-functions) now uses conditional execution.

diff --git a/JsonToolsNppPlugin/Properties/AssemblyInfo.cs b/JsonToolsNppPlugin/Properties/AssemblyInfo.cs
@@ -28,5 +28,5 @@
 //      Build Number
 //      Revision
 //
-[assembly: AssemblyVersion("6.1.1.2")]
-[assembly: AssemblyFileVersion("6.1.1.2")]
+[assembly: AssemblyVersion("6.1.1.3")]
+[assembly: AssemblyFileVersion("6.1.1.3")]
diff --git a/JsonToolsNppPlugin/Tests/RemesPathTests.cs b/JsonToolsNppPlugin/Tests/RemesPathTests.cs
@@ -265,8 +265,8 @@ public static bool Test()
                 new Query_DesiredResult("j`[{\"a\": 0,\"b\":-1}, {\"c\":1}, {\"a\":1}, {\"a\":3,\"b\":-4}, 4]`[:3]!.a", "[{\"b\": -1},{\"c\":1}]"),
                 // ufunction tests
                 new Query_DesiredResult("len(@)", fooLen.ToString()),
-                new Query_DesiredResult("s_mul(@.bar.b, 2)", "[\"a`ga`g\", \"bahbah\"]"),
-                new Query_DesiredResult("s_mul(@.bar.b{foo: s_slice(@[0], 2), bar: s_slice(@[1], :2), baz: a}, len(@.foo))", "{\"foo\": \"ggg\", \"bar\": \"bababa\", \"baz\": \"aaa\"}"), // vectorized arg functions on objects where second and subsequent args are functions of input
+                new Query_DesiredResult("s_mul(@.bar.b, len(@.foo) - 1)", "[\"a`ga`g\", \"bahbah\"]"), // vectorized arg function on *arrays* at least one non-first arg is a function of input
+                new Query_DesiredResult("s_mul(@.bar.b{foo: s_slice(@[0], 2), bar: s_slice(@[1], :2), baz: a}, len(@.foo))", "{\"foo\": \"ggg\", \"bar\": \"bababa\", \"baz\": \"aaa\"}"), // vectorized arg function on *objects* where at least one non-first arg is a function of input
                 new Query_DesiredResult("s_lpad(@{ab, aba, d*7}, cd, 5)", "[\"cdcdab\", \"cdaba\", \"ddddddd\"]"),
                 new Query_DesiredResult("s_lpad(@{ab, c*4}, c, 4)", "[\"ccab\", \"cccc\"]"),
                 new Query_DesiredResult("s_rpad(@{ab, aba, d*7}, cd, 5)", "[\"abcdcd\", \"abacd\", \"ddddddd\"]"),
@@ -706,7 +706,7 @@ public static bool Test()
                 }
             }
             // the rand() and randint() functions require a special test because their outputs are nondeterministic
-            ii += 6;
+            ii += 7;
             bool test_failed = false;
             string randints1argQuery = "flatten(@.foo)[:]->randint(1000)";
             string randints2argQuery = "range(9)[:]->randint(-700, 800)";
@@ -799,6 +799,26 @@ public static bool Test()
                     Npp.AddLine($"Expected remesparser.Search(j`[1,2,3]`{{{{rand(@)}}}}[0] to return array of doubles that aren't equal, but instead threw" +
                                       $" an exception:\n{RemesParser.PrettifyException(ex)}");
                 }
+                // make sure that if an array has every value is equal to a variable referencing rand(),
+                //     all values in the array are the same (because they all reference the same variable)
+                string q = "var foo = range(3); var bar = rand(); foo = bar; foo";
+                try
+                {
+                    result = remesparser.Search(q, foo);
+                    if (!(result is JArray arr && arr[0].value is double d1 && d1 >= 0 && d1 <= 1 && arr.children.All(x => x.value is double xd && xd == d1)))
+                    {
+                        test_failed = true;
+                        tests_failed++;
+                        Npp.AddLine($"Expected remesparser.Search(\"{q}\", foo) to return an array where every value is the same double between 0 and 1, but instead got result {result.ToString()}");
+                    }
+                }
+                catch (Exception ex)
+                {
+                    test_failed = true;
+                    tests_failed++;
+                    Npp.AddLine($"Expected remesparser.Search(\"{q}\", foo) to return an array where every value is the same double between 0 and 1, but instead got exception {RemesParser.PrettifyException(ex)}");
+                }
+                // randint tests
                 if (firstRandintsIfElse is null || firstRandints1arg is null || firstRandints2args is null)
                     continue;
                 try

diff --git a/RemesPath UDL.xml b/RemesPath UDL.xml
@@ -0,0 +1,64 @@
+<NotepadPlus>
+    <UserLang name="RemesPath" ext="rpath remes remespath" udlVersion="2.1">
+        <Settings>
+            <Global caseIgnored="no" allowFoldOfComments="no" foldCompact="no" forcePureLC="2" decimalSeparator="0" />
+            <Prefix Keywords1="no" Keywords2="no" Keywords3="yes" Keywords4="no" Keywords5="no" Keywords6="no" Keywords7="no" Keywords8="no" />
+        </Settings>
+        <KeywordLists>
+            <Keywords name="Comments">00var 00for 00end 01 02= 02= 02; 03 04</Keywords>
+            <Keywords name="Numbers, prefix1"></Keywords>
+            <Keywords name="Numbers, prefix2"></Keywords>
+            <Keywords name="Numbers, extras1"></Keywords>
+            <Keywords name="Numbers, extras2"></Keywords>
+            <Keywords name="Numbers, suffix1"></Keywords>
+            <Keywords name="Numbers, suffix2"></Keywords>
+            <Keywords name="Numbers, range"></Keywords>
+            <Keywords name="Operators1"></Keywords>
+            <Keywords name="Operators2"></Keywords>
+            <Keywords name="Folders in code1, open"></Keywords>
+            <Keywords name="Folders in code1, middle"></Keywords>
+            <Keywords name="Folders in code1, close"></Keywords>
+            <Keywords name="Folders in code2, open"></Keywords>
+            <Keywords name="Folders in code2, middle"></Keywords>
+            <Keywords name="Folders in code2, close"></Keywords>
+            <Keywords name="Folders in comment, open"></Keywords>
+            <Keywords name="Folders in comment, middle"></Keywords>
+            <Keywords name="Folders in comment, close"></Keywords>
+            <Keywords name="Keywords1"></Keywords>
+            <Keywords name="Keywords2">=</Keywords>
+            <Keywords name="Keywords3">@</Keywords>
+            <Keywords name="Keywords4"></Keywords>
+            <Keywords name="Keywords5"></Keywords>
+            <Keywords name="Keywords6"></Keywords>
+            <Keywords name="Keywords7"></Keywords>
+            <Keywords name="Keywords8"></Keywords>
+            <Keywords name="Delimiters">00` 01\ 02` 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23</Keywords>
+        </KeywordLists>
+        <Styles>
+            <WordsStyle name="DEFAULT" fgColor="000000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
+            <WordsStyle name="COMMENTS" fgColor="FF00FF" bgColor="FFFFFF" fontStyle="7" nesting="0" />
+            <WordsStyle name="LINE COMMENTS" fgColor="FF00FF" bgColor="FFFFFF" fontStyle="7" nesting="0" />
+            <WordsStyle name="NUMBERS" fgColor="E80000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
+            <WordsStyle name="KEYWORDS1" fgColor="000000" bgColor="FFFFFF" fontStyle="1" nesting="0" />
+            <WordsStyle name="KEYWORDS2" fgColor="0000FF" bgColor="FFFFFF" fontStyle="3" nesting="0" />
+            <WordsStyle name="KEYWORDS3" fgColor="F47A00" bgColor="FFFFFF" fontStyle="1" nesting="0" />
+            <WordsStyle name="KEYWORDS4" fgColor="000000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
+            <WordsStyle name="KEYWORDS5" fgColor="000000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
+            <WordsStyle name="KEYWORDS6" fgColor="000000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
+            <WordsStyle name="KEYWORDS7" fgColor="000000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
+            <WordsStyle name="KEYWORDS8" fgColor="000000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
+            <WordsStyle name="OPERATORS" fgColor="000000" bgColor="FFFFFF" fontStyle="1" nesting="0" />
+            <WordsStyle name="FOLDER IN CODE1" fgColor="000000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
+            <WordsStyle name="FOLDER IN CODE2" fgColor="000000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
+            <WordsStyle name="FOLDER IN COMMENT" fgColor="000000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
+            <WordsStyle name="DELIMITERS1" fgColor="999900" bgColor="FFFFFF" fontStyle="0" nesting="0" />
+            <WordsStyle name="DELIMITERS2" fgColor="000000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
+            <WordsStyle name="DELIMITERS3" fgColor="000000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
+            <WordsStyle name="DELIMITERS4" fgColor="000000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
+            <WordsStyle name="DELIMITERS5" fgColor="000000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
+            <WordsStyle name="DELIMITERS6" fgColor="000000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
+            <WordsStyle name="DELIMITERS7" fgColor="000000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
+            <WordsStyle name="DELIMITERS8" fgColor="000000" bgColor="FFFFFF" fontStyle="0" nesting="0" />
+        </Styles>
+    </UserLang>
+</NotepadPlus>
diff --git a/docs/RemesPath.md b/docs/RemesPath.md
@@ -103,13 +103,15 @@ In general, binary operators *should* raise an exception when two objects of une
 
 Starting in [v5.4.0](/CHANGELOG.md#540---2023-07-04), all arithmetic operations can accept a boolean as one or both of the arguments. For example, prior to 5.4.0, `true * 3 - (false / 2.5)` was a type error, but since then it is valid. 
 
-*Beginning in [v5.1.0](/CHANGELOG.md#510---2023-06-02), the `*` operator in supports multiplication of strings by integers (but not integers by strings). For example, `["a", "b", "c"] * [1,2,3]` will return `["a", "bb", "ccc"]`. Starting in [5.4.0](/CHANGELOG.md), multiplication of a string by a boolean or a negative integer is valid.
+Beginning in [v5.1.0](/CHANGELOG.md#510---2023-06-02), the `*` operator in supports multiplication of strings by integers (but not integers by strings). For example, `["a", "b", "c"] * [1,2,3]` will return `["a", "bb", "ccc"]`. Starting in [5.4.0](/CHANGELOG.md), multiplication of a string by a boolean or a negative integer is valid.
 
 If you find that a binary operator can operate on a number and a non-number without raising an exception, *this is a bug in my implementation.*
 
 ### Unary operators ###
 
-As in normal math, the unary minus operator (e.g., `-5`) has lower precedence than exponentiation. Starting in [5.4.0](/CHANGELOG.md#540---2023-07-04), the unary `+` operator has the same precedence as the unary minus operator. Unary `+` is a no-op on floats and ints, but it converts `true` and `false` to `1` and `0` respectively.
+As in normal math, the unary minus operator (e.g., `-5`) has lower precedence than exponentiation and higher precedence than everything else.
+
+Starting in [5.4.0](/CHANGELOG.md#540---2023-07-04), the unary `+` operator has the same precedence as the unary minus operator. Unary `+` is a no-op on floats and ints, but it converts `true` and `false` to `1` and `0` respectively.
 
 The `not` operator introduced in [5.4.0](/CHANGELOG.md#540---2023-07-04) (which replaced [the older function of the same name](#vectorized-functions)) is very similar to the Python operator of the same name:
 * `not true = false`, `not false = true`
@@ -1183,7 +1185,7 @@ If the RHS is a function and the LHS is an iterable, the RHS is applied separate
 ### Limitations ###
 1. Until further notice, you __cannot__ mutate an object or array, other than to change its scalar elements
     * For example, the query `@ = len(@)` on JSON `[[1, 2, 3]]` will fail, because this ends up trying to mutate the subarray `[1, 2, 3]`.
-2. You also cannot mutate a non-array or non-object into an array or object. For example, the query ``@[0] = j`[1]` ``on the input `[0]` will fail because you're trying to convert a scalar (the integer `1`) to an array (`[1]`).
+2. You also cannot mutate a non-array or non-object into an array or object. For example, the query ``@[0] = j`[1]` ``on the input `[0]` will fail because you're trying to convert a scalar (the integer `0`) to an array (`[1]`).
 
 An assignment expression mutates the input and then returns the input.