Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/new filter #1068

Merged
merged 57 commits into from
Jan 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
7ebad1a
created new classes for new filter-method
kloss-o Nov 10, 2021
2ff6774
created new classes for new filter-method
kloss-o Nov 16, 2021
cc9f648
added new filters. Ready for testing
kloss-o Nov 22, 2021
ac68e8d
finished a few simple tests for new filter-objects
kloss-o Nov 22, 2021
7b9210c
added more tests
Nov 30, 2021
e464fbc
old filter-functionaltiy added again + tests
Dec 7, 2021
1d9bd8b
added doc
Dec 14, 2021
5587582
added doctrings to sphinx documentation
Feb 1, 2022
03c4253
renamed all FilterConditions to match python standards
Feb 4, 2022
06d9b55
pep8 standards
Feb 4, 2022
fd34add
fixed remaining pep8 issues
Feb 8, 2022
3692618
shortening lines
Feb 8, 2022
2d1af00
Update neo/core/container.py
kloss-o Feb 10, 2022
62558c8
Update neo/core/container.py
kloss-o Feb 10, 2022
8b7bd40
Update neo/core/container.py
kloss-o Feb 10, 2022
528514c
Update neo/test/coretest/test_container.py
kloss-o Feb 17, 2022
4e98709
changed a few things based on feedback on pull request
Feb 17, 2022
75acb73
fixed last pep8speaks issue
Feb 17, 2022
417fcad
moved filters to separate module and renamed them
Jun 14, 2022
f2f560c
added changes from branch enh/fastFilter
Jun 14, 2022
fcc8836
Merge remote-tracking branch 'origin/master' into feature/newFilter
Jun 14, 2022
8b53e11
fixed filtering error
Jun 28, 2022
e11a7d6
review
Jul 19, 2022
e21e447
refactor names for limits
Jul 19, 2022
ff6f631
remove brackets
Jul 19, 2022
98e0a3d
remove __init__
Jul 19, 2022
0d734d0
add module description
Jul 19, 2022
ac957b2
re-add __init__ for subclasses
Jul 21, 2022
270f8a8
undo renaming for InRange FilterCondition
Jul 21, 2022
7856030
re-ad __init__ to Equal FilterCondition
Jul 21, 2022
80606b9
add comment to imports in __init__ to enable review
Jul 21, 2022
b3800ae
Merge pull request #42 from INM-6/review_new_filter
Moritz-Alexander-Kern Jul 21, 2023
17666ac
Merge branch 'master' into feature/newFilter
Jul 21, 2023
bf1c51d
do not import filters individually into top level namespace
Jul 28, 2023
4e1df3b
revert to previous version of docstring of docstring for filterdata
Jul 28, 2023
f9e1d29
remove whitespace changes
Jul 28, 2023
21f829a
changed class names for GreaterThanEqual, LessThanEqual and Equal
Jul 28, 2023
e46e9ed
adapt unittests according to new class names
Jul 28, 2023
f183572
eliminate duplicates with a dict comprehension, using id() as the key…
Jul 28, 2023
f50f222
remove else
Jul 28, 2023
dd9d2f0
replace list with generator before returning tuple
Jul 28, 2023
56c2caf
use isinstance to check for type
Jul 28, 2023
f8a56ea
change names, add docstrings to classes
Jul 28, 2023
54373b9
use abstract base class from the abc module for the FilterCondition c…
Jul 28, 2023
4616987
fix typo in docstring
Jul 28, 2023
812ed39
Merge branch 'NeuralEnsemble:master' into feature/newFilter
Moritz-Alexander-Kern Jul 28, 2023
afdc1f5
undo returns for tuples
Jul 28, 2023
d124f1b
add module docstring for filters.py
Jul 28, 2023
d50cb3b
fix doctrings and class names
Jul 31, 2023
ca82dcd
refactor unit tests
Jul 31, 2023
70a425e
add tuples and sets to IsIn class
Jul 31, 2023
dd5ec83
simplify evaluate in IsIn
Jul 31, 2023
2407353
further simplification of unit tests
Jul 31, 2023
fd32d6a
add typehints
Jul 31, 2023
d655ac6
add docstring to unit test
Jul 31, 2023
9c30bde
refactor naming for x to compare and z to control
Jul 31, 2023
dedfc1b
fix typo and refactor filterdata
Jul 31, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/authors.rst
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ and may not be the current affiliation of a contributor.
* Etienne Combrisson [6]
* Ben Dichter [24]
* Elodie Legouée [21]
* Oliver Kloss [13]
* Heberto Mayorquin [24]
* Thomas Perret [25]
* Kyle Johnsen [26, 27]
Expand Down
5 changes: 5 additions & 0 deletions neo/core/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@
Classes:

.. autoclass:: Block
.. automethod:: Block.filter
.. autoclass:: Segment
.. automethod:: Segment.filter
.. autoclass:: Group

.. autoclass:: AnalogSignal
Expand All @@ -35,6 +37,9 @@
from neo.core.analogsignal import AnalogSignal
from neo.core.irregularlysampledsignal import IrregularlySampledSignal

# Import FilterClasses
from neo.core import filters

from neo.core.event import Event
from neo.core.epoch import Epoch

Expand Down
61 changes: 40 additions & 21 deletions neo/core/container.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
"""

from copy import deepcopy

from neo.core import filters
from neo.core.baseneo import BaseNeo, _reference_name, _container_name
from neo.core.objectlist import ObjectList
from neo.core.spiketrain import SpikeTrain
Expand All @@ -21,24 +23,25 @@ def unique_objs(objs):
return [obj for obj in objs
if id(obj) not in seen and not seen.add(id(obj))]


def filterdata(data, targdict=None, objects=None, **kwargs):
"""
Return a list of the objects in data matching *any* of the search terms
in either their attributes or annotations. Search terms can be
provided as keyword arguments or a dictionary, either as a positional
argument after data or to the argument targdict. targdict can also
be a list of dictionaries, in which case the filters are applied
sequentially. If targdict and kwargs are both supplied, the
targdict filters are applied first, followed by the kwarg filters.
A targdict of None or {} and objects = None corresponds to no filters
applied, therefore returning all child objects.
Default targdict and objects is None.
argument after data or to the argument targdict.
A key of a provided dictionary is the name of the requested annotation
and the value is a FilterCondition object.
E.g.: Equal(x), LessThan(x), InRange(x, y).

targdict can also
be a list of dictionaries, in which case the filters are applied
sequentially.

objects (optional) should be the name of a Neo object type,
a neo object class, or a list of one or both of these. If specified,
only these objects will be returned.
A list of dictionaries is handled as follows: [ { or } and { or } ]
If targdict and kwargs are both supplied, the
targdict filters are applied first, followed by the kwarg filters.
A targdict of None or {} corresponds to no filters applied, therefore
returning all child objects. Default targdict is None.
"""

# if objects are specified, get the classes
Expand Down Expand Up @@ -72,20 +75,26 @@ def filterdata(data, targdict=None, objects=None, **kwargs):
else:
# do the actual filtering
results = []
for key, value in sorted(targdict.items()):
for obj in data:
if (hasattr(obj, key) and getattr(obj, key) == value and
all([obj is not res for res in results])):
for obj in data:
for key, value in sorted(targdict.items()):
if hasattr(obj, key) and getattr(obj, key) == value:
results.append(obj)
elif (key in obj.annotations and obj.annotations[key] == value and
all([obj is not res for res in results])):
break
if isinstance(value, filters.FilterCondition) and key in obj.annotations:
if value.evaluate(obj.annotations[key]):
results.append(obj)
break
if key in obj.annotations and obj.annotations[key] == value:
results.append(obj)
break

# remove duplicates from results
results = list({ id(res): res for res in results }.values())

# keep only objects of the correct classes
if objects:
results = [result for result in results if
result.__class__ in objects or
result.__class__.__name__ in objects]
result.__class__ in objects or result.__class__.__name__ in objects]

if results and all(isinstance(obj, SpikeTrain) for obj in results):
return SpikeTrainList(results)
Expand Down Expand Up @@ -366,9 +375,17 @@ def filter(self, targdict=None, data=True, container=False, recursive=True,
Return a list of child objects matching *any* of the search terms
in either their attributes or annotations. Search terms can be
provided as keyword arguments or a dictionary, either as a positional
argument after data or to the argument targdict. targdict can also
argument after data or to the argument targdict.
A key of a provided dictionary is the name of the requested annotation
and the value is a FilterCondition object.
E.g.: equal(x), less_than(x), InRange(x, y).

targdict can also
be a list of dictionaries, in which case the filters are applied
sequentially. If targdict and kwargs are both supplied, the
sequentially.

A list of dictionaries is handled as follows: [ { or } and { or } ]
If targdict and kwargs are both supplied, the
targdict filters are applied first, followed by the kwarg filters.
A targdict of None or {} corresponds to no filters applied, therefore
returning all child objects. Default targdict is None.
Expand All @@ -391,6 +408,8 @@ def filter(self, targdict=None, data=True, container=False, recursive=True,
>>> obj.filter(name="Vm")
>>> obj.filter(objects=neo.SpikeTrain)
>>> obj.filter(targdict={'myannotation':3})
>>> obj.filter(name=neo.core.filters.Equal(5))
>>> obj.filter({'name': neo.core.filters.LessThan(5)})
"""

if isinstance(targdict, str):
Expand Down
173 changes: 173 additions & 0 deletions neo/core/filters.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
"""
This module implements :class:`FilterCondition`, which enables use of different filter conditions
for neo.core.container.filter.

Classes:
- :class:`FilterCondition`: Abstract base class for defining filter conditions.
- :class:`Equals`: Filter condition to check if a value is equal to the control value.
- :class:`IsNot`: Filter condition to check if a value is not equal to the control value.
- :class:`LessThanOrEquals`: Filter condition to check if a value is less than or equal to the
control value.
- :class:`GreaterThanOrEquals`: Filter condition to check if a value is greater than or equal to
the control value.
- :class:`LessThan`: Filter condition to check if a value is less than the control value.
- :class:`GreaterThan`: Filter condition to check if a value is greater than the control value.
- :class:`IsIn`: Filter condition to check if a value is in a list or equal to the control
value.
- :class:`InRange`: Filter condition to check if a value is in a specified range.

The provided classes allow users to select filter conditions and use them with
:func:`neo.core.container.filter()` to perform specific filtering operations on data.
"""
from abc import ABC, abstractmethod
from numbers import Number
from typing import Union, Any


class FilterCondition(ABC):
"""
FilterCondition object is given as parameter to container.filter():

Usage:
segment.filter(my_annotation=<FilterCondition>) or
segment=filter({'my_annotation': <FilterCondition>})
"""
@abstractmethod
def __init__(self, control: Any) -> None:
"""
Initialize new FilterCondition object.

Parameters:
control: Any - The control value to be used for filtering.

This is an abstract base class and should not be instantiated directly.
"""

@abstractmethod
def evaluate(self, compare: Any) -> bool:
"""
Evaluate the filter condition for given value.

Parameters:
compare: Any - The value to be compared with the control value.

Returns:
bool: True if the condition is satisfied, False otherwise.

This method should be implemented in subclasses.
"""


class Equals(FilterCondition):
"""
Filter condition to check if target value is equal to the control value.
"""
def __init__(self, control: Any) -> None:
self.control = control

def evaluate(self, compare: Any) -> bool:
return compare == self.control


class IsNot(FilterCondition):
"""
Filter condition to check if target value is not equal to the control value.
"""
def __init__(self, control: Any) -> None:
self.control = control

def evaluate(self, compare: Any) -> bool:
return compare != self.control


class LessThanOrEquals(FilterCondition):
"""
Filter condition to check if target value is less than or equal to the control value.
"""
def __init__(self, control: Number) -> None:
self.control = control

def evaluate(self, compare: Number) -> bool:
return compare <= self.control


class GreaterThanOrEquals(FilterCondition):
"""
Filter condition to check if target value is greater than or equal to the control value.
"""
def __init__(self, control: Number) -> None:
self.control = control

def evaluate(self, compare: Number) -> bool:
return compare >= self.control


class LessThan(FilterCondition):
"""
Filter condition to check if target value is less than the control value.
"""
def __init__(self, control: Number) -> None:
self.control = control

def evaluate(self, compare: Number) -> bool:
return compare < self.control


class GreaterThan(FilterCondition):
"""
Filter condition to check if target value is greater than the control value.
"""
def __init__(self, control: Number) -> None:
self.control = control

def evaluate(self, compare: Number) -> bool:
return compare > self.control


class IsIn(FilterCondition):
"""
Filter condition to check if target is in control.
"""
def __init__(self, control: Union[list, tuple, set, int]) -> None:
self.control = control

def evaluate(self, compare: Any) -> bool:
if isinstance(self.control, (list, tuple, set)):
return compare in self.control
if isinstance(self.control, int):
return compare == self.control

raise SyntaxError('parameter not of type list, tuple, set or int')


class InRange(FilterCondition):
"""
Filter condition to check if a value is in a specified range.

Usage:
InRange(upper_bound, upper_bound, left_closed=False, right_closed=False)

Parameters:
lower_bound: int - The lower bound of the range.
upper_bound: int - The upper bound of the range.
left_closed: bool - If True, the range includes the lower bound (lower_bound <= compare).
right_closed: bool - If True, the range includes the upper bound (compare <= upper_bound).
"""
def __init__(self, lower_bound: Number, upper_bound: Number,
left_closed: bool=False, right_closed: bool=False) -> None:
if not isinstance(lower_bound, Number) or not isinstance(upper_bound, Number):
raise ValueError("parameter is not a number")

self.lower_bound = lower_bound
self.upper_bound = upper_bound
self.left_closed = left_closed
self.right_closed = right_closed

def evaluate(self, compare: Number) -> bool:
if not self.left_closed and not self.right_closed:
return self.lower_bound <= compare <= self.upper_bound
if not self.left_closed and self.right_closed:
return self.lower_bound <= compare < self.upper_bound
if self.left_closed and not self.right_closed:
return self.lower_bound < compare <= self.upper_bound
return self.lower_bound < compare < self.upper_bound
Loading