Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactoring of the main classes #10

Merged
merged 188 commits into from
Jul 26, 2024
Merged

Refactoring of the main classes #10

merged 188 commits into from
Jul 26, 2024

Conversation

lauraporta
Copy link
Member

@lauraporta lauraporta commented Jul 18, 2024

Description

What is this PR

  • Bug fix
  • Addition of a new feature
  • Other

Why is this PR needed?
Refactor the main analysis code.

What does this PR do?
Legend:
❌: removes
💭: changes
➕: adds

  1. ❌ Removes a lot of modules that were useful in the exploration stages;
  2. ❌ Removes contrast enhancement (see: Contrast enhancement and registration #11 ) from the pipeline and related methods;
  3. ❌ Removes "adjust increment" (1) boolean as I think I should not give the user this flexibility, increments should be always corrected to achieve a full 360 rotation, as this improves the overall derotation;
  4. 💭 Renames a few methods;
  5. 💭 Replaces np.nonzero with np.where (just personal preference, they seem equivalent for my use-case);
  6. 💭 Changes handling of number of rotations. Now it is estimated from the rotation signal;
  7. ➕ Adds automated creation of folders for saving purposes;
  8. ➕ Adds a few comments to separate the methods in groups (signal processing, derotation, saving);
  9. 💭 Improve usage of pandas syntax
  10. ➕ Overwrites the method check_rotation_number_after_interpolation in the IncrementalPipeline class

I can provide some datasets to test the pipeline locally.

Explanations

(1) : with "increments" I refer to the angle resolution of the motor, i.e. the smallest increment in angles that motor is able to report. 0.2 deg in our case.

How has this PR been tested?

Tested locally with tests that were made before.

Is this a breaking change?

No

Does this PR require an update to the documentation?

Not for now.

Checklist:

  • The code has been tested locally
  • Tests have been added to cover all new functionality
  • The documentation has been updated to reflect any changes
  • The code has been formatted with pre-commit

@lauraporta lauraporta linked an issue Jul 19, 2024 that may be closed by this pull request
@lauraporta lauraporta marked this pull request as ready for review July 19, 2024 13:05
@JoeZiminski
Copy link
Member

JoeZiminski commented Jul 25, 2024

(derotation-ev) C:\Users\Joe\work\git-repos\derotation>python examples/derotate.py
2024-07-25 14:42:50 PM INFO     2024-07-25 14:42:50 PM - INFO - MainProcess fancylog.py:326 - Starting logging                                                                                                          fancylog.py:326
                       INFO     2024-07-25 14:42:50 PM - INFO - MainProcess fancylog.py:327 - Not logging multiple processes                                                                                            fancylog.py:327
                       INFO     2024-07-25 14:42:50 PM - INFO - MainProcess full_rotation_pipeline.py:122 - Loading data...                                                                               full_rotation_pipeline.py:122
(derotation-ev) C:\Users\Joe\work\git-repos\derotation>python examples/derotate.py
2024-07-25 14:42:58 PM INFO     2024-07-25 14:42:58 PM - INFO - MainProcess fancylog.py:326 - Starting logging                                                                                                          fancylog.py:326
                       INFO     2024-07-25 14:42:58 PM - INFO - MainProcess fancylog.py:327 - Not logging multiple processes                                                                                            fancylog.py:327
                       INFO     2024-07-25 14:42:58 PM - INFO - MainProcess full_rotation_pipeline.py:122 - Loading data...                                                                               full_rotation_pipeline.py:122
2024-07-25 14:42:59 PM WARNING  2024-07-25 14:42:59 PM - WARNING - MainProcess tifffile.py:4539 - <tifffile.TiffFile 'rotation_00001.tif'> <asarray> failed to reshape (8258, 256, 256) to (8500, 256, 256), raised    tifffile.py:4539
                                ValueError('cannot reshape array of size 541196288 into shape (8500,256,256)')
Traceback (most recent call last):
  File "C:\Users\Joe\work\git-repos\derotation\examples\derotate.py", line 3, in <module>
    derotate = FullPipeline("full_rotation")
  File "C:\Users\Joe\work\git-repos\derotation\derotation\analysis\full_rotation_pipeline.py", line 46, in __init__
    self.load_data()
  File "C:\Users\Joe\work\git-repos\derotation\derotation\analysis\full_rotation_pipeline.py", line 144, in load_data
    ) = get_analog_signals(
  File "C:\Users\Joe\work\git-repos\derotation\derotation\load_data\custom_data_loaders.py", line 72, in get_analog_signals
    data = convert_to_volts(data)
  File "C:\Users\Joe\work\git-repos\derotation\derotation\load_data\custom_data_loaders.py", line 49, in convert_to_volts
    data = -10 + 20 * (data + 2**15) / 2**16
OverflowError: Python integer 32768 out of bounds for int16
Config file
paths_read:
  path_to_randperm: "C:/Users/Joe/work/git-repos/derotation/joe-example-data/example_derotation_data/stimlus_randperm.mat"
  path_to_aux: "C:/Users/Joe/work/git-repos/derotation/joe-example-data/example_derotation_data/aux_stim/230802_CAA_1120182_rotation_1_001.bin"
  path_to_tif: "C:/Users/Joe/work/git-repos/derotation/joe-example-data/example_derotation_data/imaging/rotation_00001.tif"
paths_write:
  debug_plots_folder: "C:/Users/Joe/work/git-repos/derotation/joe-example-data/output/debug_plots/"
  logs_folder: "C:/Users/Joe/work/git-repos/derotation/joe-example-data/output/logs/"
  derotated_tiff_folder: "C:/Users/Joe/work/git-repos/derotation/joe-example-data/output/data_folder/"
  saving_name: "derotated_image_stack"


channel_names: [
    "camera",
    "scanimage_frameclock",
    "scanimage_lineclock",
    "photodiode2",
    "PI_rotCW",
    "PI_rotCCW",
]

# camera,scanimage_frameclock,scanimage_lineclock,photodiode2,PI_rotCW,PI_rotCCW
rotation_increment: 0.2
adjust_increment: True
rot_deg: 360

debugging_plots: True

contrast_enhancement: 0.35

analog_signals_processing:
  find_rotation_ticks_peaks:
    height: 4
    distance: 20
  squared_pulse_k: 0
  inter_rotation_interval_min_len: 1000
  angle_interpolation_artifact_threshold: 0.15

interpolation:
  line_use_start: True
  frame_use_start: True

@lauraporta
Copy link
Member Author

Thank you Joe, it seems that this issue might be related to a newer version of numpy. I just upgraded it and found the same exception.

@JoeZiminski
Copy link
Member

Otherwise was working okay on numpy 1.23.1, will try on 2.0 after the fix. I had a couple of warnings I will dump here in case they are relevant:

2024-07-25 16:18:18 PM INFO     2024-07-25 16:18:18 PM - INFO - MainProcess fancylog.py:326 - Starting logging                                                                                                          fancylog.py:326
                       INFO     2024-07-25 16:18:18 PM - INFO - MainProcess fancylog.py:327 - Not logging multiple processes                                                                                            fancylog.py:327
                       INFO     2024-07-25 16:18:18 PM - INFO - MainProcess full_rotation_pipeline.py:121 - Loading data...                                                                               full_rotation_pipeline.py:121
                       WARNING  2024-07-25 16:18:18 PM - WARNING - MainProcess tifffile.py:4539 - <tifffile.TiffFile 'rotation_00001.tif'> <asarray> failed to reshape (8258, 256, 256) to (8500, 256, 256), raised    tifffile.py:4539
                                ValueError('cannot reshape array of size 541196288 into shape (8500,256,256)')
2024-07-25 16:18:20 PM INFO     2024-07-25 16:18:20 PM - INFO - MainProcess full_rotation_pipeline.py:170 - Dataset rotation_00001 loaded                                                                 full_rotation_pipeline.py:170
                       INFO     2024-07-25 16:18:20 PM - INFO - MainProcess full_rotation_pipeline.py:171 - Filename: derotated_image_stack                                                               full_rotation_pipeline.py:171
                       INFO     2024-07-25 16:18:20 PM - INFO - MainProcess full_rotation_pipeline.py:249 - Finding rotation ticks peaks...                                                               full_rotation_pipeline.py:249
2024-07-25 16:18:22 PM INFO     2024-07-25 16:18:22 PM - INFO - MainProcess full_rotation_pipeline.py:324 - Cleaning start and end rotation signal...                                                     full_rotation_pipeline.py:324
                       INFO     2024-07-25 16:18:22 PM - INFO - MainProcess full_rotation_pipeline.py:349 - Creating signed rotation array...                                                             full_rotation_pipeline.py:349
                       INFO     2024-07-25 16:18:22 PM - INFO - MainProcess full_rotation_pipeline.py:371 - Dropping ticks outside of the rotation period...                                              full_rotation_pipeline.py:371
2024-07-25 16:18:25 PM INFO     2024-07-25 16:18:25 PM - INFO - MainProcess full_rotation_pipeline.py:407 - Ticks dropped: 8.                                                                             full_rotation_pipeline.py:407
                                Ticks remaining: 137958
                       INFO     2024-07-25 16:18:25 PM - INFO - MainProcess full_rotation_pipeline.py:436 - Number of rotations is as expected                                                            full_rotation_pipeline.py:436
                       WARNING  2024-07-25 16:18:25 PM - WARNING - MainProcess full_rotation_pipeline.py:457 - Number of ticks is not as expected: 137958.                                                full_rotation_pipeline.py:457
                                Expected ticks: 144000.0
                                Delta: -6042.0
                       INFO     2024-07-25 16:18:25 PM - INFO - MainProcess full_rotation_pipeline.py:509 - New increment example: 0.209                                                                  full_rotation_pipeline.py:509
                       INFO     2024-07-25 16:18:25 PM - INFO - MainProcess full_rotation_pipeline.py:522 - Interpolating angles...                                                                       full_rotation_pipeline.py:522
2024-07-25 16:18:27 PM INFO     2024-07-25 16:18:27 PM - INFO - MainProcess full_rotation_pipeline.py:562 - Cleaning interpolated angles...                                                               full_rotation_pipeline.py:562
2024-07-25 16:18:29 PM INFO     2024-07-25 16:18:29 PM - INFO - MainProcess full_rotation_pipeline.py:616 - Calculating angles by line and frame...                                                       full_rotation_pipeline.py:616
                       INFO     2024-07-25 16:18:29 PM - INFO - MainProcess full_rotation_pipeline.py:688 - Plotting rotation ticks and rotation on signal...                                             full_rotation_pipeline.py:688
2024-07-25 16:18:34 PM INFO     2024-07-25 16:18:34 PM - INFO - MainProcess full_rotation_pipeline.py:732 - Plotting rotation angles...                                                                   full_rotation_pipeline.py:732
2024-07-25 16:18:35 PM INFO     2024-07-25 16:18:35 PM - INFO - MainProcess full_rotation_pipeline.py:236 - ✨ Analog signals processed ✨                                                                full_rotation_pipeline.py:236  
                       INFO     2024-07-25 16:18:35 PM - INFO - MainProcess full_rotation_pipeline.py:822 - Starting derotation by line...                                                                full_rotation_pipeline.py:822
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2114195/2114195 [10:20<00:00, 3408.86it/s] 
2024-07-25 16:28:55 PM INFO     2024-07-25 16:28:55 PM - INFO - MainProcess full_rotation_pipeline.py:832 - ✨ Image stack rotated ✨                                                                     full_rotation_pipeline.py:832
2024-07-25 16:28:57 PM INFO     2024-07-25 16:28:57 PM - INFO - MainProcess full_rotation_pipeline.py:941 - Masked image saved in                                                                         full_rotation_pipeline.py:941
                                C:/Users/Joe/work/git-repos/derotation/joe-example-data/output/data_folder/

Copy link
Member

@JoeZiminski JoeZiminski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @lauraporta all looks good to me, its very clear and easy to follow. I made a couple of suggestions but they are not critical so feel free to ignore!

derotation/analysis/full_rotation_pipeline.py Show resolved Hide resolved
derotation/analysis/full_rotation_pipeline.py Show resolved Hide resolved
for i, (start, end) in enumerate(
zip(rotation_start[1:], rotation_end[:-1])
):
for start, end in zip(rotation_start[1:], rotation_end[:-1]):
self.interpolated_angles[end:start] = 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just checking this is end:start and not start:end, its like the end of the first rotation to the start of the second?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that's what it is 😅 do you find it confusing?

Copy link
Member

@JoeZiminski JoeZiminski Jul 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think its so confusing and makes sense in the context, but for a reader not particularly familiar with the procedure like myself it might be initially a little unexpected (i.e. just seeing an array indexed end:start). Maybe just a brief clarifying comment above the line? or maybe the variables could be last_rotation_end, next_rotation_start if not too verbose.

derotation/analysis/full_rotation_pipeline.py Show resolved Hide resolved
derotation/analysis/full_rotation_pipeline.py Show resolved Hide resolved
derotation/analysis/full_rotation_pipeline.py Show resolved Hide resolved
derotation/analysis/full_rotation_pipeline.py Show resolved Hide resolved
derotation/analysis/full_rotation_pipeline.py Show resolved Hide resolved
@lauraporta lauraporta merged commit b01e5fd into main Jul 26, 2024
11 checks passed
@lauraporta lauraporta deleted the feature/refactoring branch July 26, 2024 10:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

General refactoring of the classes
3 participants