No simple and uniform way to visualize orientation and retardance-orientation overlays

[talonchandler]

@ziw-liu @mattersoflight and I discussed a few possible direction for recOrder visualizations:

Current behavior: reconstructions can be dragged and dropped into napari (thanks iohub!) , but there isn't an easy way to color the orientation channel or generate overlays.

Possible solutions:

1. Use a custom reader plugin - this won't allow us to generate overlays, so it won't work by itself.
2. Write RGB overlays alongside the reconstructions and use a custom reader plugin - this meets our requirements, but requires a lot of unnecessary data to be stored.
3. Use a CLI like recOrder view to generate RGB overlays in a napari window - this will work, and it could be unified with the recOrder GUI, but it requires a user to use the CLI...no drag and drop option.
4. @ziw-liu's latest idea - Use the napari event loop to generate an overlay whenever layers named "Retardance" and "Orientation" are added to the viewer with the recOrder-napari plugin running. This solution allows drag-and-drop visualization, and it seems intuitive to me that this will only work when you have the recOrder plugin open. This approach unifies our visualizations, doesn't bump our storage requirements, works for arbitrarily shaped arrays, and is a reasonably non-invasive change to make.

I really like option 4. A potential drawback is that it seems a bit "magic" (where did that overlay come from and why is it happening now?), but we might alleviate this by printing messages to the user.

[ziw-liu]

1. Use a custom reader plugin - this won't allow us to generate overlays, so it won't work by itself.

Minor clarification: This does allow for generating overlays. Just that it cannot do so lazily for very large datasets.

Edit: with some Dask delayed compute it might also work lazily, but not guaranteed.

[talonchandler]

After discussing we decided to start with option 4, and we'll explore some combination of 2 & 3 when we have a better sense of the workflows they enable.

[mattersoflight]

@ziw-liu Option 4 is compatible with #329. We can add fields to control the overlay to a tab of the recOrder plugin (the tab where we show color overlays).

[ziw-liu]

We can add fields to control the overlay to a tab of the recOrder plugin (the tab where we show color overlays).

There's actually existing code to do similar things in that tab. We hid it because it was buggy:

recOrder/recOrder/plugin/main_widget.py

Lines 2176 to 2293 in 802cc62

	# Commenting for 0.3.0. Consider debugging or deleting for 1.0.0.
	# @Slot(int)
	# def update_sat_scale(self):
	# idx = self.ui.cb_saturation.currentIndex()
	# if idx != -1:
	# layer = self.ui.cb_saturation.itemText(idx)
	# data = self.viewer.layers[layer].data
	# min_, max_ = np.min(data), np.max(data)
	# self.ui.slider_saturation.setMinimum(min_)
	# self.ui.slider_saturation.setMaximum(max_)
	# self.ui.slider_saturation.setSingleStep((max_ - min_) / 250)
	# self.ui.slider_saturation.setValue((min_, max_))
	# self.ui.le_sat_max.setText(str(np.round(max_, 3)))
	# self.ui.le_sat_min.setText(str(np.round(min_, 3)))
	# @Slot(int)
	# def update_value_scale(self):
	# idx = self.ui.cb_value.currentIndex()
	# if idx != -1:
	# layer = self.ui.cb_value.itemText(idx)
	# data = self.viewer.layers[layer].data
	# min_, max_ = np.min(data), np.max(data)
	# self.ui.slider_value.setMinimum(min_)
	# self.ui.slider_value.setMaximum(max_)
	# self.ui.slider_value.setSingleStep((max_ - min_) / 250)
	# self.ui.slider_value.setValue((min_, max_))
	# self.ui.le_val_max.setText(str(np.round(max_, 3)))
	# self.ui.le_val_min.setText(str(np.round(min_, 3)))
	# @Slot(bool)
	# def create_overlay(self):
	# """
	# Creates HSV or JCh overlay with the specified channels from the combo boxes. Will compute and then
	# display the overlay in napari.
	# Returns
	# -------
	# """
	# if (
	# self.ui.cb_hue.count() == 0
	# or self.ui.cb_saturation.count() == 0
	# or self.ui.cb_value == 0
	# ):
	# raise ValueError(
	# "Cannot create overlay until all 3 combo boxes are populated"
	# )
	# # Gather channel data
	# H = self.viewer.layers[
	# self.ui.cb_hue.itemText(self.ui.cb_hue.currentIndex())
	# ].data
	# S = self.viewer.layers[
	# self.ui.cb_saturation.itemText(
	# self.ui.cb_saturation.currentIndex()
	# )
	# ].data
	# V = self.viewer.layers[
	# self.ui.cb_value.itemText(self.ui.cb_value.currentIndex())
	# ].data
	# # TODO: this is a temp fix which handles on data with n-dimensions of 4, 3, or 2 which automatically
	# # chooses the first timepoint
	# if H.ndim > 2 or S.ndim > 2 or V.ndim > 2:
	# if H.ndim == 4:
	# # assumes this is a (T, Z, Y, X) array read from napari-ome-zarr
	# H = (
	# H[0, self.display_slice]
	# if not self.use_full_volume
	# else H[0]
	# )
	# if S.ndim == 4:
	# S = (
	# S[0, self.display_slice]
	# if not self.use_full_volume
	# else S[0]
	# )
	# if V.ndim == 4:
	# V = (
	# V[0, self.display_slice]
	# if not self.use_full_volume
	# else V[0]
	# )
	# if H.ndim == 3:
	# # assumes this is a (Z, Y, X) array collected from acquisition module
	# H = H[self.display_slice] if not self.use_full_volume else H
	# if S.ndim == 3:
	# S = S[self.display_slice] if not self.use_full_volume else S
	# if S.ndim == 3:
	# S = S[self.display_slice] if not self.use_full_volume else S
	# mode = "2D" if not self.use_full_volume else "3D"
	# H_name = self.ui.cb_hue.itemText(self.ui.cb_hue.currentIndex())
	# H_scale = (
	# (np.min(H), np.max(H))
	# if "Orientation" not in H_name
	# else (0, np.pi)
	# )
	# S_scale = self.ui.slider_saturation.value()
	# V_scale = self.ui.slider_value.value()
	# hsv_image = generic_hsv_overlay(
	# H, S, V, H_scale, S_scale, V_scale, mode=mode
	# )
	# # Create overlay layer name
	# idx = 0
	# while f"HSV_Overlay_{idx}" in self.viewer.layers:
	# idx += 1
	# # add overlay image to napari
	# self.viewer.add_image(hsv_image, name=f"HSV_Overlay_{idx}", rgb=True)

Edit:
After some thought, options 1 and 4 will probably be very similar in terms of the implementation for large datasets. The main difference is that the later allows for interactive control of the rendering.

[ziw-liu]

@talonchandler @edyoshikun @mattersoflight
Can you remind me if we made any decision wrt the viewer's behavior for multi-position datasets? It brings up the complexity quite a bit if it needs to handle delayed colormap compute for the many potential rendering modes (position-as-layer, stitched grid, stacked slider etc).

For context, currently (0.3.0) recOrder's reader contribution simply falls back to napari-ome-zarr if an HCS dataset is given. But the performance will probably be pretty bad if we try to compute the colormap for the whole stitched volume.

[mattersoflight]

We decided that layers should represent channels and the positions will map to dimensional slider.

…

On Tue, Apr 25, 2023, 7:43 PM Ziwen Liu ***@***.***> wrote: @talonchandler <https://github.com/talonchandler> @edyoshikun <https://github.com/edyoshikun> @mattersoflight <https://github.com/mattersoflight> Can you remind me if we made any decision wrt the viewer's behavior for multi-position datasets? It brings up the complexity quite a bit if it needs to handle delayed colormap compute for the many potential rendering modes (position-as-layer, stitched grid, stacked slider etc). — Reply to this email directly, view it on GitHub <#348 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAWMLJ2NL2WK4IOULOWKLUTXDCDWZANCNFSM6AAAAAAXKGWBDY> . You are receiving this because you were mentioned.Message ID: ***@***.***>