Problem with substituting sdata.table with external AnnData

[gboscagli]

Hi all,
I'm starting to use SpatialData framework but I'm still missing something.
I'm working on this public CosMx dataset. I'm able to read it as a SpatialData object and to visualize it using Napari, annotations included. However, I have done previous work on the same dataset and stored some annotations (louvain clusters, cell type annotations, etc...) in an AnnData object. At a first glance, I naively tried to substitute sdata.table with my AnnData (I saw you "recommended" it in scverse/spatialdata#270): it kinda works, except for the fact that when opened in Napari the table isn't recognized anymore in the "Tables annotating layer" menu, and thus I'm not able to plot any annotation. Maybe some references and metadata linking labels to annotations got lost during the transition: I thought of the missing region, region_key and instance_key metadata, but they are missing from the SpatialData table too. For the moment, I worked around it by concatenating specific columns from AnnData to SpatialData table, but I don't think this is the ideal solution. I also guess that adding my AnnData as additional table wouldn't solve my problem.

Do you have some recommendations for my issue?

[LucaMarconato]

Hi, thanks for your interest in the library.

Please you can find an extended discussion on how to add additional annotation layers in the SpatialData object in this tutorial: https://spatialdata.scverse.org/en/latest/tutorials/notebooks/notebooks/examples/tables.html.

In your case, you can adjust the metadata via TableModel.parse() or via set_table_annotates_spatialelement(). You will find an example of both in the tutorial.

Please let me know if you have still question on this.

[gboscagli]

Hi Luca,

thanks for your precious support. With set_table_annotates_spatialelement() I was able to visualize all the annotation tables in Napari. Unfortunately, with the external AnnData only, each annotation appears to be "flat", i.e. plotted with a uniform color (as the attached image shows), while for the original SpatialData table it all works well. I guess there's something wrong with the colormap or some other metadata is missing.

Below is the code I used. It's all pretty much straightforward, except for the helper function I wrote from scratch to prepare the table for the visualization.

from pathlib import Path
import anndata as ad
import spatialdata_io
from napari_spatialdata import Interactive
from spatialdata.models import get_table_keys
from spatialdata.models import TableModel

filename = "Run5810_Case2"
nanostring_dir = Path().resolve() / "nanostring_data"
sample_dir = nanostring_dir / filename

adata = ad.read_h5ad(f"./R_analysis/adata_se_45.h5ad") #annotated anndata object
sdata = spatialdata_io.cosmx(path=sample_dir, dataset_id=f"{filename}") #spatialdata object from raw folder

def process_sdata(adata, sdata, table_name):
    '''Adds annotated AnnData to SpatialData object, creates fov_labels column and transfers coordinate slots'''
    sdata[table_name] = adata
    sdata[table_name].obs = sdata[table_name].obs.rename(columns={'cell_id': 'cell_ID'})
    sdata[table_name].obs["fov_labels"] = (sdata[table_name].obs["fov"].astype(str) + "_labels").astype("category")
    sdata[table_name].obsm["spatial"] = sdata["table"].obsm["spatial"]
    sdata[table_name].obsm["global"] = sdata["table"].obsm["global"]
    return sdata

sdata = process_sdata(adata, sdata, "table1")

region, region_key, instance_key = get_table_keys(sdata["table"])
sdata.set_table_annotates_spatialelement("table1", region=region, region_key=region_key, instance_key=instance_key)

interactive = Interactive(sdata)
interactive.run()

[gboscagli]

UPDATE: I noticed that every time I select a different annotation in Napari, in the notebook this warning is thrown:

FutureWarning: Labels.color is deprecated since 0.4.19 and will be removed in 0.5.0, please set Labels.colormap directly with an instance of napari.utils.colormaps.DirectLabelColormap instead!

In my mind, this just confirms the idea that the problem resides in the colormap. The question is: can I manually set the Labels.colormap, since the Interactive object doesn't seem to have a module to access the label layers, or even transfer the colormap from the native AnnData table that is regularly visualized?

I found a very similar PR in the github of BiAPoL, who developed a Napari plugin as well. I have napari v.0.4.19.post1, if it can help.

[LucaMarconato]

Thanks for the update @gboscagli and for the hint that the problem could be related to the colormap in napari. To make it easier to investigate this, could you please reproduce the bug using this dataset: blobs dataset.

from spatialdata.datasets import blobs
sdata = blobs()

You can also use blobs_annotating_element for more control:

from spatialdata.datasets import blobs_annotating_element
sdata = blobs_annotating_element('blobs_labels')

Thank you!

[gboscagli]

Hi @LucaMarconato,
I used the following code to reproduce the error, don't know if it's what you meant me to do. It all worked well, even if I tried to del sdata["table"] and then set it again.

from spatialdata.datasets import blobs
sdata = blobs()

adata = sdata["table"]
sdata["table1"] = adata

interactive = Interactive(sdata)
interactive.run()

I think @melonora is aware of this, as we spoke about the same problem in an Image.sc issue. With my public CosMx dataset, I also tried manually adding annotation columns from external AnnData to the sdata["table"] I read from the raw folder: Napari isn't able to color just the plugged annotations, while with the native ones it works.

By now, I circumvented the problem by saving AnnData obs slot as the metadata_file.csv, and then read the folder as usual with spatialdata_io.cosmx. Napari now recognizes the colormap for all annotations, but I noticed another problem that I'm tracking in another issue so that I'm not making any more mess than I'm already making.

Could it depend on the dtype of the columns I'm trying to plot? They actually come from a SpatialExperiment object, converted into an AnnData object with an in-house function. I guess it's a very specific problem that could probably depend on my data, but maybe is not.

[LucaMarconato]

Yes in principle it could depend on the dtype.

but I noticed another problem that I'm tracking in another issue so that I'm not making any more mess than I'm already making.

which is the other tracked issue? I could have a look if helpful.

Regarding the reproducible example, if I got it right you couldn't reproduce the bug with the blobs dataset, correct? Unfortunately without access to the data and without a reproducible example, investigating this bug becomes more time consuming and we are trying to prioritize other GitHub issues.

@melonora is unfortunately currently unavailable/with limited capacity for the next two weeks because of a pressing deadline.

[gboscagli]

which is the other tracked issue? I could have a look if helpful.

I haven't opened it yet. Basically, when I plot an obs annotation, a color is not "unique" for a certain discrete value. To be clearer, in the following image, pinkish cells can refer to several different annotations. Also, an annotation can be found with more than one color (i.e., a mural cell could be either pink or another color). I'm quite sure it is a Colormap problem, but I don't know how could be related to the original issue.

Unfortunately without access to the data and without a reproducible example, investigating this bug becomes more time consuming and we are trying to prioritize other GitHub issues.

The dataset is public (link above), I could just send my annotated AnnData object to be plugged and you can try to reproduce the bug. But I understand it's not as handy as the blobs dataset is.

[LucaMarconato]

The dataset is public (link above), I could just send my annotated AnnData object to be plugged and you can try to reproduce the bug. But I understand it's not as handy as the blobs dataset is.

Sounds great!

(i.e., a mural cell could be either pink or another color). I'
Please check the code in this PR: https://github.com/scverse/napari-spatialdata/pull/304/files this is how one can assign custom colors (this is the same as what scanpy uses). I think that what you are experiencing is the result of a random colormap where some colors are too similar or collide.

[gboscagli]

Sounds great!

Where do you prefer the .h5ad file to be sent?

Please check the code in this PR: https://github.com/scverse/napari-spatialdata/pull/304/files this is how one can assign custom colors (this is the same as what scanpy uses). I think that what you are experiencing is the result of a random colormap where some colors are too similar or collide.

I will check the PR and keep you updated.

[LucaMarconato]

Via Zulip thanks: https://scverse.zulipchat.com/#user/480560

I will check the PR and keep you updated.

sounds great, please if there are further comments on it let's follow up on a dedicated GitHub Issue.

[gboscagli]

UPDATE: I found out that the problem resided somewhere in my external AnnData. It was created from a SpatialExperiment object using reticulate in R with a converter in-house function that probably lost track of some metadata. Now I just transferred annotations as a csv file instead of a complete AnnData, and used it to read the object from scratch: when plugged inside a SpatialData object, Napari doesn't complain anymore and finally shows the colormap.

I'm closing this!