Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

i.eodag: remove pandas dependency and save results in json/geojson files #1104

Merged
merged 37 commits into from
Jul 1, 2024

Conversation

HamedElgizery
Copy link
Contributor

@HamedElgizery HamedElgizery commented Jun 17, 2024

This PR will:

  • Remove Pandas dependency
  • Add filtering with eodag crunch [1]
  • Add sorting
  • Add DOCSTRINGS
  • Add JSON support
  • Save search results as a geojson file, which can later be read by i.eodag.

Note: Reading geojson containing search results is not yet supported, and it will be addressed in a future PR.

Added filtering options:

  • area_relation
  • minimum_overlap
  • filtering by dates (useful when reading products from a file)

Sorting parameters:

  • cloudcover
  • ingestiondate

Both can be set to ascending or descending.

[1] https://eodag.readthedocs.io/en/stable/api_reference/searchresult.html#crunch

@HamedElgizery HamedElgizery mentioned this pull request Jun 20, 2024
6 tasks
@HamedElgizery HamedElgizery marked this pull request as ready for review June 22, 2024 12:16
Co-authored-by: Stefan Blumentrath <stefan.blumentrath@gmx.de>
Copy link
Member

@ninsbl ninsbl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Lots of useful improvements!
I added a couple of suggestions for your consideration. Feel free to reject or incorporate as you feel for it. I leave an approving review anyways...

src/imagery/i.eodag/i.eodag.py Outdated Show resolved Hide resolved
src/imagery/i.eodag/i.eodag.py Outdated Show resolved Hide resolved
src/imagery/i.eodag/i.eodag.py Show resolved Hide resolved
src/imagery/i.eodag/i.eodag.py Show resolved Hide resolved
src/imagery/i.eodag/i.eodag.py Outdated Show resolved Hide resolved
src/imagery/i.eodag/i.eodag.py Show resolved Hide resolved
src/imagery/i.eodag/i.eodag.py Outdated Show resolved Hide resolved
src/imagery/i.eodag/i.eodag.py Outdated Show resolved Hide resolved
src/imagery/i.eodag/i.eodag.py Show resolved Hide resolved
echoix and others added 3 commits June 24, 2024 17:20
Co-authored-by: Stefan Blumentrath <stefan.blumentrath@gmx.de>
Co-authored-by: Stefan Blumentrath <stefan.blumentrath@gmx.de>
src/imagery/i.eodag/i.eodag.html Outdated Show resolved Hide resolved
src/imagery/i.eodag/i.eodag.html Outdated Show resolved Hide resolved
Comment on lines +90 to +97
Search for scenes with a list of IDs text file, and filter the results with the
provided parameters:

<div class="code"><pre>
i.eodag -l file=ids_list.txt \
start=2022-05-25 \
area_relation=Contains clouds=3
<pre></div>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This example or functionality does not make sense to me. If users provides a file with IDs is because they already did the search and filtering and they want to download the scenes found (i.e. IDs in the file). I cannot imagine to provide a text file with IDs and eodag doing a reverse search and filter... Is that what it does?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it takes the text file with a list of IDs, search for the scenes that is associated with each ID, and then it filters these founded scenes to make sure all of them satisfies the rest of the parameters.

I think a use case would be, if for example, a user did a general search (i.e. without setting the cloud coverage percentage), and then they want to pick scenes from this list of products with a certain cloud coverage percentage...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyways IMO, I can imagine that in most use cases products found by searching can be saved as a geojson file, and that geojson file can later be parsed by the module again. What is good about these geojson files, is that you don't need to do the searching again, as you already have the products ready to be downloaded after parsing the geojson file.
For reference: https://eodag.readthedocs.io/en/stable/notebooks/api_user_guide/5_serialize_deserialize.html?highlight=serialize

And for the text files it would just be a collection of IDs that is collected apart of the i.eodag module, and instead of typing them manually by hand in the shell they are just written in a text file... That is how I think it could be optimally used.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HamedElgizery ok your explanation could make sense for me, Your idea it is not a common behaviour, people usually look for the data and later they download them. @veroandreo what do you think?

This seems the last open point before accept the pull request

Copy link
Contributor

@veroandreo veroandreo Jul 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm at a eodag workshop in FOSS4G, and now I see the potential utility of this. @HamedElgizery is this similar to what they do with the crunch functionality, that it filters a search object?

In any case, I think this can be merged.

src/imagery/i.eodag/i.eodag.html Outdated Show resolved Hide resolved
src/imagery/i.eodag/i.eodag.py Outdated Show resolved Hide resolved
src/imagery/i.eodag/i.eodag.py Outdated Show resolved Hide resolved
src/imagery/i.eodag/i.eodag.py Outdated Show resolved Hide resolved
src/imagery/i.eodag/i.eodag.py Outdated Show resolved Hide resolved
src/imagery/i.eodag/i.eodag.py Outdated Show resolved Hide resolved
src/imagery/i.eodag/i.eodag.py Outdated Show resolved Hide resolved
HamedElgizery and others added 10 commits June 27, 2024 12:17
Co-authored-by: Luca Delucchi <lucadeluge@gmail.com>
Co-authored-by: Veronica Andreo <veroandreo@gmail.com>
Co-authored-by: Veronica Andreo <veroandreo@gmail.com>
Co-authored-by: Veronica Andreo <veroandreo@gmail.com>
Co-authored-by: Veronica Andreo <veroandreo@gmail.com>
Co-authored-by: Veronica Andreo <veroandreo@gmail.com>
Co-authored-by: Veronica Andreo <veroandreo@gmail.com>
Co-authored-by: Veronica Andreo <veroandreo@gmail.com>
Co-authored-by: Veronica Andreo <veroandreo@gmail.com>
Co-authored-by: Luca Delucchi <lucadeluge@gmail.com>
@HamedElgizery
Copy link
Contributor Author

Thanks for the feedback! @ninsbl @veroandreo @lucadelu

I will be adding footprint option in the next PR, because I think this one already contains off-topic additions, right?
Also I will probably be adding more qureyables for searching in the next PR, or the one after.

@HamedElgizery HamedElgizery changed the title i.eodag: add sorting and filtering options i.eodag: remove pandas dependency and save results in json/geojson files Jun 27, 2024
@veroandreo veroandreo merged commit dbf06f1 into OSGeo:grass8 Jul 1, 2024
7 checks passed
@HamedElgizery HamedElgizery deleted the eodag_filter_sort branch July 1, 2024 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants