Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

i.eodag: remove pandas dependency and save results in json/geojson files #1104

Merged
merged 37 commits into from
Jul 1, 2024
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
3d7fc8a
Add sorting with ingestiondate and cloudcover
HamedElgizery Jun 17, 2024
0492e66
Change filters to eodag crunch
HamedElgizery Jun 20, 2024
46ba518
Remove pandas dependency
HamedElgizery Jun 20, 2024
08312e8
Add minimum_overlap option
HamedElgizery Jun 20, 2024
2e82fce
Refactor messages
HamedElgizery Jun 20, 2024
6ea3795
Remove duplicates
HamedElgizery Jun 22, 2024
8229e48
Update manual
HamedElgizery Jun 22, 2024
0e1f405
Add filtering by date
HamedElgizery Jun 22, 2024
00c3b40
Add docstrings
HamedElgizery Jun 22, 2024
72cfdc9
Merge branch 'grass8' into eodag_filter_sort
HamedElgizery Jun 22, 2024
01d5a76
Update src/imagery/i.eodag/i.eodag.py
HamedElgizery Jun 23, 2024
1c4fdbb
Merge branch 'grass8' into eodag_filter_sort
echoix Jun 24, 2024
e3569ca
Update src/imagery/i.eodag/i.eodag.py
HamedElgizery Jun 24, 2024
7983e0e
Update src/imagery/i.eodag/i.eodag.py
HamedElgizery Jun 24, 2024
e017d2c
Update src/imagery/i.eodag/i.eodag.py
HamedElgizery Jun 24, 2024
c6fd7eb
Remove exclusive relation between area_relation and minimum_overlap
HamedElgizery Jun 24, 2024
362769f
Add optional timezone to start and end options, all dates/time are co…
HamedElgizery Jun 25, 2024
ebc9884
Limit order to a single value asc or desc, instead of an order value …
HamedElgizery Jun 25, 2024
66e84c7
Fix ingestiondate to be publicationDate instead of startTimeAscending…
HamedElgizery Jun 25, 2024
2e06408
Add JSON output format and save to json/geojson files options
HamedElgizery Jun 26, 2024
a0df73a
Update src/imagery/i.eodag/i.eodag.html
HamedElgizery Jun 27, 2024
88b31ae
Update src/imagery/i.eodag/i.eodag.py
HamedElgizery Jun 27, 2024
a08eceb
Update src/imagery/i.eodag/i.eodag.py
HamedElgizery Jun 27, 2024
592f0e4
Update src/imagery/i.eodag/i.eodag.py
HamedElgizery Jun 27, 2024
e529aef
Update src/imagery/i.eodag/i.eodag.py
HamedElgizery Jun 27, 2024
c1f391d
Update src/imagery/i.eodag/i.eodag.py
HamedElgizery Jun 27, 2024
7a10a7d
Update src/imagery/i.eodag/i.eodag.py
HamedElgizery Jun 27, 2024
4096a9a
Update src/imagery/i.eodag/i.eodag.html
HamedElgizery Jun 27, 2024
d1f91ee
Update src/imagery/i.eodag/i.eodag.html
HamedElgizery Jun 27, 2024
f4d313c
Update src/imagery/i.eodag/i.eodag.html
HamedElgizery Jun 27, 2024
280c94b
Fix minimum_overlap description
HamedElgizery Jun 27, 2024
40c8e06
Update src/imagery/i.eodag/i.eodag.py
HamedElgizery Jun 27, 2024
1ffa680
Update i.eodag.py
HamedElgizery Jun 27, 2024
7934454
Update i.eodag.html
HamedElgizery Jun 27, 2024
b8b35f2
Reformat warning message setup_envrionment_variables function
HamedElgizery Jun 27, 2024
23cfee6
Merge branch 'eodag_filter_sort' of github.com:HamedElgizery/grass-ad…
HamedElgizery Jun 27, 2024
6d99e7f
Change ingestionDate to be startTimeFromAscendingNode
HamedElgizery Jun 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 37 additions & 7 deletions src/imagery/i.eodag/i.eodag.html
Original file line number Diff line number Diff line change
Expand Up @@ -63,24 +63,56 @@ <h2>EXAMPLES</h2>
v.extract input=urbanarea where="NAME = 'Durham'" output=durham

i.eodag -l start=2022-05-25 end=2022-06-01 \
map=durham dataset=S2_MSI_L2A provider=cop_dataspace
map=durham dataset=S2_MSI_L2A provider=cop_dataspace \
sort=cloudcover,ingestiondate order=asc,desc
<pre></div>

Download all available scenes in the tmp directory, with Cloud Coverage not exceeding 50%:
Search and list the available Sentinel 2 scenes in the Copernicus Data Space
Ecosystem, with at least 70% of the AOI covered:
<div class="code"><pre>
v.extract input=urbanarea where="NAME = 'Durham'" output=durham

i.eodag -l start=2021-05-25 end=2022-06-01 \
HamedElgizery marked this conversation as resolved.
Show resolved Hide resolved
dataset=S2_MSI_L2A provider=cop_dataspace \
clouds=50 map=durham minimum_overlap=70
<pre></div>

Sort results by <b>cloudcover</b>, and then by <b>ingestiondate</b> descendingly
(latest to earliest):<br><em> Note that sorting with <b>cloudcover</b> use
unrounded values, while they are rounded to the nearest intger when listing.</em>
HamedElgizery marked this conversation as resolved.
Show resolved Hide resolved

<div class="code"><pre>
i.eodag -l start=2021-05-25 end=2022-06-01 \
HamedElgizery marked this conversation as resolved.
Show resolved Hide resolved
dataset=S2_MSI_L2A provider=cop_dataspace \
sort=cloudcover,ingestiondate order=asc,desc
HamedElgizery marked this conversation as resolved.
Show resolved Hide resolved
<pre></div>

Search for scenes with a list of IDs text file, and filter the results with the
provided parameters:

<div class="code"><pre>
i.eodag -l file=ids_list.txt \
start=2022-05-25 \
area_relation=Contains clouds=3
<pre></div>
Comment on lines +90 to +97
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This example or functionality does not make sense to me. If users provides a file with IDs is because they already did the search and filtering and they want to download the scenes found (i.e. IDs in the file). I cannot imagine to provide a text file with IDs and eodag doing a reverse search and filter... Is that what it does?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it takes the text file with a list of IDs, search for the scenes that is associated with each ID, and then it filters these founded scenes to make sure all of them satisfies the rest of the parameters.

I think a use case would be, if for example, a user did a general search (i.e. without setting the cloud coverage percentage), and then they want to pick scenes from this list of products with a certain cloud coverage percentage...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyways IMO, I can imagine that in most use cases products found by searching can be saved as a geojson file, and that geojson file can later be parsed by the module again. What is good about these geojson files, is that you don't need to do the searching again, as you already have the products ready to be downloaded after parsing the geojson file.
For reference: https://eodag.readthedocs.io/en/stable/notebooks/api_user_guide/5_serialize_deserialize.html?highlight=serialize

And for the text files it would just be a collection of IDs that is collected apart of the i.eodag module, and instead of typing them manually by hand in the shell they are just written in a text file... That is how I think it could be optimally used.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HamedElgizery ok your explanation could make sense for me, Your idea it is not a common behaviour, people usually look for the data and later they download them. @veroandreo what do you think?

This seems the last open point before accept the pull request

Copy link
Contributor

@veroandreo veroandreo Jul 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm at a eodag workshop in FOSS4G, and now I see the potential utility of this. @HamedElgizery is this similar to what they do with the crunch functionality, that it filters a search object?

In any case, I think this can be merged.


Download all available scenes in the tmp directory, with Cloud Coverage not
exceeding 50%:
HamedElgizery marked this conversation as resolved.
Show resolved Hide resolved

<div class="code"><pre>
i.eodag start=2022-05-25 end=2022-06-01 \
dataset=S2_MSI_L2A provider=cop_dataspace clouds=50
<pre></div>

Download only selected scenes from a text file of IDs, using the Copernicus Data Space Ecosystem as the provider:
Download only selected scenes from a text file of IDs, using the Copernicus Data
Space Ecosystem as the provider:

<div class="code"><pre>
i.eodag file=ids_list.txt provider=cop_dataspace
<pre></div>

Download and extract only selected scenes into the <em>download_here</em> directory,
using a custom config file:
Download and extract only selected scenes into the <em>download_here</em>
directory, using a custom config file:

<div class="code"><pre>
i.eodag -e provider=cop_dataspace \
Expand All @@ -95,8 +127,6 @@ <h2>REQUIREMENTS</h2>
<ul>
<li><a href="https://eodag.readthedocs.io/en/stable/getting_started_guide/install.html">EODAG library</a>
(install with <code>pip install eodag</code>)</li>
<li><a href="https://pandas.pydata.org/">Pandas</a>
(install with <code>pip install pandas</code>)</li>
</ul>

<h2>SEE ALSO</h2>
Expand Down
Loading
Loading