i.eodag: add files support

[HamedElgizery]

This pull request will add:

• Download datasets from a text file with a list of IDs.
• Map option, to select a Vector map to act as an AOI.
• Add clouds options (Cloud Coverage) for filtering searching.
• Save search results to a Pandas DataFrame, to allow saving to different formats (e.g. CSV).
• Reformat the listing option output.
• Fix DEBUG & setup_logging bug.

[ninsbl]

Thanks @HamedElgizery Here are some comments regarding the proposed changes. Since I was not present at the last meeting, @lucadelu and @veroandreo please feel free to dismiss comments you do not agree with...

[veroandreo]

I can't get the module to see the eodag library that I injected in the jupyter env where I'm testing (pipx inject jupyterlab eodag). I get the error that eodag cannot be imported. If I import eodag in the same notebook, it works... no idea how to solve this, but it is a showstopper for me...

In any case, i.eodag -l should work without any other parameter, currently it requires provider. I'd suggest to use peps as default for provider since it is the one not requiring credentials for the search.

[veroandreo]

After activating the eodag venv and injecting pandas there, I get:

(eodag) GRASS nc_spm_08_grass7/PERMANENT:grass-addons > python ./src/imagery/i.eodag/i.eodag.py -l dataset=S2_MSI_L2A clouds=30 provider=peps start=2023-01-01 end=2024-01-01
WARNING: Experimental Version...
WARNING: This module is still under development, and its behaviour is not
         guaranteed to be reliable
Skipping error while searching for peps QueryStringSearch instance: 
Traceback (most recent call last):
  File "/home/vandreo/.local/pipx/venvs/eodag/lib/python3.11/site-packages/eodag/plugins/search/qssearch.py", line 887, in _request
    response.raise_for_status()
  File "/home/vandreo/.local/pipx/venvs/eodag/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://peps.cnes.fr/resto/api/collections/S2_MSI_L2A/search.json?cloudCover=%5B0,30%5D&startDate=2023-01-01T00:00:00&completionDate=2024-01-01T00:00:00&geometry=POLYGON%20((-78.7743%2035.6877,%20-78.7743%2035.8097,%20-78.6083%2035.8097,%20-78.6083%2035.6877,%20-78.7743%2035.6877))&productType=S2_MSI_L2A&maxRecords=40&page=1
Error while searching on provider peps (ignored):
Traceback (most recent call last):
  File "/home/vandreo/.local/pipx/venvs/eodag/lib/python3.11/site-packages/eodag/plugins/search/qssearch.py", line 887, in _request
    response.raise_for_status()
  File "/home/vandreo/.local/pipx/venvs/eodag/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://peps.cnes.fr/resto/api/collections/S2_MSI_L2A/search.json?cloudCover=%5B0,30%5D&startDate=2023-01-01T00:00:00&completionDate=2024-01-01T00:00:00&geometry=POLYGON%20((-78.7743%2035.6877,%20-78.7743%2035.8097,%20-78.6083%2035.8097,%20-78.6083%2035.6877,%20-78.7743%2035.6877))&productType=S2_MSI_L2A&maxRecords=40&page=1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vandreo/.local/pipx/venvs/eodag/lib/python3.11/site-packages/eodag/api/core.py", line 1685, in _do_search
    res, nb_res = search_plugin.query(count=count, auth=auth_plugin, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vandreo/.local/pipx/venvs/eodag/lib/python3.11/site-packages/eodag/plugins/search/qssearch.py", line 520, in query
    provider_results = self.do_search(items_per_page=items_per_page, **kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vandreo/.local/pipx/venvs/eodag/lib/python3.11/site-packages/eodag/plugins/search/qssearch.py", line 629, in do_search
    response = self._request(
               ^^^^^^^^^^^^^^
  File "/home/vandreo/.local/pipx/venvs/eodag/lib/python3.11/site-packages/eodag/plugins/search/qssearch.py", line 902, in _request
    raise RequestError(str(err))
eodag.utils.exceptions.RequestError: 500 Server Error: Internal Server Error for url: https://peps.cnes.fr/resto/api/collections/S2_MSI_L2A/search.json?cloudCover=%5B0,30%5D&startDate=2023-01-01T00:00:00&completionDate=2024-01-01T00:00:00&geometry=POLYGON%20((-78.7743%2035.6877,%20-78.7743%2035.8097,%20-78.6083%2035.8097,%20-78.6083%2035.6877,%20-78.7743%2035.6877))&productType=S2_MSI_L2A&maxRecords=40&page=1
No result could be obtained from any available provider
Applying filters...
0 product(s) found.

With default S1 product it works:

(eodag) GRASS nc_spm_08_grass7/PERMANENT:grass-addons > python ./src/imagery/i.eodag/i.eodag.py -l provider=peps
WARNING: Experimental Version...
WARNING: This module is still under development, and its behaviour is not
         guaranteed to be reliable
Applying filters...
5 product(s) found.
S1A_IW_GRDH_1SDV_20240607T231439_20240607T231504_054224_069864_1A5E 2024-06-07T23:14:39Z cloudcover_NA GRD
S1A_IW_GRDH_1SDV_20240526T231440_20240526T231505_054049_069253_4EF3 2024-05-26T23:14:40Z cloudcover_NA GRD
S1A_IW_GRDH_1SDV_20240514T231439_20240514T231504_053874_068C4B_5E79 2024-05-14T23:14:39Z cloudcover_NA GRD
S1A_IW_GRDH_1SDV_20240502T231440_20240502T231505_053699_0685E6_9DB5 2024-05-02T23:14:40Z cloudcover_NA GRD
S1A_IW_GRDH_1SDV_20240420T231440_20240420T231505_053524_067F05_72A0 2024-04-20T23:14:40Z cloudcover_NA GRD

[HamedElgizery]

@veroandreo
That is because peps doesn't provide S2_MSI_L2A products, the reason you are getting the error is probably you have set the GRASS DEBUG environment variable to some variable other than zero... So it is in debugging mode, that is why you got a full trace of the error.

If you set the DEBUG environment variable to zero, the error should be gone.
For reference: https://eodag.readthedocs.io/en/stable/getting_started_guide/product_types.html#product-types-information-csv

[lucadelu]

I think we should set the default options values to be peps for the provider, and some other product type that is provided by peps, right? So users don't get 0 results by default.

Instead I'm a little bit confused by provider option. I think providers are defined by the user in the EODAG config file, it should be enough.

[HamedElgizery]

Instead I'm a little bit confused by provider option. I think providers are defined by the user in the EODAG config file, it should be enough.

It should be enough, agree yes. But, the provider option is to constraint the module, and to tell it where to search exactly... if it didn't find the scene I am looking for with the provider being set, then stop and don't look in other providers... If it was to depend on the config file, it will try to look for the provider with highest priority, if it didn't find any scenes it will search using the next priority provider, and so on. So, I think constraining it and having it working with one and only one provider, makes it more reliable and expected...

I think we could make the provider optional as well, and let eodag use the config file for searching when the provider is not set, so we can have both ways for searching, right?

[lucadelu]

Instead I'm a little bit confused by provider option. I think providers are defined by the user in the EODAG config file, it should be enough.

It should be enough, agree yes. But, the provider option is to constraint the module, and to tell it where to search exactly... if it didn't find the scene I am looking for with the provider being set, then stop and don't look in other providers... If it was to depend on the config file, it will try to look for the provider with highest priority, if it didn't find any scenes it will search using the next priority provider, and so on. So, I think constraining it and having it working with one and only one provider, makes it more reliable and expected...

I think we could make the provider optional as well, and let eodag use the config file for searching when the provider is not set, so we can have both ways for searching, right?

Yes I think this is the most useful way, keep it as optional. It is true that it is more reliable and expected but if I want to download that images I'm not interested where they come from since they should be identical in each provider

[veroandreo]

Now, if we make the provider optional, that I agree, do we need a default product and provider, so when users do i.eodag -l, they get an output? For me it makes sense to set S2_MSI_LC1 as default product and peps as default provider as the latter allows the search without credentials. What do you think guys?

[HamedElgizery]

Now, if we make the provider optional, that I agree, do we need a default product and provider, so when users do i.eodag -l, they get an output? For me it makes sense to set S2_MSI_LC1 as default product and peps as default provider as the latter allows the search without credentials. What do you think guys?

I think we can make the provider optional without setting a default one, and so if none is provided it will then use the config file to figure out which provider to use (or use multiple providers till it find the scenes)... And for the default product I agree that we can make it S2_MSI_L1C.

[veroandreo]

Works as expected in the tests I did, ready to merge once doc changes are applied

[ninsbl]

Hi @HamedElgizery , here are some comments on the code. I hope I do not come across too picky. Please feel free to dismiss the more stylistic comments and suggestions. What I think is important to discuss is the usage of sort filter capabilities in EODAG vs. downstream adjustmens / re-implementations... But the latter can be tackled in the next PR or the PR thereafter....

[HamedElgizery]

Thanks for the comment, @ninsbl.
I have fixed the refactoring and styling issues, and I will solve the "messages" and "filtering/sorting with the EODAG capabilities as you mentioned" in the next PRs.

[echoix]

Note when ready to merge, or if it is somewhat possible to fix: the last commit, 6c27785 says that it adds "panadas" as a requirement, but it rather should be "pandas". The actual contents seem fine though.