Merge branch 'grass8' into i_sentinel_download

.github/workflows/clang-format-check.yml

@@ -18,10 +18,10 @@ jobs:
       - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
         with:
           persist-credentials: false
-      - uses: DoozyX/clang-format-lint-action@11b773b1598aa4ae3b32f023701bca5201c3817d # v0.17
+      - uses: DoozyX/clang-format-lint-action@caa179272c6ee7f1d25dfb503ee0c410c26ebd98 # v0.18.1
         with:
           source: "."
-          clangFormatVersion: 17
+          clangFormatVersion: 18.1.8
           inplace: True
       - name: Create and uploads code suggestions to apply
         id: diff

.github/workflows/super-linter.yml

@@ -32,7 +32,7 @@ jobs:
           # list of files that changed across commits
           fetch-depth: 0
       - name: Lint code base
-        uses: super-linter/super-linter@b4515bd4ad9d0aa4681960e053916ab991bdbe96 # v6.8.0
+        uses: super-linter/super-linter@1fa6ba58a88783e9714725cf89ac26d53e80c148 # v6.9.0
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           # By default, super-linter expect all linters to have their config

.pre-commit-config.yaml

@@ -37,7 +37,7 @@ repos:
     hooks:
       - id: flake8
   - repo: https://github.com/pre-commit/mirrors-clang-format
-    rev: v17.0.6
+    rev: v18.1.8
     hooks:
       - id: clang-format
         exclude_types:

src/imagery/i.eodag/i.eodag.html

@@ -52,10 +52,11 @@ <h4>Searching using scenes IDs</h4>
 Users have a set of scenes IDs that they want to search for and download.
 They can either use the <b>id</b> option or use the <b>file</b> option
 and pass a text file, with one ID per line.
-In both cases, specifying a provider is optional. In case users do
-not specify the provider, products might be offered
-from different providers as long as the user provides the
-credentials in the configuration file.
+When searching with ID, users must specify the product type that the IDs
+belong to. Specifying a provider is not mandatory, but it is recommended.
+In case users do not specify the provider, each scene might be offered
+from a different provider. Note that all scenes IDs have to belong
+to the same product type.
 
 <h4>Reading products from a GeoJSON file</h4>
 <p>
@@ -286,14 +287,14 @@ <h4>Creodias</h4>
 To register to Creodias, users should create an account
 <a href="https://portal.creodias.eu/register.php">here</a>, and
 then use their username and password in the eodag configuration file.
-Users will also need TOTP, a 6-digits temporary one time password, 
-to be able to authenticate and download scenes (product search 
-within creodias can be done without registering). 
+Users will also need TOTP, a 6-digits temporary one time password,
+to be able to authenticate and download scenes (product search
+within creodias can be done without registering).
 This TOTP is only valid for very short time, i.e., 30 to 60 seconds, so it shall
 not be set through the eodag configuration file.
 When <em>i.eodag</em> attempts to download a scene from creodias,
 users will be prompted to input the TOTP. If they prefer to discard these
-scenes, they should enter <b>"-"</b> instead. Note that interactive 
+scenes, they should enter <b>"-"</b> instead. Note that interactive
 prompt does not work in the graphical user interface.
 
 <p>
@@ -359,6 +360,21 @@ <h2>EXAMPLES</h2>
     pattern="LC09.*T1"
 <pre></div>
 
+Search and list Sentinel 2 scenes using IDs from Copernicus Data Space Ecosystem:<br>
+To download the selected scenes, simply remove the l flag.
+<div class="code"><pre>
+i.eodag -l provider=cop_dataspace producttype=S2_MSI_L2A \
+    id="S2A_MSIL2A_20240728T154941_N0511_R054_T17SPV_20240728T235651,
+    S2A_MSIL2A_20240618T154941_N0510_R054_T17SPV_20240618T222157"
+<pre></div>
+
+Search and list for a Landsat scene using its ID from USGS:<br>
+To download the selected scene, remove the l flag.
+<div class="code"><pre>
+i.eodag -l provider=usgs producttype=LANDSAT_C2L2 \
+    id=LC08_L2SP_016035_20240715_20240722_02_T1
+<pre></div>
+
 Download all available scenes with cloud coverage not exceeding 50%
 in the /tmp directory:
 
@@ -374,13 +390,16 @@ <h2>EXAMPLES</h2>
 i.eodag file=ids_list.txt provider=cop_dataspace
 <pre></div>
 
+
+
 Download only selected scenes into the <em>download_here</em>
 directory, using a custom config file:
 
 <div class="code"><pre>
 i.eodag provider=cop_dataspace \
     id="S2B_MSIL2A_20240526T080609_N0510_R078_T37SDD_20240526T094753,
     S2B_MSIL2A_20240529T081609_N0510_R121_T37SED_20240529T124818" \
+    producttype=S2_MSI_L2A \
     config=full/path/to/eodag/config.yaml \
     output=download_here
 <pre></div>
@@ -421,12 +440,21 @@ <h2>EXAMPLES</h2>
 i.eodag print=config provider=cop_dataspace
 <pre></div>
 
+Print query summary:
+<div class="code"><pre>
+i.eodag -lp provider=usgs producttype=LANDSAT_C2L2 area_relation=IsWithin clouds=30
+<pre></div>
+
 
 <h2>REQUIREMENTS</h2>
 
 <ul>
     <li><a href="https://eodag.readthedocs.io/en/stable/getting_started_guide/install.html">EODAG library</a>
     (install with <code>pip install eodag</code>)</li>
+    <li>For EODAG 3.0.0 and later, some of the providers have additonal dependencies that needs to be
+    installed, e.g. <code>pip install eodag[usgs]</code>, for more info see
+    <a href="https://eodag.readthedocs.io/en/stable/getting_started_guide/install.html">installation page</a>.
+    To install all dependencies use <code>pip install eodag[all]</code></li>
 </ul>
 
 <h2>SEE ALSO</h2>

src/imagery/i.eodag/i.eodag.py

@@ -54,6 +54,12 @@
 # % guisection: Filter
 # %end
 
+# %flag
+# % key: p
+# % description: Print compiled query string and exit
+# % guisection: Print
+# %end
+
 # OPTIONS
 # %option
 # % key: producttype
@@ -397,9 +403,20 @@ def search_by_ids(products_ids):
     search_result = []
     for query_id in products_ids:
         gs.info(_("Searching for {}".format(query_id)))
-        product, count = dag.search(
-            id=query_id, provider=options["provider"] or None, count=True
-        )
+        if int(eodag.__version__.split(".")[0]) < 3:
+            product, count = dag.search(
+                id=query_id, provider=options["provider"] or None
+            )
+        else:
+            if options["producttype"] is None:
+                gs.warning(_("The producttype option is not set"))
+            product = dag.search(
+                id=query_id,
+                provider=options["provider"] or None,
+                productType=options["producttype"] or None,
+                count=True,
+            )
+            count = product.number_matched
         if count > 1:
             gs.warning(
                 _("{}\nCould not be uniquely identified. Skipping...".format(query_id))
@@ -462,8 +479,17 @@ def no_fallback_search(search_parameters, provider):
     :rtype: class:'eodag.api.search_result.SearchResult'
     """
     try:
-        server_poke = dag.search(**search_parameters, provider=provider)
-        if server_poke[1] == 0:
+        if int(eodag.__version__.split(".")[0]) < 3:
+            server_poke = dag.search(**search_parameters, provider=provider)
+        elif (
+            int(eodag.__version__.split(".")[0]) >= 3
+            and server_poke.number_matched == 0
+        ):
+            server_poke = dag.search(**search_parameters, provider=provider, count=True)
+        if (int(eodag.__version__.split(".")[0]) < 3 and server_poke[1] == 0) or (
+            int(eodag.__version__.split(".")[0]) >= 3
+            and server_poke.number_matched == 0
+        ):
             gs.verbose(_("No products found"))
             return SearchResult([])
     except Exception as e:
@@ -1114,6 +1140,30 @@ def get_range(info):
     print(json.dumps(queryables_dict, indent=4))
 
 
+def print_query(geometry, queryables, **kwargs):
+    print(f"flags: {''.join([f for f in flags if flags[f] and f != 'p'])}")
+    print(f"AOI: {geometry}")
+    print(f"provider: {kwargs['provider'] if options['provider'] else ANY}")
+    print(f"producttype: {kwargs['producttype'] if options['producttype'] else ANY}")
+    print(
+        f"area_relation: {kwargs['area_relation'] if options['area_relation'] else 'ANY'}"
+    )
+    print(
+        f"minimum_overlap: {kwargs['minimum_overlap'] if options['minimum_overlap'] else 'ANY'}"
+    )
+    print(f"pattern: {kwargs['pattern'] if options['pattern'] else 'ANY'}")
+    print(f"start (ge): {kwargs['start'] if options['start'] else 'ANY'}")
+    print(f"end (le): {kwargs['end'] if options['end'] else 'ANY'}")
+    print(f"limit: {kwargs['limit'] if options['limit'] else 'ANY'}")
+    if kwargs["clouds"]:
+        print(f"cloudCover (le): {kwargs['clouds']}")
+    DEFAULT_OPERATOR = "eq"
+    for k, v in queryables:
+        for value in v:
+            operator = value[1] if value[1] else DEFAULT_OPERATOR
+            print(f"{k} ({operator}): {value[0]}")
+
+
 def main():
     # Products: https://github.com/CS-SI/eodag/blob/develop/eodag/resources/product_types.yml
 
@@ -1122,6 +1172,8 @@ def main():
     dag = EODataAccessGateway()
     if options["provider"]:
         dag.set_preferred_provider(options["provider"])
+    geometry = get_aoi(options["map"])
+    gs.verbose(_("AOI: {}".format(geometry)))
 
     queryables = parse_query(options["query"])
     for queryable, values in queryables:
@@ -1180,10 +1232,12 @@ def main():
         ids_set.discard(str())
         gs.message(_("Found {} distinct ID(s).".format(len(ids_set))))
         gs.message("\n".join(ids_set))
-
         # Search for products found from options["file"] or options["id"]
+        options["limit"] = len(ids_set)  # Disable limit option
+        if flags["p"]:
+            print_query(geometry, queryables, **options)
+            return
         search_result = search_by_ids(ids_set)
-        limit = len(search_result)  # Disable limit option
     elif "search_result" not in locals():
         dates_to_iso_format()
         items_per_page = 40
@@ -1192,8 +1246,6 @@ def main():
         product_type = options["producttype"]
 
         # HARDCODED VALUES FOR TESTING { "lonmin": 1.9, "latmin": 43.9, "lonmax": 2, "latmax": 45, }
-        geometry = get_aoi(options["map"])
-        gs.verbose(_("AOI: {}".format(geometry)))
 
         search_parameters = {
             "items_per_page": items_per_page,
@@ -1206,6 +1258,11 @@ def main():
 
         search_parameters["start"] = options["start"]
         search_parameters["end"] = options["end"]
+        if not options["area_relation"]:
+            options["area_relation"] = "Intersects"
+        if flags["p"]:
+            print_query(geometry, queryables, **options)
+            return
         if options["provider"]:
             search_result = no_fallback_search(search_parameters, options["provider"])
         else:

src/imagery/i.landsat/i.landsat.download/i.landsat.download.html

@@ -72,14 +72,6 @@ <h2>DESCRIPTION</h2>
 </tbody>
 </table>
 
-<p>
-To connect to EODAG the credentials of the data source of interest
-(either USGS or Planetary Computer) have to be filled in the EODAG config file.
-See EODAG Configuration section in <a href="i.eodag.html">i.eodag</a> and
-<a href="https://eodag.readthedocs.io/en/stable/getting_started_guide/configure.html">EODAG Configuration</a>.
-The configuration file is stored by default in $HOME/.config/eodag/eodag.yml.
-If it is not there, it will be created after running the module.
-<p>
 
 <p>
 By default, only products which footprint intersects current computation
@@ -103,29 +95,92 @@ <h2>DESCRIPTION</h2>
 aborting (default is 300 seconds).
 
 <h2>NOTES</h2>
-USGS is still under development and can't be used for searching nor downloading.
+
+<h3>Settings</h3>
+<p>
+<em>i.landsat.download</em> reads the user credentials either from the
+terminal or from the <b>settings</b> file. This file must contain two
+lines in case of USGS provider, or one line in case of Planetary Computer provider.
+
+<p>
+User credentials can be also defined interactively when <b>settings=-</b>
+is given. Note that interactive prompt does not work in the graphical
+user interface.
+
+<p>
+Alternatively, if the settings option is not specified <em>i.landsat.download</em>
+will attempt to use credentials stored in the default EODAG configuration file in
+<code>$HOME/.config/eodag/eodag.yml</code>. Note any parameter set in the configuration
+file, e.g. 'extract' to specify whether to extract the downloaded scenes or not,
+will be used by <em>i.landsat.download</em>, unless overriden by user options.
+<b>Exception:</b> output parameter in EODAG configuration file is not used by <em>i.landsat.download</em>,
+and the default output directory is the current working directory.
+
+<h3>USGS EarthExplorer</h3>
+<p>
+To connect to EarthExplorer both a <em>username</em> and a <em>password</em>
+are required. See the <a href="https://ers.cr.usgs.gov/register">register</a>
+page for signing up.
+
+<p>
+<b>USGS Settings File</b>
+<div class="code"><pre>
+myusername
+mypassword
+</pre></div>
+
+<p>
+<b>USGS Interactive Settings</b>
+<div class="code"><pre>
+Insert username: myusername
+Insert password:
+</pre></div>
+
+
+<h3>Planetary Computer</h3>
+<p>
+Most datasets are anonymously accessible, but a subscription
+key may be needed to
+<a href="https://planetarycomputer.microsoft.com/docs/concepts/sas/#rate-limits-and-access-restrictions">increase rate limits and access private datasets.</a>
+Users can create an account <a href="https://planetarycomputer.microsoft.com/account/request">here</a>,
+and then view their keys by signing in with their Microsoft account
+<a href="https://planetarycomputer.developer.azure-api.net/">here</a>.
+<p>
+<b>Planetary Computer Settings File</b>
+<div class="code"><pre>
+apikey
+</pre></div>
+
+<b>Planetary Computer Interactive Settings</b>
+<div class="code"><pre>
+Insert API key:
+</pre></div>
+Note: to use Plantery Computer anonymously the settings option should not be used.
+
 
 <h2>EXAMPLES</h2>
 
 Search available scenes:
 
 <div class="code"><pre>
-i.landsat.download -l dataset=landsat_8_ot_c2_l2 \
-    clouds=15 start='2018-08-24' end='2018-12-21'
+i.landsat.download -l settings=credentials.txt \
+    dataset=landsat_8_ot_c2_l2 clouds=15 \
+    datasource=usgs start='2018-08-24' end='2018-12-21'
 <pre></div>
 
 Download all available scenes:
 
 <div class="code"><pre>
-i.landsat.download dataset=landsat_8_ot_c2_l2 \
-    clouds=15 start='2018-08-24' end='2018-12-21' \
+i.landsat.download settings=credentials.txt \
+    dataset=landsat_8_ot_c2_l2 clouds=15 \
+    datasource=usgs start='2018-08-24' end='2018-12-21' \
     timeout=600
 <pre></div>
 
-Download selected scenes by ID:
+Download selected scenes by ID anonymously from Planetary Computer:
 
 <div class="code"><pre>
-i.landsat.download \
+i.landsat.download datasource=planetary_computer \
     id=LC09_L2SP_015035_20240529_02_T1,LC09_L2SP_015035_20240614_02_T1 \
     output=/tmp
 <pre></div>