Check physical forecast #92
Labels
good first issue
Good for newcomers
Comments
|
can I take this issue up? |
|
yes please thanks @glitch401 |
|
apologies for circling back on this issue so late, but in the given resources to get this project running. I'm facing issues in successfully testing this library: After installation of requirements as mentioned in the Readme: conda install python=3.10
conda install -c conda-forge xesmf esmpy h5py -y
pip install torch==2.2.0 --index-url https://download.pytorch.org/whl/cpu
pip install git+https://github.com/SheffieldSolar/PV_Live-API#pvlive_api
pip install torch==2.2.0 torchvision --index-url https://download.pytorch.org/whl/cpu
pip install -e .I'm getting the following error on running pytest: ERROR tests/test_app.py::test_app - docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
ERROR tests/test_app.py::test_app_day_ahead_model - docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))@peterdudfield what am i missing, can you help me navigate this? |
|
I think you need to have docker server running locally aswell |
|
makes sense, will try running it |
|
just curious if I'm running the docker right: (.conda) neil@Indranils-MBP pvnet_app % docker run -p 8000:8000 pvnet_app
/opt/conda/lib/python3.10/site-packages/pydantic/_internal/_config.py:321: UserWarning: Valid config keys have changed in V2:
* 'allow_population_by_field_name' has been renamed to 'populate_by_name'
* 'orm_mode' has been renamed to 'from_attributes'
* 'underscore_attrs_are_private' has been removed
warnings.warn(message, UserWarning)
[2024-07-20 01:47:05,376] {/app/pvnet_app/app.py:214} INFO - Using `pvnet` library version: 3.0.47
[2024-07-20 01:47:05,376] {/app/pvnet_app/app.py:215} INFO - Using `pvnet_app` library version: 2.3.12
[2024-07-20 01:47:05,376] {/app/pvnet_app/app.py:216} INFO - Using 10 workers
[2024-07-20 01:47:05,376] {/app/pvnet_app/app.py:223} INFO - Using adjduster: True
[2024-07-20 01:47:05,376] {/app/pvnet_app/app.py:224} INFO - Saving GSP sum: False
[2024-07-20 01:47:05,376] {/app/pvnet_app/app.py:239} INFO - Making forecast for init time: 2024-07-20 01:30:00
[2024-07-20 01:47:05,377] {/app/pvnet_app/app.py:240} INFO - Making forecast for GSP IDs: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317]
[2024-07-20 01:47:05,377] {/app/pvnet_app/app.py:246} INFO - Loading GSP metadata
[2024-07-20 01:47:05,377] {/opt/conda/lib/python3.10/site-packages/ocf_datapipes/load/gsp/database.py:133} INFO - Loading GSP data from database
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /app/pvnet_app/app.py:248 in app │
│ │
│ 245 │ # Make pands Series of most recent GSP effective capacities │
│ 246 │ logger.info("Loading GSP metadata") │
│ 247 │ │
│ ❱ 248 │ ds_gsp = next(iter(OpenGSPFromDatabase())) │
│ 249 │ │
│ 250 │ # Get capacities from the database │
│ 251 │ db_connection = DatabaseConnection(url=os.getenv("DB_URL"), base=B │
│ │
│ ╭───────────────────────────── locals ─────────────────────────────╮ │
│ │ day_ahead_model_used = False │ │
│ │ gsp_ids = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ... +307] │ │
│ │ num_workers = 10 │ │
│ │ t0 = Timestamp('2024-07-20 01:30:00') │ │
│ │ temp_dir = <TemporaryDirectory '/tmp/tmpt72lhax6'> │ │
│ │ write_predictions = True │ │
│ ╰──────────────────────────────────────────────────────────────────╯ │
│ │
│ /opt/conda/lib/python3.10/site-packages/torch/utils/data/datapipes/_hook_ite │
│ rator.py:181 in wrap_generator │
│ │
│ 178 │ │ │ │ │ with profiler_record_fn_context(datapipe): │
│ 179 │ │ │ │ │ │ response = gen.send(None) │
│ 180 │ │ │ │ else: │
│ ❱ 181 │ │ │ │ │ response = gen.send(None) │
│ 182 │ │ │ │ │
│ 183 │ │ │ │ while True: │
│ 184 │ │ │ │ │ datapipe._number_of_samples_yielded += 1 │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ _profiler_enabled = False │ │
│ │ args = (OpenGSPFromDatabaseIterDataPipe,) │ │
│ │ datapipe = OpenGSPFromDatabaseIterDataPipe │ │
│ │ full_msg = 'thrown by __iter__ of │ │
│ │ OpenGSPFromDatabaseIterDataPipe(interpolat… │ │
│ │ lo'+41 │ │
│ │ func = <function │ │
│ │ OpenGSPFromDatabaseIterDataPipe.__iter__ at │ │
│ │ 0xffff2221c790> │ │
│ │ gen = <generator object │ │
│ │ OpenGSPFromDatabaseIterDataPipe.__iter__ at │ │
│ │ 0xffff01a49850> │ │
│ │ iterator_id = 0 │ │
│ │ kwargs = {} │ │
│ │ msg = 'thrown by __iter__ of' │ │
│ │ profiler_record_fn_context = <function │ │
│ │ hook_iterator.<locals>.profiler_record_fn_… │ │
│ │ at 0xffff2221c820> │ │
│ │ single_iterator_msg = 'single iterator per IterDataPipe │ │
│ │ constraint' │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /opt/conda/lib/python3.10/site-packages/ocf_datapipes/load/gsp/database.py:7 │
│ 3 in __iter__ │
│ │
│ 70 │ │ │ gsp_pv_power_mw_df, │
│ 71 │ │ │ gsp_nominal_capacity, │
│ 72 │ │ │ gsp_effective_capacity, │
│ ❱ 73 │ │ ) = get_gsp_power_from_database( │
│ 74 │ │ │ history_duration=self.history_duration, │
│ 75 │ │ │ interpolate_minutes=self.interpolate_minutes, │
│ 76 │ │ │ load_extra_minutes=self.load_extra_minutes, │
│ │
│ ╭──────────────── locals ────────────────╮ │
│ │ self = OpenGSPFromDatabaseIterDataPipe │ │
│ ╰────────────────────────────────────────╯ │
│ │
│ /opt/conda/lib/python3.10/site-packages/ocf_datapipes/load/gsp/database.py:1 │
│ 55 in get_gsp_power_from_database │
│ │
│ 152 │ │
│ 153 │ # make database connection │
│ 154 │ url = os.getenv("DB_URL") │
│ ❱ 155 │ db_connection = DatabaseConnection(url=url, base=Base_Forecast) │
│ 156 │ │
│ 157 │ with db_connection.get_session() as session: │
│ 158 │ │ # We minus 1 second just to make sure we don't that value │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ empty_df = Empty DataFrame │ │
│ │ Columns: [] │ │
│ │ Index: [2024-07-19 23:00:00+00:00, 2024-07-19 │ │
│ │ 23:30:00+00:00, 2024-07-20 00:00:00+00:00, │ │
│ │ 2024-07-20 00:30:00+00:00, 2024-07-20 │ │
│ │ 01:00:00+00:00, 2024-07-20 01:30:00+00:00] │ │
│ │ extra_duration = datetime.timedelta(seconds=3600) │ │
│ │ gsp_ids = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ... +307] │ │
│ │ history_duration = datetime.timedelta(seconds=5400) │ │
│ │ interpolate_minutes = 60 │ │
│ │ load_extra_minutes = 60 │ │
│ │ now = Timestamp('2024-07-20 01:30:00+0000', tz='UTC') │ │
│ │ start_utc = Timestamp('2024-07-20 00:00:00+0000', tz='UTC') │ │
│ │ start_utc_extra = Timestamp('2024-07-19 23:00:00+0000', tz='UTC') │ │
│ │ url = None │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /opt/conda/lib/python3.10/site-packages/nowcasting_datamodel/connection.py:2 │
│ 7 in __init__ │
│ │
│ 24 │ │ self.url = url │
│ 25 │ │ self.base = base │
│ 26 │ │ │
│ ❱ 27 │ │ self.engine = create_engine(self.url, echo=echo) │
│ 28 │ │ │
│ 29 │ │ self.Session = sessionmaker(bind=self.engine) │
│ 30 │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ base = <class 'sqlalchemy.orm.decl_api.Base'> │ │
│ │ echo = True │ │
│ │ self = <nowcasting_datamodel.connection.DatabaseConnection object at │ │
│ │ 0xffff01a6f940> │ │
│ │ url = None │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ in create_engine:2 │
│ ╭──────── locals ─────────╮ │
│ │ kwargs = {'echo': True} │ │
│ │ url = None │ │
│ ╰─────────────────────────╯ │
│ │
│ /opt/conda/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py:281 │
│ in warned │
│ │
│ 278 │ │ │ │ │ │ version_warnings[m], │
│ 279 │ │ │ │ │ │ stacklevel=3, │
│ 280 │ │ │ │ │ ) │
│ ❱ 281 │ │ │ return fn(*args, **kwargs) # type: ignore[no-any-return] │
│ 282 │ │ │
│ 283 │ │ doc = fn.__doc__ is not None and fn.__doc__ or "" │
│ 284 │ │ if doc: │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ args = (None,) │ │
│ │ check_any_kw = 'kwargs' │ │
│ │ check_defaults = () │ │
│ │ check_kw = { │ │
│ │ │ 'implicit_returning', │ │
│ │ │ 'empty_in_strategy', │ │
│ │ │ 'strategy' │ │
│ │ } │ │
│ │ fn = <function create_engine at 0xffff476aa9e0> │ │
│ │ kwargs = {'echo': True} │ │
│ │ m = 'strategy' │ │
│ │ messages = { │ │
│ │ │ 'strategy': 'The create_engine.strategy keyword │ │
│ │ is deprecated, and the only argument accepted'+177, │ │
│ │ │ 'empty_in_strategy': 'The │ │
│ │ create_engine.empty_in_strategy keyword is │ │
│ │ deprecated, and no longer has any'+181, │ │
│ │ │ 'implicit_returning': 'The │ │
│ │ create_engine.implicit_returning parameter is │ │
│ │ deprecated and will be removed'+22 │ │
│ │ } │ │
│ │ version_warnings = { │ │
│ │ │ 'strategy': <class │ │
│ │ 'sqlalchemy.exc.SADeprecationWarning'>, │ │
│ │ │ 'empty_in_strategy': <class │ │
│ │ 'sqlalchemy.exc.SADeprecationWarning'>, │ │
│ │ │ 'implicit_returning': <class │ │
│ │ 'sqlalchemy.exc.SADeprecationWarning'> │ │
│ │ } │ │
│ │ versions = { │ │
│ │ │ 'strategy': '1.4', │ │
│ │ │ 'empty_in_strategy': '1.4', │ │
│ │ │ 'implicit_returning': '2.0' │ │
│ │ } │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /opt/conda/lib/python3.10/site-packages/sqlalchemy/engine/create.py:546 in │
│ create_engine │
│ │
│ 543 │ kwargs.pop("empty_in_strategy", None) │
│ 544 │ │
│ 545 │ # create url.URL object │
│ ❱ 546 │ u = _url.make_url(url) │
│ 547 │ │
│ 548 │ u, plugins, kwargs = u._instantiate_plugins(kwargs) │
│ 549 │
│ │
│ ╭──────── locals ─────────╮ │
│ │ kwargs = {'echo': True} │ │
│ │ url = None │ │
│ ╰─────────────────────────╯ │
│ │
│ /opt/conda/lib/python3.10/site-packages/sqlalchemy/engine/url.py:846 in │
│ make_url │
│ │
│ 843 │ elif not isinstance(name_or_url, URL) and not hasattr( │
│ 844 │ │ name_or_url, "_sqla_is_testing_if_this_is_a_mock_object" │
│ 845 │ ): │
│ ❱ 846 │ │ raise exc.ArgumentError( │
│ 847 │ │ │ f"Expected string or URL object, got {name_or_url!r}" │
│ 848 │ │ ) │
│ 849 │ else: │
│ │
│ ╭────── locals ──────╮ │
│ │ name_or_url = None │ │
│ ╰────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
ArgumentError: Expected string or URL object, got None
This exception is thrown by __iter__ of
OpenGSPFromDatabaseIterDataPipe(interpolate_minutes=60, load_extra_minutes=60,
national_only=False) |
|
Sorry you won't be able to run this app. You should be able to run the tests though? |
|
Gotcha. I was trying to run but i'm facing the following error: ======================================================================= short test summary info ========================================================================
ERROR tests/test_app.py - AttributeError: module 'pandas.core.indexers' has no attribute 'objects'
ERROR tests/test_data.py - AttributeError: module 'pandas.core.indexers' has no attribute 'objects'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================================================== 23 warnings, 2 errors in 8.04s ==================================================================== |
|
yea, i would try to use pandas 2.2.2, or the one just before 2.0.0 |
|
is ERROR: pip's dependency resolver dosent currently take into account all the packages that are installed. This behavior is the source of the following dependency conflicts.
torch-geometric-temporal 0.54.0 requires pandas<=1.3.5, but you have pandas 1.5.3 which is incompatible.
pvnet-summation 0.3.0 requires huggingface-hub==0.20.*, but you have huggingface-hub 0.23.5 which is incompatible.
pvnet-app 2.3.12 requires requests<2.32.0, but you have requests 2.32.3 which is incompatible. Can you give me a pip freeze of your env? |
|
please look in a recent CI, you'll be able to see the dependencies that were installed |
|
Was checking out the recent release __________________________________________________________ ERROR collecting tests/test_app.py ___________________________________________________________
ImportError while importing test module '/Users/neil/Documents/GitHub/pvnet_app/tests/test_app.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../miniconda3/envs/pvnet_env/lib/python3.10/site-packages/xesmf/util.py:8: in <module>
import esmpy as ESMF
E ModuleNotFoundError: No module named 'esmpy'
During handling of the above exception, another exception occurred:
../../../miniconda3/envs/pvnet_env/lib/python3.10/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_app.py:13: in <module>
from pvnet_app.data import sat_5_path, sat_15_path
pvnet_app/data.py:4: in <module>
import xesmf as xe
../../../miniconda3/envs/pvnet_env/lib/python3.10/site-packages/xesmf/__init__.py:3: in <module>
from . import data, util
../../../miniconda3/envs/pvnet_env/lib/python3.10/site-packages/xesmf/util.py:10: in <module>
import ESMF
E ModuleNotFoundError: No module named 'ESMF'and my envs look like: |
|
you might have to install with |
|
i was trying to avoid conda and get a hang on the dependencies and their versions. But this helped resolve the dependencies: conda install -c conda-forge xesmf esmpy -y
pip install -e . there's a new error: tests/test_app.py:13: in <module>
from pvnet_app.data import sat_5_path, sat_15_path
pvnet_app/data.py:10: in <module>
from ocf_datapipes.config.load import load_yaml_configuration
../../../miniconda3/lib/python3.11/site-packages/ocf_datapipes/__init__.py:4: in <module>
import ocf_datapipes.convert
../../../miniconda3/lib/python3.11/site-packages/ocf_datapipes/convert/__init__.py:3: in <module>
from .coordinates import ConvertGeostationaryToLonLatIterDataPipe as ConvertGeostationaryToLonLat
../../../miniconda3/lib/python3.11/site-packages/ocf_datapipes/convert/coordinates.py:8: in <module>
from ocf_datapipes.utils.geospatial import (
../../../miniconda3/lib/python3.11/site-packages/ocf_datapipes/utils/geospatial.py:9: in <module>
import pvlib
../../../miniconda3/lib/python3.11/site-packages/pvlib/__init__.py:3: in <module>
from pvlib import ( # noqa: F401
../../../miniconda3/lib/python3.11/site-packages/pvlib/spectrum/__init__.py:2: in <module>
from pvlib.spectrum.mismatch import ( # noqa: F401
../../../miniconda3/lib/python3.11/site-packages/pvlib/spectrum/mismatch.py:12: in <module>
from scipy.interpolate import interp1d
../../../miniconda3/lib/python3.11/site-packages/scipy/interpolate/__init__.py:167: in <module>
from ._interpolate import *
../../../miniconda3/lib/python3.11/site-packages/scipy/interpolate/_interpolate.py:12: in <module>
from . import _fitpack_py
../../../miniconda3/lib/python3.11/site-packages/scipy/interpolate/_fitpack_py.py:8: in <module>
from ._fitpack_impl import bisplrep, bisplev, dblint # noqa: F401
../../../miniconda3/lib/python3.11/site-packages/scipy/interpolate/_fitpack_impl.py:103: in <module>
'iwrk': array([], dfitpack_int), 'u': array([], float),
../../../miniconda3/lib/python3.11/site-packages/numpy/core/_dtype.py:46: in __repr__
arg_str = _construction_repr(dtype, include_align=False)
../../../miniconda3/lib/python3.11/site-packages/numpy/core/_dtype.py:100: in _construction_repr
return _scalar_str(dtype, short=short)
../../../miniconda3/lib/python3.11/site-packages/numpy/core/_dtype.py:143: in _scalar_str
elif np.issubdtype(dtype, np.number):
../../../miniconda3/lib/python3.11/site-packages/numpy/core/numerictypes.py:417: in issubdtype
arg1 = dtype(arg1).type
../../../miniconda3/lib/python3.11/site-packages/numpy/core/_dtype.py:46: in __repr__
arg_str = _construction_repr(dtype, include_align=False)
E RecursionError: maximum recursion depth exceeded
!!! Recursion detected (same locals & position)``` |
|
any hints on this? @peterdudfield |
|
oh sorry for my slow reply, what version of numpy are you using? There has been a new release and it sometimes causes some issues |
|
Thats alright :D |
|
Can you try numpy 2 |
|
alright, new day new errors So, from last time trying to solve the environment. I figured to pull the package versions from pvnet-app latest release which has upgraded the numpy and scipy and the environments looks like: envs.txt the new error looks like something to do with the docker..: |
|
do you have docker running? |
|
docker as a service, yes but the pvnet-app container, no |
|
|
sorry im not sure, are you still getting the same error? |
|
yeah getting the same thing. i'll try again removing everything and begin the process back from recloning |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Detailed Description
Would be good to check the forecast is a physical solar forecast. This is tricky to define but there are some quick wins we can do
Context
Possible Implementation
The text was updated successfully, but these errors were encountered: