Skip to content

Commit

Permalink
Merge pull request #267 from esmf-org/esmpy_support_large_arrays
Browse files Browse the repository at this point in the history
ESMPy: Fix type cast to support large Arrays

In a few places, the size of an Array was being cast to a c_int. This caused an overflow if the size exceeded 32 bits. An example of the issue is reported here, and I have reproduced this:
https://github.com/orgs/esmf-org/discussions/263

From some reading, I think the correct cast here should be to ssize_t, which resolves to a long int (64 bits) on a 64-bit architecture. I am not 100% positive of this, but this seems consistent with what I have found; e.g., see:

Argument list in the PyMemoryView_FromMemory function: https://docs.python.org/3/c-api/memoryview.html
https://docs.python.org/3/c-api/intro.html#c.Py_ssize_t
https://stackoverflow.com/questions/20987390/cython-why-when-is-it-preferable-to-use-py-ssize-t-for-indexing
  • Loading branch information
billsacks authored Jul 31, 2024
2 parents e8332f8 + fe50b8f commit 75cee0a
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions src/addon/esmpy/src/esmpy/util/esmpyarray.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ def ndarray_from_esmf(data, dtype, shape):
if sys.version_info[0] >= 3:
buffer = ct.pythonapi.PyMemoryView_FromMemory
buffer.restype = ct.py_object
buffer = buffer(data, ct.c_int(size), 0x200)
buffer = buffer(data, ct.c_ssize_t(size), 0x200)
else:
buffer = np.core.multiarray.int_asbuffer(
ct.addressof(data.contents), size)
Expand Down Expand Up @@ -65,7 +65,7 @@ def __new__(cls, data, mask, dtype, shape):
if sys.version_info[0] >= 3:
buffer = ct.pythonapi.PyMemoryView_FromMemory
buffer.restype = ct.py_object
buffer = buffer(data, ct.c_int(size), 0x200)
buffer = buffer(data, ct.c_ssize_t(size), 0x200)
else:
buffer = np.core.multiarray.int_asbuffer(
ct.addressof(data.contents), size)
Expand Down Expand Up @@ -108,7 +108,7 @@ def __new__(cls, data, dtype, shape):
if sys.version_info[0] >= 3:
buffer = ct.pythonapi.PyMemoryView_FromMemory
buffer.restype = ct.py_object
buffer = buffer(data, ct.c_int(size), 0x200)
buffer = buffer(data, ct.c_ssize_t(size), 0x200)
else:
buffer = np.core.multiarray.int_asbuffer(
ct.addressof(data.contents), size)
Expand Down

0 comments on commit 75cee0a

Please sign in to comment.