DeePMD-kit C API Error: DeePMD-kit Error: DeePMD-kit PyTorch backend error: Method 'has_message_passing' is not defined.

[link89]

I get the following error when build C++ modules and lammps plugin from source with deempd v3.0.0b. Is there anything wrong in my build step? Thank you.

  use deepmd-kit at:  /public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/opt/deepmd
  source:             v3.0.0b0
  source branch:      HEAD
  source commit:      29db791
  source commit at:   2024-07-03 14:09:23 +0000
  build float prec:   double
  build with tf inc:  /public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/lib/python3.11/site-packages/tensorflow/include;/public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/lib/python3.11/site-packages/tensorflow/include
  build with tf lib:  /public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/lib/python3.11/site-packages/tensorflow/libtensorflow_cc.so.2
load model from: unified-dpa2.pth to gpu 0
ERROR on proc 0: DeePMD-kit C API Error: DeePMD-kit Error: DeePMD-kit PyTorch backend error: Method 'has_message_passing' is not defined.
Exception raised from get_method at /public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/include/torch/csrc/jit/api/object.h:110 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xaa (0x2b6adb40d87a in /public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xf3 (0x2b6adb3c2c7e in /public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: torch::jit::Object::get_method(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const + 0x274 (0x2b6aed4dea74 in /public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/opt/deepmd/lib/libdeepmd_cc.so)
frame #3: c10::IValue torch::jit::Object::run_method<>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x25 (0x2b6aed4dec95 in /public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/opt/deepmd/lib/libdeepmd_cc.so)
frame #4: deepmd::DeepPotPT::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x2c8 (0x2b6aed4da568 in /public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/opt/deepmd/lib/libdeepmd_cc.so)
frame #5: deepmd::DeepPotPT::translate_error(std::function<void ()>) + 0x13 (0x2b6aed4d8ad3 in /public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/opt/deepmd/lib/libdeepmd_cc.so)
frame #6: deepmd::DeepPotPT::DeepPotPT(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xf2 (0x2b6aed4d9532 in /public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/opt/deepmd/lib/libdeepmd_cc.so)
frame #7: deepmd::DeepPot::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x14b (0x2b6aed4d3d9b in /public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/opt/deepmd/lib/libdeepmd_cc.so)
frame #8: deepmd::DeepPot::DeepPot(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x19 (0x2b6aed4d3ef9 in /public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/opt/deepmd/lib/libdeepmd_cc.so)
frame #9: DP_NewDeepPotWithParam2 + 0xc4 (0x2b6adb187d94 in /public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/opt/deepmd/lib/libdeepmd_c.so)
frame #10: LAMMPS_NS::PairDeepMD::settings(int, char**) + 0x122d (0x2b6adaf4fbfd in /public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/opt/deepmd/lib/deepmd_lmp/dpplugin.so)
frame #11: LAMMPS_NS::Input::execute_command() + 0xbce (0x2b6ac335144e in /public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/opt/lammps/lib/liblammps.so.0)
frame #12: LAMMPS_NS::Input::file() + 0x13e (0x2b6ac335167e in /public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/opt/lammps/lib/liblammps.so.0)
frame #13: main + 0x4d (0x400e9d in lmp)
frame #14: __libc_start_main + 0xf5 (0x2b6ac53f9445 in /lib64/libc.so.6)
frame #15: lmp() [0x400f0e]
 (/public/groups/ai4ec/libs/conda/deepmd/3.0.0b0-cuda118/source/deepmd-kit/source/lmp/pair_deepmd.cpp:969)

The key build commands are:

cmake -DUSE_TF_PYTHON_LIBS=TRUE -DENABLE_PYTORCH=TRUE -DENABLE_TENSORFLOW=TRUE -DCMAKE_INSTALL_PREFIX=$deepmd_root -DCAFFE2_USE_CUDNN=TRUE -DCAFFE2_USE_CUSPARSELT=TRUE -DUSE_CUDA_TOOLKIT=TRUE -DLAMMPS_SOURCE_ROOT=$lammps_source_dir -DCMAKE_PREFIX_PATH=`python3 -c 'import torch;print(torch.utils.cmake_prefix_path)'` -DCUDNN_ROOT=$CONDA_PREFIX -DCUSPARSELT_ROOT=$CONDA_PREFIX ..
make -j8
make install

The full build process can be find here: https://gist.github.com/link89/51cbdcb15cdc69bc55727db63cfbe5f1

Here is the build log:
deepmd-build.txt

[njzjz]

Was your model trained by v3.0.0a0? We don't promise the backward compatibility for v3.0.0a0, as mentioned in its release note.

[link89]

I see, it's a model trained by 2024Q1 branch. I was confused by the error message which looks like kind of compilation or runtime error.