diff --git a/README.md b/README.md index 0633245..eca14f7 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,7 @@ $ ils ``` ## Requirements + - iRODS v4.3.0 - irods-dev package - irods-runtime package @@ -50,6 +51,7 @@ $ ils - irods-externals-json package ## Compiling + ```bash $ git clone https://github.com/irods/irods_rule_engine_plugin_logical_quotas $ cd irods_rule_engine_plugin_logical_quotas @@ -64,6 +66,7 @@ irods-rule-engine-plugin-logical-quotas---. ``` ## Installing + Ubuntu: ```bash $ sudo dpkg -i irods-rule-engine-plugin-logical-quotas-*.deb @@ -79,6 +82,7 @@ should be similar to the following: ``` ## Configuration + To enable, prepend the following plugin configuration to the list of rule engines in `/etc/irods/server_config.json`. ```javascript "rule_engines": [ @@ -115,13 +119,16 @@ The plugin configuration must be placed ahead of all plugins that define any of - pep_api_data_obj_put_pre - pep_api_data_obj_rename_post - pep_api_data_obj_rename_pre -- pep_api_data_obj_unlink_ppost +- pep_api_data_obj_unlink_post - pep_api_data_obj_unlink_pre +- pep_api_mod_avu_metadata_pre - pep_api_replica_close_post - pep_api_replica_close_pre - pep_api_replica_open_pre - pep_api_rm_coll_post - pep_api_rm_coll_pre +- pep_api_touch_post +- pep_api_touch_pre Even though this plugin will process PEPs first due to its positioning, subsequent Rule Engine Plugins (REP) will still be allowed to process the same PEPs without any issues. @@ -137,6 +144,7 @@ The _data_size_ specific query may result in an overcount of bytes on an activel data object having different sizes. For this situation, consider using slightly larger quota limits. ## How To Use + **IMPORTANT NOTE:** To invoke rules provided by the plugin, the only requirement is that the user be a *rodsadmin*. The *rodsadmin* user does not need permissions set on the target collection. @@ -155,6 +163,7 @@ The following operations are supported: - logical_quotas_unset_total_size_in_bytes ### Invoking operations via the Plugin + To invoke an operation through the plugin, JSON must be passed using the following structure: ```javascript { @@ -203,12 +212,14 @@ The JSON output will be printed to the terminal and have the following structure The **keys** are derived from the **namespace** and **metadata_attribute_names** defined by the plugin configuration. ### Invoking operations via the Native Rule Language + Here, we demonstrate how to start monitoring a collection just like in the section above. ```bash $ irule -r irods_rule_engine_plugin-irods_rule_language-instance 'logical_quotas_start_monitoring_collection(*col)' '*col=/tempZone/home/rods' ruleExecOut ``` ## Stream Operations + With previous iterations of this plugin, changes in data were tracked and checked for violations across all stream-based operations in real-time. However, with the introduction of intermediate replicas and logical locking in iRODS v4.2.9, maintaining this behavior became complex. Due to the complexity, the handling of quotas has been @@ -220,3 +231,17 @@ relaxed. The most important changes are as follows: These changes have the following effects: - The plugin allows stream-based writes to violate the maximum bytes quota once. - Subsequent stream-based creates and writes will be denied until the quotas are out of violation. + +## Questions and Answers + +### Sometimes, the total number of bytes for my collection doesn't change when I remove a data object. Why? + +When it comes to tracking the total number of bytes in use, only **good** replicas are considered. If the data object being removed has no **good** replicas, the plugin will leave the total number of bytes as is. The reason for this is due to there not being a clear path forward for determining which replica's data size should be used for the update. Therefore, the recommendation is for administrators to recalculate the quota totals periodically. + +Remember, the plugin is designed to track the totals of **good** replicas only. + +### What are the rules around shared monitored nested collections? + +Anytime a user performs an operation that results in a quota update, that user MUST have **modify_object** permissions on ALL monitored parent collections. To reduce the management complexity of this, consider the following: +- Avoid monitoring collections that have parent collections which are already being monitored +- Use groups to simplify permission management diff --git a/packaging/test_rule_engine_plugin_logical_quotas.py b/packaging/test_rule_engine_plugin_logical_quotas.py index 86b6442..a1eca7e 100644 --- a/packaging/test_rule_engine_plugin_logical_quotas.py +++ b/packaging/test_rule_engine_plugin_logical_quotas.py @@ -198,7 +198,7 @@ def test_put_collection(self): # Test: No quota violations on put of a non-empty collection. self.logical_quotas_set_maximum_size_in_bytes(sandbox, '100') - self.admin1.assert_icommand(['iput', '-rf', dir_path], 'STDOUT', ['pre-scan']) + self.admin1.assert_icommand(['iput', '-rf', dir_path]) expected_number_of_objects = 3 expected_size_in_bytes = 60 self.assert_quotas(sandbox, expected_number_of_objects, expected_size_in_bytes) @@ -272,7 +272,7 @@ def test_copy_collection(self): dir_path = os.path.join(self.admin1.local_session_dir, 'col.a') file_size = 1 self.make_directory(dir_path, ['foo.txt'], file_size) - self.admin1.assert_icommand(['iput', '-r', dir_path], 'STDOUT', ['pre-scan']) + self.admin1.assert_icommand(['iput', '-r', dir_path]) # Monitor first collection. col1 = os.path.join(self.admin1.session_collection, 'col.a') @@ -831,6 +831,92 @@ def test_quotas_do_not_block_non_administrators_from_creating_or_writing_data_ob self.assert_quotas(sandbox, expected_number_of_objects = 1, expected_size_in_bytes = file_size) + @unittest.skipIf(test.settings.RUN_IN_TOPOLOGY, "Skip for Topology Testing") + def test_data_objects_containing_single_quotes_in_data_name_can_be_removed__issue_94(self): + config = IrodsConfig() + + with lib.file_backed_up(config.server_config_path): + self.enable_rule_engine_plugin(config) + + col = self.user.session_collection + self.logical_quotas_start_monitoring_collection(col) + + # Create a non-empty data object. + data_object = f"{col}/test_data_objects_containing_single_quote_can_be_removed__issue_94'" + contents = 'test_data_objects_containing_single_quote_can_be_removed__issue_94' + self.user.assert_icommand(['istream', 'write', data_object], input=contents) + self.user.assert_icommand(['ils', '-l', col], 'STDOUT') # Debugging. + self.assert_quotas(col, expected_number_of_objects=1, expected_size_in_bytes=len(contents)) + self.user.assert_icommand(['imeta', 'ls', '-C', col], 'STDOUT') # Debugging. + + # Show the data object was removed. + self.user.assert_icommand(['irm', '-f', data_object]) + self.user.assert_icommand(['ils', '-l', col], 'STDOUT') # Debugging. + self.assertFalse(lib.replica_exists(self.user, data_object, 0)) + # TODO(#113): expected_size_in_bytes will need to be updated once issue is resolved. + self.assert_quotas(col, expected_number_of_objects=0, expected_size_in_bytes=len(contents)) + self.user.assert_icommand(['imeta', 'ls', '-C', col], 'STDOUT') # Debugging. + + @unittest.skipIf(test.settings.RUN_IN_TOPOLOGY, "Skip for Topology Testing") + def test_data_objects_having_only_stale_replicas_can_be_removed__issue_111(self): + config = IrodsConfig() + + with lib.file_backed_up(config.server_config_path): + self.enable_rule_engine_plugin(config) + + col = self.user.session_collection + self.logical_quotas_start_monitoring_collection(col) + + # Create a non-empty data object. + data_object = f'{col}/test_data_objects_having_only_stale_replicas_can_be_removed__issue_111.txt' + contents = 'test_data_objects_having_only_stale_replicas_can_be_removed__issue_111' + self.user.assert_icommand(['istream', 'write', data_object], input=contents) + self.user.assert_icommand(['ils', '-l', col], 'STDOUT') # Debugging. + self.assert_quotas(col, expected_number_of_objects=1, expected_size_in_bytes=len(contents)) + self.user.assert_icommand(['imeta', 'ls', '-C', col], 'STDOUT') # Debugging. + + # Mark the replica stale. + lib.set_replica_status(self.admin1, data_object, 0, 0) + self.user.assert_icommand(['ils', '-l', col], 'STDOUT') # Debugging. + self.assertEqual(lib.get_replica_status(self.user, os.path.basename(data_object), 0), '0') + + # Show the data object was removed and the total data size was left as is. + self.user.assert_icommand(['irm', '-f', data_object]) + self.user.assert_icommand(['ils', '-l', col], 'STDOUT') # Debugging. + self.assertFalse(lib.replica_exists(self.user, data_object, 0)) + self.assert_quotas(col, expected_number_of_objects=0, expected_size_in_bytes=len(contents)) + self.user.assert_icommand(['imeta', 'ls', '-C', col], 'STDOUT') # Debugging. + + @unittest.skipIf(test.settings.RUN_IN_TOPOLOGY, "Skip for Topology Testing") + def test_rodsadmin_does_not_need_permission_on_the_collection_to_start_monitoring_it__issue_76(self): + config = IrodsConfig() + + with lib.file_backed_up(config.server_config_path): + self.enable_rule_engine_plugin(config) + + test_collection = f'{self.admin1.session_collection}/lq_col76' + self.admin1.assert_icommand(['imkdir', test_collection]) + + try: + test_group = 'lq_group76' + self.admin1.assert_icommand(['iadmin', 'mkgroup', test_group]) + + # Make it so that the group is the only entity that has permissions on the collection. + self.admin1.assert_icommand(['ichmod', 'own', test_group, test_collection]) + self.admin1.assert_icommand(['ichmod', 'null', self.admin1.username, test_collection]) + self.admin1.assert_icommand(['ils', '-A', test_collection], 'STDOUT') # Debugging. + + # Show the rodsadmin is able to instruct the plugin to start monitoring the collection + # even though they don't have permissions set on the data object. + json_string = json.dumps({'operation': 'logical_quotas_start_monitoring_collection', 'collection': test_collection}) + self.admin1.assert_icommand(['irule', '-r', 'irods_rule_engine_plugin-logical_quotas-instance', json_string, 'null', 'null']) + self.admin1.assert_icommand(['imeta', 'ls', '-C', test_collection], 'STDOUT') # Debugging. + + finally: + self.admin1.run_icommand(['iadmin', 'rmgroup', test_group]) + self.admin1.run_icommand(['ichmod', '-M', 'own', self.admin1.username, test_collection]) + self.admin1.run_icommand(['irmdir', test_collection]) + # # Utility Functions # diff --git a/src/attributes.hpp b/src/attributes.hpp index eec7de6..88afc6e 100644 --- a/src/attributes.hpp +++ b/src/attributes.hpp @@ -7,34 +7,34 @@ namespace irods { - class attributes final - { - public: - attributes(const std::string& _namespace, - const std::string& _maximum_number_of_data_objects, - const std::string& _maximum_size_in_bytes, - const std::string& _total_number_of_data_objects, - const std::string& _total_size_in_bytes) - : maximum_number_of_data_objects_{fmt::format("{}::{}", _namespace, _maximum_number_of_data_objects)} - , maximum_size_in_bytes_{fmt::format("{}::{}", _namespace, _maximum_size_in_bytes)} - , total_number_of_data_objects_{fmt::format("{}::{}", _namespace, _total_number_of_data_objects)} - , total_size_in_bytes_{fmt::format("{}::{}", _namespace, _total_size_in_bytes)} - { - } + class attributes final + { + public: + attributes(const std::string& _namespace, + const std::string& _maximum_number_of_data_objects, + const std::string& _maximum_size_in_bytes, + const std::string& _total_number_of_data_objects, + const std::string& _total_size_in_bytes) + : maximum_number_of_data_objects_{fmt::format("{}::{}", _namespace, _maximum_number_of_data_objects)} + , maximum_size_in_bytes_{fmt::format("{}::{}", _namespace, _maximum_size_in_bytes)} + , total_number_of_data_objects_{fmt::format("{}::{}", _namespace, _total_number_of_data_objects)} + , total_size_in_bytes_{fmt::format("{}::{}", _namespace, _total_size_in_bytes)} + { + } - // clang-format off - const std::string& maximum_number_of_data_objects() const { return maximum_number_of_data_objects_; } - const std::string& maximum_size_in_bytes() const { return maximum_size_in_bytes_; } - const std::string& total_number_of_data_objects() const { return total_number_of_data_objects_; } - const std::string& total_size_in_bytes() const { return total_size_in_bytes_; } - // clang-format on + // clang-format off + const std::string& maximum_number_of_data_objects() const { return maximum_number_of_data_objects_; } + const std::string& maximum_size_in_bytes() const { return maximum_size_in_bytes_; } + const std::string& total_number_of_data_objects() const { return total_number_of_data_objects_; } + const std::string& total_size_in_bytes() const { return total_size_in_bytes_; } + // clang-format on - private: - std::string maximum_number_of_data_objects_; - std::string maximum_size_in_bytes_; - std::string total_number_of_data_objects_; - std::string total_size_in_bytes_; - }; // class attributes + private: + std::string maximum_number_of_data_objects_; + std::string maximum_size_in_bytes_; + std::string total_number_of_data_objects_; + std::string total_size_in_bytes_; + }; // class attributes } // namespace irods #endif // IRODS_LOGICAL_QUOTAS_ATTRIBUTES_HPP diff --git a/src/handler.cpp b/src/handler.cpp index 2df7c8c..65f8842 100644 --- a/src/handler.cpp +++ b/src/handler.cpp @@ -41,264 +41,258 @@ namespace { - // clang-format off - namespace fs = irods::experimental::filesystem; - namespace log = irods::experimental::log; + // clang-format off + namespace fs = irods::experimental::filesystem; + namespace log = irods::experimental::log; - using size_type = irods::handler::size_type; - using quotas_info_type = std::unordered_map; - using file_position_map_type = std::unordered_map; - // clang-format on + using size_type = irods::handler::size_type; + using quotas_info_type = std::unordered_map; + using file_position_map_type = std::unordered_map; + // clang-format on - // - // Classes - // + // + // Classes + // - class parent_path - { - public: - explicit parent_path(const fs::path& _p) - : p_{_p} - { - } + class parent_path + { + public: + explicit parent_path(const fs::path& _p) + : p_{_p} + { + } + + parent_path(const parent_path&) = delete; + auto operator=(const parent_path&) -> parent_path& = delete; + + auto of(const fs::path& _child) -> bool + { + if (p_ == _child) { + return false; + } + + auto p_iter = std::begin(p_); + auto p_last = std::end(p_); + auto c_iter = std::begin(_child); + auto c_last = std::end(_child); + + for (; p_iter != p_last && c_iter != c_last && *p_iter == *c_iter; ++p_iter, ++c_iter) + ; + + return (p_iter == p_last); + } + + private: + const fs::path& p_; + }; // class parent_path + + // + // Function Prototypes + // + + auto get_monitored_collection_info(rsComm_t& _conn, const irods::attributes& _attrs, const fs::path& _p) + -> quotas_info_type; + + auto throw_if_maximum_number_of_data_objects_violation(const irods::attributes& _attrs, + const quotas_info_type& _tracking_info, + size_type _delta) -> void; + + auto throw_if_maximum_size_in_bytes_violation(const irods::attributes& _attrs, + const quotas_info_type& _tracking_info, + size_type _delta) -> void; + + auto is_monitored_collection(rsComm_t& _conn, const irods::attributes& _attrs, const fs::path& _p) -> bool; + + auto get_monitored_parent_collection(rsComm_t& _conn, const irods::attributes& _attrs, fs::path _p) + -> std::optional; + + auto compute_data_object_count_and_size(rsComm_t& _conn, fs::path _p) -> std::tuple; + + auto update_data_object_count_and_size(rsComm_t& _conn, + const irods::attributes& _attrs, + const fs::path& _collection, + const quotas_info_type& _info, + size_type _data_objects_delta, + size_type _size_in_bytes_delta) -> void; + + auto unset_metadata_impl(const std::string& _instance_name, + std::list& _rule_arguments, + irods::callback& _effect_handler, + std::unordered_map& _instance_configs, + std::function(const irods::attributes& _attrs)> _func) + -> irods::error; + + template + auto get_pointer(std::list& _rule_arguments, int _index = 2) -> T*; + + template + auto for_each_monitored_collection(rsComm_t& _conn, + const irods::attributes& _attrs, + fs::path _collection, + Function _func) -> void; + + template + auto get_attribute_value(const Map& _map, std::string_view _key) -> Value; + + auto get_instance_config(const irods::instance_configuration_map& _map, std::string_view _key) + -> const irods::instance_configuration&; + + auto make_unique_id(fs::path _p) -> std::string; + + auto throw_if_string_cannot_be_cast_to_an_integer(const std::string& s, const std::string& error_msg) -> void; + + auto is_group(rsComm_t& _conn, const std::string_view _entity_name) -> bool; + + auto log_logical_quotas_exception(const irods::logical_quotas_error& e, irods::callback& _effect_handler) + -> irods::error; + + auto log_irods_exception(const irods::exception& e, irods::callback& _effect_handler) -> irods::error; + + auto log_exception(const std::exception& e, irods::callback& _effect_handler) -> irods::error; + + // + // Function Implementations + // - parent_path(const parent_path&) = delete; - auto operator=(const parent_path&) -> parent_path& = delete; + auto get_monitored_collection_info(rsComm_t& _conn, const irods::attributes& _attrs, const fs::path& _p) + -> quotas_info_type + { + quotas_info_type info; + + const auto gql = + fmt::format("select META_COLL_ATTR_NAME, META_COLL_ATTR_VALUE where COLL_NAME = '{}'", _p.c_str()); + + for (auto&& row : irods::query{&_conn, gql}) { + // clang-format off + if (_attrs.maximum_number_of_data_objects() == row[0]) { info[_attrs.maximum_number_of_data_objects()] = std::stoll(row[1]); } + else if (_attrs.maximum_size_in_bytes() == row[0]) { info[_attrs.maximum_size_in_bytes()] = std::stoll(row[1]); } + else if (_attrs.total_number_of_data_objects() == row[0]) { info[_attrs.total_number_of_data_objects()] = std::stoll(row[1]); } + else if (_attrs.total_size_in_bytes() == row[0]) { info[_attrs.total_size_in_bytes()] = std::stoll(row[1]); } + // clang-format on + } + + return info; + } - auto of(const fs::path& _child) -> bool - { - if (p_ == _child) { - return false; - } + auto throw_if_maximum_number_of_data_objects_violation(const irods::attributes& _attrs, + const quotas_info_type& _tracking_info, + size_type _delta) -> void + { + const auto& max_attr_name = _attrs.maximum_number_of_data_objects(); + + if (const auto iter = _tracking_info.find(max_attr_name); iter != std::end(_tracking_info)) { + const auto total = get_attribute_value(_tracking_info, _attrs.total_number_of_data_objects()); - auto p_iter = std::begin(p_); - auto p_last = std::end(p_); - auto c_iter = std::begin(_child); - auto c_last = std::end(_child); + if (total + _delta > iter->second) { + throw irods::logical_quotas_error{ + "Logical Quotas Policy Violation: Adding object exceeds maximum number of objects limit", + SYS_NOT_ALLOWED}; + } + } + } + + auto throw_if_maximum_size_in_bytes_violation(const irods::attributes& _attrs, + const quotas_info_type& _tracking_info, + size_type _delta) -> void + { + const auto& max_attr_name = _attrs.maximum_size_in_bytes(); + + if (const auto iter = _tracking_info.find(max_attr_name); iter != std::end(_tracking_info)) { + const auto total = get_attribute_value(_tracking_info, _attrs.total_size_in_bytes()); + + if (total + _delta > iter->second) { + throw irods::logical_quotas_error{ + "Logical Quotas Policy Violation: Adding object exceeds maximum data size in bytes limit", + SYS_NOT_ALLOWED}; + } + } + } - for (; p_iter != p_last && c_iter != c_last && *p_iter == *c_iter; ++p_iter, ++c_iter); + auto is_monitored_collection(rsComm_t& _conn, const irods::attributes& _attrs, const fs::path& _p) -> bool + { + const auto gql = + fmt::format("select META_COLL_ATTR_NAME where COLL_NAME = '{}' and META_COLL_ATTR_NAME = '{}' || = '{}'", + _p.c_str(), + _attrs.total_number_of_data_objects(), + _attrs.total_size_in_bytes()); + + for (auto&& row : irods::query{&_conn, gql}) { + return true; + } - return (p_iter == p_last); - } + return false; + } - private: - const fs::path& p_; - }; // class parent_path + auto get_monitored_parent_collection(rsComm_t& _conn, const irods::attributes& _attrs, fs::path _p) + -> std::optional + { + for (; !_p.empty(); _p = _p.parent_path()) { + if (is_monitored_collection(_conn, _attrs, _p)) { + return _p; + } + else if ("/" == _p) { + break; + } + } - // - // Function Prototypes - // + return std::nullopt; + } - auto get_collection_username(rsComm_t& _conn, fs::path _p) -> std::optional; + auto compute_data_object_count_and_size(rsComm_t& _conn, fs::path _p) -> std::tuple + { + size_type objects = 0; + size_type bytes = 0; - auto get_monitored_collection_info(rsComm_t& _conn, - const irods::attributes& _attrs, - const fs::path& _p) -> quotas_info_type; + const auto gql = + fmt::format("select count(DATA_NAME), sum(DATA_SIZE) where COLL_NAME = '{0}' || like '{0}/%'", _p.c_str()); - auto throw_if_maximum_number_of_data_objects_violation(const irods::attributes& _attrs, - const quotas_info_type& _tracking_info, - size_type _delta) -> void; + for (auto&& row : irods::query{&_conn, gql}) { + objects = !row[0].empty() ? std::stoll(row[0]) : 0; + bytes = !row[1].empty() ? std::stoll(row[1]) : 0; + } - auto throw_if_maximum_size_in_bytes_violation(const irods::attributes& _attrs, - const quotas_info_type& _tracking_info, - size_type _delta) -> void; + return {objects, bytes}; + } - auto is_monitored_collection(rsComm_t& _conn, - const irods::attributes& _attrs, - const fs::path& _p) -> bool; + auto update_data_object_count_and_size(rsComm_t& _conn, + const irods::attributes& _attrs, + const fs::path& _collection, + const quotas_info_type& _info, + size_type _data_objects_delta, + size_type _size_in_bytes_delta) -> void + { + if (0 != _data_objects_delta) { + const auto& objects_attr = _attrs.total_number_of_data_objects(); - auto get_monitored_parent_collection(rsComm_t& _conn, - const irods::attributes& _attrs, - fs::path _p) -> std::optional; + if (const auto iter = _info.find(objects_attr); std::end(_info) != iter) { + const auto new_object_count = std::to_string(iter->second + _data_objects_delta); + fs::server::set_metadata(_conn, _collection, {objects_attr, new_object_count}); + } + } - auto compute_data_object_count_and_size(rsComm_t& _conn, fs::path _p) -> std::tuple; + if (0 != _size_in_bytes_delta) { + const auto& size_attr = _attrs.total_size_in_bytes(); - auto update_data_object_count_and_size(rsComm_t& _conn, - const irods::attributes& _attrs, - const fs::path& _collection, - const quotas_info_type& _info, - size_type _data_objects_delta, - size_type _size_in_bytes_delta) -> void; + if (const auto iter = _info.find(size_attr); std::end(_info) != iter) { + const auto new_size_in_bytes = std::to_string(iter->second + _size_in_bytes_delta); + fs::server::set_metadata(_conn, _collection, {size_attr, new_size_in_bytes}); + } + } + } - auto unset_metadata_impl(const std::string& _instance_name, - std::list& _rule_arguments, - irods::callback& _effect_handler, - std::unordered_map& _instance_configs, - std::function (const irods::attributes& _attrs)> _func) -> irods::error; - - template - auto get_pointer(std::list& _rule_arguments, int _index = 2) -> T*; - - template - auto for_each_monitored_collection(rsComm_t& _conn, - const irods::attributes& _attrs, - fs::path _collection, - Function _func) -> void; - - template - auto get_attribute_value(const Map& _map, std::string_view _key) -> Value; - - auto get_instance_config(const irods::instance_configuration_map& _map, - std::string_view _key) -> const irods::instance_configuration&; - - auto make_unique_id(fs::path _p) -> std::string; - - auto throw_if_string_cannot_be_cast_to_an_integer(const std::string& s, const std::string& error_msg) -> void; - - auto is_group(rsComm_t& _conn, const std::string_view _entity_name) -> bool; - - auto log_logical_quotas_exception(const irods::logical_quotas_error& e, irods::callback& _effect_handler) -> irods::error; - - auto log_irods_exception(const irods::exception& e, irods::callback& _effect_handler) -> irods::error; - - auto log_exception(const std::exception& e, irods::callback& _effect_handler) -> irods::error; - - // - // Function Implementations - // - - auto get_collection_username(rsComm_t& _conn, fs::path _p) -> std::optional - { - const auto gql = fmt::format("select COLL_OWNER_NAME where COLL_NAME = '{}'", _p.c_str()); + auto unset_metadata_impl(const std::string& _instance_name, + const irods::instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + irods::callback& _effect_handler, + std::function(const irods::attributes& _attrs)> _func) + -> irods::error + { + try { + auto args_iter = std::begin(_rule_arguments); + const auto& path = *boost::any_cast(*args_iter); - for (auto&& row : irods::query{&_conn, gql}) { - return row[0]; - } - - return std::nullopt; - } - - auto get_monitored_collection_info(rsComm_t& _conn, const irods::attributes& _attrs, const fs::path& _p) -> quotas_info_type - { - quotas_info_type info; - - const auto gql = fmt::format("select META_COLL_ATTR_NAME, META_COLL_ATTR_VALUE where COLL_NAME = '{}'", _p.c_str()); - - for (auto&& row : irods::query{&_conn, gql}) { - // clang-format off - if (_attrs.maximum_number_of_data_objects() == row[0]) { info[_attrs.maximum_number_of_data_objects()] = std::stoll(row[1]); } - else if (_attrs.maximum_size_in_bytes() == row[0]) { info[_attrs.maximum_size_in_bytes()] = std::stoll(row[1]); } - else if (_attrs.total_number_of_data_objects() == row[0]) { info[_attrs.total_number_of_data_objects()] = std::stoll(row[1]); } - else if (_attrs.total_size_in_bytes() == row[0]) { info[_attrs.total_size_in_bytes()] = std::stoll(row[1]); } - // clang-format on - } - - return info; - } - - auto throw_if_maximum_number_of_data_objects_violation(const irods::attributes& _attrs, - const quotas_info_type& _tracking_info, - size_type _delta) -> void - { - const auto& max_attr_name = _attrs.maximum_number_of_data_objects(); - - if (const auto iter = _tracking_info.find(max_attr_name); iter != std::end(_tracking_info)) { - const auto total = get_attribute_value(_tracking_info, _attrs.total_number_of_data_objects()); - - if (total + _delta > iter->second) { - throw irods::logical_quotas_error{"Logical Quotas Policy Violation: Adding object exceeds maximum number of objects limit", - SYS_NOT_ALLOWED}; - } - } - } - - auto throw_if_maximum_size_in_bytes_violation(const irods::attributes& _attrs, - const quotas_info_type& _tracking_info, - size_type _delta) -> void - { - const auto& max_attr_name = _attrs.maximum_size_in_bytes(); - - if (const auto iter = _tracking_info.find(max_attr_name); iter != std::end(_tracking_info)) { - const auto total = get_attribute_value(_tracking_info, _attrs.total_size_in_bytes()); - - if (total + _delta > iter->second) { - throw irods::logical_quotas_error{"Logical Quotas Policy Violation: Adding object exceeds maximum data size in bytes limit", - SYS_NOT_ALLOWED}; - } - } - } - - auto is_monitored_collection(rsComm_t& _conn, const irods::attributes& _attrs, const fs::path& _p) -> bool - { - const auto gql = fmt::format("select META_COLL_ATTR_NAME where COLL_NAME = '{}' and META_COLL_ATTR_NAME = '{}' || = '{}'", - _p.c_str(), - _attrs.total_number_of_data_objects(), - _attrs.total_size_in_bytes()); - - for (auto&& row : irods::query{&_conn, gql}) { - return true; - } - - return false; - } - - auto get_monitored_parent_collection(rsComm_t& _conn, const irods::attributes& _attrs, fs::path _p) -> std::optional - { - for (; !_p.empty(); _p = _p.parent_path()) { - if (is_monitored_collection(_conn, _attrs, _p)) { - return _p; - } - else if ("/" == _p) { - break; - } - } - - return std::nullopt; - } - - auto compute_data_object_count_and_size(rsComm_t& _conn, fs::path _p) -> std::tuple - { - size_type objects = 0; - size_type bytes = 0; - - const auto gql = fmt::format("select count(DATA_NAME), sum(DATA_SIZE) where COLL_NAME = '{0}' || like '{0}/%'", _p.c_str()); - - for (auto&& row : irods::query{&_conn, gql}) { - objects = !row[0].empty() ? std::stoll(row[0]) : 0; - bytes = !row[1].empty() ? std::stoll(row[1]) : 0; - } - - return {objects, bytes}; - } - - auto update_data_object_count_and_size(rsComm_t& _conn, - const irods::attributes& _attrs, - const fs::path& _collection, - const quotas_info_type& _info, - size_type _data_objects_delta, - size_type _size_in_bytes_delta) -> void - { - if (0 != _data_objects_delta) { - const auto& objects_attr = _attrs.total_number_of_data_objects(); - - if (const auto iter = _info.find(objects_attr); std::end(_info) != iter) { - const auto new_object_count = std::to_string(iter->second + _data_objects_delta); - fs::server::set_metadata(_conn, _collection, {objects_attr, new_object_count}); - } - } - - if (0 != _size_in_bytes_delta) { - const auto& size_attr = _attrs.total_size_in_bytes(); - - if (const auto iter = _info.find(size_attr); std::end(_info) != iter) { - const auto new_size_in_bytes = std::to_string(iter->second + _size_in_bytes_delta); - fs::server::set_metadata(_conn, _collection, {size_attr, new_size_in_bytes}); - } - } - } - - auto unset_metadata_impl(const std::string& _instance_name, - const irods::instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - irods::callback& _effect_handler, - std::function (const irods::attributes& _attrs)> _func) -> irods::error - { - try { - auto args_iter = std::begin(_rule_arguments); - const auto& path = *boost::any_cast(*args_iter); - - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); const auto info = get_monitored_collection_info(conn, attrs, path); @@ -312,112 +306,113 @@ namespace } } catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return SUCCESS(); - } - - template - auto get_pointer(std::list& _rule_arguments, int _index) -> T* - { - return boost::any_cast(*std::next(std::begin(_rule_arguments), _index)); - } - - template - auto for_each_monitored_collection(rsComm_t& _conn, - const irods::attributes& _attrs, - fs::path _collection, - Function _func) -> void - { - for (auto collection = get_monitored_parent_collection(_conn, _attrs, _collection.parent_path()); - collection; - collection = get_monitored_parent_collection(_conn, _attrs, collection->parent_path())) - { - auto info = get_monitored_collection_info(_conn, _attrs, *collection); - _func(*collection, info); - } - } - - template - auto get_attribute_value(const Map& _map, std::string_view _key) -> Value - { - if (const auto iter = _map.find(_key.data()); std::end(_map) != iter) { - return iter->second; - } - - throw std::runtime_error{fmt::format("Logical Quotas Policy: Failed to find metadata [{}]", _key)}; - } - - auto get_instance_config(const irods::instance_configuration_map& _map, - std::string_view _key) -> const irods::instance_configuration& - { - try { - return _map.at(_key.data()); - } - catch (const std::out_of_range&) { - throw std::runtime_error{fmt::format("Logical Quotas Policy: Failed to find configuration for " - "rule engine plugin instance [{}]", _key)}; - } - } - - auto make_unique_id(fs::path _p) -> std::string - { - std::string id = "irods_logical_quotas-"; - id += std::to_string(std::hash{}(_p.c_str())); - id += "-"; - id += std::to_string(getpid()); - - return id; - } - - auto throw_if_string_cannot_be_cast_to_an_integer(const std::string& s, const std::string& error_msg) -> void - { - try { - std::stoll(s); // TODO Could be replaced with std::from_chars when it is available. - } - catch (const std::invalid_argument&) { - throw std::invalid_argument{error_msg}; - } - catch (const std::out_of_range&) { - throw std::out_of_range{error_msg}; - } - } - - auto is_group(rsComm_t& _conn, const std::string_view _entity_name) -> bool - { - const auto gql = fmt::format("select USER_TYPE where USER_NAME = '{}'", _entity_name); - - for (auto&& row : irods::query{&_conn, gql}) { - return "rodsgroup" == row[0]; - } - - return false; - } - - auto log_logical_quotas_exception(const irods::logical_quotas_error& e, irods::callback& _effect_handler) -> irods::error - { - log::rule_engine::error(e.what()); - addRErrorMsg(&get_rei(_effect_handler).rsComm->rError, e.error_code(), e.what()); - return ERROR(e.error_code(), e.what()); - } - - auto log_irods_exception(const irods::exception& e, irods::callback& _effect_handler) -> irods::error - { - log::rule_engine::error(e.what()); - addRErrorMsg(&get_rei(_effect_handler).rsComm->rError, e.code(), e.client_display_what()); - return e; - } - - auto log_exception(const std::exception& e, irods::callback& _effect_handler) -> irods::error - { - log::rule_engine::error(e.what()); - addRErrorMsg(&get_rei(_effect_handler).rsComm->rError, RE_RUNTIME_ERROR, e.what()); - return ERROR(RE_RUNTIME_ERROR, e.what()); - } + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return SUCCESS(); + } + + template + auto get_pointer(std::list& _rule_arguments, int _index) -> T* + { + return boost::any_cast(*std::next(std::begin(_rule_arguments), _index)); + } + + template + auto for_each_monitored_collection(rsComm_t& _conn, + const irods::attributes& _attrs, + fs::path _collection, + Function _func) -> void + { + for (auto collection = get_monitored_parent_collection(_conn, _attrs, _collection.parent_path()); collection; + collection = get_monitored_parent_collection(_conn, _attrs, collection->parent_path())) + { + auto info = get_monitored_collection_info(_conn, _attrs, *collection); + _func(*collection, info); + } + } + + template + auto get_attribute_value(const Map& _map, std::string_view _key) -> Value + { + if (const auto iter = _map.find(_key.data()); std::end(_map) != iter) { + return iter->second; + } + + throw std::runtime_error{fmt::format("Logical Quotas Policy: Failed to find metadata [{}]", _key)}; + } + + auto get_instance_config(const irods::instance_configuration_map& _map, std::string_view _key) + -> const irods::instance_configuration& + { + try { + return _map.at(_key.data()); + } + catch (const std::out_of_range&) { + throw std::runtime_error{fmt::format("Logical Quotas Policy: Failed to find configuration for " + "rule engine plugin instance [{}]", + _key)}; + } + } + + auto make_unique_id(fs::path _p) -> std::string + { + std::string id = "irods_logical_quotas-"; + id += std::to_string(std::hash{}(_p.c_str())); + id += "-"; + id += std::to_string(getpid()); + + return id; + } + + auto throw_if_string_cannot_be_cast_to_an_integer(const std::string& s, const std::string& error_msg) -> void + { + try { + std::stoll(s); // TODO Could be replaced with std::from_chars when it is available. + } + catch (const std::invalid_argument&) { + throw std::invalid_argument{error_msg}; + } + catch (const std::out_of_range&) { + throw std::out_of_range{error_msg}; + } + } + + auto is_group(rsComm_t& _conn, const std::string_view _entity_name) -> bool + { + const auto gql = fmt::format("select USER_TYPE where USER_NAME = '{}'", _entity_name); + + for (auto&& row : irods::query{&_conn, gql}) { + return "rodsgroup" == row[0]; + } + + return false; + } + + auto log_logical_quotas_exception(const irods::logical_quotas_error& e, irods::callback& _effect_handler) + -> irods::error + { + log::rule_engine::error(e.what()); + addRErrorMsg(&get_rei(_effect_handler).rsComm->rError, e.error_code(), e.what()); + return ERROR(e.error_code(), e.what()); + } + + auto log_irods_exception(const irods::exception& e, irods::callback& _effect_handler) -> irods::error + { + log::rule_engine::error(e.what()); + addRErrorMsg(&get_rei(_effect_handler).rsComm->rError, e.code(), e.client_display_what()); + return e; + } + + auto log_exception(const std::exception& e, irods::callback& _effect_handler) -> irods::error + { + log::rule_engine::error(e.what()); + addRErrorMsg(&get_rei(_effect_handler).rsComm->rError, RE_RUNTIME_ERROR, e.what()); + return ERROR(RE_RUNTIME_ERROR, e.what()); + } auto get_quota_value_for_collection(RcComm& _conn, const std::string& _coll_path, const std::string& _quota_name) -> std::tuple @@ -471,25 +466,29 @@ namespace namespace irods::handler { - auto logical_quotas_get_collection_status(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - try { - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); - - auto args_iter = std::begin(_rule_arguments); - const auto& path = *boost::any_cast(*args_iter); - - if (!is_monitored_collection(conn, attrs, path)) { - THROW(SYS_INVALID_INPUT_PARAM, fmt::format("Logical Quotas Policy: [{}] is not a monitored collection.", path)); - } - - auto quota_status = nlohmann::json::object(); // Holds the current quota values. + auto logical_quotas_get_collection_status(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + try { + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); + + auto args_iter = std::begin(_rule_arguments); + const auto& path = *boost::any_cast(*args_iter); + + if (!is_monitored_collection(conn, attrs, path)) { + auto msg = fmt::format("Logical Quotas Policy: [{}] is not a monitored collection.", path); + log::rule_engine::error(msg); + constexpr auto ec = SYS_INVALID_INPUT_PARAM; + addRErrorMsg(&get_rei(_effect_handler).rsComm->rError, ec, msg.c_str()); + return ERROR(ec, std::move(msg)); + } + + auto quota_status = nlohmann::json::object(); // Holds the current quota values. // Fetch the current quota values for the collection. irods::experimental::client_connection client_conn; @@ -511,89 +510,95 @@ namespace irods::handler // exec_rule_expression, this parameter will point to a valid object. The exec_rule_text/expression // functions reply on this parameter to return information back to the client. if (_ms_param_array) { - if (auto* msp = getMsParamByLabel(_ms_param_array, "ruleExecOut"); msp) { - // Free any resources previously associated with the parameter. - if (msp->type) { std::free(msp->type); } - if (msp->inOutStruct) { std::free(msp->inOutStruct); } - - // Set the correct type information and allocate enough memory for that type. - msp->type = strdup(ExecCmdOut_MS_T); - msp->inOutStruct = std::malloc(sizeof(ExecCmdOut)); - - auto* out = static_cast(msp->inOutStruct); - std::memset(out, 0, sizeof(ExecCmdOut)); - - // Copy the JSON string into the output object. - const auto json_string = quota_status.dump(); - const auto buffer_size = json_string.size() + 1; - out->stdoutBuf.len = buffer_size; - out->stdoutBuf.buf = std::malloc(sizeof(char) * buffer_size); - std::memcpy(out->stdoutBuf.buf, json_string.data(), buffer_size); - } - else { - auto* out = static_cast(std::malloc(sizeof(ExecCmdOut))); - std::memset(out, 0, sizeof(ExecCmdOut)); - - // Copy the JSON string into the output object. - const auto json_string = quota_status.dump(); - const auto buffer_size = json_string.size() + 1; - out->stdoutBuf.len = buffer_size; - out->stdoutBuf.buf = std::malloc(sizeof(char) * buffer_size); - std::memcpy(out->stdoutBuf.buf, json_string.data(), buffer_size); - - addMsParamToArray(_ms_param_array, "ruleExecOut", ExecCmdOut_MS_T, out, nullptr, 0); - } - } - // If "_ms_param_array" is not set, then the rule must have been invoked via exec_rule. The client must provide - // a second variable so that the results can be returned. - else if (_rule_arguments.size() == 2) { - *boost::any_cast(*std::next(args_iter)) = quota_status.dump(); - } - else { - return ERROR(RE_UNABLE_TO_WRITE_VAR, "Logical Quotas Policy: Missing output variable for status."); - } - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return SUCCESS(); - } - - auto logical_quotas_start_monitoring_collection(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - return logical_quotas_recalculate_totals(_instance_name, _instance_configs, _rule_arguments, _ms_param_array, _effect_handler); - } - - auto logical_quotas_stop_monitoring_collection(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - return unset_metadata_impl(_instance_name, _instance_configs, _rule_arguments, _effect_handler, [](const auto& _attrs) { - return std::vector{&_attrs.total_number_of_data_objects(), &_attrs.total_size_in_bytes()}; - }); - } - - auto logical_quotas_count_total_number_of_data_objects(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - try { - auto args_iter = std::begin(_rule_arguments); - const auto& path = *boost::any_cast(*args_iter); - - auto& rei = get_rei(_effect_handler); + if (auto* msp = getMsParamByLabel(_ms_param_array, "ruleExecOut"); msp) { + // Free any resources previously associated with the parameter. + if (msp->type) { + std::free(msp->type); + } + if (msp->inOutStruct) { + std::free(msp->inOutStruct); + } + + // Set the correct type information and allocate enough memory for that type. + msp->type = strdup(ExecCmdOut_MS_T); + msp->inOutStruct = std::malloc(sizeof(ExecCmdOut)); + + auto* out = static_cast(msp->inOutStruct); + std::memset(out, 0, sizeof(ExecCmdOut)); + + // Copy the JSON string into the output object. + const auto json_string = quota_status.dump(); + const auto buffer_size = json_string.size() + 1; + out->stdoutBuf.len = buffer_size; + out->stdoutBuf.buf = std::malloc(sizeof(char) * buffer_size); + std::memcpy(out->stdoutBuf.buf, json_string.data(), buffer_size); + } + else { + auto* out = static_cast(std::malloc(sizeof(ExecCmdOut))); + std::memset(out, 0, sizeof(ExecCmdOut)); + + // Copy the JSON string into the output object. + const auto json_string = quota_status.dump(); + const auto buffer_size = json_string.size() + 1; + out->stdoutBuf.len = buffer_size; + out->stdoutBuf.buf = std::malloc(sizeof(char) * buffer_size); + std::memcpy(out->stdoutBuf.buf, json_string.data(), buffer_size); + + addMsParamToArray(_ms_param_array, "ruleExecOut", ExecCmdOut_MS_T, out, nullptr, 0); + } + } + // If "_ms_param_array" is not set, then the rule must have been invoked via exec_rule. The client must + // provide a second variable so that the results can be returned. + else if (_rule_arguments.size() == 2) { + *boost::any_cast(*std::next(args_iter)) = quota_status.dump(); + } + else { + return ERROR(RE_UNABLE_TO_WRITE_VAR, "Logical Quotas Policy: Missing output variable for status."); + } + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return SUCCESS(); + } + + auto logical_quotas_start_monitoring_collection(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + return logical_quotas_recalculate_totals( + _instance_name, _instance_configs, _rule_arguments, _ms_param_array, _effect_handler); + } + + auto logical_quotas_stop_monitoring_collection(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + return unset_metadata_impl( + _instance_name, _instance_configs, _rule_arguments, _effect_handler, [](const auto& _attrs) { + return std::vector{&_attrs.total_number_of_data_objects(), &_attrs.total_size_in_bytes()}; + }); + } + + auto logical_quotas_count_total_number_of_data_objects(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + try { + auto args_iter = std::begin(_rule_arguments); + const auto& path = *boost::any_cast(*args_iter); + + auto& rei = get_rei(_effect_handler); std::vector args{path + '%'}; auto query = irods::experimental::query_builder{} @@ -613,26 +618,26 @@ namespace irods::handler fs::admin, conn, path, {attrs.total_number_of_data_objects(), objects.empty() ? "0" : objects}); } catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return SUCCESS(); - } - - auto logical_quotas_count_total_size_in_bytes(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - try { - auto args_iter = std::begin(_rule_arguments); - const auto& path = *boost::any_cast(*args_iter); - - auto& rei = get_rei(_effect_handler); + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return SUCCESS(); + } + + auto logical_quotas_count_total_size_in_bytes(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + try { + auto args_iter = std::begin(_rule_arguments); + const auto& path = *boost::any_cast(*args_iter); + + auto& rei = get_rei(_effect_handler); std::vector args{path + '%'}; auto query = irods::experimental::query_builder{} @@ -651,42 +656,44 @@ namespace irods::handler fs::client::set_metadata(fs::admin, conn, path, {attrs.total_size_in_bytes(), bytes.empty() ? "0" : bytes}); } catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return SUCCESS(); - } - - auto logical_quotas_recalculate_totals(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - auto functions = {logical_quotas_count_total_number_of_data_objects, - logical_quotas_count_total_size_in_bytes}; - - for (auto&& f : functions) { - if (const auto error = f(_instance_name, _instance_configs, _rule_arguments, _ms_param_array, _effect_handler); !error.ok()) { - return error; - } - } - - return SUCCESS(); - } - - auto logical_quotas_set_maximum_number_of_data_objects(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - try { - auto args_iter = std::begin(_rule_arguments); - const auto& path = *boost::any_cast(*args_iter); + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return SUCCESS(); + } + + auto logical_quotas_recalculate_totals(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + auto functions = {logical_quotas_count_total_number_of_data_objects, logical_quotas_count_total_size_in_bytes}; + + for (auto&& f : functions) { + if (const auto error = + f(_instance_name, _instance_configs, _rule_arguments, _ms_param_array, _effect_handler); + !error.ok()) + { + return error; + } + } + + return SUCCESS(); + } + + auto logical_quotas_set_maximum_number_of_data_objects(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + try { + auto args_iter = std::begin(_rule_arguments); + const auto& path = *boost::any_cast(*args_iter); const auto& max_objects = *boost::any_cast(*++args_iter); const auto msg = fmt::format( @@ -699,24 +706,24 @@ namespace irods::handler fs::admin, client_conn, path, {attrs.maximum_number_of_data_objects(), max_objects}); } catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return SUCCESS(); - } - - auto logical_quotas_set_maximum_size_in_bytes(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - try { - auto args_iter = std::begin(_rule_arguments); - const auto& path = *boost::any_cast(*args_iter); + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return SUCCESS(); + } + + auto logical_quotas_set_maximum_size_in_bytes(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + try { + auto args_iter = std::begin(_rule_arguments); + const auto& path = *boost::any_cast(*args_iter); const auto& max_bytes = *boost::any_cast(*++args_iter); const auto msg = @@ -728,897 +735,940 @@ namespace irods::handler fs::client::set_metadata(fs::admin, client_conn, path, {attrs.maximum_size_in_bytes(), max_bytes}); } catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return SUCCESS(); - } - - auto logical_quotas_unset_maximum_number_of_data_objects(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - return unset_metadata_impl(_instance_name, _instance_configs, _rule_arguments, _effect_handler, [](const auto& _attrs) { - return std::vector{&_attrs.maximum_number_of_data_objects()}; - }); - } - - auto logical_quotas_unset_maximum_size_in_bytes(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - return unset_metadata_impl(_instance_name, _instance_configs, _rule_arguments, _effect_handler, [](const auto& _attrs) { - return std::vector{&_attrs.maximum_size_in_bytes()}; - }); - } - - auto logical_quotas_unset_total_number_of_data_objects(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - return unset_metadata_impl(_instance_name, _instance_configs, _rule_arguments, _effect_handler, [](const auto& _attrs) { - return std::vector{&_attrs.total_number_of_data_objects()}; - }); - } - - auto logical_quotas_unset_total_size_in_bytes(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - return unset_metadata_impl(_instance_name, _instance_configs, _rule_arguments, _effect_handler, [](const auto& _attrs) { - return std::vector{&_attrs.total_size_in_bytes()}; - }); - } - - auto pep_api_data_obj_copy::reset() noexcept -> void - { - data_objects_ = 0; - size_in_bytes_ = 0; - } - - auto pep_api_data_obj_copy::pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - reset(); // Not needed necessarily, but here for completeness. - - try { - auto* input = get_pointer(_rule_arguments); - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); - - if (const auto status = fs::server::status(conn, input->srcDataObjInp.objPath); fs::server::is_data_object(status)) { - data_objects_ = 1; - size_in_bytes_ = fs::server::data_object_size(conn, input->srcDataObjInp.objPath); - } - else if (fs::server::is_collection(status)) { - std::tie(data_objects_, size_in_bytes_) = compute_data_object_count_and_size(conn, input->srcDataObjInp.objPath); - } - else { - throw logical_quotas_error{"Logical Quotas Policy: Invalid object type", INVALID_OBJECT_TYPE}; - } - - for_each_monitored_collection(conn, attrs, input->destDataObjInp.objPath, [&conn, &attrs](auto& _collection, const auto& _info) { - throw_if_maximum_number_of_data_objects_violation(attrs, _info, data_objects_); - throw_if_maximum_size_in_bytes_violation(attrs, _info, size_in_bytes_); - }); - } - catch (const logical_quotas_error& e) { - return log_logical_quotas_exception(e, _effect_handler); - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_data_obj_copy::post(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - try { - auto* input = get_pointer(_rule_arguments); - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); - - for_each_monitored_collection(conn, attrs, input->destDataObjInp.objPath, [&conn, &attrs](const auto& _collection, const auto& _info) { - update_data_object_count_and_size(conn, attrs, _collection, _info, data_objects_, size_in_bytes_); - }); - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_data_obj_create_pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - try { - auto* input = get_pointer(_rule_arguments); - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); - - for_each_monitored_collection(conn, attrs, input->objPath, [&attrs, input](auto&, auto& _info) { - throw_if_maximum_number_of_data_objects_violation(attrs, _info, 1); - }); - } - catch (const logical_quotas_error& e) { - return log_logical_quotas_exception(e, _effect_handler); - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_data_obj_create_post(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - try { - auto* input = get_pointer(_rule_arguments); - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); - - for_each_monitored_collection(conn, attrs, input->objPath, [&conn, &attrs, input](const auto& _collection, const auto& _info) { - update_data_object_count_and_size(conn, attrs, _collection, _info, 1, 0); - }); - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_data_obj_put::reset() noexcept -> void - { - size_diff_ = 0; - forced_overwrite_ = false; - } - - auto pep_api_data_obj_put::pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - reset(); - - try { - auto* input = get_pointer(_rule_arguments); - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); - - if (fs::server::exists(*rei.rsComm, input->objPath)) { - forced_overwrite_ = true; - const size_type existing_size = fs::server::data_object_size(conn, input->objPath); - size_diff_ = static_cast(input->dataSize) - existing_size; - - for_each_monitored_collection(conn, attrs, input->objPath, [&conn, &attrs, input](const auto& _collection, auto& _info) { - throw_if_maximum_size_in_bytes_violation(attrs, _info, size_diff_); - }); - } - else { - for_each_monitored_collection(conn, attrs, input->objPath, [&attrs, input](auto&, auto& _info) { - throw_if_maximum_number_of_data_objects_violation(attrs, _info, 1); - throw_if_maximum_size_in_bytes_violation(attrs, _info, input->dataSize); - }); - } - } - catch (const logical_quotas_error& e) { - return log_logical_quotas_exception(e, _effect_handler); - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_data_obj_put::post(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - try { - auto* input = get_pointer(_rule_arguments); - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); - - if (forced_overwrite_) { - for_each_monitored_collection(conn, attrs, input->objPath, [&conn, &attrs, input](const auto& _collection, const auto& _info) { - update_data_object_count_and_size(conn, attrs, _collection, _info, 0, size_diff_); - }); - } - else { - for_each_monitored_collection(conn, attrs, input->objPath, [&conn, &attrs, input](const auto& _collection, const auto& _info) { - update_data_object_count_and_size(conn, attrs, _collection, _info, 1, input->dataSize); - }); - } - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_data_obj_rename::reset() noexcept -> void - { - data_objects_ = 0; - size_in_bytes_ = 0; - } - - auto pep_api_data_obj_rename::pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - reset(); - - try { - auto* input = get_pointer(_rule_arguments); - - // The parent of both paths are the same, then this operation is simply a rename of the - // source data object or collection. In this case, there is nothing to do. - if (fs::path{input->srcDataObjInp.objPath}.parent_path() == fs::path{input->destDataObjInp.objPath}.parent_path()) { - return CODE(RULE_ENGINE_CONTINUE); - } - - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); - - if (const auto status = fs::server::status(conn, input->srcDataObjInp.objPath); fs::server::is_data_object(status)) { - data_objects_ = 1; - size_in_bytes_ = fs::server::data_object_size(conn, input->srcDataObjInp.objPath); - } - else if (fs::server::is_collection(status)) { - std::tie(data_objects_, size_in_bytes_) = compute_data_object_count_and_size(conn, input->srcDataObjInp.objPath); - } - else { - throw logical_quotas_error{"Logical Quotas Policy: Invalid object type", INVALID_OBJECT_TYPE}; - } - - const auto in_violation = [&](const auto&, const auto& _info) - { - throw_if_maximum_number_of_data_objects_violation(attrs, _info, data_objects_); - throw_if_maximum_size_in_bytes_violation(attrs, _info, size_in_bytes_); - }; - - auto src_path = get_monitored_parent_collection(conn, attrs, input->srcDataObjInp.objPath); - auto dst_path = get_monitored_parent_collection(conn, attrs, input->destDataObjInp.objPath); - - if (src_path && dst_path) { - if (*src_path == *dst_path) { - return CODE(RULE_ENGINE_CONTINUE); - } - - // Moving object(s) from a parent collection to a child collection. - if (parent_path{*src_path}.of(*dst_path)) { - for_each_monitored_collection(conn, attrs, input->destDataObjInp.objPath, [&](const auto& _collection, const auto& _info) { - // Return immediately if "_collection" is equal to "*src_path". At this point, - // there is no need to check if any quotas will be violated. The totals will not - // change for parents of the source collection. - if (_collection == *src_path) { - return; - } - - throw_if_maximum_number_of_data_objects_violation(attrs, _info, data_objects_); - throw_if_maximum_size_in_bytes_violation(attrs, _info, size_in_bytes_); - }); - } - // Moving object(s) from a child collection to a parent collection. - else if (parent_path{*dst_path}.of(*src_path)) { - for_each_monitored_collection(conn, attrs, input->destDataObjInp.objPath, in_violation); - } - // Moving objects(s) between unrelated collection trees. - else { - for_each_monitored_collection(conn, attrs, input->destDataObjInp.objPath, in_violation); - } - } - else if (dst_path) { - using namespace std::string_literals; - for_each_monitored_collection(conn, attrs, input->destDataObjInp.objPath, in_violation); - } - } - catch (const logical_quotas_error& e) { - return log_logical_quotas_exception(e, _effect_handler); - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_data_obj_rename::post(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - // There is no change in state, therefore return immediately. - if (0 == data_objects_ && 0 == size_in_bytes_) { - return CODE(RULE_ENGINE_CONTINUE); - } - - try { - auto* input = get_pointer(_rule_arguments); - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); - auto src_path = get_monitored_parent_collection(conn, attrs, input->srcDataObjInp.objPath); - auto dst_path = get_monitored_parent_collection(conn, attrs, input->destDataObjInp.objPath); - - // Cases - // ~~~~~ - // * src_path and dst_path are monitored paths. - // - src_path and dst_path are the same path - // + Do nothing - // - // - src_path is the parent of dst_path - // + Update dst_path's metadata - // - // - dst_path is the parent of src_path - // + Update the src_path's metadata - // - // * src_path is monitored, but dst_path is not. - // - Update the src_path's metadata - // - // * dst_path is monitored, but src_path is not. - // - Update the dst_path's metadata - // - // * src_path and dst_path are not monitored paths. - // - Do nothing - - if (src_path && dst_path) { - if (*src_path == *dst_path) { - return CODE(RULE_ENGINE_CONTINUE); - } - - // Moving object(s) from a parent collection to a child collection. - if (parent_path{*src_path}.of(*dst_path)) { - auto info = get_monitored_collection_info(conn, attrs, *dst_path); - update_data_object_count_and_size(conn, attrs, *dst_path, info, data_objects_, size_in_bytes_); - } - // Moving object(s) from a child collection to a parent collection. - else if (parent_path{*dst_path}.of(*src_path)) { - auto info = get_monitored_collection_info(conn, attrs, *src_path); - update_data_object_count_and_size(conn, attrs, *src_path, info, -data_objects_, -size_in_bytes_); - } - // Moving objects(s) between unrelated collection trees. - else { - for_each_monitored_collection(conn, attrs, input->destDataObjInp.objPath, [&](const auto& _collection, const auto& _info) { - update_data_object_count_and_size(conn, attrs, _collection, _info, data_objects_, size_in_bytes_); - }); - - for_each_monitored_collection(conn, attrs, input->srcDataObjInp.objPath, [&](const auto& _collection, const auto& _info) { - update_data_object_count_and_size(conn, attrs, _collection, _info, -data_objects_, -size_in_bytes_); - }); - } - } - else if (src_path) { - for_each_monitored_collection(conn, attrs, input->srcDataObjInp.objPath, [&](const auto& _collection, const auto& _info) { - update_data_object_count_and_size(conn, attrs, _collection, _info, -data_objects_, -size_in_bytes_); - }); - } - else if (dst_path) { - for_each_monitored_collection(conn, attrs, input->destDataObjInp.objPath, [&](const auto& _collection, const auto& _info) { - update_data_object_count_and_size(conn, attrs, _collection, _info, data_objects_, size_in_bytes_); - }); - } - } - catch (const logical_quotas_error& e) { - return log_logical_quotas_exception(e, _effect_handler); - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_data_obj_unlink::reset() noexcept -> void - { - size_in_bytes_ = 0; - } - - auto pep_api_data_obj_unlink::pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - reset(); - - try { - auto* input = get_pointer(_rule_arguments); - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); - - if (auto collection = get_monitored_parent_collection(conn, attrs, input->objPath); collection) { - size_in_bytes_ = fs::server::data_object_size(conn, input->objPath); - } - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_data_obj_unlink::post(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - try { - auto* input = get_pointer(_rule_arguments); - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); - - for_each_monitored_collection(conn, attrs, input->objPath, [&conn, &attrs, input](const auto& _collection, const auto& _info) { - update_data_object_count_and_size(conn, attrs, _collection, _info, -1, -size_in_bytes_); - }); - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_data_obj_open_pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - try { - auto* input = get_pointer(_rule_arguments); - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); - - if (O_CREAT == (input->openFlags & O_CREAT)) { - if (!fs::server::exists(*rei.rsComm, input->objPath)) { - for_each_monitored_collection(conn, attrs, input->objPath, [&attrs, input](auto&, auto& _info) { - throw_if_maximum_number_of_data_objects_violation(attrs, _info, 1); - }); - } - } - // Opening an existing data object for reading is fine as long as it does not result in - // the creation of a new data object. - else if (O_RDONLY == (input->openFlags & O_ACCMODE)) { - return CODE(RULE_ENGINE_CONTINUE); - } - - // Because streaming operations can result in byte quotas being exceeded, the REP must - // verify that the quotas have not been violated by a previous streaming operation. This - // is because the REP does not track bytes written during streaming operations. - for_each_monitored_collection(conn, attrs, input->objPath, [&attrs, input](auto&, auto& _info) { - // We only need to check the byte count here. If the rest of the REP is implemented - // correctly, then the data object count should be in line already. - throw_if_maximum_size_in_bytes_violation(attrs, _info, 0); - }); - } - catch (const logical_quotas_error& e) { - return log_logical_quotas_exception(e, _effect_handler); - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_data_obj_close::reset() noexcept -> void - { - path_.clear(); - } - - auto pep_api_data_obj_close::pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - reset(); - - try { - auto* input = get_pointer(_rule_arguments); - const auto& l1desc = irods::get_l1desc(input->l1descInx); - - // Return immediately if the client opened an existing data object for reading. - // This avoids unnecessary catalog updates. - if (const auto flags = l1desc.dataObjInp->openFlags; - O_RDONLY == (flags & O_ACCMODE) && O_CREAT != (flags & O_CREAT)) - { - return CODE(RULE_ENGINE_CONTINUE); - } - - path_ = l1desc.dataObjInfo->objPath; - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_data_obj_close::post(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - try { - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); - - // If the path is empty, either the pre-PEP detected that the client opened an - // existing data object for reading and returned early, or an error occurred. - // This avoids unnecessary catalog updates. - if (path_.empty()) { - return CODE(RULE_ENGINE_CONTINUE); - } - - for_each_monitored_collection(conn, attrs, path_, [&](auto& _collection, const auto& _info) { - std::string p = fs::path{path_}.parent_path(); - std::list args{&p}; - const auto err = logical_quotas_recalculate_totals(_instance_name, - _instance_configs, - args, - _ms_param_array, - _effect_handler); - - if (!err.ok()) { - THROW(err.code(), err.result()); - } - }); - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_mod_avu_metadata_pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - try { - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - const auto* input = get_pointer(_rule_arguments); - - if (std::string_view{"add"} != input->arg0 || !fs::server::is_collection(conn, input->arg2)) { - return CODE(RULE_ENGINE_CONTINUE); - } - - const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); - - const auto attr_list = {&attrs.maximum_number_of_data_objects(), - &attrs.maximum_size_in_bytes(), - &attrs.total_number_of_data_objects(), - &attrs.total_size_in_bytes()}; - - const auto iter = std::find_if(std::begin(attr_list), std::end(attr_list), - [attr_name = std::string_view{input->arg3}](const std::string* _attr) { - return *_attr == attr_name; - }); - - if (iter != std::end(attr_list)) { - const auto gql = fmt::format("select META_COLL_ATTR_NAME " - "where COLL_NAME = '{}' and META_COLL_ATTR_NAME = '{}'", - input->arg2, **iter); - - if (irods::query{&conn, gql}.size() > 0) { - return ERROR(SYS_NOT_ALLOWED, "Logical Quotas Policy: Metadata attribute name already defined."); - } - } - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_replica_close::reset() noexcept -> void - { - path_.clear(); - } - - auto pep_api_replica_close::pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - reset(); - - try { - auto* input = get_pointer(_rule_arguments); - const auto json_input = nlohmann::json::parse(std::string_view(static_cast(input->buf), input->len)); - const auto& l1desc = irods::get_l1desc(json_input.at("fd").get()); - - // Return immediately if the client opened an existing data object for reading. - // This avoids unnecessary catalog updates. - if (const auto flags = l1desc.dataObjInp->openFlags; - O_RDONLY == (flags & O_ACCMODE) && O_CREAT != (flags & O_CREAT)) - { - return CODE(RULE_ENGINE_CONTINUE); - } - - path_ = l1desc.dataObjInfo->objPath; - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_replica_close::post(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - try { - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); - - // If the path is empty, either the pre-PEP detected that the client opened an - // existing data object for reading and returned early, or an error occurred. - // This avoids unnecessary catalog updates. - if (path_.empty()) { - return CODE(RULE_ENGINE_CONTINUE); - } - - for_each_monitored_collection(conn, attrs, path_, [&](auto& _collection, const auto& _info) { - std::string p = fs::path{path_}.parent_path(); - std::list args{&p}; - const auto err = logical_quotas_recalculate_totals(_instance_name, - _instance_configs, - args, - _ms_param_array, - _effect_handler); - - if (!err.ok()) { - THROW(err.code(), err.result()); - } - }); - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_rm_coll::reset() noexcept -> void - { - data_objects_ = 0; - size_in_bytes_ = 0; - } - - auto pep_api_rm_coll::pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - reset(); - - try { - auto* input = get_pointer(_rule_arguments); - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); - - if (auto collection = get_monitored_parent_collection(conn, attrs, input->collName); collection) { - std::tie(data_objects_, size_in_bytes_) = compute_data_object_count_and_size(conn, input->collName); - } - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_rm_coll::post(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - try { - auto* input = get_pointer(_rule_arguments); - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); - - for_each_monitored_collection(conn, attrs, input->collName, [&conn, &attrs, input](const auto& _collection, const auto& _info) { - update_data_object_count_and_size(conn, attrs, _collection, _info, -data_objects_, -size_in_bytes_); - }); - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_touch::reset() noexcept -> void - { - path_.clear(); - exists_ = false; - } - - auto pep_api_touch::pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - reset(); - - try { - auto* input = get_pointer(_rule_arguments); - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - - const auto json_input = nlohmann::json::parse(std::string_view(static_cast(input->buf), input->len)); - path_ = json_input.at("logical_path").get(); - exists_ = fs::server::exists(conn, path_); - } - catch (const fs::filesystem_error& e) { - rodsLog(LOG_ERROR, e.what()); - addRErrorMsg(&get_rei(_effect_handler).rsComm->rError, e.code().value(), e.what()); - return ERROR(e.code().value(), e.what()); - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto pep_api_touch::post(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error - { - try { - auto& rei = get_rei(_effect_handler); - auto& conn = *rei.rsComm; - - // Verify that the target object was created. This is necessary because the touch API - // does not always result in a new data object (i.e. no_create JSON option). - if (!exists_ && fs::server::exists(conn, path_)) { - const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); - - for_each_monitored_collection(conn, attrs, path_, [&conn, &attrs](const auto& _collection, const auto& _info) { - update_data_object_count_and_size(conn, attrs, _collection, _info, 1, 0); - }); - } - } - catch (const fs::filesystem_error& e) { - rodsLog(LOG_ERROR, e.what()); - addRErrorMsg(&get_rei(_effect_handler).rsComm->rError, e.code().value(), e.what()); - return ERROR(e.code().value(), e.what()); - } - catch (const irods::exception& e) { - return log_irods_exception(e, _effect_handler); - } - catch (const std::exception& e) { - return log_exception(e, _effect_handler); - } - - return CODE(RULE_ENGINE_CONTINUE); - } -} // namespace irods::handler + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return SUCCESS(); + } + + auto logical_quotas_unset_maximum_number_of_data_objects(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + return unset_metadata_impl( + _instance_name, _instance_configs, _rule_arguments, _effect_handler, [](const auto& _attrs) { + return std::vector{&_attrs.maximum_number_of_data_objects()}; + }); + } + + auto logical_quotas_unset_maximum_size_in_bytes(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + return unset_metadata_impl( + _instance_name, _instance_configs, _rule_arguments, _effect_handler, [](const auto& _attrs) { + return std::vector{&_attrs.maximum_size_in_bytes()}; + }); + } + + auto logical_quotas_unset_total_number_of_data_objects(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + return unset_metadata_impl( + _instance_name, _instance_configs, _rule_arguments, _effect_handler, [](const auto& _attrs) { + return std::vector{&_attrs.total_number_of_data_objects()}; + }); + } + + auto logical_quotas_unset_total_size_in_bytes(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + return unset_metadata_impl( + _instance_name, _instance_configs, _rule_arguments, _effect_handler, [](const auto& _attrs) { + return std::vector{&_attrs.total_size_in_bytes()}; + }); + } + + auto pep_api_data_obj_copy::reset() noexcept -> void + { + data_objects_ = 0; + size_in_bytes_ = 0; + } + + auto pep_api_data_obj_copy::pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + reset(); // Not needed necessarily, but here for completeness. + + try { + auto* input = get_pointer(_rule_arguments); + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); + + if (const auto status = fs::server::status(conn, input->srcDataObjInp.objPath); + fs::server::is_data_object(status)) { + data_objects_ = 1; + size_in_bytes_ = fs::server::data_object_size(conn, input->srcDataObjInp.objPath); + } + else if (fs::server::is_collection(status)) { + std::tie(data_objects_, size_in_bytes_) = + compute_data_object_count_and_size(conn, input->srcDataObjInp.objPath); + } + else { + throw logical_quotas_error{"Logical Quotas Policy: Invalid object type", INVALID_OBJECT_TYPE}; + } + + for_each_monitored_collection( + conn, attrs, input->destDataObjInp.objPath, [&conn, &attrs](auto& _collection, const auto& _info) { + throw_if_maximum_number_of_data_objects_violation(attrs, _info, data_objects_); + throw_if_maximum_size_in_bytes_violation(attrs, _info, size_in_bytes_); + }); + } + catch (const logical_quotas_error& e) { + return log_logical_quotas_exception(e, _effect_handler); + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + + auto pep_api_data_obj_copy::post(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + try { + auto* input = get_pointer(_rule_arguments); + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); + + for_each_monitored_collection(conn, + attrs, + input->destDataObjInp.objPath, + [&conn, &attrs](const auto& _collection, const auto& _info) { + update_data_object_count_and_size( + conn, attrs, _collection, _info, data_objects_, size_in_bytes_); + }); + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + auto pep_api_data_obj_create_pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + try { + auto* input = get_pointer(_rule_arguments); + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); + + for_each_monitored_collection(conn, attrs, input->objPath, [&attrs, input](auto&, auto& _info) { + throw_if_maximum_number_of_data_objects_violation(attrs, _info, 1); + }); + } + catch (const logical_quotas_error& e) { + return log_logical_quotas_exception(e, _effect_handler); + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + + auto pep_api_data_obj_create_post(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + try { + auto* input = get_pointer(_rule_arguments); + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); + + for_each_monitored_collection( + conn, attrs, input->objPath, [&conn, &attrs, input](const auto& _collection, const auto& _info) { + update_data_object_count_and_size(conn, attrs, _collection, _info, 1, 0); + }); + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + + auto pep_api_data_obj_put::reset() noexcept -> void + { + size_diff_ = 0; + forced_overwrite_ = false; + } + + auto pep_api_data_obj_put::pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + reset(); + + try { + auto* input = get_pointer(_rule_arguments); + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); + + if (fs::server::exists(*rei.rsComm, input->objPath)) { + forced_overwrite_ = true; + const size_type existing_size = fs::server::data_object_size(conn, input->objPath); + size_diff_ = static_cast(input->dataSize) - existing_size; + + for_each_monitored_collection( + conn, attrs, input->objPath, [&conn, &attrs, input](const auto& _collection, auto& _info) { + throw_if_maximum_size_in_bytes_violation(attrs, _info, size_diff_); + }); + } + else { + for_each_monitored_collection(conn, attrs, input->objPath, [&attrs, input](auto&, auto& _info) { + throw_if_maximum_number_of_data_objects_violation(attrs, _info, 1); + throw_if_maximum_size_in_bytes_violation(attrs, _info, input->dataSize); + }); + } + } + catch (const logical_quotas_error& e) { + return log_logical_quotas_exception(e, _effect_handler); + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + + auto pep_api_data_obj_put::post(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + try { + auto* input = get_pointer(_rule_arguments); + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); + + if (forced_overwrite_) { + for_each_monitored_collection( + conn, attrs, input->objPath, [&conn, &attrs, input](const auto& _collection, const auto& _info) { + update_data_object_count_and_size(conn, attrs, _collection, _info, 0, size_diff_); + }); + } + else { + for_each_monitored_collection( + conn, attrs, input->objPath, [&conn, &attrs, input](const auto& _collection, const auto& _info) { + update_data_object_count_and_size(conn, attrs, _collection, _info, 1, input->dataSize); + }); + } + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + + auto pep_api_data_obj_rename::reset() noexcept -> void + { + data_objects_ = 0; + size_in_bytes_ = 0; + } + + auto pep_api_data_obj_rename::pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + reset(); + + try { + auto* input = get_pointer(_rule_arguments); + + // The parent of both paths are the same, then this operation is simply a rename of the + // source data object or collection. In this case, there is nothing to do. + if (fs::path{input->srcDataObjInp.objPath}.parent_path() == + fs::path{input->destDataObjInp.objPath}.parent_path()) { + return CODE(RULE_ENGINE_CONTINUE); + } + + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); + + if (const auto status = fs::server::status(conn, input->srcDataObjInp.objPath); + fs::server::is_data_object(status)) { + data_objects_ = 1; + size_in_bytes_ = fs::server::data_object_size(conn, input->srcDataObjInp.objPath); + } + else if (fs::server::is_collection(status)) { + std::tie(data_objects_, size_in_bytes_) = + compute_data_object_count_and_size(conn, input->srcDataObjInp.objPath); + } + else { + throw logical_quotas_error{"Logical Quotas Policy: Invalid object type", INVALID_OBJECT_TYPE}; + } + + const auto in_violation = [&](const auto&, const auto& _info) { + throw_if_maximum_number_of_data_objects_violation(attrs, _info, data_objects_); + throw_if_maximum_size_in_bytes_violation(attrs, _info, size_in_bytes_); + }; + + auto src_path = get_monitored_parent_collection(conn, attrs, input->srcDataObjInp.objPath); + auto dst_path = get_monitored_parent_collection(conn, attrs, input->destDataObjInp.objPath); + + if (src_path && dst_path) { + if (*src_path == *dst_path) { + return CODE(RULE_ENGINE_CONTINUE); + } + + // Moving object(s) from a parent collection to a child collection. + if (parent_path{*src_path}.of(*dst_path)) { + for_each_monitored_collection( + conn, attrs, input->destDataObjInp.objPath, [&](const auto& _collection, const auto& _info) { + // Return immediately if "_collection" is equal to "*src_path". At this point, + // there is no need to check if any quotas will be violated. The totals will not + // change for parents of the source collection. + if (_collection == *src_path) { + return; + } + + throw_if_maximum_number_of_data_objects_violation(attrs, _info, data_objects_); + throw_if_maximum_size_in_bytes_violation(attrs, _info, size_in_bytes_); + }); + } + // Moving object(s) from a child collection to a parent collection. + else if (parent_path{*dst_path}.of(*src_path)) { + for_each_monitored_collection(conn, attrs, input->destDataObjInp.objPath, in_violation); + } + // Moving objects(s) between unrelated collection trees. + else { + for_each_monitored_collection(conn, attrs, input->destDataObjInp.objPath, in_violation); + } + } + else if (dst_path) { + using namespace std::string_literals; + for_each_monitored_collection(conn, attrs, input->destDataObjInp.objPath, in_violation); + } + } + catch (const logical_quotas_error& e) { + return log_logical_quotas_exception(e, _effect_handler); + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + + auto pep_api_data_obj_rename::post(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + // There is no change in state, therefore return immediately. + if (0 == data_objects_ && 0 == size_in_bytes_) { + return CODE(RULE_ENGINE_CONTINUE); + } + + try { + auto* input = get_pointer(_rule_arguments); + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); + auto src_path = get_monitored_parent_collection(conn, attrs, input->srcDataObjInp.objPath); + auto dst_path = get_monitored_parent_collection(conn, attrs, input->destDataObjInp.objPath); + + // Cases + // ~~~~~ + // * src_path and dst_path are monitored paths. + // - src_path and dst_path are the same path + // + Do nothing + // + // - src_path is the parent of dst_path + // + Update dst_path's metadata + // + // - dst_path is the parent of src_path + // + Update the src_path's metadata + // + // * src_path is monitored, but dst_path is not. + // - Update the src_path's metadata + // + // * dst_path is monitored, but src_path is not. + // - Update the dst_path's metadata + // + // * src_path and dst_path are not monitored paths. + // - Do nothing + + if (src_path && dst_path) { + if (*src_path == *dst_path) { + return CODE(RULE_ENGINE_CONTINUE); + } + + // Moving object(s) from a parent collection to a child collection. + if (parent_path{*src_path}.of(*dst_path)) { + auto info = get_monitored_collection_info(conn, attrs, *dst_path); + update_data_object_count_and_size(conn, attrs, *dst_path, info, data_objects_, size_in_bytes_); + } + // Moving object(s) from a child collection to a parent collection. + else if (parent_path{*dst_path}.of(*src_path)) { + auto info = get_monitored_collection_info(conn, attrs, *src_path); + update_data_object_count_and_size(conn, attrs, *src_path, info, -data_objects_, -size_in_bytes_); + } + // Moving objects(s) between unrelated collection trees. + else { + for_each_monitored_collection( + conn, attrs, input->destDataObjInp.objPath, [&](const auto& _collection, const auto& _info) { + update_data_object_count_and_size( + conn, attrs, _collection, _info, data_objects_, size_in_bytes_); + }); + + for_each_monitored_collection( + conn, attrs, input->srcDataObjInp.objPath, [&](const auto& _collection, const auto& _info) { + update_data_object_count_and_size( + conn, attrs, _collection, _info, -data_objects_, -size_in_bytes_); + }); + } + } + else if (src_path) { + for_each_monitored_collection( + conn, attrs, input->srcDataObjInp.objPath, [&](const auto& _collection, const auto& _info) { + update_data_object_count_and_size( + conn, attrs, _collection, _info, -data_objects_, -size_in_bytes_); + }); + } + else if (dst_path) { + for_each_monitored_collection( + conn, attrs, input->destDataObjInp.objPath, [&](const auto& _collection, const auto& _info) { + update_data_object_count_and_size( + conn, attrs, _collection, _info, data_objects_, size_in_bytes_); + }); + } + } + catch (const logical_quotas_error& e) { + return log_logical_quotas_exception(e, _effect_handler); + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + + auto pep_api_data_obj_unlink::reset() noexcept -> void + { + size_in_bytes_ = 0; + } + + auto pep_api_data_obj_unlink::pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + reset(); + + try { + auto* input = get_pointer(_rule_arguments); + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); + + if (auto collection = get_monitored_parent_collection(conn, attrs, input->objPath); collection) { + try { + size_in_bytes_ = fs::server::data_object_size(conn, input->objPath); + } + catch (const fs::filesystem_error& e) { + // The filesystem library's data_object_size() function will throw an exception + // if the data object does not have any good replicas. Because the REP is designed + // to track the data size of good replicas, the only reasonable step is to leave + // the data size as is. + // + // Attempting to define rules for handling data objects without good replicas + // contains too many challenges. Relying on the admin to recalculate the totals is + // the best/safest approach. + if (e.code().value() == SYS_NO_GOOD_REPLICA) { + log::rule_engine::info("Logical Quotas: Removal of data object [{}] will not affect total " + "number of bytes. Data object does not have any good replicas.", + input->objPath); + } + else { + log::rule_engine::error(e.what()); + addRErrorMsg(&get_rei(_effect_handler).rsComm->rError, e.code().value(), e.what()); + return ERROR(e.code().value(), e.what()); + } + } + } + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + + auto pep_api_data_obj_unlink::post(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + try { + auto* input = get_pointer(_rule_arguments); + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); + + for_each_monitored_collection( + conn, attrs, input->objPath, [&conn, &attrs, input](const auto& _collection, const auto& _info) { + update_data_object_count_and_size(conn, attrs, _collection, _info, -1, -size_in_bytes_); + }); + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + + auto pep_api_data_obj_open_pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + try { + auto* input = get_pointer(_rule_arguments); + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); + + if (O_CREAT == (input->openFlags & O_CREAT)) { + if (!fs::server::exists(*rei.rsComm, input->objPath)) { + for_each_monitored_collection(conn, attrs, input->objPath, [&attrs, input](auto&, auto& _info) { + throw_if_maximum_number_of_data_objects_violation(attrs, _info, 1); + }); + } + } + // Opening an existing data object for reading is fine as long as it does not result in + // the creation of a new data object. + else if (O_RDONLY == (input->openFlags & O_ACCMODE)) { + return CODE(RULE_ENGINE_CONTINUE); + } + + // Because streaming operations can result in byte quotas being exceeded, the REP must + // verify that the quotas have not been violated by a previous streaming operation. This + // is because the REP does not track bytes written during streaming operations. + for_each_monitored_collection(conn, attrs, input->objPath, [&attrs, input](auto&, auto& _info) { + // We only need to check the byte count here. If the rest of the REP is implemented + // correctly, then the data object count should be in line already. + throw_if_maximum_size_in_bytes_violation(attrs, _info, 0); + }); + } + catch (const logical_quotas_error& e) { + return log_logical_quotas_exception(e, _effect_handler); + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + + auto pep_api_data_obj_close::reset() noexcept -> void + { + path_.clear(); + } + + auto pep_api_data_obj_close::pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + reset(); + + try { + auto* input = get_pointer(_rule_arguments); + const auto& l1desc = irods::get_l1desc(input->l1descInx); + + // Return immediately if the client opened an existing data object for reading. + // This avoids unnecessary catalog updates. + if (const auto flags = l1desc.dataObjInp->openFlags; + O_RDONLY == (flags & O_ACCMODE) && O_CREAT != (flags & O_CREAT)) { + return CODE(RULE_ENGINE_CONTINUE); + } + + path_ = l1desc.dataObjInfo->objPath; + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + + auto pep_api_data_obj_close::post(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + try { + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); + + // If the path is empty, either the pre-PEP detected that the client opened an + // existing data object for reading and returned early, or an error occurred. + // This avoids unnecessary catalog updates. + if (path_.empty()) { + return CODE(RULE_ENGINE_CONTINUE); + } + + for_each_monitored_collection(conn, attrs, path_, [&](auto& _collection, const auto& _info) { + std::string p = fs::path{path_}.parent_path(); + std::list args{&p}; + const auto err = logical_quotas_recalculate_totals( + _instance_name, _instance_configs, args, _ms_param_array, _effect_handler); + + if (!err.ok()) { + THROW(err.code(), err.result()); + } + }); + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + + auto pep_api_mod_avu_metadata_pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + try { + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + const auto* input = get_pointer(_rule_arguments); + + if (std::string_view{"add"} != input->arg0 || !fs::server::is_collection(conn, input->arg2)) { + return CODE(RULE_ENGINE_CONTINUE); + } + + const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); + + const auto attr_list = {&attrs.maximum_number_of_data_objects(), + &attrs.maximum_size_in_bytes(), + &attrs.total_number_of_data_objects(), + &attrs.total_size_in_bytes()}; + + const auto iter = std::find_if( + std::begin(attr_list), + std::end(attr_list), + [attr_name = std::string_view{input->arg3}](const std::string* _attr) { return *_attr == attr_name; }); + + if (iter != std::end(attr_list)) { + const auto gql = fmt::format("select META_COLL_ATTR_NAME " + "where COLL_NAME = '{}' and META_COLL_ATTR_NAME = '{}'", + input->arg2, + **iter); + + if (irods::query{&conn, gql}.size() > 0) { + return ERROR(SYS_NOT_ALLOWED, "Logical Quotas Policy: Metadata attribute name already defined."); + } + } + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + + auto pep_api_replica_close::reset() noexcept -> void + { + path_.clear(); + } + + auto pep_api_replica_close::pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + reset(); + + try { + auto* input = get_pointer(_rule_arguments); + const auto json_input = nlohmann::json::parse(std::string_view(static_cast(input->buf), input->len)); + const auto& l1desc = irods::get_l1desc(json_input.at("fd").get()); + + // Return immediately if the client opened an existing data object for reading. + // This avoids unnecessary catalog updates. + if (const auto flags = l1desc.dataObjInp->openFlags; + O_RDONLY == (flags & O_ACCMODE) && O_CREAT != (flags & O_CREAT)) { + return CODE(RULE_ENGINE_CONTINUE); + } + + path_ = l1desc.dataObjInfo->objPath; + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + + auto pep_api_replica_close::post(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + try { + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); + + // If the path is empty, either the pre-PEP detected that the client opened an + // existing data object for reading and returned early, or an error occurred. + // This avoids unnecessary catalog updates. + if (path_.empty()) { + return CODE(RULE_ENGINE_CONTINUE); + } + + for_each_monitored_collection(conn, attrs, path_, [&](auto& _collection, const auto& _info) { + std::string p = fs::path{path_}.parent_path(); + std::list args{&p}; + const auto err = logical_quotas_recalculate_totals( + _instance_name, _instance_configs, args, _ms_param_array, _effect_handler); + + if (!err.ok()) { + THROW(err.code(), err.result()); + } + }); + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + + auto pep_api_rm_coll::reset() noexcept -> void + { + data_objects_ = 0; + size_in_bytes_ = 0; + } + + auto pep_api_rm_coll::pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + reset(); + + try { + auto* input = get_pointer(_rule_arguments); + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); + + if (auto collection = get_monitored_parent_collection(conn, attrs, input->collName); collection) { + std::tie(data_objects_, size_in_bytes_) = compute_data_object_count_and_size(conn, input->collName); + } + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + + auto pep_api_rm_coll::post(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + try { + auto* input = get_pointer(_rule_arguments); + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); + + for_each_monitored_collection( + conn, attrs, input->collName, [&conn, &attrs, input](const auto& _collection, const auto& _info) { + update_data_object_count_and_size(conn, attrs, _collection, _info, -data_objects_, -size_in_bytes_); + }); + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + + auto pep_api_touch::reset() noexcept -> void + { + path_.clear(); + exists_ = false; + } + + auto pep_api_touch::pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + reset(); + + try { + auto* input = get_pointer(_rule_arguments); + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + + const auto json_input = nlohmann::json::parse(std::string_view(static_cast(input->buf), input->len)); + path_ = json_input.at("logical_path").get(); + exists_ = fs::server::exists(conn, path_); + } + catch (const fs::filesystem_error& e) { + rodsLog(LOG_ERROR, e.what()); + addRErrorMsg(&get_rei(_effect_handler).rsComm->rError, e.code().value(), e.what()); + return ERROR(e.code().value(), e.what()); + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } + + auto pep_api_touch::post(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error + { + try { + auto& rei = get_rei(_effect_handler); + auto& conn = *rei.rsComm; + + // Verify that the target object was created. This is necessary because the touch API + // does not always result in a new data object (i.e. no_create JSON option). + if (!exists_ && fs::server::exists(conn, path_)) { + const auto& attrs = get_instance_config(_instance_configs, _instance_name).attributes(); + + for_each_monitored_collection( + conn, attrs, path_, [&conn, &attrs](const auto& _collection, const auto& _info) { + update_data_object_count_and_size(conn, attrs, _collection, _info, 1, 0); + }); + } + } + catch (const fs::filesystem_error& e) { + rodsLog(LOG_ERROR, e.what()); + addRErrorMsg(&get_rei(_effect_handler).rsComm->rError, e.code().value(), e.what()); + return ERROR(e.code().value(), e.what()); + } + catch (const irods::exception& e) { + return log_irods_exception(e, _effect_handler); + } + catch (const std::exception& e) { + return log_exception(e, _effect_handler); + } + + return CODE(RULE_ENGINE_CONTINUE); + } +} // namespace irods::handler diff --git a/src/handler.hpp b/src/handler.hpp index aec65c2..7370ef8 100644 --- a/src/handler.hpp +++ b/src/handler.hpp @@ -13,294 +13,295 @@ namespace irods::handler { - // clang-format off - using size_type = std::int64_t; - using file_position_type = std::int64_t; - // clang-format on - - auto logical_quotas_get_collection_status(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - auto logical_quotas_start_monitoring_collection(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - auto logical_quotas_stop_monitoring_collection(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - auto logical_quotas_count_total_number_of_data_objects(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - auto logical_quotas_count_total_size_in_bytes(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - auto logical_quotas_recalculate_totals(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - auto logical_quotas_set_maximum_number_of_data_objects(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - auto logical_quotas_set_maximum_size_in_bytes(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - auto logical_quotas_unset_maximum_number_of_data_objects(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - auto logical_quotas_unset_maximum_size_in_bytes(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - auto logical_quotas_unset_total_number_of_data_objects(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - auto logical_quotas_unset_total_size_in_bytes(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - class pep_api_data_obj_copy final - { - public: - pep_api_data_obj_copy() = delete; - - static auto reset() noexcept -> void; - - static auto pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - static auto post(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - private: - inline static size_type data_objects_ = 0; - inline static size_type size_in_bytes_ = 0; - }; // class pep_api_data_obj_copy - - auto pep_api_data_obj_create_pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - auto pep_api_data_obj_create_post(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - class pep_api_data_obj_put final - { - public: - pep_api_data_obj_put() = delete; - - static auto reset() noexcept -> void; - - static auto pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - static auto post(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - private: - inline static size_type size_diff_ = 0; - inline static bool forced_overwrite_ = false; - }; // class pep_api_data_obj_put - - class pep_api_data_obj_rename final - { - public: - pep_api_data_obj_rename() = delete; - - static auto reset() noexcept -> void; - - static auto pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - static auto post(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - private: - inline static size_type data_objects_ = 0; - inline static size_type size_in_bytes_ = 0; - }; // class pep_api_data_obj_rename - - class pep_api_data_obj_unlink final - { - public: - pep_api_data_obj_unlink() = delete; - - static auto reset() noexcept -> void; - - static auto pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - static auto post(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - private: - inline static size_type size_in_bytes_ = 0; - }; // class pep_api_data_obj_unlink - - auto pep_api_data_obj_open_pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - class pep_api_data_obj_close final - { - public: - pep_api_data_obj_close() = delete; - - static auto reset() noexcept -> void; - - static auto pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - static auto post(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - private: - inline static std::string path_; - }; // class pep_api_data_obj_close - - auto pep_api_mod_avu_metadata_pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - class pep_api_replica_close final - { - public: - pep_api_replica_close() = delete; - - static auto reset() noexcept -> void; - - static auto pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - static auto post(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - private: - inline static std::string path_; - }; // class pep_api_replica_close - - class pep_api_rm_coll final - { - public: - pep_api_rm_coll() = delete; - - static auto reset() noexcept -> void; - - static auto pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - static auto post(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - private: - inline static size_type data_objects_ = 0; - inline static size_type size_in_bytes_ = 0; - }; // class pep_api_rm_coll - - class pep_api_touch final - { - public: - pep_api_touch() = delete; - - static auto reset() noexcept -> void; - - static auto pre(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - static auto post(const std::string& _instance_name, - const instance_configuration_map& _instance_configs, - std::list& _rule_arguments, - MsParamArray* _ms_param_array, - irods::callback& _effect_handler) -> irods::error; - - private: - inline static std::string path_; - inline static bool exists_ = false; - }; // class pep_api_touch + // clang-format off + using size_type = std::int64_t; + using file_position_type = std::int64_t; + // clang-format on + + auto logical_quotas_get_collection_status(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + auto logical_quotas_start_monitoring_collection(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + auto logical_quotas_stop_monitoring_collection(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + auto logical_quotas_count_total_number_of_data_objects(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + auto logical_quotas_count_total_size_in_bytes(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + auto logical_quotas_recalculate_totals(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + auto logical_quotas_set_maximum_number_of_data_objects(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + auto logical_quotas_set_maximum_size_in_bytes(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + auto logical_quotas_unset_maximum_number_of_data_objects(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + auto logical_quotas_unset_maximum_size_in_bytes(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + auto logical_quotas_unset_total_number_of_data_objects(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + auto logical_quotas_unset_total_size_in_bytes(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + class pep_api_data_obj_copy final + { + public: + pep_api_data_obj_copy() = delete; + + static auto reset() noexcept -> void; + + static auto pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + static auto post(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + private: + inline static size_type data_objects_ = 0; + inline static size_type size_in_bytes_ = 0; + }; // class pep_api_data_obj_copy + + auto pep_api_data_obj_create_pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + auto pep_api_data_obj_create_post(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + class pep_api_data_obj_put final + { + public: + pep_api_data_obj_put() = delete; + + static auto reset() noexcept -> void; + + static auto pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + static auto post(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + private: + inline static size_type size_diff_ = 0; + inline static bool forced_overwrite_ = false; + }; // class pep_api_data_obj_put + + class pep_api_data_obj_rename final + { + public: + pep_api_data_obj_rename() = delete; + + static auto reset() noexcept -> void; + + static auto pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + static auto post(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + private: + inline static size_type data_objects_ = 0; + inline static size_type size_in_bytes_ = 0; + }; // class pep_api_data_obj_rename + + class pep_api_data_obj_unlink final + { + public: + pep_api_data_obj_unlink() = delete; + + static auto reset() noexcept -> void; + + static auto pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + static auto post(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + private: + inline static size_type size_in_bytes_ = 0; + }; // class pep_api_data_obj_unlink + + auto pep_api_data_obj_open_pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + class pep_api_data_obj_close final + { + public: + pep_api_data_obj_close() = delete; + + static auto reset() noexcept -> void; + + static auto pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + static auto post(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + private: + inline static std::string path_; + }; // class pep_api_data_obj_close + + auto pep_api_mod_avu_metadata_pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + class pep_api_replica_close final + { + public: + pep_api_replica_close() = delete; + + static auto reset() noexcept -> void; + + static auto pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + static auto post(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + private: + inline static std::string path_; + }; // class pep_api_replica_close + + class pep_api_rm_coll final + { + public: + pep_api_rm_coll() = delete; + + static auto reset() noexcept -> void; + + static auto pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + static auto post(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + private: + inline static size_type data_objects_ = 0; + inline static size_type size_in_bytes_ = 0; + }; // class pep_api_rm_coll + + class pep_api_touch final + { + public: + pep_api_touch() = delete; + + static auto reset() noexcept -> void; + + static auto pre(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + static auto post(const std::string& _instance_name, + const instance_configuration_map& _instance_configs, + std::list& _rule_arguments, + MsParamArray* _ms_param_array, + irods::callback& _effect_handler) -> irods::error; + + private: + inline static std::string path_; + inline static bool exists_ = false; + }; // class pep_api_touch } // namespace irods::handler #endif // IRODS_LOGICAL_QUOTAS_HANDLER_HPP - diff --git a/src/instance_configuration.hpp b/src/instance_configuration.hpp index 106456e..241c7b9 100644 --- a/src/instance_configuration.hpp +++ b/src/instance_configuration.hpp @@ -8,24 +8,24 @@ namespace irods { - class instance_configuration final - { - public: - instance_configuration(attributes _attrs) noexcept - : attrs_{std::move(_attrs)} - { - } + class instance_configuration final + { + public: + instance_configuration(attributes _attrs) noexcept + : attrs_{std::move(_attrs)} + { + } - const attributes& attributes() const noexcept - { - return attrs_; - } + const attributes& attributes() const noexcept + { + return attrs_; + } - private: - class attributes attrs_; - }; // class instance_config + private: + class attributes attrs_; + }; // class instance_config - using instance_configuration_map = std::unordered_map; + using instance_configuration_map = std::unordered_map; } // namespace irods #endif // IRODS_LOGICAL_QUOTAS_INSTANCE_CONFIGURATION_HPP diff --git a/src/logical_quotas_error.hpp b/src/logical_quotas_error.hpp index 9cc3475..d0b82de 100644 --- a/src/logical_quotas_error.hpp +++ b/src/logical_quotas_error.hpp @@ -5,26 +5,25 @@ namespace irods { - class logical_quotas_error - : public std::runtime_error - { - public: - using error_code_type = int; + class logical_quotas_error : public std::runtime_error + { + public: + using error_code_type = int; - logical_quotas_error(const char* _msg, error_code_type _error_code) noexcept - : std::runtime_error{_msg} - , error_code_{_error_code} - { - } + logical_quotas_error(const char* _msg, error_code_type _error_code) noexcept + : std::runtime_error{_msg} + , error_code_{_error_code} + { + } - auto error_code() const noexcept -> error_code_type - { - return error_code_; - } + auto error_code() const noexcept -> error_code_type + { + return error_code_; + } - private: - error_code_type error_code_; - }; + private: + error_code_type error_code_; + }; } // namespace irods #endif // IRODS_LOGICAL_QUOTAS_LOGICAL_QUOTAS_ERROR_HPP diff --git a/src/main.cpp b/src/main.cpp index c9ead30..6238b30 100644 --- a/src/main.cpp +++ b/src/main.cpp @@ -24,313 +24,314 @@ namespace { - // clang-format off - namespace handler = irods::handler; - namespace log = irods::experimental::log; - - using json = nlohmann::json; - - irods::instance_configuration_map instance_configs; - - using handler_type = std::function&, - MsParamArray*, - irods::callback&)>; - - using handler_map_type = std::map; - - const handler_map_type logical_quotas_handlers{ - {"logical_quotas_count_total_number_of_data_objects", handler::logical_quotas_count_total_number_of_data_objects}, - {"logical_quotas_count_total_size_in_bytes", handler::logical_quotas_count_total_size_in_bytes}, - {"logical_quotas_recalculate_totals", handler::logical_quotas_recalculate_totals}, - {"logical_quotas_set_maximum_number_of_data_objects", handler::logical_quotas_set_maximum_number_of_data_objects}, - {"logical_quotas_set_maximum_size_in_bytes", handler::logical_quotas_set_maximum_size_in_bytes}, - {"logical_quotas_start_monitoring_collection", handler::logical_quotas_start_monitoring_collection}, - {"logical_quotas_get_collection_status", handler::logical_quotas_get_collection_status}, - {"logical_quotas_stop_monitoring_collection", handler::logical_quotas_stop_monitoring_collection}, - {"logical_quotas_unset_maximum_number_of_data_objects", handler::logical_quotas_unset_maximum_number_of_data_objects}, - {"logical_quotas_unset_maximum_size_in_bytes", handler::logical_quotas_unset_maximum_size_in_bytes}, - {"logical_quotas_unset_total_number_of_data_objects", handler::logical_quotas_unset_total_number_of_data_objects}, - {"logical_quotas_unset_total_size_in_bytes", handler::logical_quotas_unset_total_size_in_bytes} - }; - - const handler_map_type pep_handlers{ - {"pep_api_data_obj_close_post", handler::pep_api_data_obj_close::post}, - {"pep_api_data_obj_close_pre", handler::pep_api_data_obj_close::pre}, - {"pep_api_data_obj_copy_post", handler::pep_api_data_obj_copy::post}, - {"pep_api_data_obj_copy_pre", handler::pep_api_data_obj_copy::pre}, - {"pep_api_data_obj_create_and_stat_post", handler::pep_api_data_obj_create_post}, - {"pep_api_data_obj_create_and_stat_pre", handler::pep_api_data_obj_create_pre}, - {"pep_api_data_obj_create_post", handler::pep_api_data_obj_create_post}, - {"pep_api_data_obj_create_pre", handler::pep_api_data_obj_create_pre}, - {"pep_api_data_obj_open_and_stat_pre", handler::pep_api_data_obj_open_pre}, - {"pep_api_data_obj_open_pre", handler::pep_api_data_obj_open_pre}, - {"pep_api_data_obj_put_post", handler::pep_api_data_obj_put::post}, - {"pep_api_data_obj_put_pre", handler::pep_api_data_obj_put::pre}, - {"pep_api_data_obj_rename_post", handler::pep_api_data_obj_rename::post}, - {"pep_api_data_obj_rename_pre", handler::pep_api_data_obj_rename::pre}, - {"pep_api_data_obj_unlink_post", handler::pep_api_data_obj_unlink::post}, - {"pep_api_data_obj_unlink_pre", handler::pep_api_data_obj_unlink::pre}, - {"pep_api_mod_avu_metadata_pre", handler::pep_api_mod_avu_metadata_pre}, - {"pep_api_replica_close_post", handler::pep_api_replica_close::post}, - {"pep_api_replica_close_pre", handler::pep_api_replica_close::pre}, - {"pep_api_replica_open_pre", handler::pep_api_data_obj_open_pre}, - {"pep_api_rm_coll_post", handler::pep_api_rm_coll::post}, - {"pep_api_rm_coll_pre", handler::pep_api_rm_coll::pre}, - {"pep_api_touch_post", handler::pep_api_touch::post}, - {"pep_api_touch_pre", handler::pep_api_touch::pre} - }; - // clang-format on - - // - // Rule Engine Plugin - // - - template - using operation = std::function; - - auto start(irods::default_re_ctx&, const std::string& _instance_name) -> irods::error - { - std::string config_path; - - if (auto error = irods::get_full_path_for_config_file("server_config.json", config_path); - !error.ok()) - { - const char* msg = "Server configuration not found"; - - // clang-format off - log::rule_engine::error({{"rule_engine_plugin", "logical_quotas"}, - {"rule_engine_plugin_function", __func__}, - {"log_message", msg}}); - // clang-format on - - return ERROR(SYS_CONFIG_FILE_ERR, msg); - } - - // clang-format off - log::rule_engine::trace({{"rule_engine_plugin", "logical_quotas"}, - {"rule_engine_plugin_function", __func__}, - {"log_message", "Reading plugin configuration ..."}}); - // clang-format on - - json config; - - { - std::ifstream config_file{config_path}; - config_file >> config; - } - - try { - const auto get_prop = [](const json& _config, auto&& _name) -> std::string - { - using name_type = decltype(_name); - - try { - return _config.at(std::forward(_name)).template get(); - } - catch (...) { - throw std::runtime_error{fmt::format("Logical Quotas Policy: Failed to find rule engine " - "plugin configuration property [{}]", std::forward(_name))}; - } - }; - - for (const auto& re : config.at(irods::KW_CFG_PLUGIN_CONFIGURATION).at(irods::KW_CFG_PLUGIN_TYPE_RULE_ENGINE)) { - if (_instance_name == re.at(irods::KW_CFG_INSTANCE_NAME).get()) { - const auto& plugin_config = re.at(irods::KW_CFG_PLUGIN_SPECIFIC_CONFIGURATION); - - const auto& attr_names = [&plugin_config] { - try { - return plugin_config.at("metadata_attribute_names"); - } - catch (...) { - throw std::runtime_error{fmt::format("Logical Quotas Policy: Failed to find rule engine " - "plugin configuration property [metadata_attribute_names]")}; - } - }(); - - irods::instance_configuration instance_config{{get_prop(plugin_config, "namespace"), - get_prop(attr_names, "maximum_number_of_data_objects"), - get_prop(attr_names, "maximum_size_in_bytes"), - get_prop(attr_names, "total_number_of_data_objects"), - get_prop(attr_names, "total_size_in_bytes")}}; - - instance_configs.insert_or_assign(_instance_name, instance_config); - - return SUCCESS(); - } - } - } - catch (const std::exception& e) { - // clang-format off - log::rule_engine::error({{"rule_engine_plugin", "logical_quotas"}, - {"rule_engine_plugin_function", __func__}, - {"log_message", "Bad rule engine plugin configuration"}}); - // clang-format on - - return ERROR(SYS_CONFIG_FILE_ERR, e.what()); - } - - return ERROR(SYS_CONFIG_FILE_ERR, "[logical_quotas] Bad rule engine plugin configuration"); - } - - auto rule_exists(const std::string& _instance_name, - irods::default_re_ctx&, - const std::string& _rule_name, - bool& _exists) -> irods::error - { - _exists = (logical_quotas_handlers.find(_rule_name) != std::end(logical_quotas_handlers) || - pep_handlers.find(_rule_name) != std::end(pep_handlers)); - return SUCCESS(); - } - - auto list_rules(irods::default_re_ctx&, std::vector& _rules) -> irods::error - { - std::transform(std::begin(logical_quotas_handlers), - std::end(logical_quotas_handlers), - std::back_inserter(_rules), - [](auto _v) { return std::string{_v.first}; }); - - std::transform(std::begin(pep_handlers), std::end(pep_handlers), std::back_inserter(_rules), [](auto _v) { - return std::string{_v.first}; - }); - - return SUCCESS(); - } - - auto exec_rule(const std::string& _instance_name, - irods::default_re_ctx&, - const std::string& _rule_name, - std::list& _rule_arguments, - irods::callback _effect_handler) -> irods::error - { - if (const auto iter = pep_handlers.find(_rule_name); iter != std::end(pep_handlers)) { - return (iter->second)(_instance_name, instance_configs, _rule_arguments, nullptr, _effect_handler); - } - - if (const auto iter = logical_quotas_handlers.find(_rule_name); iter != std::end(logical_quotas_handlers)) { - return (iter->second)(_instance_name, instance_configs, _rule_arguments, nullptr, _effect_handler); - } - - log::rule_engine::error(fmt::format("Rule not supported in rule engine plugin [rule => {}]", _rule_name)); - - return CODE(RULE_ENGINE_CONTINUE); - } - - auto exec_rule_text_impl(const std::string& _instance_name, - std::string_view _rule_text, - MsParamArray* _ms_param_array, - irods::callback _effect_handler) -> irods::error - { - log::rule_engine::debug("_rule_text => [{}]", _rule_text); - - // irule - if (_rule_text.find("@external rule {") != std::string_view::npos) { - const auto start = _rule_text.find_first_of('{') + 1; - const auto end = _rule_text.rfind(" }"); - - if (end == std::string_view::npos) { - auto msg = fmt::format("Received malformed rule text. " - "Expected closing curly brace following rule text [{}].", - _rule_text); - log::rule_engine::error(msg); - return ERROR(SYS_INVALID_INPUT_PARAM, std::move(msg)); - } - - _rule_text = _rule_text.substr(start, end - start); - } - // irule -F