Merge pull request #24 from plone/fix-blob-cache-size

fix wrong handling of blob-cache-size

CHANGES.rst

@@ -1,6 +1,18 @@
 Changelog
 =========
 
+2.1.1
+-----
+
+- Fix broken handling of `blob-cache-size` and `blob-cache-size-check`.
+  [@jensens, 2024-07-01]
+
+- Fix: Introduce consistent naming of all `db_blob_*` and deprecate all `db_blobs_*`.
+  [@jensens, 2024-07-01]
+
+- Add a footer to all generated files to make clear where they come from, include version of `cookiecutter-zope-instance`.
+  [@jensens, 2024-07-01]
+
 2.1.0
 -----
 

README.rst

@@ -9,7 +9,7 @@ It bakes configuration for Zope 5
 Features
 ========
 
-- Creates basic file-system structure with ``zope.conf``, ``zope.ini``, ``site.zcml`` and  inital user.
+- Creates basic file-system structure with ``zope.conf``, ``zope.ini``, ``site.zcml`` and  initial user.
 - Set Zope's main configuration options.
 - Configure different database backends such as local file-system storage, ``RelStorage`` or ``ZEO``.
 - Enable development options.
@@ -68,7 +68,7 @@ Base Locations
     The target directory name of the cookiecutter generated configuration.
     This is also the so called *INSTANCEHOME*.
 
-    Attention, this is relative to current directory or to cookiecutters command line options if given (``-o PATH`` or ``--output-dir PATH``).
+    Attention, this is relative to current directory or to cookiecutter command line options if given (``-o PATH`` or ``--output-dir PATH``).
 
     Default: ``instance``
 
@@ -240,7 +240,7 @@ Zope/Plone offers different ZODB storage backends for different environments and
 - For development a simple local file based *direct* storage is all you need (aka filestorage).
 - As soon as you want multiple application processes of Zope/Plone (horizontal scaling) you need to run a separate database server process and connect to it.
 
-  - We recommend to use a Postgresql database using the *RelStorage* implementation for ZODB with *psycopg2* driver as database server in production environments.
+  - We recommend to use a PostgreSQL database using the *RelStorage* implementation for ZODB with *psycopg2* driver as database server in production environments.
     RelStorage supports very well MySQL (and derivatives), Oracle and SQLite 3 as database servers.
   - Zope and ZODB comes with *ZEO* (Zope Enterprise Objects). This more lightweight storage server is supported here too. It is widely used in production environment.
 
@@ -311,7 +311,7 @@ Blobs Settings
 
 The blob settings are valid for all storages.
 
-``db_blobs_mode``
+``db_blob_mode``
     Set if blobs are stored *shared* within all clients or are they stored on the storage backend and the client only operates as temporary *cache*.
     For *direct* storage only *shared* applies (operates like shared with one single client).
     Attention: Do not forget to set this to *cache* if you use RelStorage!
@@ -320,23 +320,23 @@ The blob settings are valid for all storages.
 
     Default: ``shared``
 
-``db_blobs_location``
-    The name of the directory where the ZODB blob data or cache (depends on *db_blobs_mode*) will be stored.
+``db_blob_location``
+    The name of the directory where the ZODB blob data or cache (depends on *db_blob_mode*) will be stored.
 
     Default: ``{{ cookiecutter.location_clienthome }}/blobs``.
 
-``db_blobs_cache_size``
+``db_blob_cache_size``
     Set the maximum size of the blob cache, in bytes.
     With many blobs and enough disk space on the client hardware this should be increased.
     If not set, then the cache size isn't checked and the blob directory will grow without bound.
-    Only valid for *db_blobs_mode* *cache*.
+    Only valid for *db_blob_mode* *cache*.
 
     Default: ``6312427520`` (5GB).
 
-``db_blobs_cache_size_check``
-    Set the ZEO check size as percent of ``blobss_cache_size`` (for example, ``10`` for 10%).
+``db_blob_cache_size_check``
+    Set the ZEO check size as percent of ``blobs_cache_size`` (for example, ``10`` for 10%).
     The ZEO cache size will be checked when this many bytes have been loaded into the cache.
-    Only valid for *db_blobs_mode* *cache*.
+    Only valid for *db_blob_mode* *cache*.
 
     Defaults: ``10`` (10% of the blob cache size).
 
@@ -390,7 +390,7 @@ RelStorage
 
 `RelStorage <https://pypi.org/project/RelStorage/>`_ is a storage implementation for ZODB that stores pickles in a relational database (RDBMS).
 
-Note: Please see `Database`_ and `Blobs Settings`_ , as you will have to set ``db_blobs_mode`` to ``cache``. 
+Note: Please see `Database`_ and `Blobs Settings`_ , as you will have to set ``db_blob_mode`` to ``cache``.
 Usually you will also have to set up the correct DSN for your database.
 
 General settings
@@ -526,14 +526,16 @@ The configuration for the scripts is generated as separate file:
 The file ``relstorage-pack.conf`` for the command line utility ``zobdpack`` is always generated for all RelStorage configurations.
 For usage information read `Packing Or Reference Checking A ZODB Storage: zodbpack <https://relstorage.readthedocs.io/en/latest/zodbpack.html>`_.
 
-The file ``relstorage-export.conf`` is generated if the two ``db_relstorage_export_*`` settings are given.
-The file ``relstorage-import.conf`` is generated if the two ``db_relstorage_import_*`` settings are given.
+The files
+- ``relstorage-export.conf`` is generated if the two ``db_relstorage_export_*`` settings are given, and
+- ``relstorage-import.conf`` is generated if the two ``db_relstorage_import_*`` settings are given.
+
 Both are for the command line utility ``zobdconvert``.
 For usage information read `Copying Data Between ZODB Storages: zodbconvert <https://relstorage.readthedocs.io/en/latest/zodbconvert.html>`_
 
 At the moment only the filestorage with blobs is supported.
 In future there may be more options, like converting from/to a ZEO-server or another RelStorage/Database.
-Latter would be useful to upgrade a database or convert MyQL to Postgresql or vice versa.
+Latter would be useful to upgrade a database or convert MySQL to PostgreSQL or vice versa.
 
 ``db_relstorage_import_filestorage_location``
     The filename of the filestorage to import from.
@@ -914,7 +916,7 @@ Then we set a bunch of environment variables for production:
     export INSTANCE_debug_mode=false
     export INSTANCE_verbose_security=false
     export INSTANCE_db_storage=relstorage
-    export INSTANCE_db_blobs_mode=cache
+    export INSTANCE_db_blob_mode=cache
     export INSTANCE_db_relstorage_keep_history=false
     export INSTANCE_db_relstorage=postgresql
     export INSTANCE_db_relstorage_postgresql_dsn="host='db' dbname='plone' user='plone' password='verysecret'"
@@ -927,7 +929,7 @@ all prefixed environment variables are transformed into a new configuration file
 .. code-block:: YAML
 
     default_context:
-        db_blobs_mode: cache
+        db_blob_mode: cache
         db_cache_size: '50000'
         db_cache_size_bytes: 1500MB
         db_relstorage: postgresql

cookiecutter.json

@@ -1,5 +1,5 @@
 {
-  "_version": "2.1.0",
+  "_version": "2.1.1",
   "_extensions": ["local_extensions.ZopeExtensions"],
   "target": "instance",
   "location_clienthome": "{{ cookiecutter.target }}/var",
@@ -48,8 +48,12 @@
   "db_large_record_size": "",
   "db_pool_size": "",
 
-  "db_blobs_mode": ["shared", "cache"],
-  "db_blobs_location": "{{ cookiecutter.location_clienthome }}/blobs",
+  "db_blob_mode": ["shared", "cache"],
+  "db_blobs_mode": ["","shared", "cache"],
+  "db_blob_location": "{{ cookiecutter.location_clienthome }}/blobs",
+  "db_blobs_location": "",
+  "db_blob_cache_size": "6312427520",
+  "db_blob_cache_size_check": "",
 
   "db_filestorage_location": "{{ cookiecutter.location_clienthome }}/filestorage/Data.fs",
   "db_filestorage_pack_keep_old": true,
@@ -63,7 +67,7 @@
   "db_relstorage_create_schema": true,
   "db_relstorage_commit_lock_timeout": "",
   "db_relstorage_commit_lock_id": "",
-  "db_relstorage_blob_cache_size_check_external": "",
+  "db_relstorage_blob_cache_size_check_external": false,
   "db_relstorage_blob_chunk_size": "",
   "db_relstorage_cache_local_mb": "",
   "db_relstorage_cache_local_object_max": "",

hooks/post_gen_project.py

@@ -36,7 +36,7 @@
 with work_in(basedir):
     Path("{{ cookiecutter.location_clienthome }}").mkdir(parents=True, exist_ok=True)
     Path("{{ cookiecutter.location_log }}").mkdir(parents=True, exist_ok=True)
-    Path("{{ cookiecutter.db_blobs_location }}").mkdir(parents=True, exist_ok=True)
+    Path("{{ cookiecutter.db_blob_location }}").mkdir(parents=True, exist_ok=True)
     Path("{{ cookiecutter.environment['CHAMELEON_CACHE'] }}").mkdir(
         parents=True, exist_ok=True
     )

hooks/pre_gen_project.py

@@ -4,17 +4,19 @@
 # check database mode direct and blobs not cache
 if (
     "{{ cookiecutter.db_storage }}" == "direct"
-    and "{{ cookiecutter.db_blobs_mode }}" == "cache"
+    and "{{ cookiecutter.db_blobs_mode or cookiecutter.db_blob_mode }}" == "cache"
 ):
-    print("Error: A 'direct' database must be configured with 'shared' blobs_mode!")
+    print("Error: A 'direct' database must be configured with 'shared' db_blob_mode!")
     exit(1)
 
 # check database mode not direct and blobs not shared
+db_blob_mode = "{{ cookiecutter.db_blobs_mode }}"
+if db_blob_mode == "":
+    db_blob_mode = "{{ cookiecutter.db_blob_mode }}"
 if (
-    "{{ cookiecutter.db_storage }}" == "relstorage"
-    and "{{ cookiecutter.db_blobs_mode }}" == "shared"
+    "{{ cookiecutter.db_storage }}" == "relstorage" and db_blob_mode == "shared"
 ):
-    print("Warning: A 'relstorage' database is better used with 'cache' blobs_mode!\n")
+    print("Warning: A 'relstorage' database is better used with 'cache' db_blob_mode!\n")
 
 # minimal sanity check for password
 password = "{{ cookiecutter.initial_user_password }}"
@@ -40,7 +42,17 @@
 
 if "{{ cookiecutter.verbose_security in [True, False] }}" != "True":
     upgrade_errors.append(
-        "The'verbose_security' setting must be boolean in 2.0, please fix your configuration!\n"
+        "The 'verbose_security' setting must be boolean in 2.0, please fix your configuration!\n"
+    )
+
+if "{{ cookiecutter.db_blobs_mode }}" != "":
+    upgrade_warnings.append(
+        "The 'db_blobs_mode' setting was renamed to 'db_blob_mode', please fix your configuration!"
+    )
+
+if "{{ cookiecutter.db_blobs_location }}" != "":
+    upgrade_warnings.append(
+        "The 'db_blobs_location' setting was renamed to 'db_blob_location', please fix your configuration!"
     )
 
 if upgrade_errors:
@@ -52,4 +64,5 @@
 elif upgrade_warnings:
     print("Please review the following warning messages and fix them at your earliest convenience:")
     for warning_msg in upgrade_warnings:
-        print(f" - Warning: {warning_msg}")
+        print(f" - Warning: {warning_msg}")
+    print()

templates/db.conf

@@ -1,9 +1,9 @@
 # =============================================================================
 # Filestorage (flat file with blobs)
-{% macro filestorage(filestorage_location, blobs_location, pack_keep_old, quota, packer, pack_gc) -%}
+{% macro filestorage(filestorage_location, blob_location, pack_keep_old, quota, packer, pack_gc) -%}
     <filestorage>
         path {{ filestorage_location| abspath }}
-        blob-dir {{ blobs_location | abspath }}
+        blob-dir {{ blob_location | abspath }}
         pack-keep-old {{ "true" if cookiecutter.db_filestorage_pack_keep_old else "false" }}
 {%- if quota %}
         quota {{ quota }}
@@ -20,8 +20,10 @@
 # =============================================================================
 # Relstorage
 {% macro relstorage(
-    blobs_mode,
-    blobs_location,
+    blob_mode,
+    blob_location,
+    blob_cache_size,
+    blob_cache_size_check,
     keep_history,
     read_only,
     create_schema,
@@ -56,17 +58,23 @@
     %import relstorage
 <relstorage{{ tagmarker }}>
     # blob settings
-    shared-blob-dir {{ "true" if blobs_mode == "shared" else "false" }}
-    blob-dir {{ blobs_location | abspath }}
+    shared-blob-dir {{ "true" if blob_mode == "shared" else "false" }}
+    blob-dir {{ blob_location | abspath }}
     # relstorage general settings
     keep-history {{ keep_history|lower }}
     read-only {{ read_only|lower }}
     create-schema {{ create_schema|lower}}
 {%- if commit_lock_timeout %}
     commit-lock-timeout {{ commit_lock_timeout }}
 {%- endif %}
+{%- if blob_cache_size %}
+    blob-cache-size {{ blob_cache_size }}
+{%- endif %}
+{%- if blob_cache_size_check %}
+    blob-cache-size-check {{ blob_cache_size_check }}
+{%- endif %}
 {%- if blob_cache_size_check_external %}
-    blob-cache-size-check-external {{ blob_cache_size_check_external }}
+    blob-cache-size-check-external {{ "true" if blob_cache_size_check_external else "false" }}
 {%- endif %}
 {%- if blob_chunk_size %}
     blob-chunk-size {{ blob_chunk_size }}
@@ -147,8 +155,10 @@
 # =============================================================================
 # ZEO storage
 {% macro zeo(
-    blobs_mode,
-    blobs_location,
+    blob_mode,
+    blob_location,
+    blob_cache_size,
+    blob_cache_size_check,
     server,
     name,
     client,
@@ -164,8 +174,8 @@
 )-%}
     <zeoclient>
         # blobs
-        shared-blob-dir {{ "true" if blobs_mode == "shared" else "false" }}
-        blob-dir {{ blobs_location | abspath }}
+        shared-blob-dir {{ "true" if blob_mode == "shared" else "false" }}
+        blob-dir {{ blob_location | abspath }}
         # general settings
         server {{ server }}
         name {{ name }}
@@ -179,6 +189,12 @@
 {%- if cache_size %}
         cache-size {{ cache_size }}
 {%- endif %}
+{%- if blob_cache_size %}
+    blob-cache-size {{ blob_cache_size }}
+{%- endif %}
+{%- if blob_cache_size_check %}
+    blob-cache-size-check {{ blob_cache_size_check }}
+{%- endif %}
 {%- if username %}
         # Authentication
         username {{ username }}

{{ cookiecutter.target }}/etc/cors.zcml

@@ -12,5 +12,11 @@
       allow_headers="{{ cookiecutter.cors_allow_headers }}"
       max_age="{{ cookiecutter.cors_max_age }}"
       />
+
+<!--
+  This file was generated by cookiecutter-zope-instance {{ cookiecutter._version }}.
+  for details follow https://github.com/plone/cookiecutter-zope-instance
+-->
+
 </configure>
 {%- endif %}

{{ cookiecutter.target }}/etc/relstorage-export.conf

@@ -1,14 +1,19 @@
 {% import 'db.conf' as db with context
-%}# Configuration file to pack a Relstorage database to a Filestorage database.
+%}{%- set db_blob_mode = cookiecutter.db_blobs_mode or cookiecutter.db_blob_mode%}
+{%- set db_blob_location = cookiecutter.db_blobs_location or cookiecutter.db_blob_location
+%}
+## Configuration file to pack a Relstorage database to a Filestorage database.
 # It is meant to be used with the zobdconvert command line utility from RelStorage.
 # zodbconvert is installed in the same virtual environment as the Plone site.
 # For more information consult https://relstorage.readthedocs.io/en/latest/zodbconvert.html#how-to-use-zodbconvert
 
 {%- if cookiecutter.db_storage == "relstorage" %}
 
 {{ db.relstorage(
-    cookiecutter.db_blobs_mode,
-    cookiecutter.db_blobs_location,
+    db_blob_mode,
+    db_blob_location,
+    cookiecutter.db_blob_cache_size,
+    cookiecutter.db_blob_cache_size_check,
     cookiecutter.db_relstorage_keep_history,
     cookiecutter.db_relstorage_read_only,
     cookiecutter.db_relstorage_create_schema,
@@ -48,4 +53,7 @@
 
 {% else %}
 # !!! Missing db_relstorage_export_* settings!
-{%- endif %}
+{%- endif %}
+
+# This file was generated by cookiecutter-zope-instance {{ cookiecutter._version }}.
+# for details follow https://github.com/plone/cookiecutter-zope-instance

{{ cookiecutter.target }}/etc/relstorage-import.conf

@@ -1,5 +1,7 @@
 {% import 'db.conf' as db with context
-%}# Configuration file to import a Filestorage database into a RelStorage database.
+%}{%- set db_blob_mode = cookiecutter.db_blobs_mode or cookiecutter.db_blob_mode%}
+{%- set db_blob_location = cookiecutter.db_blobs_location or cookiecutter.db_blob_location
+%}## Configuration file to import a Filestorage database into a RelStorage database.
 # It is meant to be used with the zobdconvert command line utility from RelStorage.
 # zodbconvert is installed in the same virtual environment as the Plone site.
 # For more information consult https://relstorage.readthedocs.io/en/latest/zodbconvert.html#how-to-use-zodbconvert
@@ -12,8 +14,10 @@
 </filestorage>
 
 {{ db.relstorage(
-    cookiecutter.db_blobs_mode,
-    cookiecutter.db_blobs_location,
+    db_blob_mode,
+    db_blob_location,
+    cookiecutter.db_blob_cache_size,
+    cookiecutter.db_blob_cache_size_check,
     cookiecutter.db_relstorage_keep_history,
     cookiecutter.db_relstorage_read_only,
     cookiecutter.db_relstorage_create_schema,
@@ -48,4 +52,7 @@
 
 {% else %}
 # !!! Missing db_relstorage_export_* settings!
-{%- endif %}
+{%- endif %}
+
+# This file was generated by cookiecutter-zope-instance {{ cookiecutter._version }}.
+# for details follow https://github.com/plone/cookiecutter-zope-instance