Pipeline metrics (#572)

* #556 move tracing config under parameters.telemetry.tracing * #556 use savant-rs to measure module performance * #556 implement base metrics exporter * #556 add prometheus to peoplenet detector sample * #556 add frame and object counter for whole module * #556 refactor prometheus metrics exporter * #556 add grafana to peopelenet detector * #556 collect metrics with timestamps * #556 improve grafana charts filtering * #556 move prometheus and grafana to pass-through-processing sample * #556 change units in configuration to seconds * #556 update docs * #556 add method get_runtime_metrics to pyfunc plugins * #556 fix metrics configuration * #556 remove separate thread to collect metrics for prometheus * #556 fix docker-compose for pass-through processing
insight-platform · Nov 23, 2023 · 454549c · 454549c
1 parent 3e0efce
commit 454549c
Show file tree

Hide file tree

Showing 67 changed files with 1,135 additions and 181 deletions.
diff --git a/docs/source/advanced_topics/0_dead_stream_eviction.rst b/docs/source/advanced_topics/0_dead_stream_eviction.rst
@@ -22,6 +22,6 @@ Take a look at the ``default.yml`` for details:
 
 .. literalinclude:: ../../../savant/config/default.yml
   :language: YAML
-  :lines: 133-149
+  :lines: 152-168
 
 You can override only required parameters in your module YAML configuration file. Also, take a look at corresponding environment variables helping to configure the parameters without specifying them in the module config.
diff --git a/docs/source/advanced_topics/9_open_telemetry.rst b/docs/source/advanced_topics/9_open_telemetry.rst
@@ -41,7 +41,7 @@ Trace propagation is a mechanism of passing traces between distributed, decouple
 OpenTelemetry Configuration
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Use ``params.telemetry`` to configure OpenTelemetry for the module.
+Use ``params.telemetry.tracing`` to configure OpenTelemetry for the module.
 
 .. code-block:: yaml
 
@@ -52,12 +52,13 @@ Use ``params.telemetry`` to configure OpenTelemetry for the module.
 
       # enable OpenTelemetry
       telemetry:
-        sampling_period: 100
-        root_span_name: pipeline
-        provider: jaeger
-        provider_params:
-          service_name: demo-pipeline
-          endpoint: jaeger:6831
+        tracing:
+          sampling_period: 100
+          root_span_name: pipeline
+          provider: jaeger
+          provider_params:
+            service_name: demo-pipeline
+            endpoint: jaeger:6831
 
 .. note::
 
@@ -79,4 +80,4 @@ Use ``params.telemetry`` to configure OpenTelemetry for the module.
           -p 9411:9411 \
           jaegertracing/all-in-one:1.48
 
-.. youtube:: -DXDM_wJZho
+.. youtube:: -DXDM_wJZho
diff --git a/docs/source/savant_101/12_module_definition.rst b/docs/source/savant_101/12_module_definition.rst
@@ -54,7 +54,7 @@ The following parameters are defined for a Savant module by default:
 
 .. literalinclude:: ../../../savant/config/default.yml
   :language: YAML
-  :lines: 1-132
+  :lines: 1-150
 
 .. note::
 
@@ -153,23 +153,44 @@ If the ``output_frame`` section is set, Savant adds encoded video streams to sin
 OpenTelemetry Configuration
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-The ``telemetry`` section defines the telemetry configuration. The ``endpoint`` in ``telemetry.provider_params`` is required when ``telemetry.provider`` is set to ``'jaeger'``.
+The ``telemetry.tracing`` section defines the telemetry configuration. The ``endpoint`` in ``telemetry.tracing.provider_params`` is required when ``telemetry.tracing.provider`` is set to ``'jaeger'``.
 
 Example:
 
 .. code-block:: yaml
 
     telemetry:
-      sampling_period: 100
-      append_frame_meta_to_span: false
-      root_span_name: demo-pipeline-root
-      provider: jaeger
-      provider_params:
-        service_name: demo-pipeline
-        endpoint: jaeger:6831
+      tracing:
+        sampling_period: 100
+        append_frame_meta_to_span: false
+        root_span_name: demo-pipeline-root
+        provider: jaeger
+        provider_params:
+          service_name: demo-pipeline
+          endpoint: jaeger:6831
 
 Read more on OpenTelemetry in :doc:`/advanced_topics/9_open_telemetry`.
 
+Metrics Collection Configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``telemetry.metrics`` section defines the metrics collection configuration. The ``port`` in ``telemetry.metrics.provider_params`` is required when ``telemetry.metrics.provider`` is set to ``'prometheus'``. ``labels`` in ``telemetry.metrics.provider_params`` defines extra labels added to the metrics.
+
+Example:
+
+.. code-block:: yaml
+
+    telemetry:
+      metrics:
+        frame_period: 1000
+        time_period: 1
+        history: 100
+        provider: prometheus
+        provider_params:
+          port: 8000
+          labels:
+            module_type: detector
+
 DevServer Configuration
 ^^^^^^^^^^^^^^^^^^^^^^^
 

diff --git a/docs/source/savant_101/12_pipeline.rst b/docs/source/savant_101/12_pipeline.rst
@@ -17,7 +17,7 @@ Default module configuration file already defines the :py:attr:`~savant.config.s
 
 .. literalinclude:: ../../../savant/config/default.yml
   :language: YAML
-  :lines: 135-
+  :lines: 152-
 
 It is possible to redefine them, but the encouraged operation mode assumes the use of ZeroMQ source and sink.
 

diff --git a/requirements/base.txt b/requirements/base.txt
@@ -23,3 +23,5 @@ tqdm~=4.64
 
 # ClientSDK JpegSource, PngSource
 python-magic~=0.4.27
+
+prometheus-client~=0.19.0
diff --git a/requirements/savant-rs.txt b/requirements/savant-rs.txt
@@ -1 +1 @@
-savant-rs==0.1.74
+savant-rs==0.1.83
diff --git a/samples/age_gender_recognition/docker-compose.l4t.yml b/samples/age_gender_recognition/docker-compose.l4t.yml
@@ -30,7 +30,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=sub+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
       - CODEC=raw-rgba
     runtime: nvidia
 

diff --git a/samples/age_gender_recognition/docker-compose.x86.yml b/samples/age_gender_recognition/docker-compose.x86.yml
@@ -31,7 +31,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=sub+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
     deploy:
       resources:
         reservations:

diff --git a/samples/animegan/docker-compose.x86.yml b/samples/animegan/docker-compose.x86.yml
@@ -29,7 +29,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=router+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
     deploy:
       resources:
         reservations:

diff --git a/samples/area_object_counting/docker-compose.l4t.yml b/samples/area_object_counting/docker-compose.l4t.yml
@@ -29,7 +29,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=sub+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
       - MODULE_NAME=area_object_counting
       - CODEC=raw-rgba
     runtime: nvidia

diff --git a/samples/area_object_counting/docker-compose.x86.yml b/samples/area_object_counting/docker-compose.x86.yml
@@ -29,7 +29,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=sub+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
       - MODULE_NAME=area_object_counting
     deploy:
       resources:

diff --git a/samples/conditional_video_processing/docker-compose.l4t.yml b/samples/conditional_video_processing/docker-compose.l4t.yml
@@ -30,7 +30,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=sub+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
       - CODEC=raw-rgba
     depends_on:
       etcd:

diff --git a/samples/conditional_video_processing/docker-compose.x86.yml b/samples/conditional_video_processing/docker-compose.x86.yml
@@ -30,7 +30,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=sub+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
     depends_on:
       etcd:
         condition: service_healthy

diff --git a/samples/face_reid/docker-compose.l4t.yml b/samples/face_reid/docker-compose.l4t.yml
@@ -69,7 +69,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=sub+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
       - CODEC=raw-rgba
     runtime: nvidia
     profiles:

diff --git a/samples/face_reid/docker-compose.x86.yml b/samples/face_reid/docker-compose.x86.yml
@@ -76,7 +76,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=sub+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
     deploy:
       resources:
         reservations:

diff --git a/samples/intersection_traffic_meter/docker-compose.l4t.yml b/samples/intersection_traffic_meter/docker-compose.l4t.yml
@@ -31,7 +31,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=router+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
       - CODEC=raw-rgba
     runtime: nvidia
 

diff --git a/samples/intersection_traffic_meter/docker-compose.x86.yml b/samples/intersection_traffic_meter/docker-compose.x86.yml
@@ -31,7 +31,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=router+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
     deploy:
       resources:
         reservations:

diff --git a/samples/kafka_redis_adapter/docker-compose-no-keydb.l4t.yml b/samples/kafka_redis_adapter/docker-compose-no-keydb.l4t.yml
@@ -94,7 +94,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=sub+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
       - CODEC=raw-rgba
     runtime: nvidia
 

diff --git a/samples/kafka_redis_adapter/docker-compose-no-keydb.x86.yml b/samples/kafka_redis_adapter/docker-compose-no-keydb.x86.yml
@@ -94,7 +94,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=sub+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
     deploy:
       resources:
         reservations:

diff --git a/samples/kafka_redis_adapter/docker-compose.l4t.yml b/samples/kafka_redis_adapter/docker-compose.l4t.yml
@@ -106,7 +106,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=sub+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
       - CODEC=raw-rgba
     runtime: nvidia
 

diff --git a/samples/kafka_redis_adapter/docker-compose.x86.yml b/samples/kafka_redis_adapter/docker-compose.x86.yml
@@ -106,7 +106,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=sub+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
     deploy:
       resources:
         reservations:

diff --git a/samples/keypoint_detection/docker-compose.l4t.yml b/samples/keypoint_detection/docker-compose.l4t.yml
@@ -30,7 +30,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=router+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
       - CODEC=raw-rgba
     runtime: nvidia
 

diff --git a/samples/keypoint_detection/docker-compose.x86.yml b/samples/keypoint_detection/docker-compose.x86.yml
@@ -30,7 +30,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=router+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
     deploy:
       resources:
         reservations:

diff --git a/samples/license_plate_recognition/docker-compose.l4t.yml b/samples/license_plate_recognition/docker-compose.l4t.yml
@@ -32,7 +32,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=router+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
       - CODEC=raw-rgba
     runtime: nvidia
 

diff --git a/samples/license_plate_recognition/docker-compose.x86.yml b/samples/license_plate_recognition/docker-compose.x86.yml
@@ -32,7 +32,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=router+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
     deploy:
       resources:
         reservations:

diff --git a/samples/multiple_rtsp/docker-compose.l4t.yml b/samples/multiple_rtsp/docker-compose.l4t.yml
@@ -41,7 +41,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=sub+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
       - CODEC=raw-rgba
     runtime: nvidia
 

diff --git a/samples/multiple_rtsp/docker-compose.x86.yml b/samples/multiple_rtsp/docker-compose.x86.yml
@@ -41,7 +41,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=sub+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
     deploy:
       resources:
         reservations:

diff --git a/samples/nvidia_car_classification/docker-compose.l4t.yml b/samples/nvidia_car_classification/docker-compose.l4t.yml
@@ -30,7 +30,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=sub+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
       - CODEC=raw-rgba
     runtime: nvidia
 

diff --git a/samples/nvidia_car_classification/docker-compose.x86.yml b/samples/nvidia_car_classification/docker-compose.x86.yml
@@ -30,7 +30,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=sub+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
     deploy:
       resources:
         reservations:

diff --git a/samples/opencv_cuda_bg_remover_mog2/docker-compose.l4t.yml b/samples/opencv_cuda_bg_remover_mog2/docker-compose.l4t.yml
@@ -28,7 +28,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=sub+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
       - CODEC=raw-rgba
     runtime: nvidia
 

diff --git a/samples/opencv_cuda_bg_remover_mog2/docker-compose.x86.yml b/samples/opencv_cuda_bg_remover_mog2/docker-compose.x86.yml
@@ -28,7 +28,7 @@ services:
     environment:
       - ZMQ_SRC_ENDPOINT=sub+bind:ipc:///tmp/zmq-sockets/input-video.ipc
       - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
-      - FPS_PERIOD=1000
+      - METRICS_FRAME_PERIOD=1000
     deploy:
       resources:
         reservations:

diff --git a/samples/pass_through_processing/README.md b/samples/pass_through_processing/README.md
@@ -6,6 +6,8 @@ A pipeline demonstrates how to pass frames from source to sink without content m
 - tracker;
 - draw-func.
 
+The modules performance is also stored in Prometheus and displayed on a Grafana dashboard.
+
 ![pass-through-demo.png](assets/pass-through-demo.png)
 
 Detector and tracker are running in pass-through mode (`codec: copy`). Draw-func encodes frames to H264.
@@ -26,8 +28,13 @@ git lfs pull
 # open 'rtsp://127.0.0.1:554/stream' in your player
 # or visit 'http://127.0.0.1:888/stream/' (LL-HLS)
 
+# for pre-configured Grafana dashboard visit
+# http://127.0.0.1:3000/d/a4c1f484-75c9-4375-a04d-ab5d50578239/module-performance-metrics?orgId=1&refresh=5s
+
 # Ctrl+C to stop running the compose bundle
 
 # to get back to project root
 cd ../..
 ```
+
+To create a custom Grafana dashboard, sign in with `admin\admin` credentials.