From 1c32d2598c4fa1dff410d3f928aa0bebf4b163ff Mon Sep 17 00:00:00 2001 From: Matthias Wirtz Date: Sun, 20 Oct 2024 11:51:10 +0200 Subject: [PATCH 1/4] add some notes about db performance to contribution guide --- website/docs/development.md | 43 +++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/website/docs/development.md b/website/docs/development.md index cd9457c5c6..82d30697e3 100644 --- a/website/docs/development.md +++ b/website/docs/development.md @@ -140,3 +140,46 @@ The Grafana VS Code extension allows you to open Grafana dashboards as JSON file - From the editor UI, save the updated dashboard back to the original JSON file see: [grafana-vs-code-extension](https://github.com/grafana/grafana-vs-code-extension) + +### Best Practices + +When Streaming API is enabled roughly 1 GB of data is gathered per car and 30.000km. Most of that data (95+ percent) is stored in positions table. For optimal dashboard performance these recommendations should be followed: + +- only query positions table when really needed +- if data in 15 second intervals is sufficient consider excluding streaming data by adding `ideal_battery_range_km IS NOT NULL and car_id = $car_id` as WHERE conditions + +Before opening pull requests please diagnose index usage & query performance by making use of `EXPLAIN ANALYZE`. + +### Enable _pg_stat_statements_ to collect query statistics +To quickly identify performance bottlenecks we encourage all contributors to enable the pg_stat_statements extension in their instance. For docker based installs you can follow these steps: + +- Enable the pg_stat_statements module + + ```yml + services: + database: + image: postgres:17 + ... + command: postgres -c shared_preload_libraries=pg_stat_statements + ... + ``` + +- Create Extension to enable `pg_stat_statements` view + + ```sql + CREATE EXTENSION IF NOT EXISTS `pg_stat_statements`; + ``` + +- Identify potentially slow queries (mean_exec_time) + + ```sql + SELECT query, calls, mean_exec_time, total_exec_time FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10; + ``` + +- Identify frequently executed queries (calls) + + ```sql + SELECT query, calls, mean_exec_time, total_exec_time FROM pg_stat_statements ORDER BY calls DESC LIMIT 10; + ``` + +Additional details about pg_stat_statements can be found here: https://www.postgresql.org/docs/current/pgstatstatements.html \ No newline at end of file From f9df5a23a92188728117db1d84fa0cada1e364d5 Mon Sep 17 00:00:00 2001 From: Matthias Wirtz Date: Sun, 20 Oct 2024 12:26:23 +0200 Subject: [PATCH 2/4] add docs for usage of timestamp columns --- website/docs/development.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/website/docs/development.md b/website/docs/development.md index 82d30697e3..0266e896e8 100644 --- a/website/docs/development.md +++ b/website/docs/development.md @@ -141,7 +141,21 @@ The Grafana VS Code extension allows you to open Grafana dashboards as JSON file see: [grafana-vs-code-extension](https://github.com/grafana/grafana-vs-code-extension) -### Best Practices +## Best Practices + +### Queries involving timestamp columns + +Datetime values are currently stored in columns of type `timestamp`. [This is NOT recommended](https://wiki.postgresql.org/wiki/Don't_Do_This#Don.27t_use_timestamp_.28without_time_zone.29_to_store_UTC_times). + +While [Grafana macros](https://grafana.com/docs/grafana/latest/datasources/postgres/#macros) like `$__timeFilter` & `$__timeGroup` are working PostgreSQL functions like `DATE_TRUNC()` require additional treatment. + +```sql +DATE_TRUNC('day', TIMEZONE('UTC', date)) +``` + +In addition ensure to compare either values with or without time zone. + +### Streaming API data / positions table usage in dashboard queries When Streaming API is enabled roughly 1 GB of data is gathered per car and 30.000km. Most of that data (95+ percent) is stored in positions table. For optimal dashboard performance these recommendations should be followed: From b25cd5a9defea20a281b14de4d4b15f56da38bdc Mon Sep 17 00:00:00 2001 From: Matthias Wirtz Date: Sun, 20 Oct 2024 12:37:22 +0200 Subject: [PATCH 3/4] format prettier --- website/docs/development.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/website/docs/development.md b/website/docs/development.md index 0266e896e8..704b7d2ba6 100644 --- a/website/docs/development.md +++ b/website/docs/development.md @@ -165,6 +165,7 @@ When Streaming API is enabled roughly 1 GB of data is gathered per car and 30.00 Before opening pull requests please diagnose index usage & query performance by making use of `EXPLAIN ANALYZE`. ### Enable _pg_stat_statements_ to collect query statistics + To quickly identify performance bottlenecks we encourage all contributors to enable the pg_stat_statements extension in their instance. For docker based installs you can follow these steps: - Enable the pg_stat_statements module @@ -196,4 +197,4 @@ To quickly identify performance bottlenecks we encourage all contributors to ena SELECT query, calls, mean_exec_time, total_exec_time FROM pg_stat_statements ORDER BY calls DESC LIMIT 10; ``` -Additional details about pg_stat_statements can be found here: https://www.postgresql.org/docs/current/pgstatstatements.html \ No newline at end of file +Additional details about pg_stat_statements can be found here: https://www.postgresql.org/docs/current/pgstatstatements.html From 5dcb9e4e1f1582c86e9b521b46f5821f22039887 Mon Sep 17 00:00:00 2001 From: Jakob Lichterfeld Date: Mon, 21 Oct 2024 15:47:24 +0200 Subject: [PATCH 4/4] fix: remove unneeded apostrophe --- website/docs/development.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/development.md b/website/docs/development.md index 704b7d2ba6..981241abf9 100644 --- a/website/docs/development.md +++ b/website/docs/development.md @@ -182,7 +182,7 @@ To quickly identify performance bottlenecks we encourage all contributors to ena - Create Extension to enable `pg_stat_statements` view ```sql - CREATE EXTENSION IF NOT EXISTS `pg_stat_statements`; + CREATE EXTENSION IF NOT EXISTS pg_stat_statements; ``` - Identify potentially slow queries (mean_exec_time)