- The environment is running on google cloud.
- Make sure you've installed the
gcloud
andkubectl
cli tools, and usegcloud auth login
to get access to theopen-raadsinformatie
account. - Connect to the cluster using
gcloud container clusters get-credentials ori-cluster --zone europe-west4-a --project open-raadsinformatie-52162
. If that fails, visit the cluster in google cloud console, pressactions
and search for aconnect
command.
- New municipalities are probably added to the issue tracker.
- Identify supplier (Notubiz / go / etc)
- Open the relevant
ocd_backend/sources
file. - Duplicate bottom municipality
- Set key (equal to shortname, dashes allowed)
- Allamak code: https://allmanak.nl/cat/1/Gemeenten (https://allmanak.nl/cat/32/Waterschappen), search name, get ID from URL
- For CBS code: https://www.cbs.nl/nl-nl/onze-diensten/methoden/classificaties/overig/gemeentelijke-indelingen-per-jaar/indeling-per-jaar/gemeentelijke-indeling-op-1-januari-2024
- Optionally set
source_name
if municipality name can't be properly derived from shortname. - Optionally set
municipality_prefix
if municipality has multiple suppliers per region. - The next step depends on supplier, see below
- Push to master
ssh
towolf
(ask Breyten)- sh to
redis
(see redis) select 1
for setting individual municipalitiesset "ori.{supplier}.{key}" "all daily monthly"
add municipalityexit
- Make sure the image version running in GCLoud is the latest one containing the new municipalities. You can check the backend deployment. The last number of the version should be the build number of semaphore. If this is not a match, you can manually set it in the deployment YAML file.
- see #starting-a-run below to sh into
backend-${id}
- start the extraction process for the new municipality
/opt/ori $ python manage.py extract process all --source_path=ori.notubiz.weesp
. They will be set in a list forcelery
, which means that they will be processed in time. You can scale up the amount of workers forbackend
in Google, but it's probably not necessary. - You can track the progress in the Google cloud logs.
- Update the status per municipality (importing, finished) in the github issue tracker.
- For finding
ibabs_sitename
, google foribabs ${municipality_name}
and derive it from the URL - Duplicate
- Exclcude / include are rarely required, but can be useful if one instance is shared across municipalitites
- Usually trying a
base_url
that makes sense works fine
- Go to redis (see devops), set source value to
archived
for the older one.
- Bugs are reported to Bugsnag.
kubectl get -A pods | grep redis
to get the name for the redis pod.kubectl exec -it -n production ${name-of-redis-pod} redis-cli
to open the redis CLI.select 0
for generic settings and pipelineskeys _*
see all intervals¸ e.g. the base start dateset _all.start_date
set start date for a new run (e.g. when some specific run has to be done)
select 1
for the sources (municipalities / provinces) settingsget ori.ibabs.aalsmeer
=> see current statusset "ori.ibabs.{key}" "all daily monthly"
add municipality
kubectl get -A pods | grep backend
, pick a running onekubectl exec -it -n production ${name-of-pod} sh
- Import all municipalities (takes a couple of days):
python manage.py extract process all
(see manage.py for more options and commands)
- Not enough available disk space can cause downtime. Elastic starts to have issues at 80% disk usage - it starts moving stuff to other instances. Fix this by making the disk larger and copying the contents.
- When dealing with IBabs issues, use SoapUI.
- When finding logs for a municipality, use the GCP querybuilder with
textPayload:municipality
- Go to GCP -> Kubernetes -> Snapshots
- The snapshot schedule
postgres-weekly
is actually elastic - Go to disks, create disk, standard disk, source type -> snapshot2
- Deployments for kubernetes files
- ocd_backend contains most logic
- bin e.g. kubernetes deploy script
- models Use OpenGov definitions
- sources contain municipalities config
- extractor Responsible for fetching data
- loader Responsible for writing data (e.g. elastic and linked-delta's)
- transformers Responsible for mapping data
- enrichers Extracting text from PDFs, adding locations, adding themes
This project uses cert-manager
for creating certificates.
This is running as a service in kubernetes.
kubectl exec elastic-0 -it -n production -- sh
get a shell in a running elastic container- Send an HTTP DELETE to the ID:
curl -X DELETE 0.0.0.0:9200/${index}/${type}/${id}
, e.g.curl -X DELETE 0.0.0.0:9200/ori_vlaardingen_20190809125128/_doc/1234567
Sometimes VNG wants a list of municipalities. You can use ./fetch_municipalities.sh
to fetch the data for all suppliers and output it in a format suitable for Excel sheets.