Skip to content

Commit

Permalink
added conversion sync with prometheus objects labels and annotation, (#…
Browse files Browse the repository at this point in the history
…69)

added behaviour controll labels
  • Loading branch information
f41gh7 authored Aug 22, 2020
1 parent 9618f25 commit 1023964
Show file tree
Hide file tree
Showing 7 changed files with 234 additions and 11 deletions.
4 changes: 2 additions & 2 deletions config/crd/bases/operator.victoriametrics.com_vmagents.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -544,7 +544,7 @@ spec:
type: boolean
type: object
configMaps:
description: ConfigMaps is a list of ConfigMaps in the same namespace as the vmagent object, which shall be mounted into the vmagent Pods. will be mounted at path /etc/vmagent/configs
description: ConfigMaps is a list of ConfigMaps in the same namespace as the vmagent object, which shall be mounted into the vmagent Pods. will be mounted at path /etc/vm/configs
items:
type: string
type: array
Expand Down Expand Up @@ -1108,7 +1108,7 @@ spec:
pattern: '[0-9]+(ms|s|m|h)'
type: string
secrets:
description: Secrets is a list of Secrets in the same namespace as the vmagent object, which shall be mounted into the vmagent Pods. will be mounted at path /etc/vmagent/secrets
description: Secrets is a list of Secrets in the same namespace as the vmagent object, which shall be mounted into the vmagent Pods. will be mounted at path /etc/vm/secrets
items:
type: string
type: array
Expand Down
4 changes: 2 additions & 2 deletions config/crd/bases/operator.victoriametrics.com_vmalerts.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -372,7 +372,7 @@ spec:
type: object
type: object
configMaps:
description: ConfigMaps is a list of ConfigMaps in the same namespace as the VMAlert object, which shall be mounted into the VMAlert Pods. The ConfigMaps are mounted into /etc/vmalert/configmaps/<configmap-name>.
description: ConfigMaps is a list of ConfigMaps in the same namespace as the VMAlert object, which shall be mounted into the VMAlert Pods. The ConfigMaps are mounted into /etc/vm/configs/<configmap-name>.
items:
type: string
type: array
Expand Down Expand Up @@ -1242,7 +1242,7 @@ spec:
type: object
type: object
secrets:
description: Secrets is a list of Secrets in the same namespace as the VMAlert object, which shall be mounted into the VMAlert Pods. The Secrets are mounted into /etc/vmalert/secrets/<secret-name>.
description: Secrets is a list of Secrets in the same namespace as the VMAlert object, which shall be mounted into the VMAlert Pods. The Secrets are mounted into /etc/vm/secrets/<secret-name>.
items:
type: string
type: array
Expand Down
10 changes: 5 additions & 5 deletions config/crd/bases/operator.victoriametrics.com_vmclusters.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -412,7 +412,7 @@ spec:
type: object
type: object
configMaps:
description: ConfigMaps is a list of ConfigMaps in the same namespace as the VMSelect object, which shall be mounted into the VMSelect Pods. The ConfigMaps are mounted into /etc/vmalert/configmaps/<configmap-name>.
description: ConfigMaps is a list of ConfigMaps in the same namespace as the VMSelect object, which shall be mounted into the VMSelect Pods. The ConfigMaps are mounted into /etc/vm/configs/<configmap-name>.
items:
type: string
type: array
Expand Down Expand Up @@ -605,7 +605,7 @@ spec:
description: SchedulerName - defines kubernetes scheduler name
type: string
secrets:
description: Secrets is a list of Secrets in the same namespace as the VMSelect object, which shall be mounted into the VMSelect Pods. The Secrets are mounted into /etc/vmalert/secrets/<secret-name>.
description: Secrets is a list of Secrets in the same namespace as the VMSelect object, which shall be mounted into the VMSelect Pods. The Secrets are mounted into /etc/vm/secrets/<secret-name>.
items:
type: string
type: array
Expand Down Expand Up @@ -1889,7 +1889,7 @@ spec:
description: CacheMountPath allows to add cache persistent for VMSelect
type: string
configMaps:
description: ConfigMaps is a list of ConfigMaps in the same namespace as the VMSelect object, which shall be mounted into the VMSelect Pods. The ConfigMaps are mounted into /etc/vm/configmaps/<configmap-name>.
description: ConfigMaps is a list of ConfigMaps in the same namespace as the VMSelect object, which shall be mounted into the VMSelect Pods. The ConfigMaps are mounted into /etc/vm/configs/<configmap-name>.
items:
type: string
type: array
Expand Down Expand Up @@ -3547,7 +3547,7 @@ spec:
type: object
type: object
configMaps:
description: ConfigMaps is a list of ConfigMaps in the same namespace as the VMSelect object, which shall be mounted into the VMSelect Pods. The ConfigMaps are mounted into /etc/vmalert/configmaps/<configmap-name>.
description: ConfigMaps is a list of ConfigMaps in the same namespace as the VMSelect object, which shall be mounted into the VMSelect Pods. The ConfigMaps are mounted into /etc/vm/configs/<configmap-name>.
items:
type: string
type: array
Expand Down Expand Up @@ -3740,7 +3740,7 @@ spec:
description: SchedulerName - defines kubernetes scheduler name
type: string
secrets:
description: Secrets is a list of Secrets in the same namespace as the VMSelect object, which shall be mounted into the VMSelect Pods. The Secrets are mounted into /etc/vmalert/secrets/<secret-name>.
description: Secrets is a list of Secrets in the same namespace as the VMSelect object, which shall be mounted into the VMSelect Pods. The Secrets are mounted into /etc/vm/secrets/<secret-name>.
items:
type: string
type: array
Expand Down
2 changes: 2 additions & 0 deletions controllers/vmprobe_controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ type VMProbeReconciler struct {
BaseConf *config.BaseOperatorConf
}

// Reconcile - syncs VMProbe
// +kubebuilder:rbac:groups=operator.victoriametrics.com,resources=vmprobes,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=operator.victoriametrics.com,resources=vmprobes/status,verbs=get;update;patch
func (r *VMProbeReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
Expand Down Expand Up @@ -78,6 +79,7 @@ func (r *VMProbeReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
return ctrl.Result{}, nil
}

// SetupWithManager - setups VMProbe manager
func (r *VMProbeReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&operatorv1beta1.VMProbe{}).
Expand Down
102 changes: 101 additions & 1 deletion controllers/vmprometheusconverter_controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,33 @@ import (
"k8s.io/client-go/tools/cache"
)

const (
// MetaMergeStrategyLabel merge strategy by default prefer prometheus meta labels
// but with annotation value added to VMObject:
// annotations:
// operator.victoriametrics.com/merge-api-strategy: prefer-victoriametrics
// metadata from VMObject will be preferred during merge
MetaMergeStrategyLabel = "operator.victoriametrics.com/merge-meta-strategy"
// MetaPreferVM - prefers VM object meta values, ignores prometheus
MetaPreferVM = "prefer-victoriametrics"
// MetaPreferProm - prefers prometheus
MetaPreferProm = "prefer-prometheus"
// MetaMergeLabelsVMPriority merges both label sets
// its not possible to remove values
MetaMergeLabelsVMPriority = "merge-victoriametrics-priority"
// MetaMergeLabelsPromPriority merges both label sets
// its not possible to remove values
MetaMergeLabelsPromPriority = "merge-prometheus-priority"

// IgnoreConversionLabel this annotation disables updating of corresponding VMObject
// must be added to annotation of VMObject
// annotations:
// operator.victoriametrics.com/ignore-prometheus-updates: enabled
IgnoreConversionLabel = "operator.victoriametrics.com/ignore-prometheus-updates"
// IgnoreConversion - disables updates from prometheus api
IgnoreConversion = "enabled"
)

// ConverterController - watches for prometheus objects
// and create VictoriaMetrics objects
type ConverterController struct {
Expand Down Expand Up @@ -212,8 +239,15 @@ func (c *ConverterController) UpdatePrometheusRule(old, new interface{}) {
l.Error(err, "cannot get existing VMRule")
return
}

if existingVMRule.Annotations[IgnoreConversionLabel] == IgnoreConversion {
l.Info("syncing for object was disabled by annotation", "annotation", IgnoreConversionLabel)
return
}
existingVMRule.Spec = VMRule.Spec
metaMergeStrategy := getMetaMergeStrategy(existingVMRule.Annotations)
existingVMRule.Annotations = mergeLabelsWithStrategy(existingVMRule.Annotations, VMRule.Annotations, metaMergeStrategy)
existingVMRule.Labels = mergeLabelsWithStrategy(existingVMRule.Labels, VMRule.Labels, metaMergeStrategy)

err = c.vclient.Update(ctx, existingVMRule)
if err != nil {
l.Error(err, "cannot update VMRule")
Expand Down Expand Up @@ -254,8 +288,16 @@ func (c *ConverterController) UpdateServiceMonitor(old, new interface{}) {
l.Error(err, "cannot get existing vmServiceScrape")
return
}

if existingVMServiceScrape.Annotations[IgnoreConversionLabel] == IgnoreConversion {
l.Info("syncing for object was disabled by annotation", "annotation", IgnoreConversionLabel)
return
}
existingVMServiceScrape.Spec = vmServiceScrape.Spec

metaMergeStrategy := getMetaMergeStrategy(existingVMServiceScrape.Annotations)
existingVMServiceScrape.Annotations = mergeLabelsWithStrategy(existingVMServiceScrape.Annotations, vmServiceScrape.Annotations, metaMergeStrategy)
existingVMServiceScrape.Labels = mergeLabelsWithStrategy(existingVMServiceScrape.Labels, vmServiceScrape.Labels, metaMergeStrategy)
err = c.vclient.Update(ctx, existingVMServiceScrape)
if err != nil {
l.Error(err, "cannot update")
Expand Down Expand Up @@ -295,7 +337,16 @@ func (c *ConverterController) UpdatePodMonitor(old, new interface{}) {
l.Error(err, "cannot get existing podMonitor")
return
}
if existingVMPodScrape.Annotations[IgnoreConversionLabel] == IgnoreConversion {
l.Info("syncing for object was disabled by annotation", "annotation", IgnoreConversionLabel)
return
}

existingVMPodScrape.Spec = podScrape.Spec
mergeStrategy := getMetaMergeStrategy(existingVMPodScrape.Annotations)
existingVMPodScrape.Annotations = mergeLabelsWithStrategy(existingVMPodScrape.Annotations, podScrape.Annotations, mergeStrategy)
existingVMPodScrape.Labels = mergeLabelsWithStrategy(existingVMPodScrape.Labels, podScrape.Labels, mergeStrategy)

err = c.vclient.Update(ctx, existingVMPodScrape)
if err != nil {
l.Error(err, "cannot update podScrape")
Expand All @@ -305,6 +356,46 @@ func (c *ConverterController) UpdatePodMonitor(old, new interface{}) {

}

// default merge strategy - prefer-prometheus
// old - from vm
// new - from prometheus
// by default new has priority
func mergeLabelsWithStrategy(old, new map[string]string, mergeStrategy string) map[string]string {

switch mergeStrategy {
case MetaPreferVM:
return old
case MetaPreferProm:
return new
case MetaMergeLabelsVMPriority:
old, new = new, old
case MetaMergeLabelsPromPriority:
break
}
merged := make(map[string]string)
for k, v := range old {
merged[k] = v
}
for k, v := range new {
merged[k] = v
}
return merged
}

// helper function - extracts meta merge strategy
// in the future we can introduce another merge strategies
func getMetaMergeStrategy(vmMeta map[string]string) string {
switch vmMeta[MetaMergeStrategyLabel] {
case MetaPreferVM:
return MetaPreferVM
case MetaMergeLabelsPromPriority:
return MetaMergeLabelsPromPriority
case MetaMergeLabelsVMPriority:
return MetaMergeLabelsVMPriority
}
return MetaPreferProm
}

// CreateProbe converts Probe to VMProbe
func (c *ConverterController) CreateProbe(obj interface{}) {
probe := obj.(*v1.Probe)
Expand Down Expand Up @@ -336,6 +427,15 @@ func (c *ConverterController) UpdateProbe(old, new interface{}) {
l.Error(err, "cannot get existing vmProbe")
return
}
if existingVMProbe.Annotations[IgnoreConversionLabel] == IgnoreConversion {
l.Info("syncing for object was disabled by annotation", "annotation", IgnoreConversionLabel)
return
}

mergeStrategy := getMetaMergeStrategy(existingVMProbe.Annotations)
existingVMProbe.Annotations = mergeLabelsWithStrategy(existingVMProbe.Annotations, probeNew.Annotations, mergeStrategy)
existingVMProbe.Labels = mergeLabelsWithStrategy(existingVMProbe.Labels, probeNew.Labels, mergeStrategy)

existingVMProbe.Spec = vmProbe.Spec
err = c.vclient.Update(ctx, existingVMProbe)
if err != nil {
Expand Down
81 changes: 81 additions & 0 deletions controllers/vmprometheusconverter_controller_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
package controllers

import (
"reflect"
"testing"
)

func Test_mergeLabelsWithStrategy(t *testing.T) {
type args struct {
old map[string]string
new map[string]string
mergeStrategy string
}
tests := []struct {
name string
args args
want map[string]string
}{
{
name: "delete not existing label",
args: args{
old: map[string]string{"label1": "value1", "label2": "value2", "missinglabel": "value3"},
new: map[string]string{"label1": "value1", "label2": "value4"},
mergeStrategy: MetaPreferProm,
},
want: map[string]string{"label1": "value1", "label2": "value4"},
},
{
name: "add new label",
args: args{
old: map[string]string{"label1": "value1", "label2": "value2", "missinglabel": "value3"},
new: map[string]string{"label1": "value1", "label2": "value4", "label5": "value10"},
mergeStrategy: MetaPreferProm,
},
want: map[string]string{"label1": "value1", "label2": "value4", "label5": "value10"},
},
{
name: "add new label with VM priority",
args: args{
old: map[string]string{"label1": "value1", "label2": "value2", "label5": "value3"},
new: map[string]string{"label1": "value1", "label2": "value4", "missinglabel": "value10"},
mergeStrategy: MetaPreferVM,
},
want: map[string]string{"label1": "value1", "label2": "value2", "label5": "value3"},
},
{
name: "remove all labels",
args: args{
old: nil,
new: map[string]string{"label1": "value1", "label2": "value4", "missinglabel": "value10"},
mergeStrategy: MetaPreferVM,
},
want: nil,
},
{
name: "remove keep old labels",
args: args{
old: map[string]string{"label1": "value1", "label2": "value4"},
new: nil,
mergeStrategy: MetaPreferVM,
},
want: map[string]string{"label1": "value1", "label2": "value4"},
},
{
name: "merge all labels with VMPriority",
args: args{
old: map[string]string{"label1": "value1", "label2": "value4"},
new: map[string]string{"label1": "value2", "label2": "value4", "missinglabel": "value10"},
mergeStrategy: MetaMergeLabelsVMPriority,
},
want: map[string]string{"label1": "value1", "label2": "value4", "missinglabel": "value10"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := mergeLabelsWithStrategy(tt.args.old, tt.args.new, tt.args.mergeStrategy); !reflect.DeepEqual(got, tt.want) {
t.Errorf("mergeLabelsWithStrategy() = %v, want %v", got, tt.want)
}
})
}
}
42 changes: 41 additions & 1 deletion docs/quick-start.MD
Original file line number Diff line number Diff line change
Expand Up @@ -1046,10 +1046,50 @@ VM_ENABLEDPROMETHEUSCONVERTER_SERVICESCRAPE=false
VM_ENABLEDPROMETHEUSCONVERTER_PROMETHEUSRULE=false
VM_ENABLEDPROMETHEUSCONVERTER_PROBE=false
```
Otherwise, victoriametrics-operator would try to discover prometheus-operator API and convert it.
Conversion of api objects can be controlled by annotations, added to `VMObject`s, there are following annotations:
- `operator.victoriametrics.com/merge-meta-strategy` - it controls syncing of metadata labels and annotations between
`VMObject`s and `Prometheus` api objects during updates to `Prometheus` objects. By default, it has `prefer-prometheus`.
And annotations and labels will be used from `Prometheus` objects, manually set values will be dropped.
You can set it to `prefer-victoriametrics`. In this case all labels and annotations applied to `Prometheus` object
will be ignored and `VMObject` will use own values.
Two additional strategies annotations -`merge-victoriametrics-priority` and `merge-prometheus-priority` merges labelSets into one combined labelSet, with priority.
Example:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
annotations:
meta.helm.sh/release-name: prometheus
operator.victoriametrics.com/merge-meta-strategy: prefer-victoriametrics
labels:
release: prometheus
name: prometheus-monitor
spec:
endpoints: []
```
- `operator.victoriametrics.com/ignore-prometheus-updates` - it controls updates from Prometheus api objects.
By default, it set to `disabled`. You define it to `enabled` state and all updates from Prometheus api objects will be
ignored.
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
annotations:
meta.helm.sh/release-name: prometheus
operator.victoriametrics.com/ignore-prometheus-updates: enabled
labels:
release: prometheus
name: prometheus-monitor
spec:
endpoints: []
```
## Expose the VMSingle API
Expand Down

0 comments on commit 1023964

Please sign in to comment.