Benchmark Tests

[manno]

We want to measure how Fleet performs on a given system of clusters, for a set of tests.

For a future benchmark suite, create the first test. This can be a script or executable, a Go benchmark test, etc.
The test runs against a cluster and collects several measures.

It should create 5 GitRepo resources, which contain 2 bundles each. The bundles should be targeted at 5 downstream clusters.
This will deploy 50 applications. The applications contain a medium amount of normal sized resources, e.g. 10 each.

• how long does it take for the git commit sha to show up in each GitRepo resource (duration seconds per repo)
• how long till bundle is created (per bundle)
• how long till bundledeployment is created (per bundledeployment)
• how long till agent finished install (per helm release)
• how long till gitrepo/cluster/clustergroup are updated
• number of reconciles (from prometheus)
• duration of reconciles (from prometheus)
• overall duration
• controllers memory usage before and after the test
• the amount of resources on the control plane before and after the test

Output the results in a useful format, e.g. JSON.

Future Work

• setup of test environment
• more test cases, e.g. one gitrepo, 50 gitrepos, 100 clusters
• measure the time it takes to update the Rancher UI
• measure CPU usage