-
Notifications
You must be signed in to change notification settings - Fork 247
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
help request: Tail and systemd clusterinputs not working. #855
Comments
Thanks but looking at the helm chart it looks like those filters are enabled by default?
|
@jamesla Can you provide the configuration for Fluentbit, which resides in secret |
Sure thing - assuming you mean the secret
|
Please set the
|
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
name: docker
namespace: fluent
labels:
fluentbit.fluent.io/enabled: "true"
spec:
systemd:
tag: service.*
path: /var/log/journal
parser: containerd
db: /fluent-bit/tail/systemd.db
dbSync: Normal
systemdFilter:
- _SYSTEMD_UNIT=docker.service
- _SYSTEMD_UNIT=kubelet.service Get's the following error:
|
Sorry, my mistake, it should be set in |
@jamesla If you're installing using helm, the parser should be already there. So have you installed using helm? [Input]
Name tail
Path /var/log/containers/*.log
Refresh_Interval 10
Skip_Long_Lines true
DB /fluent-bit/tail/pos.db
DB.Sync Normal
Mem_Buf_Limit 5MB
Tag kube.* |
yep so basically for this testing my workflow is: minikube start #creating a fresh cluster
helm upgrade -i --set containerRuntime=containerd fluent-operator --create-namespace -n fluent https://github.com/fluent/fluent-operator/releases/download/v2.4.0/fluent-operator.tgz
kubectl apply -f %file_containing_example_above% |
ah yes ok this deploys ok: ---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
name: tail
namespace: fluent
labels:
fluentbit.fluent.io/enabled: "true"
spec:
tail:
tag: kube.*
path: /var/log/containers/*.log
parser: containerd
refreshIntervalSeconds: 10
memBufLimit: 5MB
skipLongLines: false
db: /fluent-bit/tail/pos.db
dbSync: Normal however when I create some container noise via the following: kubectl run hello-world --image hello-world all I see in the logs is still the output from the dummy log clusterinput (via
and have confirmed that there logs that I would expect fluentbit to be collecting through the tail clusterinput via: vagrant@carverlinux ~/pa-DRS-elastic (main) [1]> kubectl run hello-world --image hello-world
pod/hello-world created
vagrant@carverlinux ~/pa-DRS-elastic (main)> kubectl logs -f hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/ |
Did you set the ContainerLogRealPath, it is the really path of the log files, if not , try to set it. |
What should i set it to? |
well i tried that steps are:
and no container logs or systemd logs are being output |
I have tried adding this and nothing.
|
Please enable debug log to provide more info, and make sure the config of fluentbit in secret |
ok now we're getting some output
unsure why it's invalid though :S |
You'll need to find the real path: ls -al /var/log/containers |
looks like the right path:
|
ls -al /var/log |
$ ls -al /var/log
total 20
drwxr-xr-x 6 root root 4096 Jul 31 21:59 .
drwxr-xr-x 8 root root 200 Jul 31 21:28 ..
drwxr-xr-x 2 root root 4096 Aug 1 01:57 containers
drwx------ 3 root root 4096 Jul 31 21:28 crio
drwxr-xr-x 2 root root 4096 Jul 31 21:59 journal
drwxr-xr-x 18 root root 4096 Aug 1 01:20 pods $ ls -al /
total 24
drwxr-xr-x 20 root root 560 Jul 31 21:28 .
drwxr-xr-x 20 root root 560 Jul 31 21:28 ..
-rw-r--r-- 1 root root 7976 Apr 3 21:34 CHANGELOG
lrwxrwxrwx 1 root root 7 Apr 3 22:58 bin -> usr/bin
drwxr-xr-x 2 root root 60 Apr 4 00:02 boot
drwxr-xr-x 2 root root 4096 Jul 31 21:28 data
drwxr-xr-x 15 root root 3060 Jul 31 21:28 dev
drwxr-xr-x 28 root root 1280 Jul 31 21:28 etc
drwxr-xr-x 3 root root 60 Apr 4 00:19 home
drwxr-xr-x 1 docker docker 4096 Jun 27 09:18 hosthome
-rwxr-xr-x 1 root root 457 Apr 3 21:34 init
lrwxrwxrwx 1 root root 7 Apr 3 22:58 lib -> usr/lib
lrwxrwxrwx 1 root root 3 Apr 3 22:58 lib64 -> lib
drwxr-xr-x 2 root root 60 Apr 4 00:19 libexec
lrwxrwxrwx 1 root root 11 Apr 3 23:19 linuxrc -> bin/busybox
drwxr-xr-x 2 root root 40 Apr 3 21:34 media
drwxr-xr-x 3 root root 60 Jul 31 21:28 mnt
drwxr-xr-x 4 root root 80 Jul 31 21:28 opt
dr-xr-xr-x 332 root root 0 Jul 31 21:28 proc
drwxr-xr-x 3 root root 60 Jul 31 21:29 root
drwxr-xr-x 27 root root 800 Jul 31 21:29 run
lrwxrwxrwx 1 root root 8 Apr 3 22:58 sbin -> usr/sbin
drwxr-xr-x 2 root root 40 Apr 3 22:58 srv
dr-xr-xr-x 12 root root 0 Jul 31 21:28 sys
drwxrwxrwt 12 root root 280 Jul 31 21:29 tmp
drwxr-xr-x 7 root root 160 Apr 4 00:19 usr
drwxr-xr-x 8 root root 200 Jul 31 21:28 var
-rw-r--r-- 1 root root 159 Apr 3 22:52 version.json $ ls -al /var
total 12
drwxr-xr-x 8 root root 200 Jul 31 21:28 .
drwxr-xr-x 20 root root 560 Jul 31 21:28 ..
-rw-r--r-- 1 root root 208 Apr 4 00:19 .updated
drwxr-xr-x 3 root root 60 Jul 31 21:28 cache
drwxr-xr-x 2 root root 40 Apr 4 00:19 empty
drwxr-xr-x 23 root root 460 Jul 31 21:59 lib
drwxr-xr-x 6 root root 4096 Jul 31 21:59 log
lrwxrwxrwx 1 root root 6 Apr 3 22:58 run -> ../run
drwxr-xr-x 2 root root 40 Jul 31 21:28 spool
drwxr-xr-x 3 root root 4096 Jul 31 21:28 tmp |
You'll need the |
vagrant@carverlinux ~/pa-DRS-elastic (main)> minikube ssh
_ _
_ _ ( ) ( )
___ ___ (_) ___ (_)| |/') _ _ | |_ __
/' _ ` _ `\| |/' _ `\| || , < ( ) ( )| '_`\ /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )( ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)
$ ls -al /var/log/containers/kube-apiserver-pa-drs-cluster_kube-system_kube-apiserver-1ba81788e9d8a48998fea25d5fe511c774f9de76eaaca81c50ff44b5960223b2.log
lrwxrwxrwx 1 root root 109 Aug 1 04:40 /var/log/containers/kube-apiserver-pa-drs-cluster_kube-system_kube-apiserver-1ba81788e9d8a48998fea25d5fe511c774f9de76eaaca81c50ff44b5960223b2.log -> /var/log/pods/kube-system_kube-apiserver-pa-drs-cluster_8b1e9057c80f383869339e73f7286d7b/kube-apiserver/0.log |
First the
For example, run this command in a k8s cluster.
So the |
so given my output is like this: $ pwd
/var/log/containers
$ ls -al
total 96
drwxr-xr-x 2 root root 4096 Aug 1 19:52 '*.log'
drwxr-xr-x 3 root root 4096 Aug 2 01:54 .
drwxr-xr-x 6 root root 4096 Aug 1 04:47 ..
lrwxrwxrwx 1 root root 123 Aug 2 01:44 cert-manager-7b5cc56d74-xbcqr_cert-manager_cert-manager-controller-6919db12205ae4d35975c9ccf8f112ccd9be6a2ffed867a8f843420274ad701d.log -> /var/log/pods/cert-manager_cert-manager-7b5cc56d74-xbcqr_5e032ec6-e277-43d0-8b14-6e8e7f8edf52/cert-manager-controller/1.log
lrwxrwxrwx 1 root root 123 Aug 2 01:50 cert-manager-7b5cc56d74-xbcqr_cert-manager_cert-manager-controller-e5a7c92a8e39f83a8f7e1139a26de6eec1fa805593c8b10d2d021adb647edb67.log -> /var/log/pods/cert-manager_cert-manager-7b5cc56d74-xbcqr_5e032ec6-e277-43d0-8b14-6e8e7f8edf52/cert-manager-controller/2.log
lrwxrwxrwx 1 root root 134 Aug 2 01:50 cert-manager-cainjector-7d948796d5-mg7rh_cert-manager_cert-manager-cainjector-2e42a041992638ff7c8dcd8f45dcee2958c867cf39b04c870e936bbcfb536699.log -> /var/log/pods/cert-manager_cert-manager-cainjector-7d948796d5-mg7rh_12a63b1f-fa1d-4b7d-a00c-5781f93c9996/cert-manager-cainjector/2.log
lrwxrwxrwx 1 root root 134 Aug 2 01:44 cert-manager-cainjector-7d948796d5-mg7rh_cert-manager_cert-manager-cainjector-49c1b268679772f9828d1556e4f2dacae1727ffef13498f4f1de04602c939a14.log -> /var/log/pods/cert-manager_cert-manager-cainjector-7d948796d5-mg7rh_12a63b1f-fa1d-4b7d-a00c-5781f93c9996/cert-manager-cainjector/1.log
lrwxrwxrwx 1 root root 128 Aug 1 04:41 cert-manager-webhook-68f677967c-wtbwt_cert-manager_cert-manager-webhook-8e5db60d6e89d8545906940fc52e1380f9c7a56ef8f492a0e2366e7794e26c3d.log -> /var/log/pods/cert-manager_cert-manager-webhook-68f677967c-wtbwt_a6f666dd-5e4a-43a7-b71e-d5cf9500d918/cert-manager-webhook/0.log
lrwxrwxrwx 1 root root 101 Aug 1 04:40 coredns-787d4945fb-qskds_kube-system_coredns-2375bae8e35839c1a7d2d4ecbaa79907ab98b0cbcedd43d1082ff74a58b55b4a.log -> /var/log/pods/kube-system_coredns-787d4945fb-qskds_3c71e561-50c4-4d08-9fa2-3d74842d6b4e/coredns/0.log
lrwxrwxrwx 1 root root 97 Aug 2 01:44 elastic-operator-0_observability_manager-57ac0e7a5214ac924bc5fedb128cf57b8519f171b82c964f8bab1a341e0c64bb.log -> /var/log/pods/observability_elastic-operator-0_10f7129e-3e5e-4084-814d-0531544f2879/manager/1.log
lrwxrwxrwx 1 root root 97 Aug 2 01:50 elastic-operator-0_observability_manager-f263c39656573064c9d9211151177510cf354ce8c577902b921ef91b3e59da14.log -> /var/log/pods/observability_elastic-operator-0_10f7129e-3e5e-4084-814d-0531544f2879/manager/2.log
lrwxrwxrwx 1 root root 89 Aug 1 04:40 etcd-pa-drs-cluster_kube-system_etcd-25704aba434667bea6bacd3b9cabda4d6980cb41293dc63b6d17adba8734be4b.log -> /var/log/pods/kube-system_etcd-pa-drs-cluster_58123f9b092065056624138e28782ab7/etcd/0.log
lrwxrwxrwx 1 root root 91 Aug 2 01:53 fluent-bit-nnjnt_fluent_fluent-bit-aa50c6ba7b680e9f62a5eca6a163043fc4e35a12b4f4dd28f24b0530f0a8591f.log -> /var/log/pods/fluent_fluent-bit-nnjnt_83ae1a44-80e2-4d68-8129-278d0a4a5211/fluent-bit/0.log
lrwxrwxrwx 1 root root 112 Aug 1 04:46 fluent-operator-55b5859c44-l6p7w_fluent_fluent-operator-57e57f0f03ebae987505275bf243f487c4c81509846ff204d7498128f3716ca0.log -> /var/log/pods/fluent_fluent-operator-55b5859c44-l6p7w_42de5edd-59b7-490a-9bc8-ad3551de736b/fluent-operator/0.log
lrwxrwxrwx 1 root root 103 Aug 1 04:46 fluent-operator-55b5859c44-l6p7w_fluent_setenv-d33d69ca07bf0e0c30e6a5af3f16a76dbc0a88cc07b8f427145c7d38cbeb9e38.log -> /var/log/pods/fluent_fluent-operator-55b5859c44-l6p7w_42de5edd-59b7-490a-9bc8-ad3551de736b/setenv/0.log
lrwxrwxrwx 1 root root 88 Aug 2 01:54 hello-world_default_hello-world-5b5e858eeb1c8cf95680f575bff079c347d4717fab5581096299d1a9c08d8782.log -> /var/log/pods/default_hello-world_22b8f980-4f3f-4790-8e90-52486ba129c0/hello-world/3.log
lrwxrwxrwx 1 root root 109 Aug 1 04:40 kube-apiserver-pa-drs-cluster_kube-system_kube-apiserver-1ba81788e9d8a48998fea25d5fe511c774f9de76eaaca81c50ff44b5960223b2.log -> /var/log/pods/kube-system_kube-apiserver-pa-drs-cluster_8b1e9057c80f383869339e73f7286d7b/kube-apiserver/0.log
lrwxrwxrwx 1 root root 127 Aug 1 04:40 kube-controller-manager-pa-drs-cluster_kube-system_kube-controller-manager-1a2feb70140ea8dfb43e13b72caf2ad7b568962afe8ce291a125a7fd1d1dc888.log -> /var/log/pods/kube-system_kube-controller-manager-pa-drs-cluster_593dc083a6f5f7623ad9a4d6adba7b77/kube-controller-manager/0.log
lrwxrwxrwx 1 root root 96 Aug 1 04:40 kube-proxy-jzx5h_kube-system_kube-proxy-a5a0b85e7556854e96343f1ff0ecf82ad5945e606fcaec3c678cc7d2c8927259.log -> /var/log/pods/kube-system_kube-proxy-jzx5h_8d2e7303-4ec2-4c28-809e-c191f40c40ad/kube-proxy/0.log
lrwxrwxrwx 1 root root 109 Aug 1 04:40 kube-scheduler-pa-drs-cluster_kube-system_kube-scheduler-ade1da48e2f2c1cce256077ef4f7659aa1fc39029f7638f542c6ca01a085ef7d.log -> /var/log/pods/kube-system_kube-scheduler-pa-drs-cluster_7127d7352f28daa3649bab78b6964aee/kube-scheduler/0.log
lrwxrwxrwx 1 root root 137 Aug 1 04:41 mongodb-kubernetes-operator-66b8b9f5cc-kvmc7_mongodb_mongodb-kubernetes-operator-37c3865e7984f2a15aa8b949092aa3a2a2b18ddca146473c083440d5a6278692.log -> /var/log/pods/mongodb_mongodb-kubernetes-operator-66b8b9f5cc-kvmc7_1189fb0c-74b2-4af0-81bf-9f125afc7f54/mongodb-kubernetes-operator/0.log
lrwxrwxrwx 1 root root 121 Aug 1 04:44 nginx-ingress-nginx-controller-594495f5df-ldbh6_nginx_controller-4b3178c6f0ef3dacf2174414e40de8ee230cfb90cdb00f958b7677a4929fc15d.log -> /var/log/pods/nginx_nginx-ingress-nginx-controller-594495f5df-ldbh6_31015c21-c2e0-42c3-a53e-f96736635689/controller/0.log
lrwxrwxrwx 1 root root 108 Aug 1 04:41 storage-provisioner_kube-system_storage-provisioner-60defc68607dd0f620c33a518d859bdd7cbdfd4ba387691803ba1481eae7c497.log -> /var/log/pods/kube-system_storage-provisioner_3abd4642-8b22-4289-a6af-cc6ecda9fe72/storage-provisioner/1.log
lrwxrwxrwx 1 root root 108 Aug 1 04:40 storage-provisioner_kube-system_storage-provisioner-c60caee3cfca6071db4c1c8bb87c662e8f9514e89d30d8a1e504b08de56e86d9.log -> /var/log/pods/kube-system_storage-provisioner_3abd4642-8b22-4289-a6af-cc6ecda9fe72/storage-provisioner/0.log Are you suggesting I update |
Ok I've tried this and it doesn't work either
Why don't you guys try it - I don't think it's my machine or configuration you simply just have to make sure minikube is instlled and minikube start #creating a fresh cluster
helm upgrade -i --set containerRuntime=containerd fluent-operator --create-namespace -n fluent https://github.com/fluent/fluent-operator/releases/download/v2.4.0/fluent-operator.tgz
kubectl apply -f %file_containing_example_above% |
I don't think you get what @wanjunlei says, you'll need to find out the real path of your env. Not copying the real path of our env directly. You'll need to understand what this is doing |
Oh yes I see $ sudo ls -al /var/log/pods/kube-system_kube-apiserver-pa-drs-cluster_8b1e9057c80f383869339e73f7286d7b/kube-apiserver/
total 12
drwxr-xr-x 2 root root 4096 Aug 1 04:40 .
drwxr-xr-x 3 root root 4096 Aug 1 04:40 ..
lrwxrwxrwx 1 root root 165 Aug 1 04:40 0.log -> /var/lib/docker/containers/1ba81788e9d8a48998fea25d5fe511c774f9de76eaaca81c50ff44b5960223b2/1ba81788e9d8a48998fea25d5fe511c774f9de76eaaca81c50ff44b5960223b2-json.log so would this mean that my container log path would be the following? ...
containerLogRealPath: /var/lib/docker/containers
... or would it be: ...
containerLogRealPath: /var/lib/docker/containers/*/*.log
... |
The
|
In addition, the |
@jamesla valid values for parser are |
That was it thank you guys for all your help! |
Describe the issue
I have a simple set of manifests based on the quickstart that simply configure a fluentbit that takes input from the dummy, container logs (tail) and kubernetes system logs (systemd) and outputs them to stdout.
Now the dummy works great, however the systemd and the tail inputs are not working.
Output looks like this even when killing pods and confirming they are writing to stdout (via
kubectl logs ....
):Any idea how to troubleshoot this?
How did you install fluent operator?
helm chart
Additional context
2.4.0
@wenchajun @benjaminhuo
The text was updated successfully, but these errors were encountered: