Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

help request: Tail and systemd clusterinputs not working. #855

Closed
jamesla opened this issue Jul 30, 2023 · 35 comments
Closed

help request: Tail and systemd clusterinputs not working. #855

jamesla opened this issue Jul 30, 2023 · 35 comments
Assignees

Comments

@jamesla
Copy link

jamesla commented Jul 30, 2023

Describe the issue

I have a simple set of manifests based on the quickstart that simply configure a fluentbit that takes input from the dummy, container logs (tail) and kubernetes system logs (systemd) and outputs them to stdout.

Now the dummy works great, however the systemd and the tail inputs are not working.

---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: FluentBit
metadata:
  name: fluent-bit
  namespace: fluent
  labels:
    app.kubernetes.io/name: fluent-bit
spec:
  image: kubesphere/fluent-bit:v2.1.7
  fluentBitConfigName: fluent-bit-config
  positionDB:
    hostPath:
      path: /var/lib/fluent-bit/
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterFluentBitConfig
metadata:
  name: fluent-bit-config
  namespace: fluent
  labels:
    app.kubernetes.io/name: fluent-bit
spec:
  inputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  outputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
  name: docker
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  systemd:
    tag: service.*
    path: /var/log/journal
    db: /fluent-bit/tail/systemd.db
    dbSync: Normal
    systemdFilter:
      - _SYSTEMD_UNIT=docker.service
      - _SYSTEMD_UNIT=kubelet.service
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
  name: tail
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  tail:
    tag: kube.*
    path: /var/log/containers/*.log
    refreshIntervalSeconds: 10
    memBufLimit: 5MB
    skipLongLines: true
    db: /fluent-bit/tail/pos.db
    dbSync: Normal
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
  name: dummy
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  dummy:
    tag: my_dummy
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  name: stdout
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  match: "*"
  stdout: {}

Output looks like this even when killing pods and confirming they are writing to stdout (via kubectl logs ....):

[0] my_dummy: [[1690707109.298142698, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707110.298840841, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707111.299267477, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707112.298684587, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707113.299942396, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707114.299799594, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707115.299161966, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707116.298993463, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707117.298655826, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707118.298470887, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707119.299251398, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707120.298838334, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707121.298885289, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707122.298978125, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707123.298685272, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707124.298576172, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707125.298208337, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707126.298342160, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707127.298894887, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707128.298762534, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707129.298756377, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707130.298633375, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707131.298095696, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707132.300729192, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690707133.299339113, {}], {"message"=>"dummy"}]

Any idea how to troubleshoot this?

How did you install fluent operator?

helm chart

Additional context

2.4.0

@wenchajun @benjaminhuo

@wanjunlei
Copy link
Collaborator

@jamesla Can you provide the configuration for Fluentbit, which resides in secret fluent-bit-conf in the namespace fluent?

@jamesla
Copy link
Author

jamesla commented Jul 31, 2023

@jamesla Can you provide the configuration for Fluentbit, which resides in secret fluent-bit-conf in the namespace fluent?

Sure thing - assuming you mean the secret fluent-bit-config

apiVersion: v1
data:
  containerd.lua: ZnVuY3Rpb24gY29udGFpbmVyZCggdGFnLCB0aW1lc3RhbXAsIHJlY29yZCkKICAgICAgIGlmKHJlY29yZFsibG9ndGFnIl1+PW5pbCkKICAgICAgIHRoZW4KICAgICAgIHRpbWVTdHIgPSBvcy5kYXRlKCIhKnQiLCAgdGltZXN0YW1wWyJzZWMiXSkKICAgICAgICB0ID0gc3RyaW5nLmZvcm1hdCgiJTRkLSUwMmQtJTAyZFQlMDJkOiUwMmQ6JTAyZC4lc1oiLAogICAgICAgIHRpbWVTdHJbInllYXIiXSwgdGltZVN0clsibW9udGgiXSwgdGltZVN0clsiZGF5Il0sCiAgICAgICAgdGltZVN0clsiaG91ciJdLCB0aW1lU3RyWyJtaW4iXSwgdGltZVN0clsic2VjIl0sCiAgICAgICAgdGltZXN0YW1wWyJuc2VjIl0pOwogICAgICAgIHJlY29yZFsidGltZSJdID0gdDsKICAgICAgICByZWNvcmRbImxvZyJdID0gcmVjb3JkWyJtZXNzYWdlIl07CiAgICAgICAgcmVjb3JkWyJtZXNzYWdlIl0gPSAgbmlsOwogICAgICAgIHJldHVybiAxLCB0aW1lc3RhbXAsIHJlY29yZAogICAgICAgIGVsc2UKICAgICAgICByZXR1cm4gMCx0aW1lc3RhbXAscmVjb3JkCiAgICAgICBlbmQKZW5k
  fluent-bit.conf: W0lucHV0XQogICAgTmFtZSAgICBzeXN0ZW1kCiAgICBQYXRoICAgIC92YXIvbG9nL2pvdXJuYWwKICAgIERCICAgIC9mbHVlbnQtYml0L3RhaWwvc3lzdGVtZC5kYgogICAgREIuU3luYyAgICBOb3JtYWwKICAgIFRhZyAgICBzZXJ2aWNlLioKICAgIFN5c3RlbWRfRmlsdGVyICAgIF9TWVNURU1EX1VOSVQ9Y29udGFpbmVyZC5zZXJ2aWNlCiAgICBTeXN0ZW1kX0ZpbHRlciAgICBfU1lTVEVNRF9VTklUPWt1YmVsZXQuc2VydmljZQogICAgU3RyaXBfVW5kZXJzY29yZXMgICAgb2ZmCiAgICBzdG9yYWdlLnR5cGUgICAgbWVtb3J5CltJbnB1dF0KICAgIE5hbWUgICAgc3lzdGVtZAogICAgUGF0aCAgICAvdmFyL2xvZy9qb3VybmFsCiAgICBEQiAgICAvZmx1ZW50LWJpdC90YWlsL3N5c3RlbWQuZGIKICAgIERCLlN5bmMgICAgTm9ybWFsCiAgICBUYWcgICAgc2VydmljZS4qCiAgICBTeXN0ZW1kX0ZpbHRlciAgICBfU1lTVEVNRF9VTklUPWRvY2tlci5zZXJ2aWNlCiAgICBTeXN0ZW1kX0ZpbHRlciAgICBfU1lTVEVNRF9VTklUPWt1YmVsZXQuc2VydmljZQpbSW5wdXRdCiAgICBOYW1lICAgIGR1bW15CiAgICBUYWcgICAgbXlfZHVtbXkKW0lucHV0XQogICAgTmFtZSAgICB0YWlsCiAgICBQYXRoICAgIC92YXIvbG9nL2NvbnRhaW5lcnMvKi5sb2cKICAgIFJlZnJlc2hfSW50ZXJ2YWwgICAgMTAKICAgIFNraXBfTG9uZ19MaW5lcyAgICB0cnVlCiAgICBEQiAgICAvZmx1ZW50LWJpdC90YWlsL3Bvcy5kYgogICAgREIuU3luYyAgICBOb3JtYWwKICAgIE1lbV9CdWZfTGltaXQgICAgNU1CCiAgICBUYWcgICAga3ViZS4qCltGaWx0ZXJdCiAgICBOYW1lICAgIGx1YQogICAgTWF0Y2ggICAga3ViZS4qCiAgICBzY3JpcHQgICAgL2ZsdWVudC1iaXQvY29uZmlnL2NvbnRhaW5lcmQubHVhCiAgICBjYWxsICAgIGNvbnRhaW5lcmQKICAgIHRpbWVfYXNfdGFibGUgICAgdHJ1ZQpbRmlsdGVyXQogICAgTmFtZSAgICBrdWJlcm5ldGVzCiAgICBNYXRjaCAgICBrdWJlLioKICAgIEt1YmVfVVJMICAgIGh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yzo0NDMKICAgIEt1YmVfQ0FfRmlsZSAgICAvdmFyL3J1bi9zZWNyZXRzL2t1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvY2EuY3J0CiAgICBLdWJlX1Rva2VuX0ZpbGUgICAgL3Zhci9ydW4vc2VjcmV0cy9rdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3Rva2VuCiAgICBMYWJlbHMgICAgZmFsc2UKICAgIEFubm90YXRpb25zICAgIGZhbHNlCltGaWx0ZXJdCiAgICBOYW1lICAgIG5lc3QKICAgIE1hdGNoICAgIGt1YmUuKgogICAgT3BlcmF0aW9uICAgIGxpZnQKICAgIE5lc3RlZF91bmRlciAgICBrdWJlcm5ldGVzCiAgICBBZGRfcHJlZml4ICAgIGt1YmVybmV0ZXNfCltGaWx0ZXJdCiAgICBOYW1lICAgIG1vZGlmeQogICAgTWF0Y2ggICAga3ViZS4qCiAgICBSZW1vdmUgICAgc3RyZWFtCiAgICBSZW1vdmUgICAga3ViZXJuZXRlc19wb2RfaWQKICAgIFJlbW92ZSAgICBrdWJlcm5ldGVzX2hvc3QKICAgIFJlbW92ZSAgICBrdWJlcm5ldGVzX2NvbnRhaW5lcl9oYXNoCltGaWx0ZXJdCiAgICBOYW1lICAgIG5lc3QKICAgIE1hdGNoICAgIGt1YmUuKgogICAgT3BlcmF0aW9uICAgIG5lc3QKICAgIFdpbGRjYXJkICAgIGt1YmVybmV0ZXNfKgogICAgTmVzdF91bmRlciAgICBrdWJlcm5ldGVzCiAgICBSZW1vdmVfcHJlZml4ICAgIGt1YmVybmV0ZXNfCltGaWx0ZXJdCiAgICBOYW1lICAgIGx1YQogICAgTWF0Y2ggICAgc2VydmljZS4qCiAgICBzY3JpcHQgICAgL2ZsdWVudC1iaXQvY29uZmlnL3N5c3RlbWQubHVhCiAgICBjYWxsICAgIGFkZF90aW1lCiAgICB0aW1lX2FzX3RhYmxlICAgIHRydWUKW091dHB1dF0KICAgIE5hbWUgICAgc3Rkb3V0CiAgICBNYXRjaCAgICAqCg==
  parsers.conf: ""
  systemd.lua: ZnVuY3Rpb24gYWRkX3RpbWUodGFnLCB0aW1lc3RhbXAsIHJlY29yZCkKICBuZXdfcmVjb3JkID0ge30KCiAgdGltZVN0ciA9IG9zLmRhdGUoIiEqdCIsIHRpbWVzdGFtcFsic2VjIl0pCiAgdCA9IHN0cmluZy5mb3JtYXQoIiU0ZC0lMDJkLSUwMmRUJTAyZDolMDJkOiUwMmQuJXNaIiwKICB0aW1lU3RyWyJ5ZWFyIl0sIHRpbWVTdHJbIm1vbnRoIl0sIHRpbWVTdHJbImRheSJdLAogIHRpbWVTdHJbImhvdXIiXSwgdGltZVN0clsibWluIl0sIHRpbWVTdHJbInNlYyJdLAogIHRpbWVzdGFtcFsibnNlYyJdKQoKICBrdWJlcm5ldGVzID0ge30KICBrdWJlcm5ldGVzWyJwb2RfbmFtZSJdID0gcmVjb3JkWyJfSE9TVE5BTUUiXQogIGt1YmVybmV0ZXNbImNvbnRhaW5lcl9uYW1lIl0gPSByZWNvcmRbIlNZU0xPR19JREVOVElGSUVSIl0KICBrdWJlcm5ldGVzWyJuYW1lc3BhY2VfbmFtZSJdID0gImt1YmUtc3lzdGVtIgoKICBuZXdfcmVjb3JkWyJ0aW1lIl0gPSB0CiAgbmV3X3JlY29yZFsibG9nIl0gPSByZWNvcmRbIk1FU1NBR0UiXQogIG5ld19yZWNvcmRbImt1YmVybmV0ZXMiXSA9IGt1YmVybmV0ZXMKCiAgcmV0dXJuIDEsIHRpbWVzdGFtcCwgbmV3X3JlY29yZAplbmQ=
kind: Secret
metadata:
  creationTimestamp: "2023-07-30T08:41:07Z"
  name: fluent-bit-config
  namespace: fluent
  ownerReferences:
  - apiVersion: fluentbit.fluent.io/v1alpha2
    blockOwnerDeletion: true
    controller: true
    kind: ClusterFluentBitConfig
    name: fluent-bit-config
    uid: ed5bcd8f-6c60-403b-b866-e023bbbf85cc
  resourceVersion: "3915"
  uid: 8a8d23aa-6e48-42cf-9ba4-7e1d2198230f
type: Opaque

@wanjunlei
Copy link
Collaborator

Please set the parser field in the ClusterInput.

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
  name: docker
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  systemd:
    tag: service.*
    path: /var/log/journal
    parser: containerd # or docker
    db: /fluent-bit/tail/systemd.db
    dbSync: Normal
    systemdFilter:
      - _SYSTEMD_UNIT=docker.service
      - _SYSTEMD_UNIT=kubelet.service
---

@jamesla
Copy link
Author

jamesla commented Jul 31, 2023

parser: containerd # or docker

---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
  name: docker
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  systemd:
    tag: service.*
    path: /var/log/journal
    parser: containerd
    db: /fluent-bit/tail/systemd.db
    dbSync: Normal
    systemdFilter:
      - _SYSTEMD_UNIT=docker.service
      - _SYSTEMD_UNIT=kubelet.service

Get's the following error:

The request is invalid: patch: Invalid value: "{\"apiVersion\":\"fluentbit.fluent.io/v1alpha2\",\"kind\":\"ClusterInput\",\"metadata\":{\"annotations\":{\"kubectl.kubernetes.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"fluentbit.fluent.io/v1alpha2\\\",\\\"kind\\\":\\\"ClusterInput\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"labels\\\":{\\\"fluentbit.fluent.io/enabled\\\":\\\"true\\\"},\\\"name\\\":\\\"docker\\\"},\\\"spec\\\":{\\\"systemd\\\":{\\\"db\\\":\\\"/fluent-bit/tail/systemd.db\\\",\\\"dbSync\\\":\\\"Normal\\\",\\\"parser\\\":\\\"containerd\\\",\\\"path\\\":\\\"/var/log/journal\\\",\\\"systemdFilter\\\":[\\\"_SYSTEMD_UNIT=docker.service\\\",\\\"_SYSTEMD_UNIT=kubelet.service\\\"],\\\"tag\\\":\\\"service.*\\\"}}}\\n\"},\"creationTimestamp\":\"2023-07-30T08:41:07Z\",\"generation\":1,\"labels\":{\"fluentbit.fluent.io/enabled\":\"true\"},\"managedFields\":[{\"apiVersion\":\"fluentbit.fluent.io/v1alpha2\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:kubectl.kubernetes.io/last-applied-configuration\":{}},\"f:labels\":{\".\":{},\"f:fluentbit.fluent.io/enabled\":{}}},\"f:spec\":{\".\":{},\"f:systemd\":{\".\":{},\"f:db\":{},\"f:dbSync\":{},\"f:path\":{},\"f:systemdFilter\":{},\"f:tag\":{}}}},\"manager\":\"kubectl-client-side-apply\",\"operation\":\"Update\",\"time\":\"2023-07-30T08:41:07Z\"}],\"name\":\"docker\",\"resourceVersion\":\"3258\",\"uid\":\"5cb02c33-c105-407e-a83d-db9a7d258798\"},\"spec\":{\"systemd\":{\"db\":\"/fluent-bit/tail/systemd.db\",\"dbSync\":\"Normal\",\"parser\":\"containerd\",\"path\":\"/var/log/journal\",\"systemdFilter\":[\"_SYSTEMD_UNIT=docker.service\",\"_SYSTEMD_UNIT=kubelet.service\"],\"tag\":\"service.*\"}}}": strict decoding error: unknown field "spec.systemd.parser"

@wanjunlei
Copy link
Collaborator

parser: containerd # or docker

---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
  name: docker
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  systemd:
    tag: service.*
    path: /var/log/journal
    parser: containerd
    db: /fluent-bit/tail/systemd.db
    dbSync: Normal
    systemdFilter:
      - _SYSTEMD_UNIT=docker.service
      - _SYSTEMD_UNIT=kubelet.service

Get's the following error:

The request is invalid: patch: Invalid value: "{\"apiVersion\":\"fluentbit.fluent.io/v1alpha2\",\"kind\":\"ClusterInput\",\"metadata\":{\"annotations\":{\"kubectl.kubernetes.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"fluentbit.fluent.io/v1alpha2\\\",\\\"kind\\\":\\\"ClusterInput\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"labels\\\":{\\\"fluentbit.fluent.io/enabled\\\":\\\"true\\\"},\\\"name\\\":\\\"docker\\\"},\\\"spec\\\":{\\\"systemd\\\":{\\\"db\\\":\\\"/fluent-bit/tail/systemd.db\\\",\\\"dbSync\\\":\\\"Normal\\\",\\\"parser\\\":\\\"containerd\\\",\\\"path\\\":\\\"/var/log/journal\\\",\\\"systemdFilter\\\":[\\\"_SYSTEMD_UNIT=docker.service\\\",\\\"_SYSTEMD_UNIT=kubelet.service\\\"],\\\"tag\\\":\\\"service.*\\\"}}}\\n\"},\"creationTimestamp\":\"2023-07-30T08:41:07Z\",\"generation\":1,\"labels\":{\"fluentbit.fluent.io/enabled\":\"true\"},\"managedFields\":[{\"apiVersion\":\"fluentbit.fluent.io/v1alpha2\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:kubectl.kubernetes.io/last-applied-configuration\":{}},\"f:labels\":{\".\":{},\"f:fluentbit.fluent.io/enabled\":{}}},\"f:spec\":{\".\":{},\"f:systemd\":{\".\":{},\"f:db\":{},\"f:dbSync\":{},\"f:path\":{},\"f:systemdFilter\":{},\"f:tag\":{}}}},\"manager\":\"kubectl-client-side-apply\",\"operation\":\"Update\",\"time\":\"2023-07-30T08:41:07Z\"}],\"name\":\"docker\",\"resourceVersion\":\"3258\",\"uid\":\"5cb02c33-c105-407e-a83d-db9a7d258798\"},\"spec\":{\"systemd\":{\"db\":\"/fluent-bit/tail/systemd.db\",\"dbSync\":\"Normal\",\"parser\":\"containerd\",\"path\":\"/var/log/journal\",\"systemdFilter\":[\"_SYSTEMD_UNIT=docker.service\",\"_SYSTEMD_UNIT=kubelet.service\"],\"tag\":\"service.*\"}}}": strict decoding error: unknown field "spec.systemd.parser"

Sorry, my mistake, it should be set in tail ClusterInput.

@benjaminhuo
Copy link
Member

@jamesla If you're installing using helm, the parser should be already there. So have you installed using helm?
https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/templates/fluentbit-clusterinput-tail.yaml#L17

[Input]
    Name    tail
    Path    /var/log/containers/*.log
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/tail/pos.db
    DB.Sync    Normal
    Mem_Buf_Limit    5MB
    Tag    kube.*

@jamesla
Copy link
Author

jamesla commented Jul 31, 2023

@jamesla If you're installing using helm, the parser should be already there. So have you installed using helm? https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/templates/fluentbit-clusterinput-tail.yaml#L17

[Input]
    Name    tail
    Path    /var/log/containers/*.log
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/tail/pos.db
    DB.Sync    Normal
    Mem_Buf_Limit    5MB
    Tag    kube.*

yep so basically for this testing my workflow is:

minikube start #creating a fresh cluster
helm upgrade -i --set containerRuntime=containerd fluent-operator --create-namespace -n fluent https://github.com/fluent/fluent-operator/releases/download/v2.4.0/fluent-operator.tgz
kubectl apply -f %file_containing_example_above%

@jamesla
Copy link
Author

jamesla commented Jul 31, 2023

parser: containerd # or docker

---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
  name: docker
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  systemd:
    tag: service.*
    path: /var/log/journal
    parser: containerd
    db: /fluent-bit/tail/systemd.db
    dbSync: Normal
    systemdFilter:
      - _SYSTEMD_UNIT=docker.service
      - _SYSTEMD_UNIT=kubelet.service

Get's the following error:

The request is invalid: patch: Invalid value: "{\"apiVersion\":\"fluentbit.fluent.io/v1alpha2\",\"kind\":\"ClusterInput\",\"metadata\":{\"annotations\":{\"kubectl.kubernetes.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"fluentbit.fluent.io/v1alpha2\\\",\\\"kind\\\":\\\"ClusterInput\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"labels\\\":{\\\"fluentbit.fluent.io/enabled\\\":\\\"true\\\"},\\\"name\\\":\\\"docker\\\"},\\\"spec\\\":{\\\"systemd\\\":{\\\"db\\\":\\\"/fluent-bit/tail/systemd.db\\\",\\\"dbSync\\\":\\\"Normal\\\",\\\"parser\\\":\\\"containerd\\\",\\\"path\\\":\\\"/var/log/journal\\\",\\\"systemdFilter\\\":[\\\"_SYSTEMD_UNIT=docker.service\\\",\\\"_SYSTEMD_UNIT=kubelet.service\\\"],\\\"tag\\\":\\\"service.*\\\"}}}\\n\"},\"creationTimestamp\":\"2023-07-30T08:41:07Z\",\"generation\":1,\"labels\":{\"fluentbit.fluent.io/enabled\":\"true\"},\"managedFields\":[{\"apiVersion\":\"fluentbit.fluent.io/v1alpha2\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:kubectl.kubernetes.io/last-applied-configuration\":{}},\"f:labels\":{\".\":{},\"f:fluentbit.fluent.io/enabled\":{}}},\"f:spec\":{\".\":{},\"f:systemd\":{\".\":{},\"f:db\":{},\"f:dbSync\":{},\"f:path\":{},\"f:systemdFilter\":{},\"f:tag\":{}}}},\"manager\":\"kubectl-client-side-apply\",\"operation\":\"Update\",\"time\":\"2023-07-30T08:41:07Z\"}],\"name\":\"docker\",\"resourceVersion\":\"3258\",\"uid\":\"5cb02c33-c105-407e-a83d-db9a7d258798\"},\"spec\":{\"systemd\":{\"db\":\"/fluent-bit/tail/systemd.db\",\"dbSync\":\"Normal\",\"parser\":\"containerd\",\"path\":\"/var/log/journal\",\"systemdFilter\":[\"_SYSTEMD_UNIT=docker.service\",\"_SYSTEMD_UNIT=kubelet.service\"],\"tag\":\"service.*\"}}}": strict decoding error: unknown field "spec.systemd.parser"

Sorry, my mistake, it should be set in tail ClusterInput.

ah yes ok this deploys ok:

---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
  name: tail
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  tail:
    tag: kube.*
    path: /var/log/containers/*.log
    parser: containerd
    refreshIntervalSeconds: 10
    memBufLimit: 5MB
    skipLongLines: false
    db: /fluent-bit/tail/pos.db
    dbSync: Normal

however when I create some container noise via the following:

kubectl run hello-world --image hello-world

all I see in the logs is still the output from the dummy log clusterinput (via kubectl -n fluent logs -f fluent-bit-598rs):

[0] my_dummy: [[1690783835.015938479, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783836.015781209, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783837.017540598, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783838.015903211, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783839.015844818, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783840.016366473, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783841.015434044, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783842.016123161, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783843.015940927, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783844.015421431, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783845.017177095, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783846.016133311, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783847.015254509, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783848.015197357, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783849.016135925, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783850.015259655, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783851.015805810, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783852.016517558, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783853.015871373, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783854.015613253, {}], {"message"=>"dummy"}]
[0] my_dummy: [[1690783855.015283042, {}], {"message"=>"dummy"}]

and have confirmed that there logs that I would expect fluentbit to be collecting through the tail clusterinput via:

vagrant@carverlinux ~/pa-DRS-elastic (main) [1]> kubectl run hello-world --image hello-world
pod/hello-world created
vagrant@carverlinux ~/pa-DRS-elastic (main)> kubectl logs -f hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

@wanjunlei
Copy link
Collaborator

Did you set the ContainerLogRealPath, it is the really path of the log files, if not , try to set it.

@benjaminhuo
Copy link
Member

kubectl apply -f %file_containing_example_above%

You don't need this step because the helm include these already
image

@jamesla
Copy link
Author

jamesla commented Jul 31, 2023

Did you set the ContainerLogRealPath, it is the really path of the log files, if not , try to set it.

What should i set it to?

@jamesla
Copy link
Author

jamesla commented Jul 31, 2023

```shell
kubectl apply -f %file_containing_example_above%

You don't need this step because the helm include these already

well i tried that steps are:

# 1. start fresh minikube
minikube start

# 2. install fluentbit operator
helm upgrade -i --set containerRuntime=containerd fluent-operator --create-namespace -n fluent https://github.com/fluent/fluent-operator/releases/download/v2.4.0/fluent-operator.tgz

#3 run container to generate output
kubectl run hello-world --image hello-world

#4 check logs 
kubectl -n fluent logs -f fluent-bit-xggqj

and no container logs or systemd logs are being output

@jamesla
Copy link
Author

jamesla commented Jul 31, 2023

Did you set the ContainerLogRealPath, it is the really path of the log files, if not , try to set it.

I have tried adding this and nothing.

---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: FluentBit
metadata:
  name: fluent-bit
  namespace: fluent
  labels:
    app.kubernetes.io/name: fluent-bit
spec:
  image: kubesphere/fluent-bit:v2.1.7
  fluentBitConfigName: fluent-bit-config
  ContainerLogRealPath: /var/log/containers/*.log
  positionDB:
    hostPath:
      path: /var/lib/fluent-bit/

@wanjunlei
Copy link
Collaborator

Please enable debug log to provide more info, and make sure the config of fluentbit in secret fluent-bit-config had changed after the debug log enabled.

@jamesla
Copy link
Author

jamesla commented Aug 1, 2023

Please enable debug log to provide more info, and make sure the config of fluentbit in secret fluent-bit-config had changed after the debug log enabled.

ok now we're getting some output

[2023/08/01 01:14:19] [debug] [input:tail:tail.1] scanning path /var/log/containers/*.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] skip (invalid) entry=/var/log/containers/cert-manager-7b5cc56d74-9qh2l_cert-manager_cert-manager-controller-9a1307b74b75f3467cf9794b0c85b04c7e1d7ddc8fa127d3297158dabbb533b9.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] skip (invalid) entry=/var/log/containers/cert-manager-cainjector-7d948796d5-frh9n_cert-manager_cert-manager-cainjector-9c185b79381e0e3bd10f57ee70b6d182b1b79e6b56e95bf7986135de8b98a331.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] skip (invalid) entry=/var/log/containers/cert-manager-webhook-68f677967c-btbqn_cert-manager_cert-manager-webhook-d7a81f518752cd7bb3bcab23e783c13d25acb66312dee551625bcc752d11a8b7.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] skip (invalid) entry=/var/log/containers/coredns-787d4945fb-q9bzl_kube-system_coredns-c2e13799615bfda1d022bccc8927aa7702fa46b42b710cc04990898f2c1751b8.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] skip (invalid) entry=/var/log/containers/elastic-operator-0_observability_manager-8fd12ca5a1661751a90a9d9215751ee9eff91f84ff25c62727abeed504e973d2.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] skip (invalid) entry=/var/log/containers/etcd-pa-drs-cluster_kube-system_etcd-58d25c6b072abf6bbab39c8f7e0b94c7527200bb90a6cc8ee98c2c9610439baf.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] skip (invalid) entry=/var/log/containers/fluent-bit-xggqj_fluent_fluent-bit-87f0f77a06ce6ee443db8596a423586295c1405bf96a0e424e8528aab458626a.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] skip (invalid) entry=/var/log/containers/fluent-operator-55b5859c44-f6g57_fluent_fluent-operator-73b2dca0c75f14d1a5f2f4e60d068e7c29e914565c1374a35f812c6c200811da.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] skip (invalid) entry=/var/log/containers/fluent-operator-55b5859c44-f6g57_fluent_setenv-a8f5b6bc08cd5fcf226d87947b92bdb3aef8d1778f79aac17fae23849c457ff5.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] skip (invalid) entry=/var/log/containers/hello-world_default_hello-world-efcbdf9bdf66f22eb706e4f9a336dee824d5946ab0d9ffef3bfb0f440bcff0ba.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] skip (invalid) entry=/var/log/containers/kube-apiserver-pa-drs-cluster_kube-system_kube-apiserver-4e6cc0b351a40c7806bf12d2c5b0ce014d05434eaf8e4d1fc6dd05a1c5e2a95e.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] skip (invalid) entry=/var/log/containers/kube-controller-manager-pa-drs-cluster_kube-system_kube-controller-manager-789e6dab5d8fa9e4c23fedeeef2cbb335e6b805b78e7439902b06ad48964c3dd.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] skip (invalid) entry=/var/log/containers/kube-proxy-gm5qv_kube-system_kube-proxy-b23219a4e2ce1096920478e1655e3774362751673ad36ed8fbb51b26dcb81b23.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] skip (invalid) entry=/var/log/containers/kube-scheduler-pa-drs-cluster_kube-system_kube-scheduler-19752f5f9dd53739d109da6b4f6d954bff8cf5c0f5afd149851de8455fe3cc5d.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] skip (invalid) entry=/var/log/containers/mongodb-kubernetes-operator-66b8b9f5cc-7zxv2_mongodb_mongodb-kubernetes-operator-7448497d4a69dd04a5fb13744a935d70a338aeb577212e22c116166d1998d6b0.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] skip (invalid) entry=/var/log/containers/nginx-ingress-nginx-controller-594495f5df-jcknv_nginx_controller-1d0f861bae6376a5ca0929cab0b56f8b6aa59b5803fb419bb8a904ab7db0728d.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] skip (invalid) entry=/var/log/containers/storage-provisioner_kube-system_storage-provisioner-2ee9333a7b75bc538754b08e455a74e8569b3c7eae7e426eca9708cf9086253c.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] skip (invalid) entry=/var/log/containers/storage-provisioner_kube-system_storage-provisioner-cbf722fbc9dffd321c87370305a5baebe4d6660f4e448a2d9a15dcc3fbb43ae4.log
[2023/08/01 01:14:19] [debug] [input:tail:tail.1] 0 new files found on path '/var/log/containers/*.log'

unsure why it's invalid though :S

@benjaminhuo
Copy link
Member

You'll need to find the real path:

ls -al /var/log/containers

@jamesla
Copy link
Author

jamesla commented Aug 1, 2023

You'll need to find the real path:

ls -al /var/log/containers

looks like the right path:

$ ls -al /var/log/containers # from the node
total 80
drwxr-xr-x 2 root root 4096 Aug  1 01:46 .
drwxr-xr-x 6 root root 4096 Jul 31 21:59 ..
lrwxrwxrwx 1 root root  123 Jul 31 21:30 cert-manager-7b5cc56d74-9qh2l_cert-manager_cert-manager-controller-9a1307b74b75f3467cf9794b0c85b04c7e1d7ddc8fa127d3297158dabbb533b9.log -> /var/log/pods/cert-manager_cert-manager-7b5cc56d74-9qh2l_36a8d503-9f38-4df2-b7fe-e57dbfb87596/cert-manager-controller/0.log
lrwxrwxrwx 1 root root  134 Jul 31 21:31 cert-manager-cainjector-7d948796d5-frh9n_cert-manager_cert-manager-cainjector-9c185b79381e0e3bd10f57ee70b6d182b1b79e6b56e95bf7986135de8b98a331.log -> /var/log/pods/cert-manager_cert-manager-cainjector-7d948796d5-frh9n_37538bbd-51b1-492a-89f0-6ffd52833d56/cert-manager-cainjector/0.log
lrwxrwxrwx 1 root root  128 Jul 31 21:30 cert-manager-webhook-68f677967c-btbqn_cert-manager_cert-manager-webhook-d7a81f518752cd7bb3bcab23e783c13d25acb66312dee551625bcc752d11a8b7.log -> /var/log/pods/cert-manager_cert-manager-webhook-68f677967c-btbqn_b0636d8f-caaf-4b04-891e-442fe7c52027/cert-manager-webhook/0.log
lrwxrwxrwx 1 root root  101 Jul 31 21:29 coredns-787d4945fb-q9bzl_kube-system_coredns-c2e13799615bfda1d022bccc8927aa7702fa46b42b710cc04990898f2c1751b8.log -> /var/log/pods/kube-system_coredns-787d4945fb-q9bzl_6950ee14-a2e7-45f2-9c5d-e89eec72fcf8/coredns/0.log
lrwxrwxrwx 1 root root   97 Jul 31 21:30 elastic-operator-0_observability_manager-8fd12ca5a1661751a90a9d9215751ee9eff91f84ff25c62727abeed504e973d2.log -> /var/log/pods/observability_elastic-operator-0_7fbcc7c1-35cc-4637-b9ed-34b4d018168e/manager/0.log
lrwxrwxrwx 1 root root   89 Jul 31 21:29 etcd-pa-drs-cluster_kube-system_etcd-58d25c6b072abf6bbab39c8f7e0b94c7527200bb90a6cc8ee98c2c9610439baf.log -> /var/log/pods/kube-system_etcd-pa-drs-cluster_ea07ccd9a98e38376dbf988c3959036c/etcd/0.log
lrwxrwxrwx 1 root root   91 Jul 31 21:59 fluent-bit-xggqj_fluent_fluent-bit-87f0f77a06ce6ee443db8596a423586295c1405bf96a0e424e8528aab458626a.log -> /var/log/pods/fluent_fluent-bit-xggqj_f90c0828-9625-41de-84a7-6857ea8c8607/fluent-bit/0.log
lrwxrwxrwx 1 root root  112 Jul 31 21:58 fluent-operator-55b5859c44-f6g57_fluent_fluent-operator-73b2dca0c75f14d1a5f2f4e60d068e7c29e914565c1374a35f812c6c200811da.log -> /var/log/pods/fluent_fluent-operator-55b5859c44-f6g57_c444e067-868b-4f69-98db-04e4887a348a/fluent-operator/0.log
lrwxrwxrwx 1 root root  103 Jul 31 21:58 fluent-operator-55b5859c44-f6g57_fluent_setenv-a8f5b6bc08cd5fcf226d87947b92bdb3aef8d1778f79aac17fae23849c457ff5.log -> /var/log/pods/fluent_fluent-operator-55b5859c44-f6g57_c444e067-868b-4f69-98db-04e4887a348a/setenv/0.log
lrwxrwxrwx 1 root root   89 Aug  1 01:46 hello-world_default_hello-world-2a06b2b97cf1802a199a4c09b9df39fee7a78966487329e1f1ab0db978f554d2.log -> /var/log/pods/default_hello-world_8341ed25-6bcd-4954-9083-317a5e7c5bd4/hello-world/10.log
lrwxrwxrwx 1 root root  109 Jul 31 21:29 kube-apiserver-pa-drs-cluster_kube-system_kube-apiserver-4e6cc0b351a40c7806bf12d2c5b0ce014d05434eaf8e4d1fc6dd05a1c5e2a95e.log -> /var/log/pods/kube-system_kube-apiserver-pa-drs-cluster_aff2d73431c0215ccd9809ea48d93ff6/kube-apiserver/0.log
...

@benjaminhuo
Copy link
Member

benjaminhuo commented Aug 1, 2023

ls -al /var/log
or
ls -al /
ls -al /var

@jamesla
Copy link
Author

jamesla commented Aug 1, 2023

$ ls -al /var/log
total 20
drwxr-xr-x  6 root root 4096 Jul 31 21:59 .
drwxr-xr-x  8 root root  200 Jul 31 21:28 ..
drwxr-xr-x  2 root root 4096 Aug  1 01:57 containers
drwx------  3 root root 4096 Jul 31 21:28 crio
drwxr-xr-x  2 root root 4096 Jul 31 21:59 journal
drwxr-xr-x 18 root root 4096 Aug  1 01:20 pods
$ ls -al /
total 24
drwxr-xr-x  20 root   root    560 Jul 31 21:28 .
drwxr-xr-x  20 root   root    560 Jul 31 21:28 ..
-rw-r--r--   1 root   root   7976 Apr  3 21:34 CHANGELOG
lrwxrwxrwx   1 root   root      7 Apr  3 22:58 bin -> usr/bin
drwxr-xr-x   2 root   root     60 Apr  4 00:02 boot
drwxr-xr-x   2 root   root   4096 Jul 31 21:28 data
drwxr-xr-x  15 root   root   3060 Jul 31 21:28 dev
drwxr-xr-x  28 root   root   1280 Jul 31 21:28 etc
drwxr-xr-x   3 root   root     60 Apr  4 00:19 home
drwxr-xr-x   1 docker docker 4096 Jun 27 09:18 hosthome
-rwxr-xr-x   1 root   root    457 Apr  3 21:34 init
lrwxrwxrwx   1 root   root      7 Apr  3 22:58 lib -> usr/lib
lrwxrwxrwx   1 root   root      3 Apr  3 22:58 lib64 -> lib
drwxr-xr-x   2 root   root     60 Apr  4 00:19 libexec
lrwxrwxrwx   1 root   root     11 Apr  3 23:19 linuxrc -> bin/busybox
drwxr-xr-x   2 root   root     40 Apr  3 21:34 media
drwxr-xr-x   3 root   root     60 Jul 31 21:28 mnt
drwxr-xr-x   4 root   root     80 Jul 31 21:28 opt
dr-xr-xr-x 332 root   root      0 Jul 31 21:28 proc
drwxr-xr-x   3 root   root     60 Jul 31 21:29 root
drwxr-xr-x  27 root   root    800 Jul 31 21:29 run
lrwxrwxrwx   1 root   root      8 Apr  3 22:58 sbin -> usr/sbin
drwxr-xr-x   2 root   root     40 Apr  3 22:58 srv
dr-xr-xr-x  12 root   root      0 Jul 31 21:28 sys
drwxrwxrwt  12 root   root    280 Jul 31 21:29 tmp
drwxr-xr-x   7 root   root    160 Apr  4 00:19 usr
drwxr-xr-x   8 root   root    200 Jul 31 21:28 var
-rw-r--r--   1 root   root    159 Apr  3 22:52 version.json
$ ls -al /var
total 12
drwxr-xr-x  8 root root  200 Jul 31 21:28 .
drwxr-xr-x 20 root root  560 Jul 31 21:28 ..
-rw-r--r--  1 root root  208 Apr  4 00:19 .updated
drwxr-xr-x  3 root root   60 Jul 31 21:28 cache
drwxr-xr-x  2 root root   40 Apr  4 00:19 empty
drwxr-xr-x 23 root root  460 Jul 31 21:59 lib
drwxr-xr-x  6 root root 4096 Jul 31 21:59 log
lrwxrwxrwx  1 root root    6 Apr  3 22:58 run -> ../run
drwxr-xr-x  2 root root   40 Jul 31 21:28 spool
drwxr-xr-x  3 root root 4096 Jul 31 21:28 tmp

@wanjunlei
Copy link
Collaborator

This error is because the file does not exist or is not a regular file. So make sure you set the right containerLogRealPath.

1690855846864

Run ls -al /var/log/pods/kube-system_kube-apiserver-pa-drs-cluster_aff2d73431c0215ccd9809ea48d93ff6/kube-apiserver/0.log to find the real path of log.

@jamesla
Copy link
Author

jamesla commented Aug 1, 2023

This error is because the file does not exist or is not a regular file. So make sure you set the right containerLogRealPath.

1690855846864 Run `ls -al /var/log/pods/kube-system_kube-apiserver-pa-drs-cluster_aff2d73431c0215ccd9809ea48d93ff6/kube-apiserver/0.log ` to find the real path of log.

I've added containerLogPath as follows however still not working.

---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: FluentBit
metadata:
  name: fluent-bit
  namespace: fluent
  labels:
    app.kubernetes.io/name: fluent-bit
spec:
  image: kubesphere/fluent-bit:v2.1.7
  fluentBitConfigName: fluent-bit-config
  containerLogRealPath: /var/log/containers/*.log
  positionDB:
    hostPath:
      path: /var/lib/fluent-bit/
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterFluentBitConfig
metadata:
  name: fluent-bit-config
  namespace: fluent
  labels:
    app.kubernetes.io/name: fluent-bit
spec:
  inputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  outputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  service:
    logLevel: debug
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
  name: docker
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  systemd:
    tag: service.*
    path: /var/log/journal
    db: /fluent-bit/tail/systemd.db
    dbSync: Normal
    systemdFilter:
      - _SYSTEMD_UNIT=docker.service
      - _SYSTEMD_UNIT=kubelet.service
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
  name: tail
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  tail:
    tag: kube.*
    path: /var/log/containers/*.log
    parser: containerd
    refreshIntervalSeconds: 10
    memBufLimit: 5MB
    skipLongLines: false
    db: /fluent-bit/tail/pos.db
    dbSync: Normal
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
  name: dummy
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  dummy:
    tag: my_dummy
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  name: stdout
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  match: "*"
  stdout: {}

container logs are definitely on that path:

vagrant@carverlinux ~/pa-DRS-elastic (main) [1]> minikube ssh
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ cd /var/log/containers/
$ ls
'*.log'
 cert-manager-7b5cc56d74-xbcqr_cert-manager_cert-manager-controller-65b521cb5a74f0264bbe78863ce8a21541397aee0d7c52559dd3f3b02bcb0546.log
 cert-manager-cainjector-7d948796d5-mg7rh_cert-manager_cert-manager-cainjector-e347348125f02ae2a2000dad0628b3b2bce147a4ce7288dccb372ba45e6abf34.log
 cert-manager-webhook-68f677967c-wtbwt_cert-manager_cert-manager-webhook-8e5db60d6e89d8545906940fc52e1380f9c7a56ef8f492a0e2366e7794e26c3d.log
 coredns-787d4945fb-qskds_kube-system_coredns-2375bae8e35839c1a7d2d4ecbaa79907ab98b0cbcedd43d1082ff74a58b55b4a.log
 elastic-operator-0_observability_manager-6c2c0f94b9d33867e88b413b3ef40850ca61b63abc71e057e1ab5e4763fd29a2.log
 etcd-pa-drs-cluster_kube-system_etcd-25704aba434667bea6bacd3b9cabda4d6980cb41293dc63b6d17adba8734be4b.log
 fluent-bit-9hc59_fluent_fluent-bit-c3a672e24f1f61c8b64a29b5cbec7a1a7fbc400e0944f736f740d50b4dc83b95.log
 fluent-operator-55b5859c44-l6p7w_fluent_fluent-operator-57e57f0f03ebae987505275bf243f487c4c81509846ff204d7498128f3716ca0.log
 fluent-operator-55b5859c44-l6p7w_fluent_setenv-d33d69ca07bf0e0c30e6a5af3f16a76dbc0a88cc07b8f427145c7d38cbeb9e38.log
 hello-world_default_hello-world-03f5ce6853b217041d607a76250b421957630a570ff8023ba4ed5754e5c36123.log
 hello-world_default_hello-world-bc37ce0cc3fbde159b4d7b44fb7786e8f3433f36ca8037ff37b4c9442834ea22.log
 kube-apiserver-pa-drs-cluster_kube-system_kube-apiserver-1ba81788e9d8a48998fea25d5fe511c774f9de76eaaca81c50ff44b5960223b2.log
 kube-controller-manager-pa-drs-cluster_kube-system_kube-controller-manager-1a2feb70140ea8dfb43e13b72caf2ad7b568962afe8ce291a125a7fd1d1dc888.log
 kube-proxy-jzx5h_kube-system_kube-proxy-a5a0b85e7556854e96343f1ff0ecf82ad5945e606fcaec3c678cc7d2c8927259.log
 kube-scheduler-pa-drs-cluster_kube-system_kube-scheduler-ade1da48e2f2c1cce256077ef4f7659aa1fc39029f7638f542c6ca01a085ef7d.log
 mongodb-kubernetes-operator-66b8b9f5cc-kvmc7_mongodb_mongodb-kubernetes-operator-37c3865e7984f2a15aa8b949092aa3a2a2b18ddca146473c083440d5a6278692.log
 nginx-ingress-nginx-controller-594495f5df-ldbh6_nginx_controller-4b3178c6f0ef3dacf2174414e40de8ee230cfb90cdb00f958b7677a4929fc15d.log
 storage-provisioner_kube-system_storage-provisioner-60defc68607dd0f620c33a518d859bdd7cbdfd4ba387691803ba1481eae7c497.log
 storage-provisioner_kube-system_storage-provisioner-c60caee3cfca6071db4c1c8bb87c662e8f9514e89d30d8a1e504b08de56e86d9.log

debug error is:

[2023/08/01 20:04:52] [debug] [input:tail:tail.2] scanning path /var/log/containers/*.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/*.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/cert-manager-7b5cc56d74-xbcqr_cert-manager_cert-manager-controller-65b521cb5a74f0264bbe78863ce8a21541397aee0d7c52559dd3f3b02bcb0546.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/cert-manager-cainjector-7d948796d5-mg7rh_cert-manager_cert-manager-cainjector-e347348125f02ae2a2000dad0628b3b2bce147a4ce7288dccb372ba45e6abf34.log
[2023/08/01 20:04:52] [debug] [out flush] cb_destroy coro_id=198
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/cert-manager-webhook-68f677967c-wtbwt_cert-manager_cert-manager-webhook-8e5db60d6e89d8545906940fc52e1380f9c7a56ef8f492a0e2366e7794e26c3d.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/coredns-787d4945fb-qskds_kube-system_coredns-2375bae8e35839c1a7d2d4ecbaa79907ab98b0cbcedd43d1082ff74a58b55b4a.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/elastic-operator-0_observability_manager-6c2c0f94b9d33867e88b413b3ef40850ca61b63abc71e057e1ab5e4763fd29a2.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/etcd-pa-drs-cluster_kube-system_etcd-25704aba434667bea6bacd3b9cabda4d6980cb41293dc63b6d17adba8734be4b.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/fluent-bit-9hc59_fluent_fluent-bit-c3a672e24f1f61c8b64a29b5cbec7a1a7fbc400e0944f736f740d50b4dc83b95.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/fluent-operator-55b5859c44-l6p7w_fluent_fluent-operator-57e57f0f03ebae987505275bf243f487c4c81509846ff204d7498128f3716ca0.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/fluent-operator-55b5859c44-l6p7w_fluent_setenv-d33d69ca07bf0e0c30e6a5af3f16a76dbc0a88cc07b8f427145c7d38cbeb9e38.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/hello-world_default_hello-world-9a2a324397ecc4e537059783e99c0ab3fa11be0a8d2bfddb95522865b8f5bc7a.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/kube-apiserver-pa-drs-cluster_kube-system_kube-apiserver-1ba81788e9d8a48998fea25d5fe511c774f9de76eaaca81c50ff44b5960223b2.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/kube-controller-manager-pa-drs-cluster_kube-system_kube-controller-manager-1a2feb70140ea8dfb43e13b72caf2ad7b568962afe8ce291a125a7fd1d1dc888.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/kube-proxy-jzx5h_kube-system_kube-proxy-a5a0b85e7556854e96343f1ff0ecf82ad5945e606fcaec3c678cc7d2c8927259.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/kube-scheduler-pa-drs-cluster_kube-system_kube-scheduler-ade1da48e2f2c1cce256077ef4f7659aa1fc39029f7638f542c6ca01a085ef7d.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/mongodb-kubernetes-operator-66b8b9f5cc-kvmc7_mongodb_mongodb-kubernetes-operator-37c3865e7984f2a15aa8b949092aa3a2a2b18ddca146473c083440d5a6278692.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/nginx-ingress-nginx-controller-594495f5df-ldbh6_nginx_controller-4b3178c6f0ef3dacf2174414e40de8ee230cfb90cdb00f958b7677a4929fc15d.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/storage-provisioner_kube-system_storage-provisioner-60defc68607dd0f620c33a518d859bdd7cbdfd4ba387691803ba1481eae7c497.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] skip (invalid) entry=/var/log/containers/storage-provisioner_kube-system_storage-provisioner-c60caee3cfca6071db4c1c8bb87c662e8f9514e89d30d8a1e504b08de56e86d9.log
[2023/08/01 20:04:52] [debug] [input:tail:tail.2] 0 new files found on path '/var/log/containers/*.log'

The path looks good and it's scanning in the correct location - any ideas?

@benjaminhuo
Copy link
Member

benjaminhuo commented Aug 2, 2023

You'll need the -al flag to find out if there is any symbol link like this:
ls -al /var/log/pods/kube-system_kube-apiserver-pa-drs-cluster_aff2d73431c0215ccd9809ea48d93ff6/kube-apiserver/0.log

@jamesla
Copy link
Author

jamesla commented Aug 2, 2023

ls -al /var/log/pods/kube-system_kube-apiserver-pa-drs-cluster_aff2d73431c0215ccd9809ea48d93ff6/kube-apiserver/0.log

vagrant@carverlinux ~/pa-DRS-elastic (main)> minikube ssh
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ ls -al /var/log/containers/kube-apiserver-pa-drs-cluster_kube-system_kube-apiserver-1ba81788e9d8a48998fea25d5fe511c774f9de76eaaca81c50ff44b5960223b2.log
lrwxrwxrwx 1 root root 109 Aug  1 04:40 /var/log/containers/kube-apiserver-pa-drs-cluster_kube-system_kube-apiserver-1ba81788e9d8a48998fea25d5fe511c774f9de76eaaca81c50ff44b5960223b2.log -> /var/log/pods/kube-system_kube-apiserver-pa-drs-cluster_8b1e9057c80f383869339e73f7286d7b/kube-apiserver/0.log

@wanjunlei
Copy link
Collaborator

First the containerLogRealPath is a directory that includes the real log file, the files in /var/log/containers/ are symlink point to the files in /var/log/pods, and the files in /var/log/pods are symlink too. As I said below, please run this command to find the real path of the log files.

ls -al /var/log/pods/kube-system_kube-apiserver-pa-drs-cluster_aff2d73431c0215ccd9809ea48d93ff6/kube-apiserver/

For example, run this command in a k8s cluster.

ls -al /var/log/pods/kube-system_kube-apiserver-worker1_6de26eaf2734fc9226119c0114a89e30/kube-apiserver/
total 16
drwxr-xr-x 2 root root 4096 Jul 31 08:34 .
drwxr-xr-x 3 root root 4096 Jul 18 16:15 ..
lrwxrwxrwx 1 root root  165 Jul 24 08:48 1.log -> /var/lib/docker/containers/5edd62c02f894c0cac40f14637f422847a46cc927fae80208ede21a1111b74bb/5edd62c02f894c0cac40f14637f422847a46cc927fae80208ede21a1111b74bb-json.log
lrwxrwxrwx 1 root root  165 Jul 31 08:34 2.log -> /var/lib/docker/containers/ad39f8d7ec4be144e0efc8739524a8e9ddcc7e4b1aa2c97e4f69c09e2929cd6d/ad39f8d7ec4be144e0efc8739524a8e9ddcc7e4b1aa2c97e4f69c09e2929cd6d-json.log

So the containerLogRealPath is /var/lib/docker/containers/.

@jamesla
Copy link
Author

jamesla commented Aug 2, 2023

First the containerLogRealPath is a directory that includes the real log file, the files in /var/log/containers/ are symlink point to the files in /var/log/pods, and the files in /var/log/pods are symlink too. As I said below, please run this command to find the real path of the log files.

ls -al /var/log/pods/kube-system_kube-apiserver-pa-drs-cluster_aff2d73431c0215ccd9809ea48d93ff6/kube-apiserver/

For example, run this command in a k8s cluster.

ls -al /var/log/pods/kube-system_kube-apiserver-worker1_6de26eaf2734fc9226119c0114a89e30/kube-apiserver/
total 16
drwxr-xr-x 2 root root 4096 Jul 31 08:34 .
drwxr-xr-x 3 root root 4096 Jul 18 16:15 ..
lrwxrwxrwx 1 root root  165 Jul 24 08:48 1.log -> /var/lib/docker/containers/5edd62c02f894c0cac40f14637f422847a46cc927fae80208ede21a1111b74bb/5edd62c02f894c0cac40f14637f422847a46cc927fae80208ede21a1111b74bb-json.log
lrwxrwxrwx 1 root root  165 Jul 31 08:34 2.log -> /var/lib/docker/containers/ad39f8d7ec4be144e0efc8739524a8e9ddcc7e4b1aa2c97e4f69c09e2929cd6d/ad39f8d7ec4be144e0efc8739524a8e9ddcc7e4b1aa2c97e4f69c09e2929cd6d-json.log

So the containerLogRealPath is /var/lib/docker/containers/.

so given my output is like this:

$ pwd
/var/log/containers
$ ls -al
total 96
drwxr-xr-x 2 root root 4096 Aug  1 19:52 '*.log'
drwxr-xr-x 3 root root 4096 Aug  2 01:54  .
drwxr-xr-x 6 root root 4096 Aug  1 04:47  ..
lrwxrwxrwx 1 root root  123 Aug  2 01:44  cert-manager-7b5cc56d74-xbcqr_cert-manager_cert-manager-controller-6919db12205ae4d35975c9ccf8f112ccd9be6a2ffed867a8f843420274ad701d.log -> /var/log/pods/cert-manager_cert-manager-7b5cc56d74-xbcqr_5e032ec6-e277-43d0-8b14-6e8e7f8edf52/cert-manager-controller/1.log
lrwxrwxrwx 1 root root  123 Aug  2 01:50  cert-manager-7b5cc56d74-xbcqr_cert-manager_cert-manager-controller-e5a7c92a8e39f83a8f7e1139a26de6eec1fa805593c8b10d2d021adb647edb67.log -> /var/log/pods/cert-manager_cert-manager-7b5cc56d74-xbcqr_5e032ec6-e277-43d0-8b14-6e8e7f8edf52/cert-manager-controller/2.log
lrwxrwxrwx 1 root root  134 Aug  2 01:50  cert-manager-cainjector-7d948796d5-mg7rh_cert-manager_cert-manager-cainjector-2e42a041992638ff7c8dcd8f45dcee2958c867cf39b04c870e936bbcfb536699.log -> /var/log/pods/cert-manager_cert-manager-cainjector-7d948796d5-mg7rh_12a63b1f-fa1d-4b7d-a00c-5781f93c9996/cert-manager-cainjector/2.log
lrwxrwxrwx 1 root root  134 Aug  2 01:44  cert-manager-cainjector-7d948796d5-mg7rh_cert-manager_cert-manager-cainjector-49c1b268679772f9828d1556e4f2dacae1727ffef13498f4f1de04602c939a14.log -> /var/log/pods/cert-manager_cert-manager-cainjector-7d948796d5-mg7rh_12a63b1f-fa1d-4b7d-a00c-5781f93c9996/cert-manager-cainjector/1.log
lrwxrwxrwx 1 root root  128 Aug  1 04:41  cert-manager-webhook-68f677967c-wtbwt_cert-manager_cert-manager-webhook-8e5db60d6e89d8545906940fc52e1380f9c7a56ef8f492a0e2366e7794e26c3d.log -> /var/log/pods/cert-manager_cert-manager-webhook-68f677967c-wtbwt_a6f666dd-5e4a-43a7-b71e-d5cf9500d918/cert-manager-webhook/0.log
lrwxrwxrwx 1 root root  101 Aug  1 04:40  coredns-787d4945fb-qskds_kube-system_coredns-2375bae8e35839c1a7d2d4ecbaa79907ab98b0cbcedd43d1082ff74a58b55b4a.log -> /var/log/pods/kube-system_coredns-787d4945fb-qskds_3c71e561-50c4-4d08-9fa2-3d74842d6b4e/coredns/0.log
lrwxrwxrwx 1 root root   97 Aug  2 01:44  elastic-operator-0_observability_manager-57ac0e7a5214ac924bc5fedb128cf57b8519f171b82c964f8bab1a341e0c64bb.log -> /var/log/pods/observability_elastic-operator-0_10f7129e-3e5e-4084-814d-0531544f2879/manager/1.log
lrwxrwxrwx 1 root root   97 Aug  2 01:50  elastic-operator-0_observability_manager-f263c39656573064c9d9211151177510cf354ce8c577902b921ef91b3e59da14.log -> /var/log/pods/observability_elastic-operator-0_10f7129e-3e5e-4084-814d-0531544f2879/manager/2.log
lrwxrwxrwx 1 root root   89 Aug  1 04:40  etcd-pa-drs-cluster_kube-system_etcd-25704aba434667bea6bacd3b9cabda4d6980cb41293dc63b6d17adba8734be4b.log -> /var/log/pods/kube-system_etcd-pa-drs-cluster_58123f9b092065056624138e28782ab7/etcd/0.log
lrwxrwxrwx 1 root root   91 Aug  2 01:53  fluent-bit-nnjnt_fluent_fluent-bit-aa50c6ba7b680e9f62a5eca6a163043fc4e35a12b4f4dd28f24b0530f0a8591f.log -> /var/log/pods/fluent_fluent-bit-nnjnt_83ae1a44-80e2-4d68-8129-278d0a4a5211/fluent-bit/0.log
lrwxrwxrwx 1 root root  112 Aug  1 04:46  fluent-operator-55b5859c44-l6p7w_fluent_fluent-operator-57e57f0f03ebae987505275bf243f487c4c81509846ff204d7498128f3716ca0.log -> /var/log/pods/fluent_fluent-operator-55b5859c44-l6p7w_42de5edd-59b7-490a-9bc8-ad3551de736b/fluent-operator/0.log
lrwxrwxrwx 1 root root  103 Aug  1 04:46  fluent-operator-55b5859c44-l6p7w_fluent_setenv-d33d69ca07bf0e0c30e6a5af3f16a76dbc0a88cc07b8f427145c7d38cbeb9e38.log -> /var/log/pods/fluent_fluent-operator-55b5859c44-l6p7w_42de5edd-59b7-490a-9bc8-ad3551de736b/setenv/0.log
lrwxrwxrwx 1 root root   88 Aug  2 01:54  hello-world_default_hello-world-5b5e858eeb1c8cf95680f575bff079c347d4717fab5581096299d1a9c08d8782.log -> /var/log/pods/default_hello-world_22b8f980-4f3f-4790-8e90-52486ba129c0/hello-world/3.log
lrwxrwxrwx 1 root root  109 Aug  1 04:40  kube-apiserver-pa-drs-cluster_kube-system_kube-apiserver-1ba81788e9d8a48998fea25d5fe511c774f9de76eaaca81c50ff44b5960223b2.log -> /var/log/pods/kube-system_kube-apiserver-pa-drs-cluster_8b1e9057c80f383869339e73f7286d7b/kube-apiserver/0.log
lrwxrwxrwx 1 root root  127 Aug  1 04:40  kube-controller-manager-pa-drs-cluster_kube-system_kube-controller-manager-1a2feb70140ea8dfb43e13b72caf2ad7b568962afe8ce291a125a7fd1d1dc888.log -> /var/log/pods/kube-system_kube-controller-manager-pa-drs-cluster_593dc083a6f5f7623ad9a4d6adba7b77/kube-controller-manager/0.log
lrwxrwxrwx 1 root root   96 Aug  1 04:40  kube-proxy-jzx5h_kube-system_kube-proxy-a5a0b85e7556854e96343f1ff0ecf82ad5945e606fcaec3c678cc7d2c8927259.log -> /var/log/pods/kube-system_kube-proxy-jzx5h_8d2e7303-4ec2-4c28-809e-c191f40c40ad/kube-proxy/0.log
lrwxrwxrwx 1 root root  109 Aug  1 04:40  kube-scheduler-pa-drs-cluster_kube-system_kube-scheduler-ade1da48e2f2c1cce256077ef4f7659aa1fc39029f7638f542c6ca01a085ef7d.log -> /var/log/pods/kube-system_kube-scheduler-pa-drs-cluster_7127d7352f28daa3649bab78b6964aee/kube-scheduler/0.log
lrwxrwxrwx 1 root root  137 Aug  1 04:41  mongodb-kubernetes-operator-66b8b9f5cc-kvmc7_mongodb_mongodb-kubernetes-operator-37c3865e7984f2a15aa8b949092aa3a2a2b18ddca146473c083440d5a6278692.log -> /var/log/pods/mongodb_mongodb-kubernetes-operator-66b8b9f5cc-kvmc7_1189fb0c-74b2-4af0-81bf-9f125afc7f54/mongodb-kubernetes-operator/0.log
lrwxrwxrwx 1 root root  121 Aug  1 04:44  nginx-ingress-nginx-controller-594495f5df-ldbh6_nginx_controller-4b3178c6f0ef3dacf2174414e40de8ee230cfb90cdb00f958b7677a4929fc15d.log -> /var/log/pods/nginx_nginx-ingress-nginx-controller-594495f5df-ldbh6_31015c21-c2e0-42c3-a53e-f96736635689/controller/0.log
lrwxrwxrwx 1 root root  108 Aug  1 04:41  storage-provisioner_kube-system_storage-provisioner-60defc68607dd0f620c33a518d859bdd7cbdfd4ba387691803ba1481eae7c497.log -> /var/log/pods/kube-system_storage-provisioner_3abd4642-8b22-4289-a6af-cc6ecda9fe72/storage-provisioner/1.log
lrwxrwxrwx 1 root root  108 Aug  1 04:40  storage-provisioner_kube-system_storage-provisioner-c60caee3cfca6071db4c1c8bb87c662e8f9514e89d30d8a1e504b08de56e86d9.log -> /var/log/pods/kube-system_storage-provisioner_3abd4642-8b22-4289-a6af-cc6ecda9fe72/storage-provisioner/0.log

Are you suggesting I update containerLogRealPath to /var/log/pods/*/*.log?

@benjaminhuo
Copy link
Member

Are you suggesting I update containerLogRealPath to /var/log/pods/*/*.log?

image

@jamesla
Copy link
Author

jamesla commented Aug 2, 2023

Ok I've tried this and it doesn't work either

---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: FluentBit
metadata:
  name: fluent-bit
  namespace: fluent
  labels:
    app.kubernetes.io/name: fluent-bit
spec:
  image: kubesphere/fluent-bit:v2.1.7
  fluentBitConfigName: fluent-bit-config
  containerLogRealPath: /var/lib/docker/containers
  positionDB:
    hostPath:
      path: /var/lib/fluent-bit/
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterFluentBitConfig
metadata:
  name: fluent-bit-config
  namespace: fluent
  labels:
    app.kubernetes.io/name: fluent-bit
spec:
  inputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
  outputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
        # service:
        #   logLevel: debug
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
  name: docker
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  systemd:
    tag: service.*
    path: /var/log/journal
    db: /fluent-bit/tail/systemd.db
    dbSync: Normal
    systemdFilter:
      - _SYSTEMD_UNIT=docker.service
      - _SYSTEMD_UNIT=kubelet.service
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
  name: tail
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  tail:
    tag: kube.*
    path: /var/lib/docker/containers
    parser: containerd
    refreshIntervalSeconds: 10
    memBufLimit: 5MB
    skipLongLines: false
    db: /fluent-bit/tail/pos.db
    dbSync: Normal
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
  name: dummy
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  dummy:
    tag: my_dummy
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  name: stdout
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  match: "*"
  stdout: {}

Why don't you guys try it - I don't think it's my machine or configuration

you simply just have to make sure minikube is instlled and

minikube start #creating a fresh cluster
helm upgrade -i --set containerRuntime=containerd fluent-operator --create-namespace -n fluent https://github.com/fluent/fluent-operator/releases/download/v2.4.0/fluent-operator.tgz
kubectl apply -f %file_containing_example_above%

@benjaminhuo
Copy link
Member

First the containerLogRealPath is a directory that includes the real log file, the files in /var/log/containers/ are symlink point to the files in /var/log/pods, and the files in /var/log/pods are symlink too. As I said below, please run this command to find the real path of the log files.

ls -al /var/log/pods/kube-system_kube-apiserver-pa-drs-cluster_aff2d73431c0215ccd9809ea48d93ff6/kube-apiserver/

For example, run this command in a k8s cluster.

ls -al /var/log/pods/kube-system_kube-apiserver-worker1_6de26eaf2734fc9226119c0114a89e30/kube-apiserver/
total 16
drwxr-xr-x 2 root root 4096 Jul 31 08:34 .
drwxr-xr-x 3 root root 4096 Jul 18 16:15 ..
lrwxrwxrwx 1 root root  165 Jul 24 08:48 1.log -> /var/lib/docker/containers/5edd62c02f894c0cac40f14637f422847a46cc927fae80208ede21a1111b74bb/5edd62c02f894c0cac40f14637f422847a46cc927fae80208ede21a1111b74bb-json.log
lrwxrwxrwx 1 root root  165 Jul 31 08:34 2.log -> /var/lib/docker/containers/ad39f8d7ec4be144e0efc8739524a8e9ddcc7e4b1aa2c97e4f69c09e2929cd6d/ad39f8d7ec4be144e0efc8739524a8e9ddcc7e4b1aa2c97e4f69c09e2929cd6d-json.log

So the containerLogRealPath is /var/lib/docker/containers/.

I don't think you get what @wanjunlei says, you'll need to find out the real path of your env. Not copying the real path of our env directly.

You'll need to understand what this is doing

@jamesla
Copy link
Author

jamesla commented Aug 2, 2023

First the containerLogRealPath is a directory that includes the real log file, the files in /var/log/containers/ are symlink point to the files in /var/log/pods, and the files in /var/log/pods are symlink too. As I said below, please run this command to find the real path of the log files.

ls -al /var/log/pods/kube-system_kube-apiserver-pa-drs-cluster_aff2d73431c0215ccd9809ea48d93ff6/kube-apiserver/

For example, run this command in a k8s cluster.

ls -al /var/log/pods/kube-system_kube-apiserver-worker1_6de26eaf2734fc9226119c0114a89e30/kube-apiserver/
total 16
drwxr-xr-x 2 root root 4096 Jul 31 08:34 .
drwxr-xr-x 3 root root 4096 Jul 18 16:15 ..
lrwxrwxrwx 1 root root  165 Jul 24 08:48 1.log -> /var/lib/docker/containers/5edd62c02f894c0cac40f14637f422847a46cc927fae80208ede21a1111b74bb/5edd62c02f894c0cac40f14637f422847a46cc927fae80208ede21a1111b74bb-json.log
lrwxrwxrwx 1 root root  165 Jul 31 08:34 2.log -> /var/lib/docker/containers/ad39f8d7ec4be144e0efc8739524a8e9ddcc7e4b1aa2c97e4f69c09e2929cd6d/ad39f8d7ec4be144e0efc8739524a8e9ddcc7e4b1aa2c97e4f69c09e2929cd6d-json.log

So the containerLogRealPath is /var/lib/docker/containers/.

I don't think you get what @wanjunlei says, you'll need to find out the real path of your env. Not copying the real path of our env directly.

You'll need to understand what this is doing

Oh yes I see

$ sudo ls -al /var/log/pods/kube-system_kube-apiserver-pa-drs-cluster_8b1e9057c80f383869339e73f7286d7b/kube-apiserver/
total 12
drwxr-xr-x 2 root root 4096 Aug  1 04:40 .
drwxr-xr-x 3 root root 4096 Aug  1 04:40 ..
lrwxrwxrwx 1 root root  165 Aug  1 04:40 0.log -> /var/lib/docker/containers/1ba81788e9d8a48998fea25d5fe511c774f9de76eaaca81c50ff44b5960223b2/1ba81788e9d8a48998fea25d5fe511c774f9de76eaaca81c50ff44b5960223b2-json.log

so would this mean that my container log path would be the following?

...
containerLogRealPath: /var/lib/docker/containers
...

or would it be:

...
containerLogRealPath: /var/lib/docker/containers/*/*.log
...

@wanjunlei
Copy link
Collaborator

The containerLogRealPath is /var/lib/docker/containers for your cluster. But why you set the parser as containerd when the cluster use docker?

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
  name: tail
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  tail:
    tag: kube.*
    path: /var/lib/docker/containers
    parser: containerd
    refreshIntervalSeconds: 10
    memBufLimit: 5MB
    skipLongLines: false
    db: /fluent-bit/tail/pos.db
    dbSync: Normal

@wanjunlei
Copy link
Collaborator

/var/log/containers/*.log

In addition, the path should be /var/log/containers/*.log.

@benjaminhuo
Copy link
Member

The containerLogRealPath is /var/lib/docker/containers for your cluster. But why you set the parser as containerd when the cluster use docker?

@jamesla valid values for parser are docker and cri, I don't know where you find containerd:

https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/templates/fluentbit-clusterinput-tail.yaml#L16-L21

image

@jamesla
Copy link
Author

jamesla commented Aug 3, 2023

That was it thank you guys for all your help!

@jamesla jamesla closed this as completed Aug 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants