[WIP] feat: add multiline parser support for fluentbit #842

ksdpmx · 2023-07-23T12:59:48Z

What this PR does / why we need it:

currently users cannot setup custom multiline parser via operator.
and there're many unsupported fields for multiline parser in filter

Which issue(s) this PR fixes:

Fixes #176 and #336

add more fields for multiline parser in filter (add multiline filter plugin #176)
add ClusterMultilineParser and MultilineParser crd for multiline parser (help request: multiline settings don't work? #336)

Does this PR introduced a user-facing change?

feat: add multiline parser support for fluentbit

Additional documentation, usage docs, etc.:

wanjunlei · 2023-07-24T06:13:07Z

apis/fluentbit/v1alpha2/clustermultilineparser_types.go

+// +genclient:nonNamespaced
+
+// ClusterMultilineParser is the Schema for the cluster-level multiline parser API
+type ClusterMultilineParser struct {


Why need to create a new CRD? I think multiline Parser is also a kind of Parser, it can use Parser CRD.

the section for multiline parser is [MULTILINE_PARSER] instead of [PARSER]

multiline parser is not located under Parsers in fluentbit doc

i agree with the idea that it's better to treat multiline parser as one kind of parsers as well.

but it seems like currently it's not implemented like this from fluentbit's perspective. it's also reasonable to follow the convention of fluentbit so that customers won't get confused.

Got it.

Does it need to add the parsers_multiline.conf to the SERVER section?

yes. so i propose to add one new field called parsersFiles to support multiple parsers_file in SERVICE section, and keep parsersFile for backward compatibility: https://github.com/fluent/fluent-operator/pull/842/files#diff-7aad9bf9eeeaa93f3a61c5ea93ea5a30d57bcfa6307ec4ea188222fdf69f438cR180 in case customers want to add parsers.conf and parsers_multiline.conf at the same time.

@wanjunlei would you review this again?

So how to set the ClusterFluentBitConfig when I define a MultilineParser and want to use it?

sorry for the late reply. i'll have an E2E test tomorrow.

fix rbac for new crd

inline for MultilineParser

how to set the ClusterFluentBitConfig when I define a MultilineParser and want to use it

apiVersion: fluentbit.fluent.io/v1alpha2 kind: ClusterFluentBitConfig metadata: labels: app.kubernetes.io/name: fluent-bit name: fluent-bit-config spec: filterSelector: matchLabels: fluentbit.fluent.io/enabled: "true" inputSelector: matchLabels: fluentbit.fluent.io/enabled: "true" multilineparserSelector: matchLabels: fluentbit.fluent.io/enabled: "true" outputSelector: matchLabels: fluentbit.fluent.io/enabled: "true" parserSelector: matchLabels: fluentbit.fluent.io/enabled: "true" service: logLevel: info parsersFile: parsers_multiline.conf

apiVersion: fluentbit.fluent.io/v1alpha2 kind: ClusterMultilineParser metadata: labels: fluentbit.fluent.io/component: logging fluentbit.fluent.io/enabled: "true" name: stacktrace spec: flushTimeout: 1000 rules: - next: cont regex: /([a-zA-Z]+ \d+ \d+\:\d+\:\d+)(.*)/ start: start_state - next: cont regex: /^\s+at.*/ start: cont type: regex

By default, the parsersFile in ClusterFluentBitConfig is parsers.conf, And it points to /fluent-bit/etc/parsers.conf which include the common parsers such as docker, cri. If it was set to parsers_multiline.conf, the fluentbit cannot find these parsers.

BTW, if the parsersFile in ClusterFluentBitConfig is a relative path, the fluentbit will find it in the directory where the fluentbit configuration file is located, which is /fluent-bit/etc/, but the parsers_multiline.conf is in /fluent-bit/config/.

Maybe you should handle the parsers_multiline.conf like parsers.conf

benjaminhuo · 2023-08-06T15:54:22Z

apis/fluentbit/v1alpha2/clusterfluentbitconfig_types.go

@@ -41,6 +41,8 @@ type FluentBitConfigSpec struct {
 	OutputSelector metav1.LabelSelector `json:"outputSelector,omitempty"`
 	// Select parser plugins
 	ParserSelector metav1.LabelSelector `json:"parserSelector,omitempty"`
+	// Select multiline parser plugins
+	MultilineParserSelector metav1.LabelSelector `json:"multilineparserSelector,omitempty"`


Suggested change

MultilineParserSelector metav1.LabelSelector `json:"multilineparserSelector,omitempty"`

MultilineParserSelector metav1.LabelSelector `json:"multilineParserSelector,omitempty"`

benjaminhuo · 2023-08-28T01:48:31Z

@ksdpmx Hi Jason, would you resolve the conflicts and conversation remains?

ksdpmx · 2023-08-28T03:55:12Z

@ksdpmx Hi Jason, would you resolve the conflicts and conversation remains?

@benjaminhuo ~~sure will check this weekend.~~ sorry for the late reply. quite busy recently:(

americans007 · 2023-09-06T07:37:59Z

@ksdpmx looking forward multiline feature, could you please find time to resolve conflicts.

JoeDerby · 2023-12-07T12:17:38Z

@ksdpmx Do you have any updates when you will have time to resolve these conflicts, this is a feature I would be very interested also. Thanks.

ksdpmx · 2023-12-09T03:42:47Z

~~thanks for the patience and understanding.. I've had some free time this month. continuing the remaining work..~~ :(

pikodd · 2023-12-19T10:09:55Z

Hi, @ksdpmx do you need any help with this task?
We really need this feature and we would like help you resolving it.

esperoz · 2024-01-09T10:22:09Z

Hi, @ksdpmx! Also interested in this feature. How can I help you?

Signed-off-by: jasonz <ksdpmx@gmail.com>

Signed-off-by: Alex Krasnov <askrasnov@gmail.com>

esperoz · 2024-01-12T10:56:28Z

Hi everybody!
@ksdpmx, would you be so kind to review and accept PR to your repo ksdpmx#1 with fixes:

Adjusted multilineparserSelector to multilineParserSelector as of comment @benjaminhuo
Added watches for ClusterMultilineParser and MultilineParser
Fixed inline import in MultilineParserSpec
Rebased to fluent:master
Checked with our team - it works. If something need to be fixed, please let me know.

wenchajun · 2024-01-16T08:38:19Z

Hi everybody! @ksdpmx, would you be so kind to review and accept PR to your repo ksdpmx#1 with fixes:

Adjusted multilineparserSelector to multilineParserSelector as of comment @benjaminhuo

Added watches for ClusterMultilineParser and MultilineParser

Fixed inline import in MultilineParserSpec

Rebased to fluent:master
Checked with our team - it works. If something need to be fixed, please let me know.

I think you can resubmit a pull request to this repository. There are too many commits here, which makes it a bit confusing. After reviewing the code, I noticed that you are treating 'multiline' as a resource to watch, and I don't think it's the best approach in this regard (it could be my misunderstanding). I suggest you submit a pull request so that we can discuss and explore alternative solutions together."

(cherry picked from commit 9005e1b)

elsnepal · 2024-03-28T10:59:54Z

@ksdpmx are you able to look at this? :)

ksdpmx · 2024-03-30T03:11:32Z

some internal team is also interested in this feature.. allow me few hours to catch with the context this time..

Signed-off-by: jasonz <ksdpmx@gmail.com>

Merge pull request #1 from shturval-tech/multiline_parser Signed-off-by: jasonz <ksdpmx@gmail.com>

ksdpmx · 2024-03-30T07:05:43Z

I think you can resubmit a pull request to this repository. There are too many commits here, which makes it a bit confusing.

two commits w/o sign-off
two rebases which makes it hard to squash
too many commits for this legacy PR

so here's the new one #1100.
@esperoz please submit another new one if there's any concern about this PR. i'm fine with the ownership of it :)

benjaminhuo · 2024-04-06T14:19:39Z

Finished in #1100, close this for now

ksdpmx force-pushed the multiline_parser branch from da012ac to deb851e Compare July 23, 2023 13:00

ksdpmx changed the title ~~feat: add multiline parser support for fluentbit~~ [WIP] feat: add multiline parser support for fluentbit Jul 23, 2023

benjaminhuo requested review from wenchajun and wanjunlei July 24, 2023 01:44

wanjunlei reviewed Jul 24, 2023

View reviewed changes

ksdpmx requested a review from wanjunlei July 24, 2023 11:21

ksdpmx force-pushed the multiline_parser branch 3 times, most recently from 1382b7a to 2b0bbbc Compare July 30, 2023 05:36

ksdpmx force-pushed the multiline_parser branch from e0ef681 to 08cb6a3 Compare August 6, 2023 14:07

benjaminhuo reviewed Aug 6, 2023

View reviewed changes

ksdpmx and others added 4 commits January 9, 2024 15:34

add multiline parser support for fluentbit

37ba06d

Signed-off-by: jasonz <ksdpmx@gmail.com>

Fix typo in multilineParserSelector field

76c5d6b

Signed-off-by: Alex Krasnov <askrasnov@gmail.com>

Add watches for ClusterMultilineParser and MultilineParser

24a97dc

Refactor MultilineParserSpec struct - fix inline derective

7e07163

Signed-off-by: Alex Krasnov <askrasnov@gmail.com>

Update emitter memory buffer limit format

b57bc5a

(cherry picked from commit 9005e1b)

ksdpmx force-pushed the multiline_parser branch 3 times, most recently from 8d3f00d to e6e7b19 Compare March 30, 2024 04:05

ksdpmx force-pushed the multiline_parser branch 2 times, most recently from 97d0608 to e0c5675 Compare March 30, 2024 04:34

feat: add multiline parser support for fluentbit

132b2df

Signed-off-by: jasonz <ksdpmx@gmail.com>

ksdpmx force-pushed the multiline_parser branch from e0c5675 to 132b2df Compare March 30, 2024 04:37

feat: add multiline parser support for fluentbit

a445c9d

Merge pull request #1 from shturval-tech/multiline_parser Signed-off-by: jasonz <ksdpmx@gmail.com>

ksdpmx force-pushed the multiline_parser branch from 4835723 to a445c9d Compare March 30, 2024 04:42

ksdpmx mentioned this pull request Mar 30, 2024

feat: add multiline parser support for fluentbit #1100

Merged

benjaminhuo closed this Apr 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] feat: add multiline parser support for fluentbit #842

[WIP] feat: add multiline parser support for fluentbit #842

ksdpmx commented Jul 23, 2023 •

edited

Loading

wanjunlei Jul 24, 2023

ksdpmx Jul 24, 2023 •

edited

Loading

wanjunlei Jul 25, 2023

ksdpmx Jul 25, 2023

benjaminhuo Jul 26, 2023

wanjunlei Jul 31, 2023

ksdpmx Aug 5, 2023 •

edited

Loading

ksdpmx Aug 6, 2023 •

edited

Loading

ksdpmx Aug 6, 2023

wanjunlei Aug 7, 2023

benjaminhuo Aug 6, 2023

benjaminhuo commented Aug 28, 2023

ksdpmx commented Aug 28, 2023 •

edited

Loading

americans007 commented Sep 6, 2023

JoeDerby commented Dec 7, 2023

ksdpmx commented Dec 9, 2023 •

edited

Loading

pikodd commented Dec 19, 2023 •

edited

Loading

esperoz commented Jan 9, 2024

esperoz commented Jan 12, 2024

wenchajun commented Jan 16, 2024

elsnepal commented Mar 28, 2024

ksdpmx commented Mar 30, 2024 •

edited

Loading

ksdpmx commented Mar 30, 2024 •

edited

Loading

benjaminhuo commented Apr 6, 2024

	MultilineParserSelector metav1.LabelSelector `json:"multilineparserSelector,omitempty"`
	MultilineParserSelector metav1.LabelSelector `json:"multilineParserSelector,omitempty"`

[WIP] feat: add multiline parser support for fluentbit #842

[WIP] feat: add multiline parser support for fluentbit #842

Conversation

ksdpmx commented Jul 23, 2023 • edited Loading

What this PR does / why we need it:

Which issue(s) this PR fixes:

Does this PR introduced a user-facing change?

Additional documentation, usage docs, etc.:

Choose a reason for hiding this comment

ksdpmx Jul 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ksdpmx Aug 5, 2023 • edited Loading

Choose a reason for hiding this comment

ksdpmx Aug 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjaminhuo commented Aug 28, 2023

ksdpmx commented Aug 28, 2023 • edited Loading

americans007 commented Sep 6, 2023

JoeDerby commented Dec 7, 2023

ksdpmx commented Dec 9, 2023 • edited Loading

pikodd commented Dec 19, 2023 • edited Loading

esperoz commented Jan 9, 2024

esperoz commented Jan 12, 2024

wenchajun commented Jan 16, 2024

elsnepal commented Mar 28, 2024

ksdpmx commented Mar 30, 2024 • edited Loading

ksdpmx commented Mar 30, 2024 • edited Loading

benjaminhuo commented Apr 6, 2024

ksdpmx commented Jul 23, 2023 •

edited

Loading

ksdpmx Jul 24, 2023 •

edited

Loading

ksdpmx Aug 5, 2023 •

edited

Loading

ksdpmx Aug 6, 2023 •

edited

Loading

ksdpmx commented Aug 28, 2023 •

edited

Loading

ksdpmx commented Dec 9, 2023 •

edited

Loading

pikodd commented Dec 19, 2023 •

edited

Loading

ksdpmx commented Mar 30, 2024 •

edited

Loading

ksdpmx commented Mar 30, 2024 •

edited

Loading