Implement image-based gadgets examples #2753

mauriciovasquezbernal · 2024-04-22T19:23:00Z

This PR provides some examples of running image-based gadgets through the Golang API

Examples included: (readme added on this PR)

Gadgets Examples

This folder contains different examples of how gadgets can be run from a Golang
application.

⚠️ These examples are work in progress. Be sure to check the release notes to
understand relevant changes on the API.

TODO: link to API and concepts documentation before

Simple Gadgets

Examples showing how to use some gadgets in their simplest configuration.

trace_open: Run the trace_open gadget and print the
events to the terminal in json format.
trace_dns: Run the trace_dns gadget with some of the
operators it requires.

Operators

Examples showing how to use some of the operators.

local_manager: Use the local manager operator
to filter and enrich events with container data.
cli: Use the CLI operator to print data to the terminal in
a pretty way.

Datasource

Examples showing how to use datasource

fields: Show how to access specific fields from the datasource.
mutate: Mutate and add fields to a datasource.

Executing Gadgets on remove Inspektor Gadget instances

Examples showing how to execute gadgets in remove instances of Inspektor Gadget
(either ig or iìg-k8s) by using the grpc runtime.

custom_operator: Run a remote gadget and print its
output to the terminal in json format by using a custom operator.

The idea of this PR is to discuss some of the pain points I found while implementing the examples and possible solutions to them. However I shouldn't be blocked on the solutions to all of them. I think the examples are valuable in their current status.

TODO

Configure CI to automatically compile them
Open issues to further discuss some "pain points"

Future TODOs

Link to document with API and documentation about concepts in examples/gadgets/README.md (Implement image-based gadgets examples #2753 (comment))
Initialize operators outside the gadget context (Implement image-based gadgets examples #2753 (comment))
How to handle SocketFilter and SchedCLS programs when localmanager is not available? (Implement image-based gadgets examples #2753 (comment))
Implement image-based gadgets examples #2753 (comment)

mauriciovasquezbernal

Some questions I found while implementing the examples.

mauriciovasquezbernal · 2024-04-22T19:23:47Z

pkg/operators/localmanager/params.go

What do you folks think about providing this structure for different operators?

/cc @flyth

examples/gadgets/simple_trace_open/main.go

flyth · 2024-04-22T19:55:04Z

examples/gadgets/fields/main.go

+			fnameF := d.GetField("fname")
+
+			d.Subscribe(func(source datasource.DataSource, data datasource.Data) error {
+				pid := pidF.Uint32(data)


Maybe we should add a generic .Interface() or .Any() (although I find that more confusing than .Interface()) to cast to whatever the underlying type is and return it as any in case you don't care about the type (e.g. when printing with %v).

mauriciovasquezbernal · 2024-04-25T20:38:38Z

I updated the PR by reorganizing the examples and adding a few more. My proposal is to start the discussion on these examples, and create a wish-list of other examples that we can add later on. Any thoughts?

examples/gadgets/grpc/custom_operator/main.go

blanquicet

Awesome! 🚀

I started reviewing the documentation and the simple/trace_dns. I'll continue tomorrow with the rest. Some discussions are long, so maybe they can become a separate issue?

examples/gadgets/README.md

examples/gadgets/simple/trace_dns/README.md

blanquicet · 2024-05-08T15:38:48Z

examples/gadgets/simple/trace_dns/main.go

+)
+
+func do() error {
+	const opPriority = 50000


How do people know the priority they need to set so that their operator is called after the other operators?

Good question. What I did was to check the priority used by other operators.

Currently it's the operator that tells that's the priority for the Subscribe, it requires operators to know the priorities of other operators (the CLI needs to be sure it has higher priority than the localmanager, formatter, etc). I'm really wondering if this is the right approach, perhaps we can move this responsibility to the caller? It could have a way to define the order of the operators and it should be sure they're called in the right order satisfying the dependencies.

I know @flyth put a lot of thinking into this. Do you have strong opinions for one or another approach?

As #2709 introduces the difference between Packet and Data (with it's own priority handling), I suggest we define only three priorities as constants for now:

PriorityDefault as 0, PriorityHigh as something like -50000 and PriorityLow as 50000 (or -PriorityHigh). If a dev/user needs something in between, this allows a lot of room. But for normal cases the default priority should work fine. Same can be used for Packets, as they're always handled after the Data subscriptions.

examples/gadgets/simple/trace_dns/main.go

blanquicet

I added some comments but LGTM already. Thanks for working on this.

examples/gadgets/simple/trace_dns/README.md

examples/gadgets/simple/trace_open/main.go

examples/gadgets/simple/trace_dns/README.md

blanquicet · 2024-05-14T21:55:22Z

examples/gadgets/simple/trace_dns/README.md

+
+Those will printed in the gadget's terminal:
+
+TODO: check why this gadget is lacking so much information compared to the built-in one.


After igjson.WithShowAll(true) in the formatter initialization, I got most of the missing fields.

examples/gadgets/simple/trace_open/main.go

blanquicet · 2024-05-14T22:44:58Z

examples/gadgets/operators/cli/main.go

+	defer runtime.Close()
+
+	params := map[string]string{
+		// columns, json, jsonpretty and yaml are supported


TODO: Link to operator documentation

examples/gadgets/operators/local_manager/README.md

examples/gadgets/operators/local_manager/main.go

examples/gadgets/grpc/custom_operator/README.md

examples/gadgets/grpc/custom_operator/main.go

eiffel-fl

Hi!

These examples are welcomed!
I am wondering nonetheless if we should not put them in the existing folder.
I will take a deeper look later, as I need to get familiar with the API first.

Best regards.

docs/core-concepts/custom-resources.md

examples/gadgets/operators/local_manager/README.md

examples/gadgets/README.md

examples/gadgets/grpc/custom_operator/main.go

mauriciovasquezbernal · 2024-05-17T21:36:53Z

I am wondering nonetheless if we should not put them in the existing folder.

Where to you mean?

eiffel-fl · 2024-05-22T08:19:02Z

I am wondering nonetheless if we should not put them in the existing folder.

Where to you mean?

In examples/, which is where you put them, so nevermind, I just got confused.

Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

Show how to: - Create a simple operator to subscribe to events - Marshall events to json - Set parameters for the gadget Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

Show how to use the localmanager to filter and enrich events with container data. Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

Show how to use the data source to get some specific fields from the event. Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

Example showing how to add and mutate fields from a datasource. Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

blanquicet · 2024-05-23T19:08:11Z

examples/gadgets/simple/trace_open/main.go

+	// TODO: documentation
+	// Our operator should be the last of the chain to print information after
+	// all operators have been run.
+	const opPriority = 50000
+	myOperator := simple.New("myHandler", simple.OnInit(func(gadgetCtx operators.GadgetContext) error {
+		// Subscribe to all datasources and print their output as json the terminal
+		// Check the datasources documentation for more information
+		// TODO: link to documentation
+		for _, d := range gadgetCtx.GetDataSources() {
+			jsonFormatter, _ := igjson.New(d,
+				// Show all fields
+				igjson.WithShowAll(true),
+
+				// Print json in a pretty format
+				igjson.WithPretty(true, "  "),
+			)
+
+			d.Subscribe(func(source datasource.DataSource, data datasource.Data) error {
+				jsonOutput := jsonFormatter.Marshal(data)
+				fmt.Printf("%s\n", jsonOutput)
+				return nil
+			}, opPriority)
+		}
+		return nil
+	}))


Now that I'm working on the combiner operator, I noticed that by subscribing to a data source in the OnInit phase there's a risk that a new operator registers a new data source afte us and we miss it. Of course, you can rely on the priority at the moment you add your operator, but what about new operator created later on by other people?

One solution is to never call GetDataSources on the OnInit phase as other data sources could be registered in operators with lower priority and instead, do it in the OnStart phase where all operators are supposed to have already registered their data sources. But again, there's no guarantee.

We need to describe in the documentation to use high priority for operators registering datasource and maybe also tell that it's a good practice to use the OnInit phase is for datasource registration and the OnStart phase for subscribing to datasources. WDYT?

cc @flyth @mauriciovasquezbernal

I drafted this earlier which contains some information on the lifecycle hooks.

Basically:

OnInit() / Instantiate(): Add & manipulate DataSources - priority of operator handles initialization order; this is what GetGadgetInfo() returns

PreStart() (optional): Subscribe to DataSources (to prevent missing some data) - priority determines handling order

Start(): Emit data, Subscriptions will be evaluated

Stop(): Stop everything

Regarding subscription priorities: I think filtering should be done before sorting on the server side. Combining should be done (on the client side only) before sorting but after filtering. So sort/filter would run on both client+server.

I hope this will work for 90% of the use cases 😄

I think I messed up the examples because there are two priorities:

operator priority: defines the order on which operators are initialized, started and stopped.

subscription priority: order in which subscriptions are called

Those two are independent, an operator with a low priority (is initialized last), could call subscribe with a high priority (its callback will be called first.

@flyth what's the design decision to have a priority for the operator and not to rely on the order used in WithDataOperators()?

what's the design decision to have a priority for the operator and not to rely on the order used in WithDataOperators()?

Explicitly adding operators using WithDataOperators() was an option added later on; most (default) operators register themselves with the registry, so at least for that I think it makes sense to have the priority mentioned inside the operator (although we could make it overridable later on - e.g. the gadget.yaml could specify the order, or we could have a WithDataOperatorsOrdered() that would honor the order in which operators are given, overriding their preset priorities).

mauriciovasquezbernal commented Apr 22, 2024

View reviewed changes

flyth reviewed Apr 22, 2024

View reviewed changes

mauriciovasquezbernal mentioned this pull request Apr 24, 2024

examples: Move gadgets to builtin-gadgets #2769

Merged

mauriciovasquezbernal changed the base branch from main to mauricio/move-examples April 24, 2024 19:56

mauriciovasquezbernal force-pushed the mauricio/move-examples branch 2 times, most recently from 9eb8702 to ebca4d5 Compare April 25, 2024 14:15

Base automatically changed from mauricio/move-examples to main April 25, 2024 14:30

mauriciovasquezbernal force-pushed the mauricio/gadgets-examples branch from 93c0222 to ec97beb Compare April 25, 2024 20:19

mauriciovasquezbernal marked this pull request as ready for review April 25, 2024 20:26

mauriciovasquezbernal force-pushed the mauricio/gadgets-examples branch 2 times, most recently from cb05414 to f253049 Compare April 25, 2024 20:30

mauriciovasquezbernal force-pushed the mauricio/gadgets-examples branch from f253049 to 36db15e Compare April 25, 2024 22:38

mauriciovasquezbernal commented Apr 25, 2024

View reviewed changes

examples/gadgets/grpc/custom_operator/main.go Show resolved Hide resolved

mauriciovasquezbernal force-pushed the mauricio/gadgets-examples branch from 36db15e to e41b66d Compare April 26, 2024 21:06

This was referenced May 2, 2024

datasources: Add Bool accessors #2796

Merged

Improve "CStrings" handling #2797

Merged

blanquicet reviewed May 8, 2024

View reviewed changes

mauriciovasquezbernal force-pushed the mauricio/gadgets-examples branch from e41b66d to 0e8e0a3 Compare May 9, 2024 17:50

blanquicet approved these changes May 14, 2024

View reviewed changes

mauriciovasquezbernal force-pushed the mauricio/gadgets-examples branch from 0e8e0a3 to 8f7009b Compare May 15, 2024 14:59

mauriciovasquezbernal requested review from eiffel-fl and mqasimsarfraz as code owners May 15, 2024 14:59

mauriciovasquezbernal force-pushed the mauricio/gadgets-examples branch from 8f7009b to bf73d38 Compare May 15, 2024 15:30

eiffel-fl reviewed May 15, 2024

View reviewed changes

docs/core-concepts/custom-resources.md Show resolved Hide resolved

examples/gadgets/operators/local_manager/README.md Outdated Show resolved Hide resolved

examples/gadgets/README.md Outdated Show resolved Hide resolved

examples/gadgets/grpc/custom_operator/main.go Show resolved Hide resolved

mauriciovasquezbernal force-pushed the mauricio/gadgets-examples branch from bf73d38 to 73b2e64 Compare May 17, 2024 21:36

mauriciovasquezbernal force-pushed the mauricio/gadgets-examples branch from 73b2e64 to 18aee98 Compare May 22, 2024 19:56

docs: Replace ⚠️ by [!WARNING]

387d41f

Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

mauriciovasquezbernal added 8 commits May 22, 2024 15:35

examples: Add simple trace open example

2fcde9a

Show how to: - Create a simple operator to subscribe to events - Marshall events to json - Set parameters for the gadget Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

examples: Add simple trace dns example

5896653

Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

examples: Add local manager example

9229f44

Show how to use the localmanager to filter and enrich events with container data. Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

examples: Add fields example

3693164

Show how to use the data source to get some specific fields from the event. Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

examples: Add cli operator example

5e46fce

Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

examples: Add mutate example

e199ed1

Example showing how to add and mutate fields from a datasource. Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

examples: Add grpc example with custom operator

5592797

Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

ci: Build new examples on the CI

43bb599

Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>

mauriciovasquezbernal force-pushed the mauricio/gadgets-examples branch from 18aee98 to 43bb599 Compare May 22, 2024 20:35

mauriciovasquezbernal merged commit a2c353e into main May 22, 2024
59 of 60 checks passed

mauriciovasquezbernal deleted the mauricio/gadgets-examples branch May 22, 2024 20:55

This was referenced May 23, 2024

Initialize operators outside the gadget context #2891

Closed

API: How to handle SchedCLS and SocketFilter programs without LocalManager #2892

Open

Golang API: Improve the parameters handling #2893

Open

blanquicet reviewed May 23, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement image-based gadgets examples #2753

Implement image-based gadgets examples #2753

mauriciovasquezbernal commented Apr 22, 2024 •

edited

mauriciovasquezbernal left a comment

mauriciovasquezbernal Apr 22, 2024

flyth Apr 22, 2024 •

edited

mauriciovasquezbernal commented Apr 25, 2024

blanquicet left a comment

blanquicet May 8, 2024

mauriciovasquezbernal May 9, 2024

flyth May 10, 2024

blanquicet left a comment

blanquicet May 14, 2024

blanquicet May 14, 2024

eiffel-fl left a comment

mauriciovasquezbernal commented May 17, 2024

eiffel-fl commented May 22, 2024

blanquicet May 23, 2024

flyth May 23, 2024

mauriciovasquezbernal May 23, 2024

flyth May 24, 2024


		Those will printed in the gadget's terminal:

		TODO: check why this gadget is lacking so much information compared to the built-in one.

Implement image-based gadgets examples #2753

Implement image-based gadgets examples #2753

Conversation

mauriciovasquezbernal commented Apr 22, 2024 • edited

Gadgets Examples

Simple Gadgets

Operators

Datasource

Executing Gadgets on remove Inspektor Gadget instances

TODO

Future TODOs

mauriciovasquezbernal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flyth Apr 22, 2024 • edited

Choose a reason for hiding this comment

mauriciovasquezbernal commented Apr 25, 2024

blanquicet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blanquicet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eiffel-fl left a comment

Choose a reason for hiding this comment

mauriciovasquezbernal commented May 17, 2024

eiffel-fl commented May 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mauriciovasquezbernal commented Apr 22, 2024 •

edited

flyth Apr 22, 2024 •

edited