Skip to content

Latest commit

 

History

History
378 lines (295 loc) · 12 KB

part44-eng.md

File metadata and controls

378 lines (295 loc) · 12 KB
Raw

How to extract info from gRPC metadata

Original video

Hello everyone, welcome back to the backend master class! In this lecture, we will learn about gRPC metadata and how to work with it. Metadata is just some information about a particular RPC call, in the form of a list of key-value pairs. It lets the client provide some extra information associated with the call to the server and vice versa. If you still remember, in the previous lectures, we've implemented the login user API, and in this API, we're creating a session and storing some useful information in the database.

However, for now, if we look at the record, you can see that the user_agent and client_ip fields are still empty.

That's because in our RPC login user handler, we're not getting and saving the correct values. We're just passing in empty strings at the moment.

So today, let's learn how to fill them with real metadata!

Extracting user_agent and client_ip from metadata

But, first, let me remind you that we're using gRPC gateway, so the requests coming to this handler function can be from both a gRPC client (such as Evans CLI), as well as an HTTP client (such as Postman).

And therefore, the metadata that they send can be stored in different formats. Alright, now I'm gonna create a new file: metadata.go inside the gapi package. Then in this file, I'm gonna add a new method to our server struct, let's call it extractMetadata. This function will take a context object as input, because all the gRPC metadata will be stored in this context, and it will return a Metadata object as output.

func (server *Server) extractMetadata(ctx context.Context) *Metadata {

}

I'm gonna define the Metadata struct up here.

type Metadata struct {
	UserAgent string
	ClientIP  string
}

This struct will keep track of all metadata we want to extract from the context. For now, we only need 2 fields, 1 for user agent, and 1 for the client's IP address.

Alright, with this struct, we can now create a new metadata object and return it.

We will learn how to fill its content in a moment.

func (server *Server) extractMetadata(ctx context.Context) *Metadata {
	mtdt := &Metadata{}
	return mtdt
}

But first, let's go back to the RPC LoginUser. Here, just before creating a session, we will call server.extractMetadata(), and pass in the context, that we receive from the input argument of the LoginUser method. Then, we just replace this empty string with mtdt.UserAgent and this one with mtdt.ClientIP.

mtdt := server.extractMetadata(ctx)
session, err := server.store.CreateSession(ctx, db.CreateSessionParams{
    ID:           refreshPayload.ID,
    Username:     user.Username,
    RefreshToken: refreshToken,
    UserAgent:    mtdt.UserAgent,
    ClientIp:     mtdt.ClientIP,
    IsBlocked:    false,
    ExpiresAt:    refreshPayload.ExpiredAt,
})

And that's it for the session info. Now let's go back to the extractMetadata() function to get the correct values from the context!

In order to do that, we have to call metadata.FromIncomingContext() and pass in the input context object.

metadata.FromIncomingContext(ctx)

Here, metadata is a subpackage of gRPC that provides functions to work with metadata. This function will return 2 outputs: a metadata.MD object and a boolean. So let's store them in 2 variables: md and ok. The ok boolean variable will tell us if the metadata is successfully retrieved or not.

if md, ok := metadata.FromIncomingContext(ctx); ok {
    
}

If it is, we will print out the context of the md object to see what's inside.

if md, ok := metadata.FromIncomingContext(ctx); ok {
    log.Printf("md: %+v\n", md)
}

Alright, let's restart the server!

make server

And go back to Postman to send the login request.

make server
go run main.go
2022/04/10 15:14:24 start gRPC server at [::]:9090
2022/04/10 15:14:24 start HTTP gateway server at [::]:8080
2022/04/10 15:14:28 md: map[grpcgateway-accept:[*/*] grpcgateway-content-type:[application/json] grpcgateway-user-agent:[PostmanRuntime/7.29.0] x-forwarded-for:[::1] x-forwarded-host:[localhost:8080]]

Here we can see that md is actually a map, where the key is a string, and the value is an array of strings.

The user agent that we're looking for must be the value of this grpcgateway-user-agent key.

So let's copy it to our code.

I'm gonna store it in a constant named grpcGatewayUserAgentHeader. Similarly, the client's IP address will be the value stored with this x-forwarded-for key. So I'm gonna declare another constant for it: xForwardedForHeader.

const (
	grpcGatewayUserAgentHeader = "grpcgateway-user-agent"
	xForwardedForHeader = "x-forwarded-for"
)

Now, in the extractMetadata() function, let's call md.Get() to get the value of the gRPC gateway user agent header. It will return an array of strings, so I'm gonna store it in the userAgents variable. Then we should check if this array is not empty. In that case, the user agent value must be the first item in the list. So I will set mddt.UserAgent to userAgents[0]. The same thing can be done for the client's IP address. We get the value of the xForwardedForHeader, store it in the clientIPs array, check if the array is not empty, and set mtdt.ClientIP = clientIPs[0].

if md, ok := metadata.FromIncomingContext(ctx); ok {
    log.Printf("md: %+v\n", md)
    if userAgents := md.Get(grpcGatewayUserAgentHeader); len(userAgents) > 0 {
        mtdt.UserAgent = userAgents[0]
    }
    
    if clientIPs := md.Get(xForwardedForHeader); len(clientIPs) > 0 {
        mtdt.ClientIP = clientIPs[0]
    }
}

That's basically it. Let's try to restart the server, and resend the login user API request in Postman.

Now, if we look at the session record in TablePlus, we will see that the user_agent and the client_ip fields are no longer empty as before.

They have been filled with the correct values we extracted from the gRPC metadata.

So it works! Excellent!

However, keep in mind that, it's just working for the HTTP client.

How about a gRPC client?

Will it be different if we send a gRPC request instead of HTTP?

Well, let's try connecting to the server with Evans CLI, and call the LoginUser RPC. Enter username and password to login.

call LoginUser
username (TYPE_STRING) => alice
password (TYPE_STRING) => secret
{
  "user": {
    "username": "alice",
    "fullName": "Alice",
    "email": "alice@email.com",
    "passwordChangedAt": "0001-01-01T00:00:00Z",
    "createdAt": "2022-04-10T12:49:05.232358Z"
  },
  "sessionId": "dd8af954-6681-4d0f-bc84-1fd4498986ab",
  "accessToken": "v2.local.R2SZJByrnVhie7anzI3Zjhpta8lkVMmQwHCG-uaDgXc7rBiBFqNQR2GcWhOhFvTlvX2SGdpqkDXJ3lJGvuk8d5XJENdZqsjtsEPYq7ojgVTjFG8VbY9p_4ub82frYATOlA0UBrQll7o09Q98ELn-6pl3FIV05nJD_tmcQVtVKSkfqWg6n53w6qUtLYiq-mvRnDPHcSOODJ8Sj8UnpuZ2HBFsuaYYKIPVCiXrcmaMYHduBh5-oxjdWv9BUgpjXAviz-hjgUvX4roY3w3jHA.bnVsbA",
  "refreshToken": "v2.local.vdRZ9VjMuWLn9JuNA3Dws1UH5r5gnz6fTMundXFI8ZlVxMjPzNlgIQSF9eX8ebnTizzlaycciYNMALe5mO9iCuMLaP-1qDDqfq0VZ59WCfjahpr8f06UQi8TXqlCZh4dJtz526Nj7Cg-pGi4trmwFIKNoO3jzSLZb1O68kGfwKW-UfknfUoY2b2mWFBd7TUu2WyES8DVgPm7AX9Lyhl_Im3THbq0jqvercvzNuHjqFRp9odbdLj-NQQqGsexXg95ao6Ujt77f1tY2SzF1w.bnVsbA",
  "accessTokenExpiresAt": "2022-04-10T13:34:47.005092Z",
  "refreshTokenExpiresAt": "2022-04-11T13:19:47.005132Z"
}

OK, the session has been created.

Let's check it out in TablePlus!

As you can see here, the user_agent and client_ip are still empty. So what we've implemented before doesn't work for a gRPC client.

Let's check the logs to find out why.

make server
go run main.go
2022/04/10 15:19:04 start gRPC server at [::]:9090
2022/04/10 15:19:04 start HTTP gateway server at [::]:8080
2022/04/10 15:19:08 md: map[grpcgateway-accept:[*/*] grpcgateway-content-type:[application/json] grpcgateway-user-agent:[PostmanRuntime/7.28.0] x-forwarded-for:[::1] x-forwarded-host:[localhost:8080]]
2022/04/10 15:19:47 md: map[:authority:[localhost:9090] content-type:[application/grpc] grpc-client:[evans] user-agent:[grpc-go/1.35.0]]

Here you can see that the metadata map looks completely different from the one above it. It's still a map of string to array, but the names of the keys are not the same. For example, the user agent key is just user-agent, not grpcgateway-user-agent as before. So let's copy this key, and declare a new constant for it in our code. I'm gonna call it just userAgentHeader.

const (
	grpcGatewayUserAgentHeader = "grpcgateway-user-agent"
	xForwardedForHeader        = "x-forwarded-for"
	userAgentHeader = "user-agent"
)

Then, in the extractMetadata() function, I'm gonna duplicate this chunk of codes,

if userAgents := md.Get(grpcGatewayUserAgentHeader); len(userAgents) > 0 {
    mtdt.UserAgent = userAgents[0]
}

and rename the header to userAgentHeader.

I'm gonna delete this log as well.

log.Printf("md: %+v\n", md)

So that's how we get the user agent.

But it seems there's no field in the metadata that stores the client's IP address. So how can we get it?

Well, although the gRPC client's IP address is not stored in the metadata, it's still stored somewhere in the context. And we can get it with a different subpackage of gRPC called peer.

All we have to do is to call peer.FromContext(), and pass in the input context object.

This function will return the peer information, together with a boolean value to tell us if the info exists or not. So we check if ok is true, then, the client's IP address will be stored in the p.Addr field. We just need to convert it to string. And that's all!

if p, ok := peer.FromContext(ctx); ok {
    mtdt.ClientIP = p.Addr.String()
}

Both user agent and client IP fields have been extracted. Let's give it a try!

Testing code using Evans and Postman

I'm gonna restart the server. Then, in the Evans client console, let's call Login User RPC again. Enter username and password.

call LoginUser
username (TYPE_STRING) => alice
password (TYPE_STRING) => secret
{
  "user": {
    "username": "alice",
    "fullName": "Alice",
    "email": "alice@email.com",
    "passwordChangedAt": "0001-01-01T00:00:00Z",
    "createdAt": "2022-04-10T12:49:05.232358Z"
  },
  "sessionId": "235c7b64-2b8e-4cd5-bfe7-4847545ebf01",
  "accessToken": "v2.local.G_Y2a5bPuRezLTSJ0A3CAnzbFhfUyEOwC--5LVOBfUzGBkVq9Of1C4PxXP5PuIU7QYb-KGvpjxxtyY6Vs3s6rDEPmjid9TulZqofjFFoPioa4gEL2S9BcEBPR_-RrFB3nM6nuW2oLVCdkoELzgxNP3C_q5nyHivTBVMuN9tzjnQd2WRKttZdq23_f8yotRfezpTtr9rVvC2NslhH4DAOEBjTZcHBarlR_cW9YdH9pHiL1R3Y55_sBxsHK_NJdqBrke9jwECrxcnnUqUeQQ.bnVsbA",
  "refreshToken": "v2.local.kwHY8Kscvi0fBw2fdantlGbYkAfnEnw0eq5gn89RZeQE6lv9WuhOHh4ITY4kvFqwcC_F1hoY_DJ4IeikN3tFPnn0TjSjgPYjF91amZyX4hkeuMzQFLWuwZlFJy1mvKKGcMYTeHaTc9kb2OMeFTDhi9-11_TblAt7Z5JOpJdn-jLkk12OoeMiY14uPKjmRW8JW4RhtreDbtE5GWDK2wIhYC89E7NfmdeVJnN5tgjJlP5W_Twr1kdh2ogZWAHTkyAbJFw5DyQPdZv0WnKU-g.bnVsbA",
  "accessTokenExpiresAt": "2022-04-10T13:37:08.498218Z",
  "refreshTokenExpiresAt": "2022-04-11T13:22:08.498455Z"
}

OK, the request is successful. This time, if we look at the session record, we will see that the user_agent and client_ip have been filled with the correct values.

So now, our code can extract that information from all requests, whether it comes from a gRPC client or an HTTP client. You can verify it again by sending an HTTP request in Postman.

It's still successful, and we have all the metadata info stored in the session table.

Pretty awesome, isn't it?

And that brings us to the end of this lecture. We've learned how to extract some useful information from the metadata of the gRPC context. I hope it's interesting and useful for you.

And by the way, I've just created a Discord channel for Tech School.

Don't forget to join it to chat with me and other students.

Thanks a lot for watching! Happy learning, and see you in the next lecture!