Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Context deadline exceeded during vppapi call #802

Open
glazychev-art opened this issue Feb 12, 2024 · 0 comments
Open

Context deadline exceeded during vppapi call #802

glazychev-art opened this issue Feb 12, 2024 · 0 comments

Comments

@glazychev-art
Copy link
Contributor

Description

Let's assume that the request timeout exceeded during some vppapi call. And at this time the Request was processed by VPP, but in our application govpp returns a context timeout error.
As a result, we have leaked vpp resources.

For example:

  1. Create a tap interface with almost expired context:
    tapCreateV2 := &tapv2.TapCreateV2{
    ID: ^uint32(0),
    UseRandomMac: true,
    NumRxQueues: 1,
    TxRingSz: 1024,
    RxRingSz: 1024,
    HostIfNameSet: true,
    HostIfName: mechanism.GetInterfaceName(),
    HostNamespaceSet: true,
    HostNamespace: nsFilename,
    TapFlags: tapv2.TAP_API_FLAG_TUN,
    }
    if conn.GetPayload() == payload.Ethernet {
    tapCreateV2.TapFlags ^= tapv2.TAP_API_FLAG_TUN
    }
    rsp, err := tapv2.NewServiceClient(vppConn).TapCreateV2(ctx, tapCreateV2)
    if err != nil {
    return errors.Wrap(err, "vppapi TapCreateV2 returned error")
    }
  2. It returns an error with context deadline exceeded
  3. But, based on &tapv2.TapCreateV2{...}, VPP creates a corresponding interface in the client namespace with a specific name.
  4. Since we received an error on the line 84, we cannot control this interface in the application. We can't delete it, for example.
  5. We have a resource leak.

Logs:

...
Feb  6 11:27:50.854�[31m [ERRO] [id:34655f24-cc23-4a94-84c3-94a0e317d25d] [type:networkService] �[0m(32.3)                                  vppapi TapCreateV2 returned error: context deadline exceeded
...
vpp# show int addr
tap6 (up):
  L2 xconnect vxlan_tunnel2
tap7 (dn):
tap8 (up):
  L2 xconnect vxlan_tunnel3

Possible solutions

  1. Make vppapi calls atomic. For example, implement transactions
  2. Check the remaining context timeout before each vppapi call
    https://github.com/networkservicemesh/vpphelper/blob/e2b961f768b67dfe0687f5aa90696ffdeffba203/connection.go#L100
  3. Create a chain element, that will check the remaining context timeout for the each endpoint (NSMgr, Forwarder, NSE...)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

No branches or pull requests

1 participant