Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent checking if model exists on every autocompletion #46

Open
dhuertas opened this issue Mar 2, 2024 · 0 comments
Open

Prevent checking if model exists on every autocompletion #46

dhuertas opened this issue Mar 2, 2024 · 0 comments

Comments

@dhuertas
Copy link

dhuertas commented Mar 2, 2024

Hi!

Thanks for coding this little wonder of extension. Kudos! I've been using it for a bit, and I have noticed that every autocompletion generates an extra request to the /api/tags endpoint in Ollama:

image

I suspect it comes from the call to ollamaCheckModel() in provideInlineCompletionItems():

let modelExists = await ollamaCheckModel(inferenceConfig.endpoint, inferenceConfig.modelName);

In my view it should not be necessary to send a request to the /api/tags endpoint every time. I am aware the latency it introduces is orders of magnitude lower than the /api/generate cat, but still ... it's extra job for the extension that (in my view) does not need to do.

I'd suggest to go for a different strategy 🤔 Perhaps do the check once and save the list of available models to check locally. Then check again whenever the configuration changes, or every now and then.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant