Add an alternative option if API rate limit is reached

[AlexandreDecan]

Hello,

Currently, according to https://github.com/natarajan-chidambaram/RABBIT/blob/main/rabbit.py#L24, if the API rate limit is reached, RABBIT waits until the quota replenishes. What about having a --no-wait option that makes RABBIT outputs all the already processed accounts and stops if the API rate limit is reached? (it will then be up to the user to compare the accounts in the input/output to see which ones were not yet processed, and to use RABBIT again on them). This will allow users to rely on multiple API keys.

[natarajan-chidambaram]

This is already part of RABBIT as an input option https://github.com/natarajan-chidambaram/RABBIT#:~:text=%2D%2Dincremental, which the user can specify if they want the results as soon as it is predicted.

For multiple API keys option - either
(i) a user can stop the execution and restart with a new key
or
(ii) the tool can accept multiple API keys and iterate over it once the limit is reached.

[AlexandreDecan]

I haven't seen anything related to the ability to use multiple API keys. Did I miss something?

Considering the first point (i): how to do that? Using CTRL+C? I've looked at the code, and it seems (see https://github.com/natarajan-chidambaram/RABBIT/blob/main/rabbit.py#L117) that you display some message to inform the user about rate limit reached... but you shouldn't do that on STDOUT but rather on STDERR, since if rabbit is in text mode, that means the output will also contain this message.

[AlexandreDecan]

Notice that, while it's convenient for a user to see a message when the rate limit is reached (so that he can either decide to wait or to stop with CTRL+C), this is not convenient when the tool is used in an automated way. Having the tool stopping itself automatically is a better option to automate its usage on thousands of accounts.

[natarajan-chidambaram]

Sorry for the confusion, the option of using multiple API keys was a proposal (should have mentioned this explicitly)

I haven't seen anything related to the ability to use multiple API keys. Did I miss something?

Considering the first point (i): how to do that? Using CTRL+C? I've looked at the code, and it seems (see https://github.com/natarajan-chidambaram/RABBIT/blob/main/rabbit.py#L117) that you display some message to inform the user about rate limit reached... but you shouldn't do that on STDOUT but rather on STDERR, since if rabbit is in text mode, that means the output will also contain this message.

Sorry for the confusion, the option of using multiple API keys was a proposal (should have mentioned this explicitly)

[natarajan-chidambaram]

Notice that, while it's convenient for a user to see a message when the rate limit is reached (so that he can either decide to wait or to stop with CTRL+C), this is not convenient when the tool is used in an automated way. Having the tool stopping itself automatically is a better option to automate its usage on thousands of accounts.

I don't want the tool to stop itself when the API rate limit is reached. If one wants to execute this tool overnight on thousands of contributors, this would be a problem.
Proposal - have an input option for the user to choose what to do when the API rate limit is reached (i) go to next key in the provided list of keys, (ii) wait (iii) exit.

[AlexandreDecan]

Having a --no-wait option makes it easy to process a lot of accounts, and to support multiple API keys without having explicit support for this. For example:

accounts = set(a, b, c, ...)
keys = itertools.cycle(set(a, b, c, ...))
while len(accounts) > 0:
  output = call_rabbit_with_no_wait(accounts, next(keys))
  accounts = accounts - list_accounts(output)