fix: missing loglines from tailing logs with GCP

[saikonen]

Fixes an issue where log tailing stops working after a point on GCP, resulting in missing log output on the terminal.

The cause is with the behaviour of google.cloud.storage blob client, where consecutive requests with identical attributes on a single instance will not yield different responses, even if the underlying blob has advanced.

Example flow to reproduce missing log lines on GCP:

from time import sleep
from metaflow import (
    FlowSpec,
    step,
    kubernetes,
)
import os

class MinimalKubeGCPLogExample(FlowSpec):
    @kubernetes
    @step
    def start(self):
        print("BUFFERED: %s" % os.getenv("PYTHONBUFFERED"))
        print("first log line so as not to have an empty file")
        for i in range(10):
            print(f"{i}\r", end="")
            sleep(1)
        print("\r\n")
        print("after progress lines")
        self.next(self.end)

    @step
    def end(self):
        pass


if __name__ == "__main__":
    MinimalKubeGCPLogExample()

also small fix to Azure log tailing as well.

[savingoyal]

can we use generator -

def _has_updates():
    while self._job.is_running:
        yield True
    yield True  # One final update after job finishes
    yield False

[saikonen]

yup this should work too, and it reduces reliance on the other status internals. I'll make the changes

[saikonen]

scratch that, it won't work without larger changes as the tail_logs has_log_updates= callable is simply used in a while loop, which won't exhaust the generator (it counts as true always)

[savingoyal]

is there any perf impact?

[saikonen]

is there any perf impact?

the tailing happens client-side so no performance impact on the actual task execution code during remote exec.
there is of course overhead with re-instantiating the blob client during every tailing poll but this didn't seem significant during testing