VM and Disk size when traing Bert in google TPU

[suchunxie]

Hello. Thank you very much for sharing your experience.
I also want to pretrain a bert with google v3-8, and I want to ask about how to choose the VM Type and Disk size to ensure a bert training.
I noticed that one example said that with 4GBs training corpus, in the create_pretraining_data step, there may cause about 400MB RAM. Is the Virtual Machine and Disk size need to bigger than the RAM maty take? (my corpus is about 1GB)

best wishes.

[suchunxie]

I solved this problem by using TPU VMs. And after furth check, if using VM instance, just choose a fitable CPU or GPU , and change the Disk size will be okay.