First of all, your code is clean and functional! I am glad that after reading your code, the reader will definitely learn a lot. I like it, so may I share some my suggestion?
1, your mlm task has only 11% acc, I think it is because your learning rate is too high, you should try 2e-5 as the google officially suggested
2, the mlm_predict=tf.matmul(x,self.embedding.token_embedding.embeddings, transpose_b=True) code may caused some warning of "variable is not traceable". the correct code should be implemented by senting a sequence of 0 to vocab_size to the embedding layer, the output is exactly the self.embedding.token_embedding.embeddings
after doing that i think you will get 99% of mlm task accuracy as me( i got 95+% acc just after 2 epoch), and the warning will go off~~
BTW, did you graduat from tju? the same to me, Good Luck to you my friend!
First of all, your code is clean and functional! I am glad that after reading your code, the reader will definitely learn a lot. I like it, so may I share some my suggestion?
1, your mlm task has only 11% acc, I think it is because your learning rate is too high, you should try 2e-5 as the google officially suggested
2, the mlm_predict=tf.matmul(x,self.embedding.token_embedding.embeddings, transpose_b=True) code may caused some warning of "variable is not traceable". the correct code should be implemented by senting a sequence of 0 to vocab_size to the embedding layer, the output is exactly the self.embedding.token_embedding.embeddings
after doing that i think you will get 99% of mlm task accuracy as me( i got 95+% acc just after 2 epoch), and the warning will go off~~
BTW, did you graduat from tju? the same to me, Good Luck to you my friend!