Add gpt2 implementations in python and c++

[ChinmayK0607]

Adds two files:

1. train.cpp -> gpt2 implementation in c++
2. train.py -> gpt2 implementation in python

Adds dataloaders as well as requirements file as well.

[kamatajinkya2]

General

1. Prefer pyproject.toml over requirements.txt. Refer to Why Should I Choose pyproject.toml over requirements.txt for managing dependencies?
2. Use a nested folder structure. Aka train.py, dataloader.py go inside src or llm101 or scripts folder. This will help in adding a test folder
3. Implement unit tests (Not applicable in this instance, but just typing out)
4. Have appropriate white spaces after a class of a function ends. You can use autoformatters like Black to achieve this.

To be continued...

[kamatajinkya2]

Use if name main pattern to make this module reusable also to prevent wonky variable scoping

[kamatajinkya2]

Suggested change

	input_file_path = os.path.join(os.path.dirname(__file__), 'input.txt')
	if not os.path.exists(input_file_path):
	data_url = 'https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt'
	with open(input_file_path, 'w', encoding='utf-8') as f:
	f.write(requests.get(data_url).text)
	with open(input_file_path, 'r', encoding='utf-8') as f:
	data = f.read()
	n = len(data)
	train_data = data[:int(n*0.9)]
	val_data = data[int(n*0.9):]
	# encode with tiktoken gpt2 bpe
	enc = tiktoken.get_encoding("gpt2")
	train_ids = enc.encode_ordinary(train_data)
	val_ids = enc.encode_ordinary(val_data)
	print(f"train has {len(train_ids):,} tokens")
	print(f"val has {len(val_ids):,} tokens")
	# export to bin files
	train_ids = np.array(train_ids, dtype=np.uint16)
	val_ids = np.array(val_ids, dtype=np.uint16)
	train_ids.tofile(os.path.join(os.path.dirname(__file__), 'train.bin'))
	val_ids.tofile(os.path.join(os.path.dirname(__file__), 'val.bin'))
	def main():
	input_file_path = os.path.join(os.path.dirname(__file__), 'input.txt')
	if not os.path.exists(input_file_path):
	data_url = 'https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt'
	with open(input_file_path, 'w', encoding='utf-8') as f:
	f.write(requests.get(data_url).text)
	with open(input_file_path, 'r', encoding='utf-8') as f:
	data = f.read()
	n = len(data)
	train_data = data[:int(n*0.9)]
	val_data = data[int(n*0.9):]
	# encode with tiktoken gpt2 bpe
	enc = tiktoken.get_encoding("gpt2")
	train_ids = enc.encode_ordinary(train_data)
	val_ids = enc.encode_ordinary(val_data)
	print(f"train has {len(train_ids):,} tokens")
	print(f"val has {len(val_ids):,} tokens")
	# export to bin files
	train_ids = np.array(train_ids, dtype=np.uint16)
	val_ids = np.array(val_ids, dtype=np.uint16)
	train_ids.tofile(os.path.join(os.path.dirname(__file__), 'train.bin'))
	val_ids.tofile(os.path.join(os.path.dirname(__file__), 'val.bin'))
	if __name__ == '__main__':
	main()

[kamatajinkya2]

Dont use global variables

[kamatajinkya2]

use if __name__ == "__main__":

[kamatajinkya2]

Move this to else block. This is causing confusion.

[kamatajinkya2]

Prefer initizliser list. This way your compiler can warn you if there are uninitialized variables

[kamatajinkya2]

I like east const. Checkout https://hackingcpp.com/cpp/design/east_vs_west_const.html

[kamatajinkya2]

Typically in C++ member variables have a m_ prefix. This prevents variable shadowing

[kamatajinkya2]

Mark this as final so no one accidently inherets.

[kamatajinkya2]

Prefer auto initialization. This prevents uninitialized garbage values