osmarks/nanogpt-experiments
1
0
Fork 0
mirror of https://github.com/osmarks/nanogpt-experiments.git synced 2024-12-18 14:10:28 +00:00
Code Issues Releases Wiki Activity

nanogpt: fix multiprocessing in load_dataset on os x

Browse Source 
The issue seems to be that _fixup_main_from_path in multiprocessing
module in python is unable to find entry point, thus, adding
```
if __name__ == '__main__'
```
This commit is contained in:
Oleksandr Kuvshynov 2023-06-17 20:35:38 -04:00
parent bb7e96754a
commit 542ac51d1f
1 changed files with 54 additions and 53 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
data/openwebtext/prepare.py
View File

@ -16,6 +16,7 @@ num_proc = 8
# it is better than 1 usually though # it is better than 1 usually though
num_proc_load_dataset = num_proc num_proc_load_dataset = num_proc
if __name__ == '__main__':
# takes 54GB in huggingface .cache dir, about 8M documents (8,013,769) # takes 54GB in huggingface .cache dir, about 8M documents (8,013,769)
dataset = load_dataset("openwebtext", num_proc=num_proc_load_dataset) dataset = load_dataset("openwebtext", num_proc=num_proc_load_dataset)