1
0
mirror of https://github.com/osmarks/nanogpt-experiments.git synced 2024-12-23 08:30:41 +00:00
nanogpt-experiments/data/openwebtext
2023-01-01 01:29:48 +00:00
..
prepare.py candidate changes to apis, have to think through more 2023-01-01 01:29:48 +00:00
readme.md first very bad commit 2022-12-28 00:58:19 +00:00

openwebtext dataset

after running prepare.py (preprocess) we get:

  • train.bin is ~17GB, val.bin ~8.5MB
  • train has ~9B tokens (9,035,582,198)
  • val has ~4M tokens (4,434,897)

this came from 8,013,769 documents in total.

references: