14 lines
891 B
Plaintext
14 lines
891 B
Plaintext
GPT-4 is a 2022/2023 [[large language model]] by [[OpenAI]]. Similarly to [[GPT-3]], the model was not released except through an [[API]]; dissimilarly, almost no technical details such as architecture, dataset and parameter count were released in the [[https://arxiv.org/abs/2303.08774|technical report]], though credible leaks later provided some of this information.
|
|
|
|
== Architecture
|
|
|
|
* 1.8 trillion parameters.
|
|
* 16-way [[Mixture of Experts]] with two used per forward pass.
|
|
* 25000 [[A100]] [[GPU]]s for ~90 days.
|
|
* ~13T tokens training data (not all unique; run for multiple epochs).
|
|
|
|
== GPT-4-base
|
|
|
|
Unlike GPT-3, a [[base model]], GPT-4 was only released to the general public in various [[instruction tuning|instruction-tuned]] forms, for "safety". GPT-4's base model is accessible under NDA. According to the limited information from those who have used it,
|
|
|
|
== Sydney |