osmarks/documentation
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Create ‘softmax_bottleneck’

Browse Source
This commit is contained in:
osmarks 2024-09-17 19:59:45 +00:00 committed by wikimind
parent ae1d055899
commit 0bda0e4b0e
1 changed files with 6 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
softmax_bottleneck.myco Normal file
View File

@ -0,0 +1,6 @@
Almost all modern [[large language model|LLMs]] map relatively low-dimensional hidden states to high-dimensional probability distributions over [[tokenizer|tokens]] using a single [[matrix]] and a [[softmax]] operation. The [[rank]] of this transformation is limited to the hidden size, so not all valid probability distributions can be represented. This has a number of [[consequences]].
References:
* https://x.com/kalomaze/status/1776341569542431150
* https://aclanthology.org/2022.acl-long.554/