osmarks/documentation
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Edit ‘meme_search_engine/reddit_dump’

Browse Source
This commit is contained in:
osmarks 2025-01-24 12:23:29 +00:00 committed by wikimind
parent 5ca4244252
commit 3cf0bc334c
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
meme_search_engine/reddit_dump.myco
View File

@ -1 +1,3 @@
osmarks.net research teams designed [[https://github.com/osmarks/meme-search-engine/blob/master/src/reddit_dump.rs|code]] to download and embed all images ever posted (and not deleted) from Reddit (excluding NSFW, ads, etc), using streaming processing to avoid having to persist intractable amounts of data to disk. Unfortunately, it is still necessary to store embeddings, so 0.8TB of storage is still required (estimated), as well as a month of compute time. This is currently in progress. Due to the unanticipated complexity of high-performance high-recall [[vector indexing]] on osmarks.net compute budgets, the project has required more development timeslices than predicted.
osmarks.net research teams designed [[https://github.com/osmarks/meme-search-engine/blob/master/src/reddit_dump.rs|code]] to download and embed all images ever posted (and not deleted) from Reddit (excluding NSFW, ads, etc), using streaming processing to avoid having to persist intractable amounts of data to disk. Unfortunately, it is still necessary to store embeddings, so 0.8TB of storage is still required (estimated), as well as a month of compute time. Due to the unanticipated complexity of high-performance high-recall [[vector indexing]] on osmarks.net compute budgets, the project required more development timeslices than predicted, but has been completed.
=> Nooscope