documentation/meme_search_engine/reddit_dump.myco

osmarks.net research teams designed [[https://github.com/osmarks/meme-search-engine/blob/master/src/reddit_dump.rs|code]] to download and embed all images ever posted (and not deleted) from Reddit (excluding NSFW, ads, etc), using streaming processing to avoid having to persist intractable amounts of data to disk. Unfortunately, it is still necessary to store embeddings, and the high cost of flash and electricity means a full run (it was tested at 0.5% sampling) has been postponed indefinitely.