osmarks/documentation
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Edit ‘osmarks.net_web_search_plan_(secret)’

Browse Source
This commit is contained in:
osmarks 2025-03-07 14:45:49 +00:00 committed by wikimind
parent f18044a7a6
commit 0726eb9ea7
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
osmarks.net_web_search_plan_(secret).myco
View File

@ -12,7 +12,9 @@ The job of a search engine is to retrieve useful information for users. This is
* Anna's Archive is ~0.5PB. This contains a substantial fraction of books and papers. These are plausibly higher-quality than the general internet.
* We do need general internet data for breadth of knowledge etc. This runs to PB (Common Crawl etc). Apparently billions of pages per month.
* There is lots of alpha in weird corners of Twitter and also Discord. It would be useful to scrape these, though people would complain.
* {There is lots of alpha in weird corners of Twitter and also Discord. It would be useful to scrape these, though people would complain.
* Also IRC, but that's logged worse.
}
* Images, PDFs, etc contain useful knowledge which hasn't been integrated properly into most things. We need* these.
= Indexing