osmarks/meme-search-engine
1
0
Fork 0
mirror of https://github.com/osmarks/meme-search-engine.git synced 2025-02-07 14:40:08 +00:00
Code Issues Releases Wiki Activity

Commit Graph

  • ec27deddbf fixed SAE export code master osmarks 2025-01-31 17:12:25 +00:00
  • 899fbb7092 fix embedding-from-URL feature osmarks 2025-01-31 14:10:15 +00:00
  • e57931d47f Multithread query server While profiling suggests that most operations are cheap and IO-bound rather than CPU-bound, the GEMM for deduplication is pretty slow. As such, use multiple threads for higher throughput. osmarks 2025-01-31 13:47:47 +00:00
  • 5215822e39 mlock osmarks 2025-01-29 14:48:15 +00:00
  • 3b2664ca98 update README osmarks 2025-01-25 16:56:54 +00:00
  • 257486678d connect metrics correctly osmarks 2025-01-24 15:23:53 +00:00
  • ee23b81444 release version osmarks 2025-01-24 09:24:28 +00:00
  • 3852d0078d integrate rating model correctly osmarks 2025-01-23 13:45:59 +00:00
  • fbf9b3421d fix lockfile osmarks 2025-01-18 19:35:21 +00:00
  • 3e568ff613 restore stashed code osmarks 2025-01-18 19:07:16 +00:00
  • fcd28a5ede add missing files osmarks 2025-01-18 17:09:44 +00:00
  • a5a6e960bb query code osmarks 2025-01-18 17:09:00 +00:00
  • 63caba2746 integrate rating model osmarks 2025-01-18 11:29:03 +00:00
  • d3fcedda09 cleanup osmarks 2025-01-18 07:25:21 +00:00
  • 0a542ef579 repurpose meme rater osmarks 2025-01-18 07:19:08 +00:00
  • 163dceca4b enforce r, oops osmarks 2025-01-16 21:36:15 +00:00
  • 9291eacead remove rogue debug code osmarks 2025-01-16 21:28:01 +00:00
  • f4376f62ed RobustVamana algorithm for big index run osmarks 2025-01-16 21:10:12 +00:00
  • d341a8c243 accursed k-means balance thing osmarks 2025-01-16 16:02:59 +00:00
  • fdc2af6f8a merge fixes into roargraph roargraph osmarks 2025-01-14 08:35:33 +00:00
  • 5ab91aa17e CLI switch for L osmarks 2025-01-14 08:31:03 +00:00
  • 2cebce1b73 new PQ training code osmarks 2025-01-14 07:46:09 +00:00
  • 4dd97631df fix entire index algorithm (very silly bug) osmarks 2025-01-12 19:48:53 +00:00
  • 0a196694b1 minor tweaks osmarks 2025-01-11 12:19:10 +00:00
  • 9334fc189c fixed roargraph things osmarks 2025-01-11 11:17:51 +00:00
  • 44ffe80102 fix gitignore osmarks 2025-01-11 07:54:23 +00:00
  • 087419f470 remove vestigal r_cap osmarks 2025-01-11 07:36:46 +00:00
  • 8ce51bcb56 correct DiskANN algorithm (silly bug with greedy search) osmarks 2025-01-11 07:35:04 +00:00
  • e9ee563381 tweak some parameters osmarks 2025-01-03 09:22:39 +00:00
  • c6303a6248 add rest of RoarGraph code osmarks 2025-01-02 21:05:32 +00:00
  • 4d08ec370b oops, dependencies osmarks 2025-01-02 21:04:50 +00:00
  • 265502f141 tweak index build, this had better work, aaa osmarks 2025-01-02 21:04:26 +00:00
  • 92396a68fb broken roargraph implementation osmarks 2025-01-02 19:57:17 +00:00
  • f1283137d6 release WIP DiskANN index orchestration code osmarks 2025-01-01 14:40:24 +00:00
  • 35df1201e2 fix overflow bug osmarks 2025-01-01 14:40:00 +00:00
  • e0cf65204b release early draft of index code osmarks 2024-12-31 23:05:48 +00:00
  • 512b776e10 use slightly worse image scaling osmarks 2024-11-13 18:31:18 +00:00
  • 8097ce8d91 improve dump processing and misc performance fixes osmarks 2024-11-11 19:43:07 +00:00
  • c277b49dc1 fix resumption, oops osmarks 2024-11-07 20:43:26 +00:00
  • b9bb629e6f performance improvements osmarks 2024-11-07 16:52:58 +00:00
  • 7fa14d45ae improve observability and fix up Reddit dump for full-scale run osmarks 2024-11-02 19:38:05 +00:00
  • 1d0ff95955 Sparse autoencoder testing osmarks 2024-10-05 17:22:44 +01:00
  • fc6d0c9409 Fix crawler (rate limit changes) osmarks 2024-07-15 18:48:55 +01:00
  • 43ff9215fb File metadata storage. The backend now knows how big images are, so the frontend can size images correctly before they're loaded. This should significantly improve the UI on slow connections. Also fix bug where videos weren't erased from the index properly. osmarks 2024-06-26 20:02:12 +01:00
  • 1ab254ff1d Adjust index storage for memory efficiency and fix SQLite interface type confusion osmarks 2024-06-25 08:23:30 +01:00
  • e7adf738f6 Fix typo, in the sense of application-killing bug. osmarks 2024-05-31 00:35:16 +01:00
  • 747058e254 misc fixes - thumbnails/OCR off was broken - problematic video files caused segfaults (I blame ffmpeg for this) osmarks 2024-05-30 19:05:54 +01:00
  • 3257521068 Video search osmarks 2024-05-30 15:58:31 +01:00
  • 74d91d52e5 probably I should do better testing osmarks 2024-05-29 21:25:50 +01:00
  • 5eae8674ce video parsing basics osmarks 2024-05-28 22:28:41 +01:00
  • 129b769a56 hackily patch horrifyingly nondeterministic-but-fast image encoder in osmarks 2024-05-27 20:21:44 +01:00
  • d4e136b6a7 AITemplate builds of the image encoder work, at great personal cost osmarks 2024-05-27 19:05:25 +01:00
  • a8329e43fc more progress on Reddit osmarks 2024-05-27 15:22:28 +01:00
  • f8d68d9d54 WIP Reddit dump loader osmarks 2024-05-24 17:47:18 +01:00
  • 978aadda6a Improved UI for sliders osmarks 2024-05-22 20:26:23 +01:00
  • d8c147df52 Predefined embedding modes in search osmarks 2024-05-22 20:17:13 +01:00
  • 14387a61a3 refactor configuration osmarks 2024-05-22 19:02:34 +01:00
  • ffc3d648a6 basic monitoring implementation osmarks 2024-05-22 18:49:32 +01:00
  • ce590298a7 concurrent index queries and fix database typo yet again osmarks 2024-05-22 18:25:50 +01:00
  • 349fe802f7 meme interpretability osmarks 2024-05-22 16:18:45 +01:00
  • bd426a30ba Port meme acquisition pipeline to new API, database Also fix a really stupid oversight in crawling code. osmarks 2024-05-22 15:43:56 +01:00
  • 30b1b72712 I really should test database queries better osmarks 2024-05-22 14:35:29 +01:00
  • 9455438bab frontend fixes osmarks 2024-05-21 20:17:23 +01:00
  • 24fbc0dd1f apparently, that quality value is too low osmarks 2024-05-21 20:09:28 +01:00
  • 63a9b3d9a6 the consequences of my own actions osmarks 2024-05-21 12:39:04 +01:00
  • e705a9db21 I hate precedence osmarks 2024-05-21 12:33:32 +01:00
  • b7010b41dd oops osmarks 2024-05-21 01:58:50 +01:00
  • 7cb42e028f Rewrite entire application (well, backend) in Rust and also Go I decided I wanted to integrate the experimental OCR thing better, so I rewrote in Go and also integrated the thumbnailer. However, Go is a bad langauge and I only used it out of spite. It turned out to have a very hard-to-fix memory leak due to some unclear interaction between libvips and both sets of bindings I tried, so I had Claude-3 transpile it to Rust then spent a while fixing the several mistakes it made and making tweaks. The new Rust version works, although I need to actually do something with the OCR data and make the index queryable concurrently. osmarks 2024-05-21 00:09:04 +01:00
  • fa863c2075 "release" unfinished scripts and miscellaneous JSON files osmarks 2024-05-18 14:34:30 +01:00
  • caa8306ff7 oops osmarks 2024-05-18 13:21:13 +01:00
  • 6491e02e88 preliminary work on OCR osmarks 2024-05-18 00:39:05 +01:00
  • a3574674d0 "documentation" osmarks 2024-04-27 17:33:24 +01:00
  • 2447e134ef There were more memes than anticipated. osmarks 2024-04-27 17:24:01 +01:00
  • 80db16d02a full pipeline osmarks 2024-04-22 18:44:29 +01:00
  • 7bae095384 accidentally lost some manual labels, oops, etc osmarks 2024-04-22 13:54:07 +01:00
  • cebb4f9d00 better evals osmarks 2024-04-22 13:43:06 +01:00
  • 58ce70bb5e meme rater model code (documentation "later") osmarks 2024-04-21 23:50:48 +01:00
  • 0b0261f625 preliminary meme rater work osmarks 2024-04-20 16:55:11 +01:00
  • e9a7493343 stop requiring internal aiosqlite patch osmarks 2024-01-25 00:01:02 +00:00
  • e3ffc426b7 Actually delete missing files osmarks 2024-01-02 14:12:26 +00:00
  • 4626f53bcb Return to OpenCLIP osmarks 2023-11-13 17:31:43 +00:00
  • 74bb1bc343 thumbnailer system osmarks 2023-10-27 15:50:21 +01:00
  • 5b5ef271aa ""documentation"" osmarks 2023-10-09 12:35:26 +01:00
  • 68a14d7da9 unignore device osmarks 2023-10-08 22:54:06 +01:00
  • 20fcc9317f forgot the README osmarks 2023-10-08 22:52:57 +01:00
  • 46fca3eb7f faster indexing, SigLIP models osmarks 2023-10-08 22:52:17 +01:00
  • 2c9ce67ab2 I really should do better testing osmarks 2023-09-30 21:10:27 +01:00
  • c235217751 read from query strings osmarks 2023-09-30 15:56:04 +01:00
  • 8abe384cd1 fix concurrency bug osmarks 2023-09-29 18:35:42 +01:00
  • 120edc1a47 oops osmarks 2023-09-29 18:34:06 +01:00
  • 33f4382146 also typo osmarks 2023-09-28 17:32:41 +01:00
  • aa6163a784 typo osmarks 2023-09-28 17:30:44 +01:00
  • 9d89e6e4f5 initial commit osmarks 2023-09-28 17:30:20 +01:00