osmarks/meme-search-engine
1
0
Fork 0
mirror of https://github.com/osmarks/meme-search-engine.git synced 2024-11-13 23:34:49 +00:00
Code Issues Releases Wiki Activity
7fa14d45ae
meme-search-engine/Cargo.toml

58 lines
1.3 KiB
TOML
Raw Normal View History

Rewrite entire application (well, backend) in Rust and also Go I decided I wanted to integrate the experimental OCR thing better, so I rewrote in Go and also integrated the thumbnailer. However, Go is a bad langauge and I only used it out of spite. It turned out to have a very hard-to-fix memory leak due to some unclear interaction between libvips and both sets of bindings I tried, so I had Claude-3 transpile it to Rust then spent a while fixing the several mistakes it made and making tweaks. The new Rust version works, although I need to actually do something with the OCR data and make the index queryable concurrently.
2024-05-20 23:09:04 +00:00
[package]
name = "meme-search-engine"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
improve observability and fix up Reddit dump for full-scale run
2024-11-02 19:38:05 +00:00
tokio = { version = "1", features = ["full", "tracing"] }
Rewrite entire application (well, backend) in Rust and also Go I decided I wanted to integrate the experimental OCR thing better, so I rewrote in Go and also integrated the thumbnailer. However, Go is a bad langauge and I only used it out of spite. It turned out to have a very hard-to-fix memory leak due to some unclear interaction between libvips and both sets of bindings I tried, so I had Claude-3 transpile it to Rust then spent a while fixing the several mistakes it made and making tweaks. The new Rust version works, although I need to actually do something with the OCR data and make the index queryable concurrently.
2024-05-20 23:09:04 +00:00
axum = "0.7"
Adjust index storage for memory efficiency and fix SQLite interface type confusion
2024-06-25 07:23:30 +00:00
image = { version = "0.25", features = ["avif", "avif-native", "nasm"] }
Rewrite entire application (well, backend) in Rust and also Go I decided I wanted to integrate the experimental OCR thing better, so I rewrote in Go and also integrated the thumbnailer. However, Go is a bad langauge and I only used it out of spite. It turned out to have a very hard-to-fix memory leak due to some unclear interaction between libvips and both sets of bindings I tried, so I had Claude-3 transpile it to Rust then spent a while fixing the several mistakes it made and making tweaks. The new Rust version works, although I need to actually do something with the OCR data and make the index queryable concurrently.
2024-05-20 23:09:04 +00:00
reqwest = { version = "0.12", features = ["multipart"] }
serde = { version = "1", features = ["derive"] }
sqlx = { version = "0.7", features = ["runtime-tokio", "sqlite"] }
walkdir = "1"
rmp-serde = "1"
serde_json = "1"
chrono = "0.4"
base64 = "0.22"
anyhow = "1"
fnv = "1"
faiss = "0.12"
ndarray = "0.15"
half = { version = "2" }
regex = "1"
improve observability and fix up Reddit dump for full-scale run
2024-11-02 19:38:05 +00:00
tracing = "0.1"
console-subscriber = "0.4"
Rewrite entire application (well, backend) in Rust and also Go I decided I wanted to integrate the experimental OCR thing better, so I rewrote in Go and also integrated the thumbnailer. However, Go is a bad langauge and I only used it out of spite. It turned out to have a very hard-to-fix memory leak due to some unclear interaction between libvips and both sets of bindings I tried, so I had Claude-3 transpile it to Rust then spent a while fixing the several mistakes it made and making tweaks. The new Rust version works, although I need to actually do something with the OCR data and make the index queryable concurrently.
2024-05-20 23:09:04 +00:00
futures-util = "0.3"
tokio-stream = "0.1"
num_cpus = "1"
serde_bytes = "0.11"
tower-http = { version = "0.5", features = ["cors"] }
tower = "0.4"
basic monitoring implementation
2024-05-22 17:49:32 +00:00
json5 = "0.4"
prometheus = "0.13"
WIP Reddit dump loader
2024-05-24 16:47:18 +00:00
lazy_static = "1"
zstd = "0.13"
url = "2"
fastrand = "2"
mimalloc = "0.1"
sonic-rs = "0.3"
video parsing basics
2024-05-28 21:28:41 +00:00
ffmpeg-the-third = "2.0"
Adjust index storage for memory efficiency and fix SQLite interface type confusion
2024-06-25 07:23:30 +00:00
compact_str = { version = "0.8.0-beta", features = ["serde"] }
improve observability and fix up Reddit dump for full-scale run
2024-11-02 19:38:05 +00:00
itertools = "0.13"
async-recursion = "1"
more progress on Reddit
2024-05-27 14:22:28 +00:00
WIP Reddit dump loader
2024-05-24 16:47:18 +00:00
[[bin]]
name = "reddit-dump"
video parsing basics
2024-05-28 21:28:41 +00:00
path = "src/reddit_dump.rs"
[[bin]]
name = "video-reader"
Adjust index storage for memory efficiency and fix SQLite interface type confusion
2024-06-25 07:23:30 +00:00
path = "src/video_reader.rs"
improve observability and fix up Reddit dump for full-scale run
2024-11-02 19:38:05 +00:00
[[bin]]
name = "dump-processor"
path = "src/dump_processor.rs"
Reference in New Issue Copy Permalink