Avi Chawla@_avichawla
open on x ↗

Researchers built a new RAG approach that: - does not need a vector DB. - does not embed data. - involves no chunking. - performs no similarity search. And it hit 98.7% accuracy on a financial benchmark (SOTA). Here's the core problem with RAG that this new approach solves: Traditional RAG chunks documents, embeds them into vectors, and retrieves based on semantic similarity. But similarity ≠ relevance. When you ask "What were the debt trends in 2023?", a vector search returns chunks that look similar. But the actual answer might be buried in some Appendix, referenced on some page, in a section that shares zero semantic overlap with your query. Traditional RAG would likely never find it. PageIndex (open-source) solves this. Instead of chunking and embedding, PageIndex builds a hierarchical tree structure from your documents, like an intelligent table of contents. Then it uses reasoning to traverse that tree. For instance, the model doesn't ask: "What text looks similar to this query?" Instead, it asks: "Based on this document's structure, where would a human expert look for this answer?" That's a fundamentally different approach with: - No arbitrary chunking that breaks context. - No vector DB infrastructure to maintain. - Traceable retrieval to see exactly why it chose a specific section. - The ability to see in-document references ("see Table 5.3") the way a human would. But here's the deeper issue that it solves. Vector search treats every query as independent. But documents have structure and logic, like sections that reference other sections and context that builds across pages. PageIndex respects that structure instead of flattening it into embeddings. Do note that this approach may not make sense in every use case since traditional vector search is still fast, simple, and works well for many applications. But for professional documents that require domain expertise and multi-step reasoning, this tree-based, reasoning-first approach shines. For instance, PageIndex achieved 98.7% accuracy on FinanceBench, significantly outperforming traditional vector-based RAG systems on complex financial document analysis. Everything is fully open-source, so you can see the full implementation in GitHub and try it yourself. I have shared the GitHub repo in the replies!

4.5K540971.3K
#ai
pinned 09 NOV 24· backfill

related

I took the @karpathy autoresearch loop and pointed it at markets. 25 AI agents debate macro, rates, commodities, sectors, and single stocks daily. Every recommendation scored against real outcomes. Worst agent by rolling Sharpe gets its prompt rewritten by the system. Keep or revert. Same loop, prompts are the weights, Sharpe is the loss function. Trained the agents on 18 months of market data. 378 iterations. 54 prompt modifications, 16 survived. The system learned which agents to trust using Darwinian weights — geopolitical, commodities, and the @BillAckman quality compounder rose to the top. The agents even figured out their own portfolio manager was the weakest link before we did! Deployed the trained agents. +22% in 173 days. Best pick: AVGO at $152, held for +128%. The final prompts are evolutionary products — shaped by market feedback, not human intuition. Now running live with my own capital. https://github.com/chrisworsey55/atlas-gic Part hedge fund, part research experiment :)@Chris_Worsey@askalphaxivTo keep analyzing pdfs, some tools are qpdf, pdfalyzer, strings. If you just want to get text out of PDFs, check out marker - https://github.com/datalab-to/marker . I'm also interested in any weird edge cases anyone else has found - let me know!@VikParuchuriAs a developer, Please Slap yourself if you are unable to explain even 10 from below terms : Indexing Clustering Denormalization Normalization Read replicas Leader–Follower replication Multi-leader replication Quorum Consensus CAP theorem BASE ACID Eventual consistency Strong consistency Snapshot isolation MVCC (Multi-Version Concurrency Control) Two-phase commit (2PC) Three-phase commit (3PC) Write-ahead logging (WAL) Checkpointing Compaction Rebalancing Resharding Data locality Hot partition Split-brain Failover High availability (HA) Horizontal scaling Vertical scaling Load balancing Connection pooling Caching Materialized views Secondary indexes Composite index Covering index Bloom filter LSM tree B-tree Query planner Cost-based optimizer Deadlock Lock escalation Optimistic locking Pessimistic locking Dirty read Phantom read Read skew Write skew Data skew Backpressure Circuit breaker Throttling Rate limiting CDC (Change Data Capture) Logical replication Physical replication Geo-replication Federation Data lake Data warehouse Columnar storage Row-based storage Time-series partitioning Hash partitioning Range partitioning Consistent hashing Data migration Schema evolution Schema registry Idempotency Exactly-once semantics@SumitM_X@prathamgrv@dhruvtwt_Crypto cards are one of the best innovations for off-ramping crypto, I have used these 3 crypto cards in India and in other countries and also online on platforms like swiggy, makemytrip, nykaa etc. How do they work? > Load card with crypto from your wallet > Spend it anywhere just like your normal cards I’ve reviewed: @Cypher_HQ_ @AviciMoney @RedotPay Based on: - Exchange rates - Daily transaction limit - Fees - Pros & cons A thread 🧵@IshitaaPandeyLow-key websites I quietly rely on 1) http://roadmap.sh Gives you a brutally clear learning path for roles like frontend, backend, DevOps, etc No fluff, just “learn this → then this → then this”. 2) http://playcode.io An online playground to quickly test HTML, CSS, JS without setting up anything locally Perfect for quick experiments and debugging ideas 3) http://usehooks.com A collection of reusable React hooks with real use cases Saves time and helps you avoid rewriting the same logic again and again 4) http://devhints.io Concise cheat sheets for languages, frameworks, and tools. Ideal when you forget syntax and don’t want to read a 20-minute blog 5) http://jsoncrack.com Turns messy JSON into a clean visual tree Makes understanding large APIs and configs way easier than staring at raw text 6) http://realtimecolors.com Lets you generate and preview color palettes instantly Useful when you want decent UI colors without guessing or copying blindly 7) http://regex101.com Build, test, and debug regex step by step with explanations Honestly, the fastest way to stop hating regex 8) http://bundlephobia.com Shows how big an npm package really is before you install it Helps you avoid bloating your app with “tiny” libraries 9) http://caniuse.com Tells you which CSS/JS features actually work across browsers Essential before using shiny new features in production 10) http://toolbox.googleapps.com Google’s own diagnostics tools for DNS, email, headers, and network issues Surprisingly useful for debugging real-world problems 👉 Which one of these do you already use and which one did you not know existed?@shekhu04@PrajwalTomar_@coreyhainesco