I Combined Dense and Sparse Vectors to Search Medical Research

A practical Qdrant experiment on when semantic search helps, when exact terminology matters, and why combining both is useful. Code: GitHub Repository When I started this project, I thought the question was simple: if someone searches biomedical research with a full sentence, surely semantic search should be enough. Then I looked at the kinds of queries doctors and researchers actually write. A query can contain a plain-language request, a disease name, an exact gene symbol, a mutation such as BRAF V600E, and a date restriction at the same time. A search system has to understand the meaning of the sentence without losing the exact notation hidden inside it. That is the experiment I built. I compared three ways to search biomedical papers: Dense search, which turns a query and a paper into vectors so it can retrieve similar meaning. Sparse BM25 search, which rewards useful word overlap and preserves exact terms. Hybrid search, which runs both searches and combines their ranked results. This is a retrieval benchmark, not a diagnosis tool and not a chatbot. The input is a search query. The output is a ranked list of PubMed papers. I deliberately stopped there so I could measure whether the system found useful evidence before a language model or a generated answer could hide a retrieval mistake. Why PubMed makes this harder than ordinary document search PubMed is the US National Library of Medicine’s public catalogue of biomedical research citations and abstracts. It is where developers building literature tools, researchers, and clinicians often start when they need to locate published evidence. The difficulty is that biomedical language mixes several kinds of information: A disease, such as melanoma, which is a type of skin cancer. A gene, such as BRAF or EGFR. Genes are named pieces of DNA that can affect how a cancer grows and responds to treatment. A mutation, such as BRAF V600E, which is a precise change in a gene. In this example, the protein has a different building block at position 600. A clinical-trial ID, often starting with NCT. It is an identifier for a registered study, not a free-form phrase. If someone searches for “treatment evidence for glioma with BRAF,” a useful system should understand the sentence. If they search for KIT L576P, it should not dilute that precise mutation into a generic search for cancer treatment. That tension is why I did not want to test one retrieval method in isolation. The three retrieval methods in plain English A retriever is simply the part of a search system that accepts a query and returns ranked documents. This project has two different retrievers. Dense search: good at meaning Dense search converts text into a list of numbers called a vector or embedding. The idea is that texts with related meaning should end up close together in this numeric space. I used sentence-transformers/all-MiniLM-L6-v2, a lightweight general-purpose text model, to produce the dense vectors. To compare two dense vectors, the system uses cosine similarity. Despite the name, it is just a way to compare the direction of two vectors. A higher score means the model considers the texts more related in meaning. This helps when wording changes. For example, a query that spells out a mutation in descriptive language can still connect to a paper that uses a shorter medical term. Dense search has a weakness: a short identifier can contribute very little to the vector. KIT L576P is only a few characters, but it carries a lot of meaning for the person searching. Sparse BM25 search: good at exact wording Sparse search keeps track of terms rather than representing the whole sentence as one dense vector. I used BM25, a long-established ranking method used in text search. It gives a document more credit when it contains important query terms, while avoiding over-rewarding common words. BM25 is useful here because it keeps exact notation visible. A paper containing BRAF V600E or NCT01234567 is easier to surface when the query uses the same notation. Its weakness is the opposite of dense search. It does not automatically know that “malignant melanocytic tumor” and “melanoma” refer to the same disease. Hybrid search: let both methods vote The hybrid method runs dense and sparse search separately, then combines their lists with reciprocal rank fusion, or RRF. RRF is a simple rule: a paper gets more credit when it appears near the top of either list, and even more credit when both methods rank it highly. This matters because dense and BM25 scores are different kinds of numbers. It would be misleading to add a cosine score directly to a BM25 score. RRF avoids that problem by combining positions in the ranked lists rather than raw scores. Why I chose Qdrant for the experiment I wanted the experiment to stay focused on retrieval instead of building vectorisation, sparse indexing, fusion, and filtering infrastructure from scratch. That is where Qdrant fits well. Qdrant is a vector database. In this project, each paper is stored once as a Qdrant point with two named representations: a dense vector for semantic search and a sparse BM25 vector for exact-term search. The same Python client can query either representation or ask Qdrant to fuse the two ranked lists with RRF. I also needed filters to run during the search. For example, a developer may want papers about a drug or gene, but only after 2020. Qdrant stores these fields as a payload, which is searchable metadata attached to the paper. That lets a query say “find papers similar to this request, but only where the year is 2020 or later and the gene is EGFR.” EGFR is a gene involved in cell growth and is often used as a treatment-relevant marker in cancer research. In short, Qdrant let me keep one corpus, two retrieval methods, metadata filters, and fusion in one small Python pipeline. For a comparison experiment, that made the moving parts easier to reason about and reproduce. The first implementation choice is where Qdrant runs. I kept that decision behind one function, so the same retrieval code can use local storage for reproduction or a Qdrant server when QDRANT_URL is configured. def get_client() -> QdrantClient: url = os.getenv("QDRANT_URL") if url: return QdrantClient( url=url, api_key=os.getenv("QDRANT_API_KEY"), local_inference_batch_size=16, ) QDRANT_PATH.parent.mkdir(parents=True, exist_ok=True) return QdrantClient( path=str(QDRANT_PATH), local_inference_batch_size=16, ) A reader can therefore run the experiment without provisioning a service. Moving the same collection to a server changes the client configuration, not the indexing or retrieval logic. client.create_collection( collection_name="trec_pm_2018_pool", vectors_config={ "dense": models.VectorParams(size=dense_size, distance=models.Distance.COSINE) }, sparse_vectors_config={ "sparse": models.SparseVectorParams(modifier=models.Modifier.IDF) }, ) The dense vector uses cosine similarity. The sparse vector uses BM25-style term weighting with inverse document frequency, which gives rarer terms more influence than common ones. Figure 1. Corpus preparation, two named vector representations, metadata payloads, and the shared Qdrant Query API The dataset: real relevance labels, controlled query wording For the benchmark, I used the TREC 2018 Precision Medicine scientific-abstract task . TREC is a long-running information-retrieval evaluation programme. In this task, precision oncologists created 50 synthetic patient cases, and physicians trained in medical informatics assessed which research records were relevant to each case. The source files are public: TREC 2018 topics : the 50 source patient cases. TREC 2018 abstract relevance judgments : the labels saying which documents were relevant to which case. PubMed : the public source for the title and abstract records fetched by the project. The downloaded corpus is not committed to Git. The repository downloads the numeric PubMed IDs found in the TREC judgment file through the NCBI API and builds the corpus locally. That makes the data path clear without redistributing a large copy of PubMed. The original judgments contain 14,946 numeric PubMed IDs. When I fetched the current title and abstract records, 12,868 usable records were available. The original TREC task also includes conference abstracts from AACR and ASCO. Those are not PubMed records, so I excluded them rather than mixing data sources. I then created seven deterministic versions of each source query. For example, the same case can be expressed as a natural-language question, an exact medical phrase, a query with a synonym, or a query containing an exact mutation. These are controlled rewrites, not new physician-written cases. They reuse the original case’s relevance labels so I can ask one narrow question: what changes when the wording changes but the information needed stays the same? The final evaluation used 210 held-out queries from 30 source cases. All seven rewrites from one source case stayed in the same split, so a near-duplicate version of a test case could not appear in training or validation. The preparation code mirrors the data flow described above. It downloads the official TREC files, parses the topics and relevance judgments, creates the controlled query forms, and extracts only numeric PubMed IDs for the NCBI fetch. download_trec_pm_sources() topics = parse_topics() source_qrels = parse_source_qrels() queries = make_all_controlled_queries(topics) pubmed_ids = ordered_pubmed_ids(source_qrels) records, _ = _fetch_batches( pubmed_ids, batch_size=400, request_delay=0.4, ) The downloader caches every XML batch before parsing it, so an interrupted run can reuse completed downloads. The generated manifest records the source checksums, corpus count, and code version used for the benchmark. Building the index Each paper becomes one Qdrant point. Its title and abstract are joined into the searchable text. The payload keeps readable fields such as the publication year, diseases, genes, drugs, and trial IDs. I used Qdrant Client’s FastEmbed integration rather than writing embedding code by hand. Passing a models.Document object tells the client which model should encode the text for each named vector. point = models.PointStruct( id=paper_id, vector={ "dense": models.Document(text=search_text, model=DENSE_MODEL), "sparse": models.Document(text=search_text, model=SPARSE_MODEL), }, payload=paper_metadata, ) The benchmark does not send all 12,868 papers in one request. It groups the prepared points into batches of 32 and waits for each upsert to finish. points = [make_point(record) for record in records] for batch in batches(points, size=32): client.upsert( collection_name=collection_name, points=batch, wait=True, ) Batching keeps each embedding and upload step bounded, while the completed-document counter makes a long indexing run easier to monitor. That is the whole indexing idea: the same paper gets two ways to be found, plus metadata for filters. Before fusing the rankings, the benchmark runs dense and sparse retrieval separately. Both branches use the same collection, result limit, payload handling, and optional filter. Only the model and named vector change. if mode == "dense": response = client.query_points( collection_name=collection_name, query=models.Document(text=query_text, model=DENSE_MODEL), using="dense", query_filter=query_filter, limit=limit, with_payload=True, ) elif mode == "sparse": response = client.query_points( collection_name=collection_name, query=models.Document(text=query_text, model=SPARSE_MODEL), using="sparse", query_filter=query_filter, limit=limit, with_payload=True, ) Keeping the surrounding query path identical matters for the benchmark: the comparison changes the retrieval representation rather than changing unrelated application behavior. For a hybrid query, I ask for more candidates from each retriever than I plan to show the user. For example, if the final UI needs five papers, pulling 20 candidates from dense search and 20 from BM25 gives RRF a wider set to combine. A paper ranked sixth in both lists can still become useful after fusion. Asking each retriever for only the final five would throw that paper away too early. response = client.query_points( collection_name=collection_name, prefetch=[ models.Prefetch( query=models.Document(text=query_text, model=DENSE_MODEL), using="dense", limit=20, ), models.Prefetch( query=models.Document(text=query_text, model=SPARSE_MODEL), using="sparse", limit=20, ), ], query=models.FusionQuery(fusion=models.Fusion.RRF), limit=5, with_payload=True, ) What I measured The main metric is Recall@20. It asks: of all the papers judged relevant for a query, how many did the system place in its first 20 results? I used 20 because a retrieval layer often hands a small evidence set to the next stage, such as a reranker or an answer-writing model. If a relevant paper is missing from those 20 results, later steps cannot use it. The implementation calculates Recall@20 per query from two sets: every document judged relevant for that query and the document IDs returned in the first 20 positions. def query_recall(ranked, judgments, depth=20): relevant = { document_id for document_id, score in judgments.items() if score > 0 } retrieved = { document_id for document_id, _ in ranked[:depth] } if not relevant: return 0.0 return len(relevant & retrieved) / len(relevant) Those per-query values are averaged for the reported Recall@20. The separate complete-miss count records the harsher case where the intersection is empty. I also counted complete misses: queries where the top 20 had no judged relevant paper. This is simpler to interpret than a single average. If a system gets a high average but completely fails for many real queries, that matters. The benchmark also records additional ranking metrics and latency measurements in the repository. A warm query latency is the time after the model and index have already been loaded into memory. It is useful for comparing repeated local queries, but it is not a production latency claim. Figure 2. Dense and sparse candidates remain independent until reciprocal rank fusion combines their ranks Results: hybrid helped most when a query mixed meaning and notation Here is the held-out Recall@20 result. Higher is better. The headline is not “hybrid wins everything.” It does not. Dense search was strongest when the query was phrased naturally or used synonyms. BM25 was particularly valuable when exact medical terms carried the intent. Hybrid did best overall and was strongest on exact terminology, mutation-like entities, and mixed queries. The practical result was the miss count. Dense had 13 held-out queries with no relevant paper in the first 20. Sparse had 14. Hybrid had 5. One example makes the trade-off concrete. When the system received a descriptive version of a melanoma question that expanded KIT L576P into words, dense search found a relevant paper at rank one. BM25 did not find relevant evidence in its first 20 because the exact notation had disappeared from the query. The opposite happened with the exact mutation form. BM25 placed a highly relevant KIT (L576P) paper first, while dense search returned no relevant paper in its first 20. Neither method was universally better. They failed in different ways. For “What treatment evidence is available for glioma with BRAF?”, a dense and sparse search each found four relevant papers in the top 20, but not the same four. Hybrid found six. That is the useful case for fusion: it expands the evidence set when the two methods bring back different relevant material. Figure 3. Held-out Recall@20 by controlled query category. The winning modality changes with query form Filtering is a separate, useful capability Search meaning and search constraints are different jobs. If a user wants papers about an EGFR-related treatment after 2020, “after 2020” is not something an embedding model should have to guess from the sentence. It is a hard constraint on a metadata field. The project stores structured fields in Qdrant’s payload and applies them at query time. query_filter = models.Filter( must=[ models.FieldCondition( key="genes", match=models.MatchValue(value="EGFR") ), models.FieldCondition( key="publication_year", range=models.Range(gte=2020) ), ] ) Creating the filter is only half of the operation. The same object is passed into the hybrid search, so Qdrant applies the hard constraints while it retrieves and fuses candidates. points = search( client, "treatment evidence for EGFR resistance", mode="hybrid", limit=20, query_filter=query_filter, collection_name="trec_pm_2018_pool", ) The returned points therefore satisfy both payload conditions before they reach any later reranking or answer-generation stage. must mean both conditions must match. The code checks filters one at a time, together, with an impossible value that should return nothing, and with exact-case matching. The point is not that filters are medically intelligent. The point is that once you have reliable metadata, you can keep hard constraints out of fuzzy semantic matching. What this experiment does not prove This is a fair comparison of three retrieval methods on a reconstructed, judged PubMed pool. It is not a full PubMed index, a clinical recommendation system, or a proof that MiniLM is the best model for oncology literature. There are three important limits: The current PubMed records were fetched after the original 2018 task. Some records may be missing or updated compared with the original collection. The seven query forms are controlled rewrites. The original cases and labels were reviewed by experts, but the rewrites were not independently reviewed medical questions. The dense model is a lightweight baseline. A biomedical model, a reranker, or a larger corpus could change the result. Those limits are why I would use this project as a starting point for a literature-retrieval pipeline, not as an end product for clinical decisions. What I would build next The next useful experiment is not to add an LLM immediately. First, I would keep this benchmark fixed and swap one retrieval component at a time: a biomedical embedding model, a reranker that reorders the retrieved papers, or a larger distractor corpus to test scale. Only after measuring retrieval and reranking separately would I put an answer-generation layer on top. That distinction matters: a fluent answer is not evidence that the search system found the right papers. My main takeaway is simple. In biomedical search, users can ask for meaning, exact notation, and hard filters in the same sentence. Dense search and BM25 each cover a different failure mode. Qdrant made it practical to test both representations against the same corpus and combine them without building the search infrastructure from scratch. References Project source code and reproduction instructions TREC 2018 Precision Medicine Track data TREC 2018 Precision Medicine task description PubMed Qdrant Hybrid and Multi-Stage Queries Qdrant FastEmbed Qdrant Filtering I Combined Dense and Sparse Vectors to Search Medical Research was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.