Searching for relevant literature is a fundamental part of research. Traditional lexical or keyword search, which relies on keyword matching, has been the go-to method for years. However, as AI technology progresses, semantic search has emerged as a powerful way to augment your search, understanding context and meaning rather than just keywords.
This post will guide you through the advantages of semantic search and how to implement it alongside lexical search to optimize your research process.
Lexical or Keyword Search: The Traditional Approach
Lexical or Keyword search involves querying databases like PubMed or Google Scholar using specific keywords. While this method is straightforward and familiar to many researchers, it has limitations:
- Keyword Dependency: Searches depend on the exact words used, which may miss relevant papers that use different terminology.
- Surface-Level Matches: Results are based on keyword frequency rather than contextual relevance.
Semantic Search: The AI-Driven Approach
Semantic search, on the other hand, leverages AI to understand the context and meaning behind your queries. It provides a deeper and more accurate set of results by interpreting the relationships between concepts rather than relying solely on keyword matches.
Comparing Lexical and Semantic Search
| Aspect | Lexical Search | Semantic Search |
|---|---|---|
| Basis of Search | Keywords | Context and meaning |
| Search Results | Matches exact words used in the query | Understands the relationship between concepts |
| Relevance | Surface-level matches based on keyword frequency | Contextual relevance, providing deeper insights |
| Flexibility | Limited to the exact terms used | More flexible, adapting to the context of queries |
My Pipeline for Semantic Search
There are multiple tools and apps in this space and it is easy to get overwhelmed. So I will share my current pipeline with you, which would include mostly apps that have a decent functionality under the free tier.
I use the following 3-step framework:
1. Exploring Research Questions using ChatGPT
I use ChatGPT to first explore the most important questions in my field of interest.
You can ask ChatGPT to provide 10 novel and significant research questions in your research field.
While this is a good starting point, you will need a bit of prompting to get exactly what you want.
NOTE: If you want to cut down this process (like me), you can access the custom chatbot we built by clicking on the link HERE along with pdf instructions. This works with the free version of ChaGPT but you will need a free account (if you don’t have one already- highly recommended!). Once you get a research topic that resonates with you, you can drill down into the subtopics as much as you want.
When I entered {psoriatic arthritis} as my research topic on the chatbot, I got the below output:
Having research experience in this field, I can vouch that all of these can be great research questions to delve into.
2. Finding Relevant Literature with Consensus and SciSpace
The next step is to use one of the other AI-powered apps, such as SciSpace or Consensus (Scite is another alternative) to find 5 to 10 key papers in your field. This can provide a solid starting point.
Below you will see an example with me using the Consensus app.
You can first copy and paste your thoroughly crafted question (from our custom GPT chatbot) into the box as shown below:
You can then export the list as a CSV or RIS file by clicking on the export button. Next, import this list into Zotero or any other reference management software of your choice.
3. Searching Related Literature on ResearchRabbit or LitMaps
If you need to find related papers or conduct a more thorough literature review (to ensure you haven’t missed any important or landmark papers), you can use an app such as ResearchRabbit (free until the publication of this post).
To use ResearchRabbit, import your collection from Zotero, then click on the “Similar Work” tab.
You will get a network of papers to explore as shown below:
If you want to only look at more recent papers, published after those on your list, you can click on the “Later Work” tab instead.
Another popular app is LitMaps, although it requires a paid subscription.
And there you have it: faster than you can say, “What would I even do research about?”, you now have a compelling research question and related literature on your question. Welcome to the possibilities that await you!
In conclusion, integrating semantic search into your research process can significantly enhance your ability to identify relevant literature and generate novel research ideas. By understanding and using both lexical and semantic search strategies, you can ensure a comprehensive and effective approach to the literature review.
This Week’s Action Step:
- Set Up: Create a Zotero folder for your literature review.
- Explore: Try out SciSpace and Consensus for your next research query.
- Integrate: Use ResearchRabbit to map out related research areas and trends.
That’s it for today. Embrace these new AI tools and take your research to the next level. See you next week with more insights!