Project Description

The Search Engine project will provide students with hands-on experience in building a functional search engine using modern technologies. The project focuses on using MongoDB for managing the reverse index, PostgreSQL for storing the crawled documents, and Node.js for implementing the various search engine components. The project will cover the entire pipeline from crawling and indexing to querying and retrieving search results.

Technologies

MongoDB

• Used for the reverse index, which maps terms to the documents they appear in.
• Suitable for handling the large-scale, unstructured data often found in search engines.

PostgreSQL

• Used for storing the actual crawled documents, metadata, and other structured data.
• Provides strong consistency and relational capabilities, complementing MongoDB’s flexibility.

Node.js

• The primary programming environment for developing the search engine components.
• Used to create scalable, asynchronous components that manage crawling, indexing, and querying.

Components

Crawler

• A Node.js component responsible for traversing the web and retrieving content from various URLs.
• The crawled documents are stored in PostgreSQL for persistence and easy retrieval.

Indexer

• Processes the crawled documents and builds a reverse index in MongoDB.
• The indexer extracts terms from documents and maps them to document IDs, storing this mapping in MongoDB for fast lookups.

Query Processor

• A Node.js component that handles search queries from users.
• It retrieves relevant documents using the reverse index in MongoDB and fetches the complete documents from PostgreSQL as needed.

Ranking Module

• Applies algorithms to rank search results based on relevance.
• This module interacts with both MongoDB and PostgreSQL to gather necessary data for ranking.

Search API

• Exposes the search engine functionality via RESTful or GraphQL APIs.
• Allows external applications or interfaces to query the search engine.

Learning Outcomes

• Understanding how to design and implement a search engine architecture.
• Gaining experience in using MongoDB and PostgreSQL together for a hybrid data storage solution.
• Developing skills in Node.js for creating scalable, asynchronous components.
• Learning about web crawling, indexing, and search algorithms.
• Practical knowledge of building RESTful or GraphQL APIs.
• This project will be a comprehensive introduction to building real-world search engines, providing a blend of theoretical knowledge and practical skills.