Abstract:

This paper compares four information retrieval methods: three semantic models (SBERT, E5-small, DistilRoBERTa-PL) and the classical lexical approach (BM25). The study is based on a corpus of Polish consumer law documents. Semantic models were used for embedding-based indexing and retrieval via FAISS, while the lexical approach used Elasticsearch with BM25. A question-answering (QA) system was built in two variants: (1) Semantic QA — retrieving legal text fragments based on meaning using sentence embeddings; (2) Lexical QA — traditional keyword-based retrieval. Both systems were evaluated on a custom set of 200 consumer law questions using standard IR metrics: precision@k, recall@k, Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (NDCG). Results show that semantic methods outperform BM25 in relevance (e.g., SBERT MRR@5 = 0.9194 vs BM25 = 0.5400), especially for queries requiring semantic understanding, with only a slight drop in absolute precision.
Index Terms—semantic search, legal information retrieval, language models, Sentence-BERT, BM25, consumer law, NLP, Polish law, FAISS, Elasticsearch