Weaviate
Weaviate is an AI-native vector database with built-in ML models for hybrid search, RAG, and semantic retrieval over scraped content.
Weaviate is an AI-native, open-source vector database with built-in machine learning model support.
Key Features:
- Built-In Vectorization - Automatically generate embeddings using integrated ML models at import time.
- Hybrid Search - Combine vector similarity with keyword search for optimal retrieval accuracy.
- GraphQL API - Query your data with a flexible, developer-friendly GraphQL interface.
- Multi-Tenancy - Isolate data for different projects or clients within a single cluster.
- Modular Architecture - Plug in custom vectorizers, rerankers, and generative modules as needed.
Whether you're building semantic search over scraped documents, powering RAG applications, or creating intelligent content discovery, Weaviate provides the AI-native foundation your search needs.
Web scraping workflow
How to use Weaviate in a web scraping pipeline
Weaviate stores objects in collections with properties and vectors derived from machine-learning models. It can support semantic retrieval over collected pages while retaining structured context about each source.
A practical workflow
- A separate crawler extracts clean passages and assigns stable page and chunk identities.
- Configure the collection and vectorization approach, then load passages with source URLs, capture times and other useful properties.
- A retrieval application searches the stored objects and presents matching passages with their provenance rather than an unexplained generated answer.
When it fits
Use this when search over a collected corpus needs semantic similarity together with structured object properties. Keep the document and embedding versions explicit so a recrawl or model change can trigger the appropriate rebuild.
Boundaries
Weaviate is not a SERP extraction service and does not imply built-in acquisition of arbitrary websites. Your pipeline still handles fetching, cleaning, chunking and obsolete-object removal. Vectorizer configuration and deployment scope matter; an available integration is not proof that the resulting retrieval is accurate for your users.
Growing a website? Track its keywords, Google rankings, Search Console insights, and share of voice with bisibility. Free during beta.
Our productRepository
weaviate/weaviate
Stars
16.9KForks
1.4KLast commit
2026-10-15License
BSD-3-Clause (outside wl/)Language
Go
Metrics recorded 2026-10-16
Repository details
License source: Repository license file
Frequently asked questions
What is Weaviate used for?
Weaviate stores objects and vectors for vector, keyword and hybrid retrieval. Applications use it for semantic search, retrieval-augmented generation and other similarity-search tasks. Model integrations can generate embeddings, depending on the configuration you choose.
Sources (checked ): Weaviate platform
Is Weaviate an open-source database?
Weaviate’s core code is available under BSD-3-Clause. Its current repository also contains separately licensed enterprise code in the wl directory. Check the license and edition of the feature you need rather than applying the core license to every component.
Sources (checked ): Weaviate core/enterprise license at verified commit 4d69259 (2026-10-01)
Is Weaviate free to use?
You can self-host the open-source core under its license, with your own infrastructure and operating costs. Weaviate Cloud provides a limited Free plan and paid options. Enterprise features and model services can have separate charges.
Sources (checked ): Weaviate core/enterprise license at verified commit 4d69259 (2026-10-01); Weaviate pricing
Which is better, Turbopuffer or Weaviate?
Both provide vector retrieval, but their storage architecture, APIs and deployment choices differ. Turbopuffer documents an object-storage-backed service; Weaviate provides its own database platform and deployment options. Compare filtering, retrieval features, latency requirements and costs on your dataset.
Sources (checked ): Weaviate platform; Turbopuffer documentation
Sources
Verified against the weaviate/weaviate repository and the PyPI package.