Our posture on crawling
NOSIBLE crawls and indexes publicly available written content and cross-references it so customers can find what was public at any point in time. We do not crawl behind paywalls or logins. We follow ethical crawling principles. This puts NOSIBLE on the public-access side of hiQ v. LinkedIn, Van Buren, and related U.S. authority.
- NOSIBLE crawls public written content. No paywalls, logins, or bypass.
- The index has 2.5B+ pages from 300,000+ sources in 150+ countries.
- Coverage spans 95 languages across news, company, government, and specialist sources.
- Every crawled domain must trace to a legal entity and pass AI vetting.
- Website checks cover redirects, blocklists, Web Risk, RDAP, robots.txt, and sitemaps.
- NOSIBLE honors robots.txt and avoids the conduct in hiQ, Power Ventures, and 3Taps.
- Fresh URLs get priority with reasonable crawl rates, ETags, and Last-Modified dates.
- Proxy networks must source IPs legally and ethically at scale, with clear sourcing.
- Published dates are verified before entering research use or model workflows.
NOSIBLE crawls public written sources for search and retrieval
NOSIBLE is focused on long-form written content from news websites, press releases, company websites, specialist publications, government pages, public records, official filings, court documents, and other vetted public sources. The product is built for discovery, retrieval, source ranking, and point-in-time verification. It is not a cached-page reader, publisher archive, marketplace mirror, social archive, or paywall bypass.
U.S. authority draws a strong distinction between public, logged-out access and attempts to enter gated systems, evade blocks, or continue after individualized revocation (Van Buren; hiQ; Power Ventures; Craigslist v. 3Taps). NOSIBLE stays in the public-page category: responsible crawling, no access-control bypass, and links back to the original source. Public, logged-out scraping also has support on contract-law grounds. Meta v. Bright Data rejected Meta's contract claims against logged-out public scraping, while Nguyen v. Barnes & Noble shows why browsewrap terms without affirmative assent are weaker than click-to-accept terms.
Crawl scope is defined before a page enters the index
| Area | NOSIBLE position |
|---|---|
| Allowed source types | News, press releases, company pages, specialist sources, government pages, public records, official filings, and court documents with useful written evidence. |
| Excluded source types | Paywalls, logged-in pages, social platforms, marketplaces, unsafe domains screened with Google Safe Browsing, inherently pornographic sites, and pages containing personally identifiable information. |
| Domain review | Every crawled domain must be traceable to a legal entity and vetted by AI before it enters the source universe or production index. |
| Website verification | Before approval, NOSIBLE records URL preflight results, homepage response metadata, redirect status, blocklist status, Google index presence, Google Web Risk status, RDAP registration signals, robots.txt, ads.txt, security.txt, legal-document paths, content sitemaps, and archive provenance. |
| URL priority | Fresh URLs are prioritized over historical URLs, with reasonable crawl rates maintained across source websites. |
| Crawl impact controls | Robots.txt, no access-control bypass, no block evasion, ETag and Last-Modified checks, cache-aware requests, and responsible crawl rates. |
| Point-in-time checks | Published dates are checked against metadata, modified dates, temporal falsification tests, corroborating sources, systems of record, archive evidence, and sensibility checks. |
Published dates are verified before they enter research workflows
A published date is not accepted as ground truth. NOSIBLE treats it as a claim that must be supported or rejected before a backtest or AI agent can use the source. Customers can use stricter evidence thresholds when confidence matters more than volume. Public-source personal data still carries privacy obligations under ICO web-scraping guidance and UK/EU GDPR (ICO; GDPR Article 6).
Common crawling questions
Does NOSIBLE crawl behind paywalls?
No. NOSIBLE does not crawl behind paywalls, logged-in areas, subscription gates, or other non-public access controls. U.S. authority treats public, logged-out web access differently from entering gated systems or bypassing access controls (Van Buren; hiQ). NOSIBLE operates on the public side of that line.
Does NOSIBLE crawl social media or marketplaces?
No. NOSIBLE does not currently index LinkedIn, Facebook, Instagram, X, Reddit, Amazon, eBay, Airbnb, or similar platforms. Those sources create a different product, risk, and permission profile. CNIL also treats social platforms, sensitivity-heavy sources, and anti-scraping signals as important factors in a lawful web-scraping assessment (CNIL).
Does NOSIBLE snapshot web pages?
No. NOSIBLE focuses on crawling evergreen content pages for search and retrieval, not visual snapshots or cached-page browsing. The product stores technical crawl artifacts so relevant passages can be found later, but it is not designed to let users browse historical page copies.
What source types does NOSIBLE crawl?
NOSIBLE crawls news stories, press releases, company websites, specialist publications, government websites, public records, official filings, court documents, niche financial publications, non-financial specialist sites, blogs, and certain high-quality forums. The common thread is that the content is public, written, source-like, and useful for search or verification.
What source types does NOSIBLE exclude before indexing?
NOSIBLE excludes insecure websites that fail Google Safe Browsing checks, social platforms, inherently pornographic websites, marketplace and listing websites, paywalled content, and pages containing personally identifiable information. The crawler stays focused on pages that improve search and avoids categories that add risk without improving the core evidence layer.
How large is the crawl and index?
NOSIBLE's index contains more than 2.5 billion pages from over 300,000 sources in more than 150 countries and 95 languages. Point-in-time verification, domain authority, source comparison, and cross-source corroboration all improve when the search engine has more independent anchors.
How much first-hand history does NOSIBLE have?
NOSIBLE has first-hand point-in-time web data going back to 2023. Earlier history is derived through evidence checks, corroborating sources, systems of record, and archive-based verification. First-hand crawls and reconstructed historical evidence carry different confidence levels, so customers can apply stricter thresholds where the workflow requires them.
How often does NOSIBLE check whitelisted sources?
NOSIBLE checks vetted sources frequently so new public pages can enter future crawl batches quickly. The crawler prioritizes freshness while maintaining reasonable crawl rates. Domains must be traceable to a legal entity and vetted by AI before they enter the source universe.
How does NOSIBLE verify a website before indexing?
NOSIBLE starts with a URL preflight check, then records the normalized scheme, host, domain, suffix, geography, homepage response, redirect behavior, status code, HTTP version, elapsed time, title, and language. The domain is checked against internal blocklists, Google Web Risk, Google index presence, RDAP registration records, robots.txt, content sitemaps, ads.txt, security.txt, privacy policy links, terms links, and archive provenance before it becomes a production source.
How does NOSIBLE identify the legal entity behind a website?
NOSIBLE looks for ownership clues in the homepage footer, copyright strings, privacy policies, and terms of service. It extracts the legal entity name and jurisdiction where the evidence is available. RDAP registration data, ads.txt owner-domain fields, and public legal documents give additional signals about who operates the site.
How does NOSIBLE use archive provenance for website verification?
NOSIBLE checks domain history through web archive evidence and Common Crawl-derived domain history where available. Those records help estimate when a domain first appeared, when it was last seen, and whether the site has a history consistent with the source being evaluated. Provenance is not a substitute for legal review, but it is useful evidence before a site enters the source universe.
How does NOSIBLE decide which URLs to crawl first?
Every domain NOSIBLE crawls must be traceable to a legal entity and vetted by AI. Fresh URLs are prioritized over historical URLs, and NOSIBLE maintains reasonable crawl rates across source websites. The crawler also looks for ETags and last modified dates so repeat checks can use cache-aware requests where the server supports them.
How does NOSIBLE reduce the burden on source websites?
NOSIBLE uses several crawl-impact controls. It looks for ETags and Last-Modified values, then uses conditional requests where the source server supports them. It also maintains reasonable crawl rates. Trespass theories require actual server damage or impairment, not mere dislike of crawling (Intel v. Hamidi); eBay v. Bidder's Edge remains a high-volume, fact-bound warning case.
Does NOSIBLE use proxies or country-aware routing?
Yes. NOSIBLE owns thousands of dedicated IP addresses and tries to access pages from the country where the website operates before using more advanced proxies. NOSIBLE depends on Bright Data for most crawling efforts and is evaluating fallback providers. Proxies are for reliable public-page access and routing, not for evading individualized blocks, access controls, or revocation notices (3Taps).
Does NOSIBLE use web archives?
Yes, but not as a substitute index. NOSIBLE uses web archives to discover URLs for recently whitelisted domains and adds those URLs to the backlog. It does not currently ingest HTML directly from web archives into the main index. Archive evidence is also part of the point-in-time roadmap where available, especially for provenance checks after 2007.
Can customers influence which sites NOSIBLE indexes?
Yes. Customers can suggest sites for NOSIBLE to add, prioritize, or deprioritize when those sources matter to a research or risk workflow. Suggestions do not change crawl scope. A requested source still needs to be public, written, reachable without access-control bypass, and consistent with the exclusion policy before it enters the index.
Does NOSIBLE wrap Bing, Brave, or another search engine?
No. NOSIBLE does not wrap Bing, Brave, or any other search API. The index is independent and built from NOSIBLE's own crawling, parsing, routing, and retrieval systems. Customers get NOSIBLE source coverage, timing controls, and retrieval logic rather than a repackaged third-party search feed.
How does NOSIBLE verify point-in-time dates?
NOSIBLE treats a published date as a claim, not a fact. The process checks HTML metadata, structured JSON, modified timestamps, temporal falsification tests, corroborating sources, authoritative systems of record such as EDGAR, archive evidence where available, and basic sensibility checks. These checks reduce future information leaking into historical research, backtests, and agent workflows.
What does temporal falsification mean?
Temporal falsification means checking whether the page contains information that should not exist at the claimed date. Examples include references to ChatGPT before 2022, COVID-19 before 2020, Oumuamua before 2017, Ozempic before 2008, or YouTube before 2005. If a page makes those kinds of impossible references, the claimed date needs to be rejected or treated with much lower confidence.
Can customers choose stricter point-in-time evidence thresholds?
Yes. Different firms may require different amounts of evidence before accepting that a page was available at a claimed point in time. NOSIBLE can dial the evidence threshold up or down. A stricter threshold improves confidence and reduces look-ahead risk, but it also reduces volume because fewer URLs will clear the bar.
Why does NOSIBLE store technical crawl artifacts?
NOSIBLE retains a slimmed-down, minified, compressed version of HTML for technical purposes. It also stores structured JSON that converts raw HTML into human-readable text and chunks. Those chunks enter the search index so NOSIBLE can retrieve relevant passages in response to customer queries. Users get retrieval infrastructure, not a full-page reading product.
What website-level metadata does NOSIBLE maintain?
NOSIBLE maintains domain-level metadata including host, title, description, language, geography, industry, topic, brand-safety status, indexed totals, first published date, last published date, sitemap status, legal-document paths, RDAP signals, archive provenance, shard routing, and content trend. The system uses this metadata to understand source quality before a query is run.
How does NOSIBLE use domain authority?
NOSIBLE uses website metadata to compute query-specific domain-authority scores. Those scores help rank sources when many pages match the same topic. Authority scoring can improve signal quality in workflows such as sentiment analysis, risk monitoring, and source comparison.