A Firehose of the Web, Already Enriched with Entities, Tickers, Sentiments, Ontologies, and Locations
Today we're sharing that Firehose, our lowest latency product, is officially in preview. Firehose is a core part of our plan to build complementary products along the latency spectrum with different levels of intelligence.
Firehose is a live feed of the web documents we crawl and enrich. We split every document into snippets, the passages of its text, and enrich every snippet on its own, so the enrichment points at the exact passage and not only at the page.
Each snippet carries:
- Named entities: the people, organisations, places and products it names, each with a type and confidence, so you can see who or what it is about.
- Locations: the places the passage is about, geocoded to a continent, region or country, so you can filter by where something happened.
- Tickers: the listed companies it matches, with identifiers such as ISIN, LEI and FIGI, plus sector and exchange, so you can route it to a position.
- Ontologies: the topics it covers, scored by probability across 15 ontology families, such as GICS sectors, emotions and event classes.
- Sentiment: real-time classification of the passage as negative, neutral or positive using model v1.2, with the probability of each.
- Forward-looking: real-time classification of whether it describes something yet to happen (forward) or not (not-forward), by model v1.2, with the probability of each.
- Embedding: the passage's own 768-dimension vector, so you can run semantic search or clustering on a passage instead of a whole page.
The page title and description are enriched the same way, and the document carries the metadata, the full text and a roll-up of everything its snippets say.
Documents are delivered over a real-time websocket, and consumers can submit filters that run server side. So you can listen to everything or just the documents that match certain metadata, topics, tickers, geolocations, named entities and ontologies. For example, stream just the documents tagged to Nvidia or Jensen Huang.
Let's do that. Our story begins 8 November, when Nvidia's chief executive, Jensen Huang, admitted to an affair. First we'll connect to the websocket and submit a server side filter. Then we'll see how documents tagged with Nvidia or Jensen Huang changed this week, and how their sentiment compares with the share price.
Why You Actually Want a Web-Scale Firehose
Your news feed publishes a story after it happens, and that is when you find out. If it breaks in a local foreign-language publication, the gap can be anywhere from a day to a week, and anyone who saw it elsewhere can already act on it.
The truth is that you don't actually want a news feed. What you actually want is a feed of the worldwide web. Here's the simple reason why:
Say 10,000 websites cover a story. One publishes first and the other 9,999 publish after it. The chance that your news feed is the first is about 1 in 10,000, so almost every time the story is already out before your feed has picked it up.
Firehose watches hundreds of thousands of websites, including local publications in 95 languages. The first one to publish is far more likely to be one of them, so you learn about the story much sooner, wherever in the world it appears.
Where Does NOSIBLE Firehose Sit in the Stack?
The NOSIBLE stack has four levels: Firehose, Search, World and Atlas. Firehose is the fastest. Each level after it is slower and more intelligent than the one before.
Traders know the same trade-off from market data. Packets carry raw messages, ticks record single trades and quote changes as they happen, intraday bars summarise a period such as an hour, and end-of-day data summarises a whole session. Each of these is slower than the one before and tells you more.
Funds trade at different frequencies, so they use different levels of market data. A daily allocator gains nothing from ticks. A fund that reacts within seconds cannot use end-of-day bars, because they say nothing about the last second. The most intelligent level cannot also be the fastest, so each suits a different decision.
Higher up the chart, more is folded into each item, so it is more intelligent. The cost is time: an item is ready after its inputs arrive.
Search, World and Atlas all answer when you ask. Firehose is the one level that pushes: a document is sent to your client when it is ready, so nothing has to poll on a timer and wait up to the length of the timer for an answer.
Firehose also does all the work before delivery that needs only one document, and it does it on every snippet: embeddings, ontologies, locations, real-time sentiment and real-time forward-looking classification and, for English documents, entities and tickers. Every subscriber receives the same result.
How the Firehose Works: Connect and Receive
Firehose is one WebSocket. You send two things, your API key and one subscribe request. The server sends back two confirmations and then a stream of documents. Your application needs WebSocket support, JSON parsing, Zstandard decompression, and durable storage for its resume cursor. This is a quick example; for everything else, see our API documentation or contact the team.
1. Connect with your API key
Open wss://firehose.nosible.world/v2.0/firehose with your API key in an X-Api-Key or Authorization: Bearer header, then wait for authenticated. If it cannot set headers, send {"type":"auth","api_key":"YOUR_API_KEY"} as a text message within 3 seconds. Your key must be approved for Firehose access.
{type, can_replay, time_ms}
2. Send a subscribe request
Within 10 seconds of authenticated, send one subscribe request. It asks for every document that names Nvidia or Jensen Huang, its chief executive from our story. Different entities in one filter combine with OR, so a single subscription follows both:
{type, subscription, resume, filters}
"filters": {entities}
"entities": [2 items]
{label, name},
{label, name}
subscription names your feed, resume says where to start, filters pick the documents, and fields (omitted) picks what you receive. The server replies with subscribed, echoing your name and start (source) and a cursor, then sends documents. A name takes 1 to 64 letters, digits, hyphens or underscores, and reusing an active name with the same key replaces the previous connection.
3. Receive documents
After subscribed, each matching document arrives as a binary message, and a text heartbeat carrying your cursor arrives every 15 seconds, even when nothing matches. Every message the server sends includes time_ms, the time it was prepared in UTC Unix milliseconds. A document message is a plain JSON header you can read without decompressing, then the zstd-compressed document: the bytes NFH1, a 4-byte big-endian header length, the header, and one zstd frame. Our Firehose reference has Python and Node decoders.
Below is a hypothetical document for this subscription, inspired by NOSIBLE World's 10 November 2026 Nvidia story. The article and publisher are invented, the tickers and topics come from the event, and every snippet's embeddings, entities, locations, sentiment, forward-looking scores and ontologies are synthetic. The header:
{cursor, arrived_ms, time_ms, facets}
"facets": {10 keys}
"languages": {en},
"entities": {GPE, ORG, PERSON},
"GPE": [1 item],
{6 keys}
"ORG": [1 item],
{6 keys}
"PERSON": [2 items]
{6 keys},
{6 keys}
"locations": [1 item],
{8 keys}
"ontologies": [7 items],
{8 keys},
{6 keys},
{6 keys},
{8 keys},
{9 keys},
{8 keys},
{6 keys}
"tickers": [1 item],
{15 keys}
"why": [1 item]
{entity, score, chain}
"chain": [1 item]
{subject, predicate, object, known_as_of}
The header says where the document sits in the stream. Its facets are flat copies of selected document fields, with the same names, so you can filter further, for example on tickers, without decompressing. cursor is the position to save. arrived_ms is when it reached our stream, not when published. facets.hash, facets.published_timestamp and facets.visited_timestamp copy request.hash, page.published_timestamp and page.visited_timestamp. The hash and the visited timestamp identify the document, so you can drop duplicates.
This is the document inside the zstd part, from the same sample:
{13 keys}
"request": {11 keys},
"page": {16 keys},
"entities": {ORG, PERSON},
"ORG": [1 item],
{name, mention_count, confidence}
"PERSON": [1 item]
{name, mention_count, confidence}
"signals": {2 keys},
"financial_sentiment": {label, probabilities},
"probabilities": {negative, neutral, positive}
"forward_looking": {label, probabilities}
"probabilities": {forward, not-forward}
"ontologies": [2 items],
{ontology, category, subcategory, event, probability},
{ontology, label, probability}
"embeddings": {embeddinggemma-2}
"languages": {en},
"full_text": "Example News - Technology Nvidia chief Jensen Hu…"(473 characters),
"snippets": {4 keys},
"FvMVztP4NAuli6lYU4Pk67Ey": {15 keys},
"statistics": {sentences, words, characters, links},
"links": {Jensen Huang},
"entities": {ORG, PERSON},
"ORG": [1 item],
{name, mention_count, confidence}
"PERSON": [1 item]
{name, mention_count, confidence}
"signals": {2 keys},
"financial_sentiment": {label, probabilities},
"probabilities": {negative, neutral, positive}
"forward_looking": {label, probabilities}
"probabilities": {forward, not-forward}
"ontologies": [2 items],
{ontology, category, subcategory, event, probability},
{ontology, label, probability}
"embeddings": {embeddinggemma-2}
"k4wlVv963sQInODe8ZcVnVEK": {15 keys},
"statistics": {sentences, words, characters, images},
"images": [1 item],
"entities": {GPE, ORG},
"GPE": [1 item],
{name, mention_count, confidence}
"ORG": [1 item]
{name, mention_count, confidence}
"locations": [1 item],
{8 keys}
"signals": {2 keys},
"financial_sentiment": {label, probabilities},
"probabilities": {negative, neutral, positive}
"forward_looking": {label, probabilities}
"probabilities": {forward, not-forward}
"ontologies": [2 items],
{ontology, category, subcategory, event, probability},
{6 keys}
"embeddings": {embeddinggemma-2}
"sXJsbSUufAhJvvVsWLNJPJ3p": {15 keys},
"statistics": {sentences, words, characters, lists},
"lists": [1 item],
[3 items]
"entities": {ORG},
"ORG": [1 item]
{name, mention_count, confidence}
"signals": {2 keys},
"financial_sentiment": {label, probabilities},
"probabilities": {negative, neutral, positive}
"forward_looking": {label, probabilities}
"probabilities": {forward, not-forward}
"ontologies": [2 items],
{ontology, category, subcategory, event, probability},
{ontology, label, probability}
"embeddings": {embeddinggemma-2}
"R0HiUrvCJ5gW8XzNqoVnyTrk": {14 keys}
"statistics": {sentences, words, characters},
"entities": {PERSON},
"PERSON": [1 item]
{name, mention_count, confidence}
"signals": {2 keys},
"financial_sentiment": {label, probabilities},
"probabilities": {negative, neutral, positive}
"forward_looking": {label, probabilities}
"probabilities": {forward, not-forward}
"ontologies": [2 items],
{ontology, category, subcategory, event, probability},
{ontology, label, probability}
"embeddings": {embeddinggemma-2}
"statistics": {12 keys},
"entities": {GPE, ORG, PERSON},
"GPE": [1 item],
{6 keys}
"ORG": [1 item],
{6 keys}
"PERSON": [2 items]
{6 keys},
{6 keys}
"tickers": [1 item],
{15 keys}
"why": [1 item]
{entity, score, chain}
"chain": [1 item]
{subject, predicate, object, known_as_of}
"ontologies": [7 items],
{8 keys},
{6 keys},
{6 keys},
{8 keys},
{9 keys},
{8 keys},
{6 keys}
"locations": [1 item],
{8 keys}
"signals": {2 keys},
"financial_sentiment": {negative, neutral, positive},
"negative": {count, min, max, mean},
"neutral": {count, min, max, mean},
"positive": {count, min, max, mean}
"forward_looking": {forward, not-forward}
"forward": {count, min, max, mean},
"not-forward": {count, min, max, mean}
A full document has 13 top-level keys:
requestholds the page address and its hash.pageholds the title, description, publish and crawl times and a 768-dimension embedding. The page is enriched like everything else:page.entities,page.locations,page.tickers,page.ontologiesandpage.signalsdescribe the title and description.full_textis the page text, andsnippetssplits it into passages.statisticscounts what the crawler found.languages,has_tickersandcompletedescribe the whole document.completeistruewhen every part has been enriched.
Every snippet is a passage of the page text with its own identity: content, the words we index, its language, statistics for sentences, words and characters, a snippet_hash, and the hashes of the snippets before and after it (prev_snippet_hash and next_snippet_hash), so you can walk the page in order. Each is enriched on its own, with the same fields as the page:
embeddings: its own 768-dimension vector. Decode it from base64 into 768 little-endian signed 16-bit integers, then divide each by 10,000.entities: the people, organisations, places and products it names, each with a confidence.locations: the places it is about, geocoded.tickers: the listed companies it matches.ontologies: its topics, scored across the 15 ontology families.signals: its own sentiment (negative, neutral or positive) and forward-looking (forward or not-forward) classification, each with a label and probabilities.
So you can ask which passage is negative, which one is forward-looking, and which one names a company, a place and a topic together.
A snippet also carries the structure around its text, attached to the passage it belongs to. Each key below is present only when the snippet has one, and statistics counts it:
links: link text mapped to its URL.images,videos,audioandfiles: URLs of the media and documents (such as PDF, Word, Excel and CSV files) beside the passage.tables: tables as column-keyed objects.lists: bulleted and numbered lists, as arrays of items.blocks: code blocks, as arrays of lines.
In the example, the first snippet has a link, the second an image and the third a list. The document-level statistics totals each of them.
The document-level entities, ontologies, signals, tickers and locations roll these up across the page and all of its snippets, so they report how often something occurred and how confident the models were:
- An entity has a
count(the passages that mention it), amention_countand the minimum, maximum and mean confidence. - An ontology has a
countand the minimum, maximum and mean probability. signalsgives, for each sentiment and forward-looking class, how many passages carry that label (count) and the minimum, maximum and mean probability of that class. In the example, four of the five passages, counting the page, are negative and the mean probability of negative is 0.82.
Accept documents with or without signals. If your request sets fields, you receive only the parts you asked for, plus request and page.visited_timestamp, and the facets still describe the full document.
Filters and Options for Your Subscription
Different filters combine with AND, options within one filter combine with OR, and a filter you leave out matches everything. Text matching ignores case, but entity names and ticker symbols must match in full, and properties in one clause must match the same row. Each filter takes up to 100 options, and each key can hold four subscriptions. The hash and the published and visited timestamps cannot be filtered. Setting has_tickers to false with a non-empty tickers filter can never match. For example, this subscription keeps English documents that matched a ticker:
{type, subscription, resume, filters}
"filters": {languages, has_tickers}
"languages": [1 item],
The subscribe reference has the full syntax.
NOSIBLE does not store your position for you. Save a cursor only after you have safely processed every document before it, and send it back as resume when you reconnect, with the same filters and fields. You can also start from a time in the last 24 hours with {"from_time_ms": 1794940000000}, a UTC Unix time in milliseconds, and once a replay catches up the server sends replay_complete and then live documents.
{type, subscription, resume}
"resume": {cursor}
Delivery is at-least-once, so always deduplicate on request.hash and page.visited_timestamp. Keep reading, because a blocked send times out after 30 seconds and can end in slow_consumer. Retry temporary failures such as upstream_unavailable with backoff and keep your cursor rather than switching to latest. Headers are capped at 513 KiB and decompressed documents at 16 MiB. The errors reference lists every error code.
Replaying Nvidia and Jensen Huang Through the Firehose
We replayed the Firehose from 7 to 13 November over wss://firehose.nosible.world/v2.0/replay, with the same API key and the same filters as a live subscription. One request kept the documents that tag Nvidia or Jensen Huang:
{type, start, end, filters}
"filters": {entities}
"entities": [2 items]
{label, name},
{label, name}
Entities are tagged on English documents, so the chart counts English documents, and we counted each title once per website to drop reposts. The chart below shows the volume of documents, their sentiment against the NVDA share price, and how often four of NOSIBLE's event classes appeared, all on the same hourly axis.
Volume stepped up in stages. It went from about 10 documents an hour to 73, 188 and 218 in the first three hours of the story on Sunday evening, settled at a few hundred an hour, and reached 1,561 in the hour from 14:00 UTC on Thursday 12 November, half an hour before the NYSE opened. Markets were closed when the spike began, so the first NVDA price came 13 hours later, at 09:00 UTC on Monday in pre-market trading, 4.9% below its last price on Friday 6 November.
Over the week NVDA fell 9.2% and touched $150.91 at 14:20 UTC on Thursday, ten minutes before the open. Where the chart shows no share price, trading was closed, and a dotted line joins the last price of one session to the first of the next. Sentiment sits in the second panel. It is the probability a document is positive minus the probability it is negative, so anything below the dotted neutral line is negative.
The archive starts on Saturday morning, when the story was already in the news, and the average across all 38,052 documents was -0.38. Sentiment had eased to +0.12 by 13:00 UTC on Wednesday, then fell to -0.15 by 16:00 UTC while NVDA held near $176. NVDA only dropped after the close, from $173.66 at 21:00 UTC to $159.37 at 23:00 UTC, and sentiment reached -0.34 and then -0.50 in those two hours. The four panels at the bottom show what the conversation was about.
Alleged sexual harassment was the biggest class, 12% of all documents, and peaked at 178 documents an hour at 22:00 UTC on Wednesday, midway through the after-hours drop. Chief executive officer departed was 6% of documents but arrived late, peaking at 400 documents an hour at 14:00 UTC on Thursday, the overall peak hour, when NVDA touched $150.91 at 14:20 UTC, closing at $166.00. Share price decreased was 7% of documents, and disclosed corporate secrets leak was 3%.
We have no class for "CEO had an affair", so the model chose the closest one, alleged sexual harassment. Documents that did not carry a sentiment signal when they arrived (about half of them, from before sentiment went live on the stream and during a short outage) were scored afterwards with the same model. Documents now carry their event classes, sentiment and forward-looking classification from the stream, so a client subscribed to Nvidia and Jensen Huang can build everything in the chart except the share price from the stream alone.
Firehose is Best for Mid-Frequency Strategies
Firehose is best for mid-frequency strategies: those that make tens or hundreds of trading decisions every day across a wide variety of capital markets. Mid-frequency here means between tick data and end-of-day data. Their problem is deciding which of the day's published documents matter to the positions they hold.
Teams use Atlas, World and Search to build their priors, meaning what they already know: how companies connect, what has happened before, and what the evidence says. Firehose makes those priors actionable in real time as stories develop.
A team subscribes to the names, topics and places behind each of its priors. Each matching document arrives as soon as it is processed, already enriched at every level: tickers in the header for routing, and on every snippet the entities, locations, ontologies, sentiment, forward-looking classification and embedding, so a team can act on the exact passage that matters instead of re-reading the page.
Get Early Access to the Firehose in Preview
Firehose is in preview until 1 December, while we test it in a live production setting with real traffic and fix any issues that turn up. When the preview ends, Firehose will become generally available.
Firehose access is reserved. Each API key is approved individually, and a key that has not been approved is queued and disconnected with firehose_access_pending. To request access, email stuart@nosible.com and tell us which documents your team needs. We will say whether the filters can select them. The Firehose reference documents every frame.