Idea and goals

This is a proof of concept to see what is possible with LLMs (text-to-text, text-to-image and image-to-text) in the context of news articles from around the world. At the core is a fairly simple crawler that attempts to get news articles and images from various websites, summarizes these articles using LLMs (text-to-text), then clusters articles together based on word relevancy. It then attempts to summarize the cluster taking all the summaries of the cluster articles together, looks for images that are close to the cluster description (image-to-text) and does a clustering of images. If there are no images available, it creates an image prompt (text-to-text) and then uses a third-party site to create the image (text-to-image). Finally it creates the HTML and deploys it. This is work in progress to test various algorithms and LLMs and thus can randomly change.

Tools in use

The pipeline is a chain of thirteen plain Node.js scripts driven by a single shell script. Each stage reads the previous stage's output from a directory of JSON files and writes its own — no database, no queue, no framework. It is deliberately simple so that any stage can be re-run for any past date on its own. Most of it was written by vibe coding, for speed.

• Crawling: each source picks one of four engines. Cheerio over plain HTTP is the default; Puppeteer (with the stealth plugin) and Playwright drive a real browser when a site needs one; and a cookie-jar engine replays the cookies of a genuine, logged-in Chrome session so subscriber articles behind bot protection can be fetched over ordinary HTTPS. RSS feeds are read directly.
• Text models: the default is a local proxy onto a ChatGPT subscription running GPT-5.4-mini. When that is unreachable or rate-limited, the pipeline fails over automatically to OpenRouter running Llama 4 Maverick. Everything textual goes through this path: per-article summaries, cluster summaries and key points, headlines, and the various judging steps.
• Embeddings: OpenAI text-embedding-3-large, served through OpenRouter. The same code can switch to embedding models hosted locally by Ollama (bge-m3, EmbeddingGemma) or on Together (multilingual-e5, GTE-ModernBERT), which is handy for comparing how much the embedding model actually matters.
• Clustering: HDBSCAN, a density-based algorithm, over the summary embeddings, with min-samples set to 2. Clusters that share enough named entities and sit close enough together are then merged, and an LLM pass moves low-confidence articles to where they belong. A second LLM judge looks at everything that landed in “In other news” and rescues what fits an existing cluster or forms a new one.
• Story arcs: each day's clusters are matched against the previous fourteen days, and candidate continuations are confirmed by the model before an edge is added. The result is a graph of multi-day stories, drawn on each story page.
• Images: real photographs from the source articles are preferred. Candidates are described by a vision model and ranked against the cluster summary by embedding similarity. Only when nothing suitable turns up does the pipeline write an image prompt and generate a picture — GPT-Image through the same local proxy, with Black Forest Labs FLUX as fallback.
• Publishing: static HTML written straight to disk, plus sitemaps, and deployed to Firebase Hosting.

Who is behind this

This is an example project by Vanguard Signals that uses vector databases and large language models (LLMs) to process and generate content.