Tokenized LLM Pro RSS: prepare feed text for a model

Choose the representation before counting

An RSS item may contain a title, summary, full HTML content, dates, and several links. Decide which fields the model actually needs. A relevance classifier may work from a title and summary, while a factual comparison may require specific passages from the full article. Count the representation you will send, rather than an unrelated copy of the original feed file.

Normalize avoidable formatting noise without erasing meaning. Preserve paragraph breaks, headings, and the relationship between a code example and its explanation. Keep source identifiers and titles as separate fields so a later response can identify its evidence. Removing every structural boundary can make a shorter input harder to interpret.

Use the matching tokenizer

OpenAI's token-counting cookbook explains how tokenizer encodings divide text. Treat token counts as dependent on the selected encoding. A count obtained for one model family should not automatically become a limit calculation for another model or provider.

Keep a few representative samples beside your preparation code: ordinary prose, a URL-heavy announcement, multilingual text, and an article with source code. Record the count before and after normalization. This shows whether a transformation saves meaningful space and whether it changes the information an editor expects to preserve.

Retain the map back to the article

• Assign each extracted passage a stable source-entry identifier.
• Record the heading or section that surrounds the passage.
• Keep the canonical URL in the source record.
• Save the normalization method and tokenizer used for the count.
• Preserve the original approved source snapshot for comparison.

If a long entry must be divided, split it at useful editorial boundaries. Keep a prerequisite with the procedure that depends on it, and retain the title of a comparison table with its rows. A mechanically equal chunk size can separate information that needs to be understood together.

Review a sample of the prepared text as a human reader. Ask whether it still explains the article accurately and whether every passage can be traced back to its source. Once preparation is reliable, the token budget guide helps decide how much of that material belongs in one request.