Anthropic Pro RSS: evaluate a Claude-oriented feed workflow

Write the evaluation question first

Choose a task that matters to the publication. For a documentation digest, the question might be whether an entry changes an installation step. For a research collection, it might be whether the supplied abstract supports inclusion in a specific topic. Record a few examples with expected decisions and the reasons behind them.

Anthropic's prompt engineering overview recommends establishing success criteria and a way to test them. Use that foundation to compare prompts against the same source material. A revised prompt should earn its place through better observed results on the task you actually need.

Prepare clearly labeled evidence

Separate the instructions from the source passages. Give each entry a short label, title, publication date, and relevant text. Retain the canonical URL in the accompanying source record. Ask the response to identify the labels supporting each finding and to name missing details when the supplied material does not resolve the question.

Keep quotations and source summaries recognizable within the input. A passage may contain instructions aimed at the original article's reader, such as a command to install a package. Those instructions describe the source material; they should not become authority to change the workflow's own behavior.

Use a practical review rubric

• Relevance: does the selected entry answer the editorial question?
• Support: does the cited passage establish the proposed claim?
• Completeness: are important conditions and exceptions retained?
• Attribution: does each source label identify the correct entry?
• Usefulness: can the intended reader act on the explanation?

Include borderline cases in the test collection. An announcement may mention a feature without explaining availability. Two articles may discuss different versions under similar titles. A good result preserves those distinctions and limits its conclusion accordingly.

Review the failures before changing the prompt. Missing source text may require better extraction, while inconsistent terminology may need a clearer task definition. Keep the accepted prompt, sample inputs, and reviewer notes together so another maintainer can repeat the comparison. The LLM evidence guide provides a provider-independent structure for that record and helps keep future implementation changes focused.