The markdown would be much more useful if you stripped headers/footers and other tags like filters etc that is not core content (i.e. low value for RAG/context). Either using tag or class-based removal from the html or using something like Mozilla's Readability or both! Highly opinionated class-based removal is risky but produces high value content and less noise.
For example a language selector in a header gets produced and should be stripped:
[Skip to main content](#main-content)
Select LanguageEnglishAfrikaansAlbanianArabicArmenianAzerbaijaniBasqueBelarusianBengaliBosnianBulgarianCatalanCebuanoChinese (Simplified)Chinese (Traditional)CroatianCzechDanishDutchEsperantoEstonianFilipinoFinnishFrenchGalicianGeorgianGermanGreekGujaratiHaitian CreoleHausaHebrewHindiHmongHungarianIcelandicIgboIndonesianIrishItalianJapaneseJavaneseKannadaKhmerKoreanLaoLatinLatvianLithuanianMacedonianMalayMalteseMaoriMarathiMongolianNepaliNorwegianPersianPolishPortuguesePunjabiRomanianRussianSerbianSlovakSlovenianSomaliSpanishSwahiliSwedishTamilTeluguThaiTurkishUkrainianUrduVietnameseWelshYiddishYorubaZulu
Here is a starter list.. should probably test against a couple thousand random pages and use an LLM like haiku with vision as judge.
const exclude = [
'header', '.header', '.top', '.navbar', '#header',
'footer', '.footer', '.bottom', '#footer',
'.sidebar', '.side', '.aside', '#sidebar',
'.modal', '.popup', '#modal', '.overlay',
'.ad', '.ads', '.advert', '#ad',
'.lang-selector', '.language', '#language-selector',
'.social', '.social-media', '.social-links', '#social',
'.menu', '.navigation', 'nav', '#nav',
'.breadcrumbs', '#breadcrumbs',
'.form', 'form', '#search-form',
'script', 'noscript'
];