Matitos

URLs Fetcher -> Inserts raw URLs
- Fetch parsing URL host
- Fetch from RSS feed
- Fetch keyword search (Google search & news, DuckDuckGo, ...) ++ Sources -> Robustness to TooManyRequests block - Selenium based - Sites change their logic, request captcha, ... - Brave Search API - Free up to X requests per day. Need credit card association (no charges) - Bing API - Subscription required - Yandex. No API? ++ Proxy / VPN? TooManyRequests, ... ++ Search per locale (nl-NL, fr-FR, en-GB)
URLs Processing -> Updates raw URLs
- Extracts title, description, content, image and video URLs, main image URL, language, keywords, authors, tags, published date
- Determines if it is a valid article content ++ Proxy / VPN? Bypass geoblock
Visualization of URLs
- Filter URLs
  - By status, search, source, language, ...
- Charts
Valid URLs
- Generate summary
  - One paragraph
  - At most three paragraphs
- Classification
  - 5W: Who, What, When, Where, Why of a Story
  - Related to child abuse?
  - ...
Content generation
- URLs Selection
  - Valid content
  - Language of interest
  - Published (or fetch) date during last_week
  - Fetched by at least N sources
  - Use classifications and summaries
- Merge summaries, ...