# Matitos

- URLs Fetcher -> Inserts raw URLs
    - Fetch parsing URL host
    - Fetch from RSS feed
    - Fetch keyword search (Google search & news, DuckDuckGo, ...)
        ++ Sources -> Robustness to TooManyRequests block
            - Selenium based
                - Sites change their logic, request captcha, ...
            - Brave Search API
                - Free up to X requests per day. Need credit card association (no charges)
            - Bing API
                - Subscription required
            - Yandex. No API?
        ++ Proxy / VPN?
            TooManyRequests, ...
        ++ Search per locale (nl-NL, fr-FR, en-GB)

- URLs Processing -> Updates raw URLs
    - Extracts title, description, content, image and video URLs, main image URL, language, keywords, authors, tags, published date
    - Determines if it is a valid article content
    ++ Proxy / VPN?
        Bypass geoblock

- Visualization of URLs
    - Filter URLs
        - By status, search, source, language, ...
    - Charts

- Valid URLs
    - Generate summary
        - One paragraph
        - At most three paragraphs
    - Classification
        - 5W: Who, What, When, Where, Why of a Story
        - Related to child abuse?
        - ...

- Content generation
    - URLs Selection
        - Valid content
        - Language of interest
        - Published (or fetch) date during last_week
        - Fetched by at least N sources
        - Use classifications and summaries
    - Merge summaries, ...