# Matitos

- Scheduled tasks
    - Fetcher -> Inserts raw URLs
        - Fetch parsing URL host
        - Fetch from RSS feed
        - Fetch keyword search (Google search & news, DuckDuckGo, ...)
            ++ Sources -> Robustness to TooManyRequests block
                - Selenium based
                    - Sites change their logic, request captcha, ...
                - Brave Search API
                    - Free up to X requests per day. Need credit card association (no charges)
                - Bing API
                    - Subscription required
                - Yandex. No API?
            ++ Proxy / VPN?
                TooManyRequests, ...
            ++ Search per locale (nl-NL, fr-FR, en-GB)
    - Process URLs -> Updates raw URLs
        - Extracts title, description, content, image and video URLs, main image URL, language, keywords, authors, tags, published date
        - Determines if it is a valid article content
        ++ Proxy / VPN?
            Bypass geoblock
    - Valid URLs
        - Generate summary
            - One paragraph
            - At most three paragraphs
        - Classification
            - 5W: Who, What, When, Where, Why of a Story
            - Related to child abuse?
            - ...

- Visualization of URLs
    - Filter URLs
        - By status, search, source, language
    - Charts

- Content generation
    - Select URLs:
        - Valid content
        - language=en
        - published_date during last_week
        - Use classifications
    - Merge summaries, ...