# Matitos

- URLs Fetcher -> Inserts raw URLs
    - Fetch parsing URL host
    - Fetch from RSS feed
    - Fetch keyword search (Google search & news, DuckDuckGo, ...)
        - TODO: More sources -> Robustness to TooManyRequests block
            - Selenium based
                - Sites change their logic, request captcha, ...
            - Brave Search API
                - Free up to X requests per day. Need credit card association (no charges)
            - Bing API
                - Subscription required
            - Yandex. No API?
        - TODO: Proxy / VPN?
            - TooManyRequests, ...
        - TODO: Search per locale (nl-NL, fr-FR, en-GB)
    - Fetch keyword search for selenium sources


- URLs Processing -> Updates raw URLs
    - Extracts title, description, content, image and video URLs, main image URL, language, keywords, authors, tags, published date
    - Determines if it is a valid article content
    - TODO: Proxy / VPN?
        - Bypass geoblock and TooManyRequests

- Visualization of URLs
    - Filter URLs
        - By fetch date, status, search, source, language, has valid content, minimum amount of sources, ...
    - Charts

- URLs selection
    - Published (or fetch) date during last_week / last 24 hrs
    - Language of interest
    - Valid content
    - Fetched by at least N sources
    - Use classifications and summaries
    - TODO: Manual inspection -> Improve automation
        - Rules or pattern for invalid articles, e.g. "youtube.com/*"
        - URL host with "priority" or "weight"

- Content generation
    - Generate summary
        - One paragraph
        - At most three paragraphs
    - Classification
        - 5W: Who, What, When, Where, Why of a Story
        - Related to child abuse?
        - ...
    - Merge similar articles?

# Deploy

* Dev mode
```
docker compose -f docker-compose-dev.yml down -v
docker compose -f docker-compose-dev.yml up --no-deps --build
```
* Prod mode
```
docker compose down -v
docker compose up -d --no-deps --build
```