0 Fork 0

T

Tests / test (pull_request) Failing after 3m29s

Details

Make optional search imports lazy to avoid GitHub dependency in CI

Wrap gnews, GoogleNews, and search_engines imports in try/except
so the module loads without those packages installed. This allows
the test suite to run on the Gitea runner without outbound GitHub
access.

Also revert requirements.txt back to the GitHub URL for the
Search-Engines-Scraper dependency.

Co-Authored-By: opencode <noreply@opencode.ai>

2026-09-02 12:00:14 -03:00

.gitea/workflows

Add unit tests and Gitea CI workflow for bug fixes

2026-09-02 09:42:07 -03:00

app_cv

Unknown instead of error for fetched urls

2025-06-19 22:43:29 +02:00

app_cv_face

CV app with fastapi, web nicegui based

2025-04-30 18:41:35 +02:00

app_img_gen

Refactor searches, env vars fetcher config, urls webpage update

2025-04-02 18:45:43 +02:00

app_selenium

Selenium docker psutil

2025-08-01 20:49:28 +02:00

app_urls

Make optional search imports lazy to avoid GitHub dependency in CI

2026-09-02 12:00:14 -03:00

utils

Schools NL tuples, traceback on notify err

2025-10-14 11:33:17 +02:00

website

Compose fix 2

2025-04-16 20:13:56 +02:00

.env.sample

Telegram bot tokens

2025-10-14 11:36:19 +02:00

.gitignore

django tasks scheduler update, .env and docker compose towards fetcher sca

2025-06-20 00:35:48 +02:00

docker-compose-base.yml

Telegram bot tokens

2025-10-14 11:36:19 +02:00

docker-compose-dev.yml

Redis cache and celery, avoid overflow (3)

2025-09-03 23:20:56 +02:00

docker-compose.yml

Redis cache and celery, avoid overflow (3)

2025-09-03 23:20:56 +02:00

README.md

Selenium based fetch of different sources

2025-07-08 18:18:26 +02:00

README.md

Matitos

URLs Fetcher -> Inserts raw URLs
- Fetch parsing URL host
- Fetch from RSS feed
- Fetch keyword search (Google search & news, DuckDuckGo, ...)
  - TODO: More sources -> Robustness to TooManyRequests block
    - Selenium based
      - Sites change their logic, request captcha, ...
    - Brave Search API
      - Free up to X requests per day. Need credit card association (no charges)
    - Bing API
      - Subscription required
    - Yandex. No API?
  - TODO: Proxy / VPN?
    - TooManyRequests, ...
  - TODO: Search per locale (nl-NL, fr-FR, en-GB)
- Fetch keyword search for selenium sources
URLs Processing -> Updates raw URLs
- Extracts title, description, content, image and video URLs, main image URL, language, keywords, authors, tags, published date
- Determines if it is a valid article content
- TODO: Proxy / VPN?
  - Bypass geoblock and TooManyRequests
Visualization of URLs
- Filter URLs
  - By fetch date, status, search, source, language, has valid content, minimum amount of sources, ...
- Charts
URLs selection
- Published (or fetch) date during last_week / last 24 hrs
- Language of interest
- Valid content
- Fetched by at least N sources
- Use classifications and summaries
- TODO: Manual inspection -> Improve automation
  - Rules or pattern for invalid articles, e.g. "youtube.com/*"
  - URL host with "priority" or "weight"
Content generation
- Generate summary
  - One paragraph
  - At most three paragraphs
- Classification
  - 5W: Who, What, When, Where, Why of a Story
  - Related to child abuse?
  - ...
- Merge similar articles?

Deploy

Dev mode

docker compose -f docker-compose-dev.yml down -v
docker compose -f docker-compose-dev.yml up --no-deps --build

Prod mode

docker compose down -v
docker compose up -d --no-deps --build