Wait db connection, login required, dev mode enable

This commit is contained in:
Luciano Gervasoni
2025-04-04 12:28:22 +02:00
parent 4dbe2e55ef
commit 76079d7bd0
8 changed files with 105 additions and 54 deletions
+8 -5
View File
@@ -4,7 +4,7 @@
- Fetcher -> Inserts raw URLs
- Fetch parsing URL host
- Fetch from RSS feed
- Fetch searching (Google search & news, DuckDuckGo, ...)
- Fetch keyword search (Google search & news, DuckDuckGo, ...)
++ Sources -> Robustness to TooManyRequests block
- Selenium based
- Sites change their logic, request captcha, ...
@@ -13,20 +13,23 @@
- Bing API
- Subscription required
- Yandex. No API?
++ Proxy / VPN?
TooManyRequests, ...
++ Search per locale (nl-NL, fr-FR, en-GB)
- Process URLs -> Updates raw URLs
- Extracts title, description, content, image and video URLs, main image URL, language, keywords, authors, tags, published date
- Determines if it is a valid article content
++ Proxy / VPN?
Bypass geoblock
- Valid URLs
- Generate summary
- One paragraph
- At most three paragraphs
- Classification
- 5W: Who, What, When, Where, Why of a Story
- Related to child abuse?
- ...
Georgia Institute of Technology
https://comm.gatech.edu resources writers
- Visualization of URLs
- Filter URLs
- By status, search, source, language