Working fetch feeds and parser, process raw and error urls

This commit is contained in:
Luciano Gervasoni
2025-03-18 14:49:12 +01:00
parent 7d7bce1e72
commit fb4b30f05e
26 changed files with 270 additions and 364 deletions

View File

@@ -0,0 +1,20 @@
# Fetcher
```
conda create -n matitos_fetcher python=3.12
conda activate matitos_fetcher
conda install -c conda-forge curl
pip install ipykernel "psycopg[binary]" git+https://github.com/ranahaani/GNews.git GoogleNews duckduckgo_search newspaper4k numpy beautifulsoup4 requests feedparser pytz redis fastapi uvicorn fastapi-utils lxml[html_clean]
```
* Fetcher app
- Contains several endpoints to perform a specific fetching type task
- For more details, check in [app.py](app.py) /{fetch_type}
* Build and run
- Important: To be deployed with other micro-services, [docker-compose.yml](../docker-compose.yml)
```
docker build -t fetch_app .
docker run --rm --name container_fetch_app fetch_app
```