[Ingest] Route scraping through an optional V2Ray/Xray proxy (Telegram in Iran)

Telegram and some sources are filtered in Iran. .NET cannot speak vmess/vless/trojan, so add an Xray sidecar (compose service 'xray', behind the 'proxy' profile) that converts the admin's config into a local SOCKS5 proxy (xray:10808). New ScrapeHttpClients provider builds a proxied or direct HttpClient (WebProxy supports socks5/socks4/http) cached per proxy URL; all five ingestion sources (Telegram/Bale/Divar/Medjobs/Websites) now use it. Admin settings gain IngestProxyEnabled + IngestProxyUrl (migration; UI under sources). Added deploy/xray/config.json template + README with vmess/vless/trojan examples. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 17:53:17 +03:30
parent 698565c460
commit cea27c8684
17 changed files with 1411 additions and 20 deletions
@@ -0,0 +1,63 @@
+# Ingestion proxy (Xray / V2Ray) — for scanning Telegram etc. from Iran
+
+The app's HttpClient can't speak `vmess` / `vless` / `trojan` directly. Instead, the **Xray
+sidecar** (compose service `xray`) reads your config and exposes a plain **SOCKS5 proxy at
+`xray:10808`** (and HTTP at `xray:10809`) on the internal compose network. The app is then
+pointed at that proxy from the admin panel, and only ingestion traffic goes through it.
+
+```
+[app ingestion] → socks5://xray:10808 → [Xray client] → vmess/vless/trojan → server → Telegram
+```
+
+## Setup
+
+1. **Put your config** in `deploy/xray/config.json`. Replace the `proxy` outbound with your
+   own vmess / vless / trojan outbound (templates below). Keep the `inbounds` and `routing`
+   sections as-is so the local SOCKS/HTTP ports stay the same.
+
+2. **Start the sidecar** (it's behind a compose profile so normal deploys don't run it):
+   ```bash
+   docker compose --profile proxy up -d xray
+   docker logs hamkadr_xray --tail 30      # should show it listening, no errors
+   ```
+
+3. **Point the app at it**: open `/Admin/Settings` → «کانال‌ها/منابع» →
+   - tick **«ارسال جمع‌آوری از طریق پروکسی»**
+   - set the proxy URL to **`socks5://xray:10808`**
+   - Save, then run ingestion (Telegram source enabled). 
+
+4. **Quick test** the proxy reaches Telegram:
+   ```bash
+   docker exec hamkadr_api sh -c "wget -q -O- --timeout=15 -e use_proxy=yes -e http_proxy=http://xray:10809 https://t.me/s/telegram | head -c 200" || true
+   ```
+
+## Where to get the config values
+If you have a share link (`vmess://…`, `vless://…`, `trojan://…`), import it into the Xray/v2rayN
+client and **export the JSON config**, or decode it and fill the templates below.
+
+### vless + ws + tls (matches the default template in config.json)
+```json
+{ "tag":"proxy","protocol":"vless","settings":{"vnext":[{"address":"HOST","port":443,
+  "users":[{"id":"UUID","encryption":"none"}]}]},
+  "streamSettings":{"network":"ws","security":"tls","tlsSettings":{"serverName":"SNI"},
+  "wsSettings":{"path":"/PATH","headers":{"Host":"SNI"}}} }
+```
+
+### vmess + ws + tls
+```json
+{ "tag":"proxy","protocol":"vmess","settings":{"vnext":[{"address":"HOST","port":443,
+  "users":[{"id":"UUID","alterId":0,"security":"auto"}]}]},
+  "streamSettings":{"network":"ws","security":"tls","tlsSettings":{"serverName":"SNI"},
+  "wsSettings":{"path":"/PATH","headers":{"Host":"SNI"}}} }
+```
+
+### trojan + tls
+```json
+{ "tag":"proxy","protocol":"trojan","settings":{"servers":[{"address":"HOST","port":443,
+  "password":"PASSWORD"}]},
+  "streamSettings":{"network":"tcp","security":"tls","tlsSettings":{"serverName":"SNI"}} }
+```
+
+> Security note: `config.json` contains your VPN credentials. It's mounted read-only into the
+> container. Do **not** commit a real config — keep the committed file as a placeholder and
+> drop the real one on the server only (or add it to `.gitignore` if you keep it locally).