Anonymous
/wsg/ - Web Scraping General 06/21/24(Fri)17:32:38 No.101090758

File: scraper.png (1.62 MB, 1892x2142)

1.62 MB PNG

/wsg/ - Web Scraping General Anonymous 06/21/24(Fri)17:32:38 No.101090758

Web Scraping General

Reverse engineering edition pt 2

QOTD: Which is easier: parsing HTML or reverse engineering priv/undocumented APIs to scrape from?

FAQ: rentry co/t6237g7x

> Captcha services
https://2captcha.com/
https://www.capsolver.com/
https://anti-captcha.com/

> Proxies
https://hproxy.com/ (no blacklist) (recommended, owned by friend of /wsg/)
https://infiniteproxies.com/ (no blacklist)
https://www.thunderproxies.com/
http://proxies.fo/ (not recommended)

> Network analysis
https://mitmproxy.org/
https://portswigger.net/burp

> Scraping tools
https://beautiful-soup-4.readthedocs.io/en/latest/
https://www.selenium.dev/documentation/
https://playwright.dev/docs/codegen
https://github.com/lwthiker/curl-impersonate
https://github.com/yifeikong/curl_cffi

Official Telegram: @scrapists
Last thread: >>101054257