Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

run.py 起來後拿不到ＩＰ #166

Open
CubatLin opened this issue May 27, 2022 · 7 comments
Open

run.py 起來後拿不到ＩＰ #166

CubatLin opened this issue May 27, 2022 · 7 comments
Assignees
@Germey
Labels
bug

Comments

@CubatLin
Copy link

HI想問一下,
透過run.py 啟動後, 用以下python code好像沒辦法成功拿到ＩＰ, 請問要怎麼透過python函式拿代理ＩＰ呢？謝謝

  • python code
>>> import requests
>>> proxypool_url = 'http://127.0.0.1:5555/random'
>>> def get_random_proxy():
...     """
...     get random proxy from proxypool
...     :return: proxy
...     """
...     return requests.get(proxypool_url).text.strip()
... 
>>> get_random_proxy()
  • Terminal logs
(crawler_env) MacBook-Pro ProxyPool-master % python3 run.py                   
2022-05-27 19:30:53.527 | INFO     | proxypool.scheduler:run:100 - starting proxypool...
2022-05-27 19:30:53.527 | INFO     | proxypool.scheduler:run:104 - starting tester, pid None...
2022-05-27 19:30:53.532 | INFO     | proxypool.scheduler:run:110 - starting getter, pid None...
2022-05-27 19:30:53.533 | INFO     | proxypool.scheduler:run:116 - starting server, pid None...
2022-05-27 19:30:53.804 | DEBUG    | proxypool.scheduler:run_tester:33 - tester loop 0 start...
 * Serving Flask app "proxypool.processors.server" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
2022-05-27 19:30:53.804 | INFO     | proxypool.processors.tester:run:74 - stating tester...
2022-05-27 19:30:53.805 | DEBUG    | proxypool.processors.tester:run:76 - 8081 proxies to test
2022-05-27 19:30:53.805 | DEBUG    | proxypool.processors.tester:run:79 - testing proxies use cursor 0, count 20
2022-05-27 19:30:53.806 | DEBUG    | proxypool.processors.tester:test:42 - testing 80.249.135.41:8080
Process Process-3:
Traceback (most recent call last):
  File "/Users/ww/miniforge3/envs/crawler_env/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/ww/miniforge3/envs/crawler_env/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/ww/Desktop/Python/Fubon/10. crawler/ProxyPool-master/proxypool/scheduler.py", line 95, in run_server
    app.run(host=API_HOST, port=API_PORT, threaded=API_THREADED)
  File "/Users/ww/miniforge3/envs/crawler_env/lib/python3.8/site-packages/flask/app.py", line 990, in run
    run_simple(host, port, self, **options)
  File "/Users/ww/miniforge3/envs/crawler_env/lib/python3.8/site-packages/werkzeug/serving.py", line 1030, in run_simple
    s.bind(server_address)
OSError: [Errno 48] Address already in use
2022-05-27 19:30:53.831 | DEBUG    | proxypool.processors.tester:test:42 - testing 165.225.196.66:10605
2022-05-27 19:30:53.831 | DEBUG    | proxypool.processors.tester:test:42 - testing 176.235.131.231:9090
2022-05-27 19:30:53.831 | DEBUG    | proxypool.processors.tester:test:42 - testing 199.60.103.241:80
2022-05-27 19:30:53.832 | DEBUG    | proxypool.processors.tester:test:42 - testing 187.115.10.50:20183
2022-05-27 19:30:53.832 | DEBUG    | proxypool.processors.tester:test:42 - testing 139.5.16.97:8080
2022-05-27 19:30:53.832 | DEBUG    | proxypool.processors.tester:test:42 - testing 31.170.174.111:53281
2022-05-27 19:30:53.832 | DEBUG    | proxypool.processors.tester:test:42 - testing 8.214.11.10:8118
2022-05-27 19:30:53.832 | DEBUG    | proxypool.processors.tester:test:42 - testing 178.22.68.113:8111
2022-05-27 19:30:53.832 | DEBUG    | proxypool.processors.tester:test:42 - testing 165.225.194.102:10605
2022-05-27 19:30:53.833 | DEBUG    | proxypool.processors.tester:test:42 - testing 58.240.110.171:8888
2022-05-27 19:30:53.833 | DEBUG    | proxypool.processors.tester:test:42 - testing 110.80.172.180:16790
2022-05-27 19:30:53.833 | DEBUG    | proxypool.processors.tester:test:42 - testing 121.139.171.22:80
2022-05-27 19:30:53.833 | DEBUG    | proxypool.processors.tester:test:42 - testing 36.91.68.149:8080
2022-05-27 19:30:53.834 | DEBUG    | proxypool.processors.tester:test:42 - testing 103.139.242.1:83
2022-05-27 19:30:53.834 | DEBUG    | proxypool.processors.tester:test:42 - testing 43.241.29.201:8080
2022-05-27 19:30:53.834 | DEBUG    | proxypool.processors.tester:test:42 - testing 119.2.41.81:8080
2022-05-27 19:30:53.834 | DEBUG    | proxypool.processors.tester:test:42 - testing 78.189.32.215:8080
2022-05-27 19:30:53.834 | DEBUG    | proxypool.processors.tester:test:42 - testing 37.110.6.178:8080
2022-05-27 19:30:53.834 | DEBUG    | proxypool.processors.tester:test:42 - testing 217.60.194.53:8080
2022-05-27 19:30:54.269 | DEBUG    | proxypool.scheduler:run_getter:48 - getter loop 0 start...
2022-05-27 19:30:54.270 | INFO     | proxypool.processors.getter:run:36 - crawler <public.daili66.Daili66Crawler object at 0x1075b38e0> to get proxy
2022-05-27 19:30:54.271 | INFO     | proxypool.crawlers.base:crawl:40 - fetching http://www.66ip.cn/1.html
2022-05-27 19:30:54.767 | INFO     | proxypool.storages.redis:decrease:84 - 199.60.103.241:80 score decrease 1, current 8.0
2022-05-27 19:30:54.767 | DEBUG    | proxypool.processors.tester:test:65 - proxy 199.60.103.241:80 is invalid, decrease score
2022-05-27 19:30:54.803 | INFO     | proxypool.storages.redis:decrease:84 - 8.214.11.10:8118 score decrease 1, current 9.0
2022-05-27 19:30:54.803 | DEBUG    | proxypool.processors.tester:test:65 - proxy 8.214.11.10:8118 is invalid, decrease score
2022-05-27 19:30:54.816 | INFO     | proxypool.storages.redis:decrease:84 - 110.80.172.180:16790 score decrease 1, current 8.0
2022-05-27 19:30:54.817 | DEBUG    | proxypool.processors.tester:test:65 - proxy 110.80.172.180:16790 is invalid, decrease score
2022-05-27 19:30:55.242 | INFO     | proxypool.storages.redis:decrease:84 - 165.225.194.102:10605 score decrease 1, current 8.0
2022-05-27 19:30:55.243 | DEBUG    | proxypool.processors.tester:test:65 - proxy 165.225.194.102:10605 is invalid, decrease score
2022-05-27 19:30:55.315 | INFO     | proxypool.storages.redis:decrease:84 - 165.225.196.66:10605 score decrease 1, current 5.0
2022-05-27 19:30:55.316 | DEBUG    | proxypool.processors.tester:test:65 - proxy 165.225.196.66:10605 is invalid, decrease score
2022-05-27 19:30:56.255 | INFO     | proxypool.storages.redis:decrease:84 - 121.139.171.22:80 score decrease 1, current 5.0
2022-05-27 19:30:56.256 | DEBUG    | proxypool.processors.tester:test:65 - proxy 121.139.171.22:80 is invalid, decrease score
2022-05-27 19:30:59.110 | ERROR    | proxypool.crawlers.base:crawl:47 - crawler <public.daili66.Daili66Crawler object at 0x1075b38e0> crawled proxy unsuccessfully, please check if target url is valid or network issue
2022-05-27 19:30:59.113 | INFO     | proxypool.processors.getter:run:36 - crawler <public.data5u.Data5UCrawler object at 0x1075b3850> to get proxy
2022-05-27 19:30:59.114 | INFO     | proxypool.crawlers.base:crawl:40 - fetching http://www.data5u.com
2022-05-27 19:31:03.269 | INFO     | proxypool.processors.getter:run:36 - crawler <public.fatezero.FatezeroCrawler object at 0x1075b3a00> to get proxy
2022-05-27 19:31:03.269 | INFO     | proxypool.crawlers.base:crawl:40 - fetching http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.528 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 3.211.17.212:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.529 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 50.16.33.219:8080 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.530 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 54.156.145.160:8080 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.531 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 159.203.127.55:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.531 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 169.57.1.85:8123 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.532 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 132.145.57.226:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.532 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 212.129.29.139:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.533 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 116.203.22.243:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.533 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 46.101.208.13:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.534 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 188.166.166.109:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.535 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 167.99.181.81:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.535 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 177.190.160.55:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.536 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 190.97.226.236:999 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.536 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 190.26.201.194:8080 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.537 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 207.148.97.123:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.537 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 51.91.157.66:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.538 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 13.125.89.72:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.538 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 178.209.51.218:7829 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.539 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 195.135.242.141:8081 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.539 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 167.71.208.92:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.539 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 104.160.189.3:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.540 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 169.57.1.85:8123 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.540 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 190.26.201.194:8080 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.540 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 188.254.0.59:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.541 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 15.165.18.201:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.541 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 122.155.165.191:3128 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.542 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 190.97.226.236:999 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.542 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 80.48.119.28:8080 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.542 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 103.149.162.195:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.543 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 113.252.54.73:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.543 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 49.85.96.80:10081 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.544 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 113.195.127.130:8085 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.544 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 103.232.215.194:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.544 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 223.96.90.216:8085 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.545 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 112.6.117.135:8085 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.545 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 139.255.116.52:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.545 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 112.6.117.178:8085 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.546 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 120.220.220.95:8085 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.546 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 39.175.75.8:30001 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.547 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 178.209.51.218:7829 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.547 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 223.68.190.136:9091 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.547 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 111.3.118.247:30001 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.548 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 95.216.249.203:83 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.548 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 36.7.108.56:9091 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.548 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 181.129.14.165:999 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.549 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 182.92.179.221:443 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.549 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 200.39.153.1:999 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.549 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 164.155.146.31:80 from http://proxylist.fatezero.org/proxy.list
2022-05-27 19:31:04.549 | INFO     | proxypool.crawlers.base:process:31 - fetched proxy 89.250.152.76:8080 from http://proxylist.fatezero.org/proxy.list
...
...
@CubatLin CubatLin added the bug label May 27, 2022
@CubatLin
Copy link
Author

想問一下用run.py 啟動後 tester測完能用的ip 要從哪邊透過python code能獲取 謝謝

@cyicz123
Copy link

你先通过浏览器http://127.0.0.1:5555/random或者http://localhost:5555/random查看能否出现代理ip。看日志，proxypool有在正常运行。

@CubatLin
Copy link
Author

運行後，localhost request後沒有收到respond, 沒有ip回傳，想請問有除了random之外能拿到ip的路由嗎

@RuoCJ
Copy link

RuoCJ commented Feb 15, 2023 •
edited

我也是同样的问题，run之后，http://127.0.0.1:5555/ 这个页面都打不开，一直转圈圈

@Germey
Copy link
Member

Germey commented Feb 15, 2023

@RuoCJ 用什么方式启动的？Docker 应该没问题

@RuoCJ
Copy link

RuoCJ commented Feb 16, 2023

@RuoCJ 用什么方式启动的？Docker 应该没问题

没有使用docker，直接用 python run.py

@Germey
Copy link
Member

Germey commented Feb 19, 2023

@RuoCJ 那估计和环境配置有关系，可以尝试下 Docker 启动。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Germey Germey

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Germey @CubatLin @cyicz123 @RuoCJ