2018 Scrapy Environment Enhance(4)Docker Service for Scrapyd and Tor Network

2018 Scrapy Environment Enhance(4)Docker Service for Scrapyd and Tor Network

Docker Service for Scrapyd
http://sillycat.iteye.com/blog/2422861

Docker Service for Tor Network
There are 3 ports and 3 different function services.
Tor 9051 Port reset IP
Tor 9050 Port to proxy
Privoxy 8118 to proxy to 9050

I put them into one Docker App, but we should be able to split them into 2 apps.
Dockerfile
#Run a Tor Network Server
#Prepare the OS
FROM ubuntu:16.04
MAINTAINER Carl Luo <luohuazju@gmail.com>
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get -qq update
RUN apt-get -qqy dist-upgrade
#Prepare the denpendencies
RUN apt-get install -qy tor
RUN apt-get install -qy netcat
RUN apt-get install -qy curl
RUN apt-get install -qy privoxy
#set up tor configuration
RUN echo "ControlPort 9051" >> /etc/tor/torrc
RUN echo "ControlListenAddress 0.0.0.0" >> /etc/tor/torrc
RUN echo HashedControlPassword $(tor --hash-password “xxxxxxxx" | tail -n 1) >> /etc/tor/torrc
#set up forward configuration
RUN echo "forward-socks5t / 127.0.0.1:9050 .">> /etc/privoxy/config
RUN echo "listen-address 0.0.0.0:8118">> /etc/privoxy/config
#set up the app
EXPOSE 9051 8118
RUN mkdir -p /app/
ADD start.sh /app/
WORKDIR /app/
CMD [ "./start.sh" ]

The command line to start, start.sh
#!/bin/sh -ex
#start the service
service tor start
service privoxy start
tail -f /dev/null

Makefile to open the PORT number
IMAGE=sillycat/public
TAG=ubuntu-tornetwork-1.0
NAME=ubuntu-tornetwork-1.0
docker-context:
build: docker-context
docker build -t $(IMAGE):$(TAG) .
run:
docker run -d -p 9051:9051 -p 8118:8118 --name $(NAME) $(IMAGE):$(TAG)
debug:
docker run -p 9051:9051 -p 8118:8118 --name $(NAME) -ti $(IMAGE):$(TAG) /bin/bash
clean:
docker stop ${NAME}
docker rm ${NAME}
logs:
docker logs ${NAME}
publish:
docker push ${IMAGE}

How to change the IP
> echo -e 'AUTHENTICATE “xxxxxxxxx"\r\nsignal NEWNYM\r\nQUIT' | nc localhost 9051

Check the IP
> curl -x localhost:8118 http://icanhazip.com/

How to change the IP from remote
> echo -e 'AUTHENTICATE “xxxxxxxxx"\r\nsignal NEWNYM\r\nQUIT' | nc ubuntu-master 9051

Check the IP from Remote
> curl -x ubuntu-master:8118 http://icanhazip.com/


References:
http://sillycat.iteye.com/blog/2418229
http://sillycat.iteye.com/blog/2418353
http://sillycat.iteye.com/blog/2422861

https://github.com/torproject/tor/blob/master/src/config/torrc.sample.in
https://github.com/osminogin/docker-tor-simple
https://github.com/dperson/torproxy
https://program-think.blogspot.com/2014/12/gfw-privoxy.html
http://www.cnblogs.com/another-wheel/archive/2011/11/16/setup-http-proxy-via-socks-and-privoxy.html
https://blog.phpgao.com/privoxy-shadowsocks.html
https://blog.csdn.net/north_eagle/article/details/8683403
https://stackoverflow.com/questions/15459170/trouble-authenticating-tor-with-python
https://gist.github.com/DusanMadar/c1155329cf6a71e4346cae271a2eafd3
https://www.torproject.org/docs/faq.html.en
https://stackoverflow.com/questions/45901892/how-to-connect-to-tor-control-port-9051-from-a-remote-host
https://stackoverflow.com/questions/25775266/how-to-keep-docker-container-running-after-starting-services/36872226