- 博客(8)
- 收藏
- 关注
RSS订阅原创 python multiprocess
import multiprocessingimport numpy as npdata_pairs = [ [3,5], [4,3], [7,3], [1,6] ]# define what to do with each data pair ( p=[3,5] ), example: calculate productdef myfunc(p): product_of_list = np.prod(p) return product_of_listif __name__
2020-05-27 17:03:02
170
原创 分布式计算学习第二坑 cudnn
andrew@1manjaro:~/mount/arch/TensorFlowOnSpark/examples/resnet#export TF_CONFIG='{"cluster": { "chief": ["localhost:2222"], "worker": ["localhost:2223"]}, "task": {"type": "chief", "index": 0}}'python resnet_cifar_main.py --data_dir=${CIFAR_DATA} --num_gp
2020-05-26 10:45:03
653
原创 分布式tensorflow学习踩坑
环境：macOS 10.15.4python 3.7.4spark 3.0.2_previewtensorflow 2.1.0temsorflowONsparkandrew@1macbook:~/tensorflowonspark/TensorFlowOnSpark/examples/resnet#${SPARK_HOME}/bin/spark-submit --master ${MASTER} --conf spark.cores.max=${TOTAL_CORES} --conf spark
2020-05-25 15:02:09
262
1
原创 Conv2DCustomBackpropInputOp only supports NHWC.
tensorflow.python.framework.errors_impl.InvalidArgumentError: Conv2DCustomBackpropInputOp only supports NHWC.[[node Conv2DBackpropInput (defined at /lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]] [Op:__inference_distributed_
2020-05-25 14:30:28
5965
4
原创 pyspark入门
In [5]: from pyspark import SparkContext ...: sc = SparkContext("local", "First App") ---------------------------------------------------------------------------ValueError
2020-05-22 21:15:42
177
原创 使用pyspark读取hive数据
之前试过pyhive直接读取hive数据，有几个依赖一直装不上，经过几天的摸索，终于使用spark自带的sql支持能够成功读取hive的数据。其中的坑很大，有很多配置文件需要写，特此记录一下。第一步全局环境变量：vim ～/.bash_profile(mac)vim ~/.bashrc(linux)配置好hadoop spark hive java的环境变量以及pyspark的默认python解释器的路径和pyspark-shell的ippython路径，不然后报错。 5 ##homebre
2020-05-20 11:24:01
3366
原创 hadoop Permission denied (publickey,password,keyboard-interactive).
andrew@localhost: Permission denied (publickey,password,keyboard-interactive).ssh localhost没有设置免密码登入ssh localhostPassword:Last login: Wed May 20 10:48:23 2020 from ::1ssh-kengen生成密码之后ssh-keygenGenerating public/private rsa key pair.Enter file in w
2020-05-20 10:56:00
2630
原创 解决远程连接mac后中文字体变？？？的问题
打开 vim ~/.zshrc在上面输入export LANG=zh_CN.UTF-8esc + ：wq 退出然会source ~/.zshrc就可以愉快是应用中文了
2020-01-05 09:48:07
409
空空如也
空空如也
TA创建的收藏夹 TA关注的收藏夹
TA关注的人