/home/wagnchogn/anaconda3/envs/deeplabcut/bin/python "/media/wagnchogn/data/dlc/demo/multi mouse.py"
2024-04-25 16:58:04.881081: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-25 16:58:05.028871: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-25 16:58:05.063917: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-04-25 16:58:05.586315: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory
2024-04-25 16:58:05.586373: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory
2024-04-25 16:58:05.586380: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Loading DLC 2.3.9...
DLC loaded in light mode; you cannot use any GUI (labeling, relabeling and standalone GUI)
Selecting multi-animal trainer
Activating limb prediction...
Config:
{'all_joints': [[0], [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11]],
'all_joints_names': ['snout',
'leftear',
'rightear',
'shoulder',
'spine1',
'spine2',
'spine3',
'spine4',
'tailbase',
'tail1',
'tail2',
'tailend'],
'alpha_r': 0.02,
'apply_prob': 0.5,
'batch_size': 8,
'contrast': {'clahe': True,
'claheratio': 0.1,
'histeq': True,
'histeqratio': 0.1},
'convolution': {'edge': False,
'emboss': {'alpha': [0.0, 1.0], 'strength': [0.5, 1.5]},
'embossratio': 0.1,
'sharpen': False,
'sharpenratio': 0.3},
'crop_pad': 0,
'crop_sampling': 'hybrid',
'crop_size': [400, 400],
'cropratio': 0.4,
'dataset': 'training-datasets/iteration-0/UnaugmentedDataSet_trimiceJun22/trimice_dlc90shuffle1.pickle',
'dataset_type': 'multi-animal-imgaug',
'decay_steps': 30000,
'deterministic': False,
'display_iters': 500,
'fg_fraction': 0.25,
'global_scale': 0.8,
'init_weights': '/home/wagnchogn/anaconda3/envs/deeplabcut/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt',
'intermediate_supervision': False,
'intermediate_supervision_layer': 12,
'location_refinement': True,
'locref_huber_loss': True,
'locref_loss_weight': 0.05,
'locref_stdev': 7.2801,
'log_dir': 'log',
'lr_init': 0.0005,
'max_input_size': 1500,
'max_shift': 0.4,
'mean_pixel': [123.68, 116.779, 103.939],
'metadataset': 'training-datasets/iteration-0/UnaugmentedDataSet_trimiceJun22/Documentation_data-trimice_90shuffle1.pickle',
'min_input_size': 64,
'mirror': False,
'multi_stage': False,
'multi_step': [[0.0001, 7500], [5e-05, 12000], [1e-05, 200000]],
'net_type': 'resnet_50',
'num_idchannel': 0,
'num_joints': 12,
'num_limbs': 66,
'optimizer': 'adam',
'pafwidth': 20,
'pairwise_huber_loss': False,
'pairwise_loss_weight': 0.1,
'pairwise_predict': False,
'partaffinityfield_graph': [[0, 1],
[0, 2],
[0, 3],
[0, 4],
[0, 5],
[0, 6],
[0, 7],
[0, 8],
[0, 9],
[0, 10],
[0, 11],
[1, 2],
[1, 3],
[1, 4],
[1, 5],
[1, 6],
[1, 7],
[1, 8],
[1, 9],
[1, 10],
[1, 11],
[2, 3],
[2, 4],
[2, 5],
[2, 6],
[2, 7],
[2, 8],
[2, 9],
[2, 10],
[2, 11],
[3, 4],
[3, 5],
[3, 6],
[3, 7],
[3, 8],
[3, 9],
[3, 10],
[3, 11],
[4, 5],
[4, 6],
[4, 7],
[4, 8],
[4, 9],
[4, 10],
[4, 11],
[5, 6],
[5, 7],
[5, 8],
[5, 9],
[5, 10],
[5, 11],
[6, 7],
[6, 8],
[6, 9],
[6, 10],
[6, 11],
[7, 8],
[7, 9],
[7, 10],
[7, 11],
[8, 9],
[8, 10],
[8, 11],
[9, 10],
[9, 11],
[10, 11]],
'partaffinityfield_predict': True,
'pos_dist_thresh': 17,
'pre_resize': [],
'project_path': '/media/wagnchogn/data/0419/trimice-dlc-2021-06-22',
'regularize': False,
'rotation': 25,
'rotratio': 0.4,
'save_iters': 10000,
'scale_jitter_lo': 0.5,
'scale_jitter_up': 1.25,
'scoremap_dir': 'test',
'shuffle': True,
'snapshot_prefix': '/media/wagnchogn/data/0419/trimice-dlc-2021-06-22/dlc-models/iteration-0/trimiceJun22-trainset90shuffle1/train/snapshot',
'stride': 8.0,
'weigh_negatives': False,
'weigh_only_present_joints': False,
'weigh_part_predictions': False,
'weight_decay': 0.0001}
Batch Size is 8
Getting specs multi-animal-imgaug 66 12
/home/wagnchogn/anaconda3/envs/deeplabcut/lib/python3.9/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:1694: UserWarning: `layer.apply` is deprecated and will be removed in a future version. Please use `layer.__call__` method instead.
warnings.warn('`layer.apply` is deprecated and '
2024-04-25 16:58:11.137485: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-25 16:58:12.571791: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2024-04-25 16:58:12.571846: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21350 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:73:00.0, compute capability: 8.6
2024-04-25 16:58:12.573202: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2024-04-25 16:58:12.573228: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 22292 MB memory: -> device: 1, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:d5:00.0, compute capability: 8.6
Loading ImageNet-pretrained resnet_50
2024-04-25 16:58:13.435951: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21350 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:73:00.0, compute capability: 8.6
2024-04-25 16:58:13.436949: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 22292 MB memory: -> device: 1, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:d5:00.0, compute capability: 8.6
2024-04-25 16:58:14.719710: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled
Max_iters overwritten as 200000
Training parameters:
{'stride': 8.0, 'weigh_part_predictions': False, 'weigh_negatives': False, 'fg_fraction': 0.25, 'mean_pixel': [123.68, 116.779, 103.939], 'shuffle': True, 'snapshot_prefix': '/media/wagnchogn/data/0419/trimice-dlc-2021-06-22/dlc-models/iteration-0/trimiceJun22-trainset90shuffle1/train/snapshot', 'log_dir': 'log', 'global_scale': 0.8, 'location_refinement': True, 'locref_stdev': 7.2801, 'locref_loss_weight': 0.05, 'locref_huber_loss': True, 'optimizer': 'adam', 'intermediate_supervision': False, 'intermediate_supervision_layer': 12, 'regularize': False, 'weight_decay': 0.0001, 'crop_pad': 0, 'scoremap_dir': 'test', 'batch_size': 8, 'dataset_type': 'multi-animal-imgaug', 'deterministic': False, 'mirror': False, 'pairwise_huber_loss': False, 'weigh_only_present_joints': False, 'partaffinityfield_predict': True, 'pairwise_predict': True, 'all_joints': [[0], [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11]], 'all_joints_names': ['snout', 'leftear', 'rightear', 'shoulder', 'spine1', 'spine2', 'spine3', 'spine4', 'tailbase', 'tail1', 'tail2', 'tailend'], 'alpha_r': 0.02, 'apply_prob': 0.5, 'contrast': {'clahe': True, 'claheratio': 0.1, 'histeq': True, 'histeqratio': 0.1, 'gamma': False, 'sigmoid': False, 'log': False, 'linear': False}, 'convolution': {'edge': False, 'emboss': {'alpha': [0.0, 1.0], 'strength': [0.5, 1.5]}, 'embossratio': 0.1, 'sharpen': False, 'sharpenratio': 0.3}, 'crop_sampling': 'hybrid', 'crop_size': [400, 400], 'cropratio': 0.4, 'dataset': 'training-datasets/iteration-0/UnaugmentedDataSet_trimiceJun22/trimice_dlc90shuffle1.pickle', 'decay_steps': 30000, 'display_iters': 500, 'init_weights': '/home/wagnchogn/anaconda3/envs/deeplabcut/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt', 'lr_init': 0.0005, 'max_input_size': 1500, 'max_shift': 0.4, 'metadataset': 'training-datasets/iteration-0/UnaugmentedDataSet_trimiceJun22/Documentation_data-trimice_90shuffle1.pickle', 'min_input_size': 64, 'multi_stage': False, 'multi_step': [[0.0001, 7500], [5e-05, 12000], [1e-05, 200000]], 'net_type': 'resnet_50', 'num_idchannel': 0, 'num_joints': 12, 'num_limbs': 66, 'pafwidth': 20, 'pairwise_loss_weight': 0.1, 'partaffinityfield_graph': [[0, 1], [0, 2], [0, 3], [0, 4], [0, 5], [0, 6], [0, 7], [0, 8], [0, 9], [0, 10], [0, 11], [1, 2], [1, 3], [1, 4], [1, 5], [1, 6], [1, 7], [1, 8], [1, 9], [1, 10], [1, 11], [2, 3], [2, 4], [2, 5], [2, 6], [2, 7], [2, 8], [2, 9], [2, 10], [2, 11], [3, 4], [3, 5], [3, 6], [3, 7], [3, 8], [3, 9], [3, 10], [3, 11], [4, 5], [4, 6], [4, 7], [4, 8], [4, 9], [4, 10], [4, 11], [5, 6], [5, 7], [5, 8], [5, 9], [5, 10], [5, 11], [6, 7], [6, 8], [6, 9], [6, 10], [6, 11], [7, 8], [7, 9], [7, 10], [7, 11], [8, 9], [8, 10], [8, 11], [9, 10], [9, 11], [10, 11]], 'pos_dist_thresh': 17, 'pre_resize': [], 'project_path': '/media/wagnchogn/data/0419/trimice-dlc-2021-06-22', 'rotation': 25, 'rotratio': 0.4, 'save_iters': 10000, 'scale_jitter_lo': 0.5, 'scale_jitter_up': 1.25, 'pseudo_threshold': 0.1, 'video_path': '', 'traintime_resize': False}
Starting multi-animal training....
2024-04-25 16:58:21.438154: I tensorflow/stream_executor/cuda/cuda_dnn.cc:384] Loaded cuDNN version 8100
2024-04-25 16:58:22.379632: I tensorflow/core/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory
2024-04-25 16:58:23.081755: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
iteration: 500 loss: 0.0326 scmap loss: 0.0274 locref loss: 0.0009 limb loss: 0.0042 lr: 0.0001
iteration: 1000 loss: 0.0120 scmap loss: 0.0105 locref loss: 0.0003 limb loss: 0.0012 lr: 0.0001
iteration: 1500 loss: 0.0091 scmap loss: 0.0080 locref loss: 0.0002 limb loss: 0.0009 lr: 0.0001
iteration: 2000 loss: 0.0080 scmap loss: 0.0071 locref loss: 0.0002 limb loss: 0.0007 lr: 0.0001
iteration: 2500 loss: 0.0073 scmap loss: 0.0065 locref loss: 0.0002 limb loss: 0.0006 lr: 0.0001
iteration: 3000 loss: 0.0067 scmap loss: 0.0060 locref loss: 0.0001 limb loss: 0.0005 lr: 0.0001
iteration: 3500 loss: 0.0065 scmap loss: 0.0058 locref loss: 0.0001 limb loss: 0.0005 lr: 0.0001
iteration: 4000 loss: 0.0062 scmap loss: 0.0056 locref loss: 0.0001 limb loss: 0.0005 lr: 0.0001
iteration: 4500 loss: 0.0059 scmap loss: 0.0053 locref loss: 0.0001 limb loss: 0.0005 lr: 0.0001
iteration: 5000 loss: 0.0058 scmap loss: 0.0052 locref loss: 0.0001 limb loss: 0.0004 lr: 0.0001
iteration: 5500 loss: 0.0058 scmap loss: 0.0053 locref loss: 0.0001 limb loss: 0.0004 lr: 0.0001
iteration: 6000 loss: 0.0053 scmap loss: 0.0048 locref loss: 0.0001 limb loss: 0.0004 lr: 0.0001
iteration: 6500 loss: 0.0053 scmap loss: 0.0048 locref loss: 0.0001 limb loss: 0.0004 lr: 0.0001
iteration: 7000 loss: 0.0052 scmap loss: 0.0047 locref loss: 0.0001 limb loss: 0.0004 lr: 0.0001
iteration: 7500 loss: 0.0051 scmap loss: 0.0046 locref loss: 0.0001 limb loss: 0.0004 lr: 0.0001
iteration: 8000 loss: 0.0045 scmap loss: 0.0041 locref loss: 0.0001 limb loss: 0.0003 lr: 5e-05
iteration: 8500 loss: 0.0044 scmap loss: 0.0040 locref loss: 0.0001 limb loss: 0.0003 lr: 5e-05
iteration: 9000 loss: 0.0043 scmap loss: 0.0039 locref loss: 0.0001 limb loss: 0.0003 lr: 5e-05
iteration: 9500 loss: 0.0044 scmap loss: 0.0040 locref loss: 0.0001 limb loss: 0.0003 lr: 5e-05
iteration: 10000 loss: 0.0043 scmap loss: 0.0039 locref loss: 0.0001 limb loss: 0.0003 lr: 5e-05
iteration: 10500 loss: 0.0041 scmap loss: 0.0038 locref loss: 0.0001 limb loss: 0.0003 lr: 5e-05
iteration: 11000 loss: 0.0041 scmap loss: 0.0037 locref loss: 0.0001 limb loss: 0.0003 lr: 5e-05
iteration: 11500 loss: 0.0041 scmap loss: 0.0037 locref loss: 0.0001 limb loss: 0.0003 lr: 5e-05
iteration: 12000 loss: 0.0040 scmap loss: 0.0037 locref loss: 0.0001 limb loss: 0.0003 lr: 5e-05
iteration: 12500 loss: 0.0037 scmap loss: 0.0034 locref loss: 0.0001 limb loss: 0.0003 lr: 1e-05
iteration: 13000 loss: 0.0036 scmap loss: 0.0033 locref loss: 0.0001 limb loss: 0.0003 lr: 1e-05
iteration: 13500 loss: 0.0035 scmap loss: 0.0032 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 14000 loss: 0.0034 scmap loss: 0.0031 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 14500 loss: 0.0036 scmap loss: 0.0032 locref loss: 0.0001 limb loss: 0.0003 lr: 1e-05
iteration: 15000 loss: 0.0035 scmap loss: 0.0031 locref loss: 0.0001 limb loss: 0.0003 lr: 1e-05
iteration: 15500 loss: 0.0034 scmap loss: 0.0031 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 16000 loss: 0.0034 scmap loss: 0.0030 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 16500 loss: 0.0034 scmap loss: 0.0031 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 17000 loss: 0.0033 scmap loss: 0.0030 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 17500 loss: 0.0033 scmap loss: 0.0030 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 18000 loss: 0.0033 scmap loss: 0.0030 locref loss: 0.0001 limb loss: 0.0003 lr: 1e-05
iteration: 18500 loss: 0.0032 scmap loss: 0.0029 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 19000 loss: 0.0032 scmap loss: 0.0029 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 19500 loss: 0.0033 scmap loss: 0.0030 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 20000 loss: 0.0032 scmap loss: 0.0029 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 20500 loss: 0.0032 scmap loss: 0.0028 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 21000 loss: 0.0031 scmap loss: 0.0028 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 21500 loss: 0.0032 scmap loss: 0.0029 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 22000 loss: 0.0031 scmap loss: 0.0028 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 22500 loss: 0.0031 scmap loss: 0.0028 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 23000 loss: 0.0031 scmap loss: 0.0028 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 23500 loss: 0.0031 scmap loss: 0.0028 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 24000 loss: 0.0031 scmap loss: 0.0028 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 24500 loss: 0.0031 scmap loss: 0.0028 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 25000 loss: 0.0030 scmap loss: 0.0027 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 25500 loss: 0.0031 scmap loss: 0.0028 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 26000 loss: 0.0030 scmap loss: 0.0027 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 26500 loss: 0.0030 scmap loss: 0.0027 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 27000 loss: 0.0030 scmap loss: 0.0027 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 27500 loss: 0.0030 scmap loss: 0.0027 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 28000 loss: 0.0030 scmap loss: 0.0027 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 28500 loss: 0.0030 scmap loss: 0.0027 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 29000 loss: 0.0030 scmap loss: 0.0027 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 29500 loss: 0.0029 scmap loss: 0.0026 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 30000 loss: 0.0029 scmap loss: 0.0026 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 30500 loss: 0.0029 scmap loss: 0.0026 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 31000 loss: 0.0029 scmap loss: 0.0026 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 31500 loss: 0.0028 scmap loss: 0.0025 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 32000 loss: 0.0028 scmap loss: 0.0025 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 32500 loss: 0.0029 scmap loss: 0.0026 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 33000 loss: 0.0028 scmap loss: 0.0025 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 33500 loss: 0.0029 scmap loss: 0.0026 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 34000 loss: 0.0028 scmap loss: 0.0025 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 34500 loss: 0.0028 scmap loss: 0.0025 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 35000 loss: 0.0028 scmap loss: 0.0025 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 35500 loss: 0.0028 scmap loss: 0.0025 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 36000 loss: 0.0027 scmap loss: 0.0025 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 36500 loss: 0.0027 scmap loss: 0.0025 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 37000 loss: 0.0027 scmap loss: 0.0024 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 37500 loss: 0.0027 scmap loss: 0.0024 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 38000 loss: 0.0027 scmap loss: 0.0024 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 38500 loss: 0.0027 scmap loss: 0.0024 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 39000 loss: 0.0027 scmap loss: 0.0024 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 39500 loss: 0.0026 scmap loss: 0.0024 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 40000 loss: 0.0027 scmap loss: 0.0024 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 40500 loss: 0.0027 scmap loss: 0.0024 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 41000 loss: 0.0026 scmap loss: 0.0024 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 41500 loss: 0.0026 scmap loss: 0.0024 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 42000 loss: 0.0026 scmap loss: 0.0023 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 42500 loss: 0.0026 scmap loss: 0.0023 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 43000 loss: 0.0027 scmap loss: 0.0024 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 43500 loss: 0.0025 scmap loss: 0.0023 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 44000 loss: 0.0026 scmap loss: 0.0024 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 44500 loss: 0.0026 scmap loss: 0.0023 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 45000 loss: 0.0026 scmap loss: 0.0024 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 45500 loss: 0.0026 scmap loss: 0.0023 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 46000 loss: 0.0026 scmap loss: 0.0023 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 46500 loss: 0.0026 scmap loss: 0.0023 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 47000 loss: 0.0026 scmap loss: 0.0023 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 47500 loss: 0.0026 scmap loss: 0.0023 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 48000 loss: 0.0025 scmap loss: 0.0023 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 48500 loss: 0.0025 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 49000 loss: 0.0026 scmap loss: 0.0023 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 49500 loss: 0.0025 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 50000 loss: 0.0025 scmap loss: 0.0023 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 50500 loss: 0.0025 scmap loss: 0.0023 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 51000 loss: 0.0025 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 51500 loss: 0.0025 scmap loss: 0.0023 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 52000 loss: 0.0025 scmap loss: 0.0023 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 52500 loss: 0.0025 scmap loss: 0.0023 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 53000 loss: 0.0025 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 53500 loss: 0.0025 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 54000 loss: 0.0025 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 54500 loss: 0.0024 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 55000 loss: 0.0025 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 55500 loss: 0.0024 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 56000 loss: 0.0024 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 56500 loss: 0.0025 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 57000 loss: 0.0024 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 57500 loss: 0.0024 scmap loss: 0.0021 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 58000 loss: 0.0024 scmap loss: 0.0021 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 58500 loss: 0.0024 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 59000 loss: 0.0024 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 59500 loss: 0.0024 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 60000 loss: 0.0024 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 60500 loss: 0.0025 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 61000 loss: 0.0024 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 61500 loss: 0.0024 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 62000 loss: 0.0024 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 62500 loss: 0.0024 scmap loss: 0.0021 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 63000 loss: 0.0024 scmap loss: 0.0022 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 63500 loss: 0.0024 scmap loss: 0.0021 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 64000 loss: 0.0024 scmap loss: 0.0021 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 64500 loss: 0.0023 scmap loss: 0.0021 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 65000 loss: 0.0024 scmap loss: 0.0021 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 65500 loss: 0.0023 scmap loss: 0.0021 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 66000 loss: 0.0023 scmap loss: 0.0021 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 66500 loss: 0.0024 scmap loss: 0.0021 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 67000 loss: 0.0023 scmap loss: 0.0021 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 67500 loss: 0.0023 scmap loss: 0.0021 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 68000 loss: 0.0023 scmap loss: 0.0021 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 68500 loss: 0.0023 scmap loss: 0.0020 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 69000 loss: 0.0023 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 69500 loss: 0.0023 scmap loss: 0.0021 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 70000 loss: 0.0023 scmap loss: 0.0020 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 70500 loss: 0.0023 scmap loss: 0.0021 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 71000 loss: 0.0023 scmap loss: 0.0021 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 71500 loss: 0.0023 scmap loss: 0.0021 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 72000 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 72500 loss: 0.0023 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 73000 loss: 0.0023 scmap loss: 0.0021 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 73500 loss: 0.0023 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 74000 loss: 0.0023 scmap loss: 0.0021 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 74500 loss: 0.0023 scmap loss: 0.0020 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 75000 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 75500 loss: 0.0023 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 76000 loss: 0.0023 scmap loss: 0.0020 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 76500 loss: 0.0023 scmap loss: 0.0020 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 77000 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 77500 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 78000 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 78500 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 79000 loss: 0.0023 scmap loss: 0.0020 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 79500 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 80000 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 80500 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 81000 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 81500 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 82000 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 82500 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 83000 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 83500 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 84000 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 84500 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 85000 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 85500 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 86000 loss: 0.0022 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 86500 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 87000 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 87500 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 88000 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 88500 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 89000 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 89500 loss: 0.0022 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 90000 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 90500 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 91000 loss: 0.0022 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 91500 loss: 0.0022 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 92000 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 92500 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 93000 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 93500 loss: 0.0022 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 94000 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0001 limb loss: 0.0002 lr: 1e-05
iteration: 94500 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 95000 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 95500 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 96000 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 96500 loss: 0.0022 scmap loss: 0.0020 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 97000 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 97500 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 98000 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 98500 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 99000 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 99500 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 100000 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 100500 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 101000 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 101500 loss: 0.0021 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 102000 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 102500 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 103000 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 103500 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 104000 loss: 0.0021 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 104500 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 105000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 105500 loss: 0.0021 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 106000 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 106500 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 107000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 107500 loss: 0.0021 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 108000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 108500 loss: 0.0021 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 109000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 109500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 110000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 110500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 111000 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 111500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 112000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 112500 loss: 0.0021 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 113000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 113500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 114000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 114500 loss: 0.0021 scmap loss: 0.0019 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 115000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 115500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 116000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 116500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 117000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 117500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 118000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 118500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 119000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 119500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 120000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 120500 loss: 0.0020 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 121000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 121500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 122000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 122500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 123000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 123500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 124000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 124500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 125000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 125500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 126000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 126500 loss: 0.0020 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 127000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 127500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 128000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 128500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 129000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 129500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 130000 loss: 0.0020 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 130500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 131000 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 131500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 132000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 132500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 133000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 133500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 134000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 134500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 135000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 135500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 136000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 136500 loss: 0.0020 scmap loss: 0.0018 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 137000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 137500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 138000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 138500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 139000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 139500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 140000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 140500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 141000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 141500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 142000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 142500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 143000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 143500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 144000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 144500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 145000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 145500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 146000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 146500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 147000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 147500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 148000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 148500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 149000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 149500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 150000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 150500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 151000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 151500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 152000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 152500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 153000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 153500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 154000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 154500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 155000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 155500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 156000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 156500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 157000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 157500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 158000 loss: 0.0018 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 158500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 159000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 159500 loss: 0.0018 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 160000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 160500 loss: 0.0018 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 161000 loss: 0.0018 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 161500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 162000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 162500 loss: 0.0018 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 163000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 163500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 164000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 164500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 165000 loss: 0.0018 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 165500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 166000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 166500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 167000 loss: 0.0018 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 167500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 168000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 168500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 169000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 169500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 170000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 170500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 171000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 171500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 172000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 172500 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 173000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 173500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 174000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 174500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 175000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 175500 loss: 0.0018 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 176000 loss: 0.0019 scmap loss: 0.0017 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 176500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 177000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 177500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 178000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 178500 loss: 0.0017 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 179000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 179500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 180000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 180500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 181000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 181500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 182000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 182500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 183000 loss: 0.0017 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 183500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 184000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 184500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 185000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0002 lr: 1e-05
iteration: 185500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 186000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 186500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 187000 loss: 0.0017 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 187500 loss: 0.0017 scmap loss: 0.0015 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 188000 loss: 0.0017 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 188500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 189000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 189500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 190000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 190500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 191000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 191500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 192000 loss: 0.0017 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 192500 loss: 0.0017 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 193000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 193500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 194000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 194500 loss: 0.0017 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 195000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 195500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 196000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 196500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 197000 loss: 0.0017 scmap loss: 0.0015 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 197500 loss: 0.0017 scmap loss: 0.0015 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 198000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 198500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 199000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 199500 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
iteration: 200000 loss: 0.0018 scmap loss: 0.0016 locref loss: 0.0000 limb loss: 0.0001 lr: 1e-05
Exception in thread Thread-1:
Traceback (most recent call last):
File "/home/wagnchogn/anaconda3/envs/deeplabcut/lib/python3.9/threading.py", line 980, in _bootstrap_inner
self.run()
File "/home/wagnchogn/anaconda3/envs/deeplabcut/lib/python3.9/threading.py", line 917, in run
self._target(*self._args, **self._kwargs)
File "/home/wagnchogn/anaconda3/envs/deeplabcut/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/core/train.py", line 86, in load_and_enqueue
sess.run(enqueue_op, feed_dict=food)
File "/home/wagnchogn/anaconda3/envs/deeplabcut/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 968, in run
result = self._run(None, fetches, feed_dict, options_ptr,
File "/home/wagnchogn/anaconda3/envs/deeplabcut/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1116, in _run
raise RuntimeError('Attempted to use a closed Session.')
RuntimeError: Attempted to use a closed Session.
The network is now trained and ready to evaluate. Use the function 'evaluate_network' to evaluate the network.
Process finished with exit code 0
train result
最新推荐文章于 2024-05-27 17:00:50 发布
6万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
