为OpenStreetMap瓦片服务器添加3弧秒全球DEM图层

在上一篇文章里，我们试着测试了局部3弧秒(即1/1200度)分辨率的 DEM等高线嵌入OSM图层的效果。下面，来分享一下批量导入全球的DEM数据。
经笔者测试，如果一次性导入全球OpenStreetMap PBF（32GB，2017-01）、全球3弧秒 DEM（hgt压缩包转为PBF 约28GB），需要一块 2TB 的 SSD。尽管最终的数据库规模可通过缩减中间表，压缩到300GB，但导入过程中会吃掉约1TB多的空间。若在导入OSM后进行一次收缩，而后导入DEM，则1TB SSD 勉强刚好。

导入全球数据的原始DEM、转换后PBF、导入后数据库镜像(pg_dump)以及完整虚拟机镜像可从ArchOSM共享地址下载。为了多线程压缩，其使用了7z16.0版本LZMA2算法。解压缩需要512MB空闲内存。我的缓存服务器位置在这里RaspberryPI OSM 缓存服务器，如果渲染服务器没有开机，仅能看到缓存的数据。

1.使用下载工具批量下载DEM压缩包

有心人会注意到，在使用phyghtmap下载局部数据时，会得到一个任务列表，位于hgt文件夹下。
viewfinderHgtIndex_3.txt
这个文件里记录了所有的原始数据地址，类似：

# VIEW3 index file, VERSION=4
...
[http://viewfinderpanoramas.org/ANTDEM3/01-15.zip]
...
S61W091
...

中括号里的都是地址。二话不说，复制所有内容到 Wps 电子表格，排序，便收集到了所有的网址。查找替换清除[]后，在下载工具里批量下载即可。

2.从本地FTP服务器创建全球高程PBF

下载完毕后，把所有ZIP文件拷贝到一个独立的文件夹，设置为FTP地址。而后，把上述文件viewfinderHgtIndex_3.txt的所有地址替换为本地FTP。这样处理后，下载全球数据就变得非常迅速啦。
修改好后的viewfinderHgtIndex_3.txt类似下面的样子。

# VIEW3 index file, VERSION=4
[ftp://192.168.1.10/zip/01-15.zip]
...
[ftp://192.168.1.10/zip/16-30.zip]
...

使用命令生成全球PBF

phyghtmap -a -180:-85:180:85 --source=view3 --pbf --max-nodes-per-tile 0

此时，该程序会自动读取已经存在的viewfinderHgtIndex_3.txt，从本地服务器下载、执行。

转换过程需要约1天，最终生成一个 PBF文件，28GB

3.导入全球PBF

留好磁盘空间，而后，进行导入：

[archosm@archosm ~]$  osm2pgsql -c -s -S"./contour/contour.style" -C24000 -dcontours --drop --flat-nodes "./cache/nodes/flat_node" ./contour/lon-180.00_180.00lat-85.00_85.00_view3.osm.pbf
osm2pgsql version 0.93.0-dev (64 bit id space)

Using built-in tag processing pipeline
Using projection SRS 3857 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=24000MB, maxblocks=384000*65536, allocation method=11
Mid: loading persistent node cache from ./cache/nodes/flat_node
Allocated space for persistent node cache file
Maximum node in persistent node cache: 0
Mid: pgsql, scale=100 cache=24000
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels

Reading in file: ./contour/lon-180.00_180.00lat-85.00_85.00_view3.osm.pbf
Using PBF parser.
Processing: Node(10198222k 6803.4k/s) Way(141003k 11.63k/s) Relation(0 0.00/s)  parse time: 13621s
Node stats: total(10198222219), max(10208222218) in 1499s
Way stats: total(141003653), max(151003652) in 12122s
Relation stats: total(0), max(0) in 0s
Maximum node in persistent node cache: 10208935935
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Mid: loading persistent node cache from ./cache/nodes/flat_node
Maximum node in persistent node cache: 10208935935
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline
Mid: loading persistent node cache from ./cache/nodes/flat_node
Maximum node in persistent node cache: 10208935935
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline
Mid: loading persistent node cache from ./cache/nodes/flat_node
Maximum node in persistent node cache: 10208935935
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline
Mid: loading persistent node cache from ./cache/nodes/flat_node
Maximum node in persistent node cache: 10208935935
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline

Going over pending ways...
        0 ways are pending

Using 4 helper-processes
Finished processing 0 ways in 0 s

Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads

Going over pending relations...
        0 relations are pending

Using 4 helper-processes
Finished processing 0 relations in 0 s

Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Sorting data and creating indexes for planet_osm_polygon
Stopping table: planet_osm_nodes
Sorting data and creating indexes for planet_osm_roads
Sorting data and creating indexes for planet_osm_point
Stopping table: planet_osm_ways
Stopping table: planet_osm_rels
Sorting data and creating indexes for planet_osm_line
Stopped table: planet_osm_rels in 0s
Stopped table: planet_osm_nodes in 0s
Copying planet_osm_polygon to cluster by geometry finished
Creating geometry index on planet_osm_polygon
Copying planet_osm_roads to cluster by geometry finished
Creating geometry index on planet_osm_roads
Creating indexes on planet_osm_polygon finished
Creating indexes on planet_osm_roads finished
Copying planet_osm_point to cluster by geometry finished
Creating geometry index on planet_osm_point
Creating indexes on planet_osm_point finished
All indexes on planet_osm_point created in 0s
Completed planet_osm_point
All indexes on planet_osm_roads created in 0s
Completed planet_osm_roads
All indexes on planet_osm_polygon created in 0s
Completed planet_osm_polygon
Stopped table: planet_osm_ways in 1s
Copying planet_osm_line to cluster by geometry finished
Creating geometry index on planet_osm_line
Creating indexes on planet_osm_line finished
All indexes on planet_osm_line created in 6804s
Completed planet_osm_line
Maximum node in persistent node cache: 10208935935
Maximum node in persistent node cache: 10208935935
Maximum node in persistent node cache: 10208935935
Maximum node in persistent node cache: 10208935935
node cache: stored: 3145729419(30.85%), storage efficiency: 100.00% (dense blocks: 384000, sparse nodes: 2432), hit rate: 30.78%

Osm2pgsql took 20429s overall

4 显示效果

显示效果还是很棒的，我们看看南美洲的山脉

连南极都有了