Those of you who know me will be familiar with the fact that I am what could be called a ‘digital packrat’. After a few years of university my DVD/Blu-Ray collection has expanded to the point that it fills multiple shelves, and I have digitised the whole thing.
那些认识我的人会熟悉这样的事实,即我就是所谓的“数字背包”。 经过几年的大学学习,我的DVD /蓝光碟片收藏已经扩展到可以装满多个架子的程度,而且我已经将整个东西数字化了。
This has turned into a more than 1.3 TB media collection, which has outgrown the 1.5 TB drive it has been stored on. For this reason, I need a more effective storage solution. This solution needs to be sustainable: I can’t keep copying the data onto ever-larger single hard drives. It also needs to be safe: I can’t rely on the occasional backup to another similarly-sized hard drive in another machine.
这已经变成了超过1.3 TB的介质集合,而该存储库的容量已经超过了其存储的1.5 TB驱动器。 因此,我需要一个更有效的存储解决方案。 该解决方案必须具有可持续性:我无法将数据复制到更大的单个硬盘驱动器上。 它还需要安全:我不能依赖偶尔备份到另一台计算机上另一个大小相似的硬盘上。
Thus, the decision was made: have a dedicated fileserver. It needs to be easily expandable, it needs to have data integrity built in, and it needs to be cheap. I’ve finally gotten around to sorting this out, and have documented the construction for posterity.
因此,决定:拥有专用的文件服务器。 它需要易于扩展,需要内置数据完整性,并且价格便宜。 我终于解决了这个问题,并记录了后代的构造。
Alright, enough chatter. Let’s get to it!
好吧,enough不休。 让我们开始吧!
建造 (The Build)
Let’s begin. First, obtain your fileserver. I got an HP Proliant Microserver from Ebuyer, which came with a £110 cashback meaning I got it dirt cheap. It has all the things you need from a fileserver: easy-access and plentiful drive bays, quiet fans and enough RAM to run a modern filesystem.
让我们开始。 首先,获取您的文件服务器。 我从Ebuyer买了HP Proliant微型服务器,并获得了110英镑的现金返还,这让我觉得价格便宜。 它具有文件服务器所需的所有功能:易于访问的大量驱动器托架,安静的风扇和足够的RAM以运行现代文件系统。
Next, obtain your disks. This is actually the part that requires the most thought. You’ll see later in this post that we will be using a filesystem that allows us to replace disks to increase the amount of space available to us, so we should plan for the short-term, with the awareness that we can expand in the long-term. For that reason, I got two 2TB WD Caviar Green drives. You can buy any drives that suit you, but bear in mind that you’ll be running this server for long periods of time, so you will probably want to get energy-efficient drives as much as possible.
接下来,获取您的磁盘。 这实际上是最需要思考的部分。 您将在本文的后面看到,我们将使用一个文件系统,该文件系统允许我们替换磁盘以增加对我们的可用空间,因此我们应该为短期计划,并意识到我们可以在长期。 因此,我得到了两个2TB WD Caviar Green驱动器。 您可以购买任何适合您的驱动器,但请记住,您将长时间运行此服务器,因此您可能希望尽可能多地获得节能驱动器。
Note also that you will need a drive for the operating system. You could mount a drive in the CD bay, or do what I did and use a USB stick to run the OS.
另请注意,您将需要用于操作系统的驱动器。 您可以将驱动器安装在CD托架中,也可以执行我所做的操作,然后使用USB记忆棒运行操作系统。
步骤1:准备服务器 (Step 1: Prepare The Server)
We first need to make the server into a condition where we can install the OS onto our system drive. I’m using a USB stick for the OS, which means I want that to be the only disk in the server on first boot. To that end, open up the server and remove any other hard drives inside it, leaving only the disk you want the OS installed to.
首先,我们需要使服务器处于可以将操作系统安装到系统驱动器上的状态。 我正在为操作系统使用USB记忆棒,这意味着我希望它成为服务器首次启动时的唯一磁盘。 为此,请打开服务器并卸下其中的所有其他硬盘驱动器,仅保留要安装操作系统的磁盘。
Don’t forget to hook up a monitor, a mouse and a keyboard to your server. It is probably possible to do a command-line install of Ubtuntu, but I really can’t be bothered to try, so it’ll be a graphical install.
不要忘记将监视器,鼠标和键盘连接到服务器。 可能可以通过命令行安装Ubtuntu,但是我真的很愿意尝试,因此它是图形安装。
Seal up your server, and plug your install USB into one of the ports on the front. You’re ready to go!
密封服务器,然后将安装的USB插入正面的端口之一。 您准备好出发了!
步骤2:安装 (Step 2: Installation)
Deep breaths now, and press the power button. You’ll immediately see a lot of pretty lights light up on the server, which is a good sign. If that doesn’t happen, then…crap. Happily, it worked for me, so I’m just going to assume it worked for you.
现在深呼吸,然后按电源按钮。 您将立即看到服务器上的许多漂亮的灯亮着,这是一个好兆头。 如果那没有发生,那就……废话。 幸运的是,它对我有用,所以我只是假设它对您有用。
Watch the POST and the prompts. Eventually you’ll see a prompt offering you the ability to press F10 to get to the BIOS: do that. When you’re in the BIOS make sure that the system time is right, because it wasn’t for me. Make sure of the date, too, although you should note that mine was set in US date/time, so was right although it appeared to be wrong.
观看POST和提示。 最终,您会看到一个提示,提示您可以按F10键进入BIOS:执行此操作。 当您进入BIOS时,请确保系统时间正确,因为这不适合我。 也要确保日期,尽管您应该注意,我的日期是在美国的日期/时间设置的,但也是如此,尽管它似乎是错误的。
DO NOT CHANGE THE BOOT ORDER. If you have only two disks in, and only one of them has an OS installed on it, you won’t have a problem booting. Save and exit the BIOS, and then allow the machine to boot. Hopefully, you’ll end up booting into a live-installation of Ubuntu.
请勿更改启动顺序。 如果您只有两个磁盘,并且其中只有一个安装了操作系统,则引导不会有问题。 保存并退出BIOS,然后让计算机启动。 希望您最终将引导到实时安装的Ubuntu中。
You’ll be asked if you want to ‘Try Ubuntu’ or ‘Install Ubuntu’. We need some of the utilities, so initially go to ‘Try Ubuntu’.
系统将询问您是否要“尝试Ubuntu”或“安装Ubuntu”。 我们需要一些实用程序,因此首先转到“尝试Ubuntu”。
The first thing we need to do is make sure our system disk is ready to take an OS. To do this, go to the Dash Home (the top icon in the dock to the left) and search ‘partition’. This should give you GParted: launch it.
我们需要做的第一件事是确保我们的系统磁盘已准备就绪可以使用操作系统。 为此,请转到Dash Home(Dash主页)(左侧扩展坞中的顶部图标),然后搜索“分区”。 这应该给您GParted:启动它。
Initially, GParted should be showing you the partitions of your boot USB. To show your future boot disk, click on the drop-down box in the top right and select the other disk (in my case, /dev/sdb
). If you’re using a new HDD, you should find it is entirelly unpartitioned space, which is great. If you’re using an off-the-shelf USB stick like me, you’ll probably find that the manufacturer has dumped some crap on your drive.
最初,GParted应该向您显示启动USB的分区。 要显示将来的启动磁盘,请单击右上角的下拉框,然后选择另一个磁盘(在我的情况下为/dev/sdb
)。 如果您使用的是新硬盘,则应该发现它完全是未分区的空间,这很棒。 如果您正在使用像我这样的现成USB随身碟,您可能会发现制造商在您的驱动器上丢了一些废话。
If your drive has any partitions, unmount them by right-clicking on them and selecting ‘Unmount’. Then, remove the partition by right-clicking on it and selecting ‘Delete’. Your device should then become entirely ‘Unallocated’. Click the green tick, and then ‘Apply’. Then close GParted.
如果您的驱动器有任何分区,请通过右键单击分区并选择“卸载”来卸载它们。 然后,通过右键单击分区并选择“删除”来删除分区。 然后,您的设备应完全变为“未分配”状态。 点击绿色的勾号,然后点击“应用”。 然后关闭GParted。
Now, double-click the Install icon on the desktop. It’s time to go through the install process. Follow the following steps:
现在,双击桌面上的“安装”图标。 现在是时候完成安装过程了。 请按照以下步骤操作:
- Select your language. Click ‘Continue’.
- If you have cabled internet access, select ‘Download Updates While Installing’. Definitely select ‘Install Third-Party Software’. Select ‘Continue’.
- Select ‘Erase disk and install Ubuntu’, then ‘Continue’.
- IMPORTANT. Make sure you have the correct drive selected. In my case,
sdb
. Then, click ‘Install Now’. - As the install proceeds, fill out the screens. Eventually, you’ll have filled everything out. At this point, go make a cup of tea, and tweet about how much fun you’re having. You should even send me a tweet! You could also do what I did, and put on some basketball (Let’s Go Opals!).
- 选择你的语言。 点击“继续”。
- 如果您有有线互联网访问,请选择“安装时下载更新”。 绝对选择“安装第三方软件”。 选择“继续”。
- 选择“擦除磁盘并安装Ubuntu”,然后选择“继续”。
- 重要提示 。 确保选择了正确的驱动器。 就我而言,是
sdb
。 然后,单击“立即安装”。 - 随着安装的进行,填写屏幕。 最终,您将填满所有内容。 在这一点上,去喝杯茶,然后在推特上分享您的乐趣。 您甚至应该给我发一条推文! 您也可以做我所做的事情,然后穿上篮球(Let's Go Opals!)。
The install will take a while (I watched a half of basketball during the install), particularly if you’re installing to a USB stick and/or downloading packages. Eventually, you’ll be asked to restart. Do that, and when the prompt tells you to remove your install USB stick and let the restart run.
安装将需要一段时间(在安装过程中我看了一半篮球),尤其是如果要安装到USB记忆棒和/或正在下载软件包。 最终,系统会要求您重新启动。 这样做,然后在提示您删除安装的USB记忆棒并让重新启动运行时提示您。
Once you’ve restarted, we want to make sure all of the system software is up to date. Open the Dash Home and type ‘update’, then select the Update Manager. Check for updates, then install everything. This, too, will take a little while. Or, you know, a long while.
重新启动后,我们要确保所有系统软件都是最新的。 打开Dash Home并键入'update',然后选择Update Manager。 检查更新,然后安装所有内容。 这也将花费一些时间。 或者,很长一段时间。
Eventually you’ll need to restart. Do it.
最终,您需要重新启动。 做吧
步骤3:准备ZFS。 (Step 3: Prepare for ZFS.)
One of the major reasons I had for using Ubuntu was because it has excellent support for ZFS. Before we get around to installing our storage drives, we should prepare our machine for use with them.
我使用Ubuntu的主要原因之一是因为它对ZFS具有出色的支持。 在开始安装存储驱动器之前,我们应该准备要与它们一起使用的机器。
Installing ZFS on Ubuntu is easy. We just want to add the ZFS-PPA to our list of software repositories. To do that, open a shell and type the following commands:
在Ubuntu上安装ZFS很容易。 我们只想将ZFS-PPA添加到我们的软件存储库列表中。 为此,请打开外壳并键入以下命令:
$ sudo add-apt-repository ppa:zfs-native/stable
$ sudo add-apt-repository ppa:zfs-native/stable
$ sudo apt-get update
$ sudo apt-get update
$ sudo apt-get install ubuntu-zfs$ sudo apt-get install ubuntu-zfs
This will, unfortunately, also take a while. Another caffeinated beverage for you, and maybe an episode of Community (Six seasons and a movie!).
不幸的是,这还需要一段时间。 另一种含咖啡因的饮料,也许是《 社区》的一集(六个季节和一部电影!)。
In the meantime, place your storage drives into two of the drive brackets. One of your drives should go in place of the 250GB drive. This is because it is currently not possible to replace a drive in a storage pool with a larger capacity one, but it is possible to extend a pool. Thus, the short term loss is a long-term gain.
同时,将存储驱动器放入两个驱动器支架中。 您的一个驱动器应代替250GB的驱动器。 这是因为当前无法替换容量更大的存储池中的驱动器,但是可以扩展存储池。 因此,短期损失是长期收益。
Put these two brackets back into your server. Ideally, they should be mounted as far as part as possible: hard drives do produce heat, and leaving them apart will keep the case cooler.
将这两个支架放回服务器中。 理想情况下,它们应安装得尽可能远:硬盘驱动器会产生热量,而将它们分开放置会使机箱温度更低。
With that done, boot your machine. While it boots, sit and idly ponder what you will want to call your drive pool. For the purposes of this guide, it will be called the intensely-boring dpool
.
完成后,启动计算机。 在启动时,请坐着并闲着思考您将如何称呼驱动器池。 出于本指南的目的,它将被称为无聊的dpool
。
步骤4:设置驱动器池。 (Step 4: Set Up Your Drive Pool.)
Once the machine has rebooted, get yourself another terminal. You’ll want to work out what the drive ids of your two new drives are. To do this, run:
机器重新启动后,让自己进入另一个终端。 您将要计算出两个新驱动器的驱动器ID。 为此,请运行:
You’ll see a whole lot of printouts. The filenames, highlighted in blue, are the IDs associated with various disks. To work out which are the two new disks you added, you want to consider their symlinks, which are shown in yellow.
您会看到很多打印输出。 以蓝色突出显示的文件名是与各种磁盘关联的ID。 要确定添加的两个新磁盘中的哪些,您需要考虑它们的符号链接,它们以黄色显示。
Your two new drives will have three IDs associated with them, all symlinked to the same place: one beginning ata-
, one beginning scsi-SATA
and one that is shorter and manufacturer specific. Your new drives should have links that point to ../../sd[a-z]
. To put it another way, their symlinks shouldn’t end in a number. On my machine, I see symlinks for sda
, sdb
, sdc
, sdc1
, sdc2
and sdc5
. From that, I can tell that sdc
is my boot disk, as it’s the only drive with partitions on it. Thus, sda
and sdb
are the two disks I added. Take a note of their IDs: I used the ones beginning with scsi-SATA
but I know of no reason you couldn’t use the others.
您的两个新驱动器将具有三个与之关联的ID,所有ID均ata-
链接到同一位置:一个开始ata-
,一个开始scsi-SATA
和一个较短且针对制造商的ID。 您的新驱动器应具有指向../../sd[az]
链接。 换句话说,它们的符号链接不应以数字结尾。 在我的机器上,我看到了sda
, sdb
, sdc
, sdc1
, sdc2
和sdc5
符号链接。 由此,我可以看出sdc
是我的启动磁盘,因为它是唯一带有分区的驱动器。 因此, sda
和sdb
是我添加的两个磁盘。 记下它们的ID:我使用了以scsi-SATA
开头的ID,但我完全不知道您不能使用其他ID。
Now, create your first drive pool. Run the following command, replacing dpool
with the name of your choice.
现在,创建您的第一个驱动器池。 运行以下命令,将dpool
替换为您选择的名称。
$ sudo zpool create dpool mirror /dev/disk/by-id/[id_of_first_disk] /dev/disk/by-id/[id_of_second_disk]$ sudo zpool create dpool mirror /dev/disk/by-id/[id_of_first_disk] /dev/disk/by-id/[id_of_second_disk]
This command, unlike so many others, will run fast. You’ll notice that, all of a sudden, /dpool
exists. (To see this, run ls /
). Ta-da! You have created your first software RAID. The eagle-eyed amongst you will note that I have created a RAID-1: that is, mirrored storage. This is deliberate. You cannot add disks to a RAIDZ, which means I do not gain the space advantage unless I have four disks in place from the start. That seems financially silly to me, so I’ll instead use two disks as a mirror for now, and add another pair when I need them.
与许多其他命令不同,该命令将快速运行。 您会发现/dpool
突然存在。 (要查看此内容,请运行ls /
)。 - 您已经创建了第一个软件RAID。 你们当中鹰眼的人会注意到我已经创建了RAID-1:即镜像存储。 这是故意的。 您无法将磁盘添加到RAIDZ,这意味着除非从头开始有四个磁盘,否则我不会获得空间优势。 这对我来说在财务上似乎很愚蠢,所以我现在暂时将两个磁盘用作镜像,并在需要时添加另一个磁盘对。
From here, you can create sub-storage locations managed by ZFS using the zfs create
command. For instance, if you (like me) want to store movies, you could run sudo zfs create dpool/movies
. If you now ls /dpool
you’ll see your new location exists.
在这里,您可以使用zfs create
命令zfs create
由ZFS管理的子存储位置。 例如,如果您(像我一样)想要存储电影,则可以运行sudo zfs create dpool/movies
。 如果现在使用ls /dpool
看到您的新位置存在。
To take a look at your new drive pool, run sudo zpool status
. To see your sub-sections, run sudo zfs list
.
要查看新的驱动器池,请运行sudo zpool status
。 要查看您的小节,请运行sudo zfs list
。
第5步:共享。 (Step 5: Sharing.)
Cool, so, I’m sick of sitting by this server now, so let’s remotely administrate it so I can lie down on the sofa watching the world’s best athletes tire themselves out.
太酷了,所以,我现在讨厌坐在这台服务器上,让我们对其进行远程管理,这样我就可以躺在沙发上,看着世界上最好的运动员感到疲倦。
I’m a reasonably security-conscious guy, so I don’t just want anyone to be able to log into my server as my user account, which will be the designated administrator. For that reason, I’m going to end up using certificate-based SSH, rather than password-interactive login. First, though, let’s make sure we can login as me at all.
我是一个相当注重安全性的人,所以我不仅希望任何人都能够以我的用户帐户(即指定的管理员)身份登录我的服务器。 因此,我将最终使用基于证书的SSH,而不是密码交互式登录。 不过,首先,请确保我们可以完全以我的身份登录。
Let’s install OpenSSH:
让我们安装OpenSSH:
Once that takes, open up /etc/ssh/sshd_config
in your favourite text editor (which should be Vim). Turn off X11Forwarding
(you’ll never use it), then save it and see if you can log in. With good luck, you’ll be able to.
完成后,在您喜欢的文本编辑器(应为Vim)中打开/etc/ssh/sshd_config
。 关闭X11Forwarding
(您将永远不会使用它),然后保存它并查看是否可以登录。祝您好运,您将可以。
Brilliant. Next, you need a private and a public SSH key. If you already have one, great, if not, on your remote machine run:
辉煌。 接下来,您需要一个专用和公用SSH密钥。 如果您已经在远程计算机上运行了一个(如果没有的话),那么很好:
$ $ cd ~/.ssh
cd ~/.ssh
$ ssh-keygen -t rsa$ ssh-keygen -t rsa
Use no passphrase and the default filename.
不使用密码短语和默认文件名。
Now you should have a certificate. To use it, first use your SSH connection to your server to run mkdir ~/.ssh
. Then, run:
现在您应该拥有证书。 要使用它,请首先使用到服务器的SSH连接运行mkdir ~/.ssh
。 然后,运行:
Awesome. Now, disable the ability to use the keyboard to login entirely. Reopen sshd_config
. Uncomment PasswordAuthentication
and change it to No. These changes will take effect next time you restart.
太棒了 现在,禁用使用键盘完全登录的功能。 重新打开sshd_config
。 取消注释PasswordAuthentication
并将其更改为No。这些更改将在您下次重新启动时生效。
Now, we want to set up file shares. The easiest way to share the files you want to as many people as possible is to use SMB, which on Linux means using Samba. Let’s install it:
现在,我们要设置文件共享。 与尽可能多的人共享文件的最简单方法是使用SMB,在Linux上这意味着使用Samba。 让我们安装它:
$ sudo apt-get install samba$ sudo apt-get install samba
When that install finishes, open up /etc/samba/smb.conf
. Uncomment security = user
. Then, scroll to the bottom of the file and add a new share. This should be a block of text that looks a bit like this:
安装完成后,打开/etc/samba/smb.conf
。 取消注释security = user
。 然后,滚动到文件底部,然后添加新共享。 这应该是一段文本,看起来像这样:
Save the file and exit it. Then, restart Samba:
保存文件并退出。 然后,重新启动Samba:
$ sudo restart smbd
$ sudo restart smbd
$ sudo restart nmbd$ sudo restart nmbd
Awesome. You can now get access to your files.
太棒了 现在,您可以访问文件了。
第6步:传输文件。 (Step 6: Transfer your files.)
If you have any files sitting anywhere that you want to put on your server, start doing it now.
如果您有任何文件要放在服务器上的任何位置,请立即开始操作。
步骤7:文件完整性。 (Step 7: File Integrity.)
One of the many advantages of using ZFS is that it provides a great deal of data integrity. However, it’s important to note that this does not entirely come for free. Data integrity issues caused during transfers or copy-on-write can be caught automatically by the filesystem, but there can be other forms of data corruption that ZFS cannot automatically catch.
使用ZFS的许多优点之一是,它提供了大量的数据完整性。 但是,需要注意的是,这并不完全免费。 文件系统可以自动捕获在传输或写时复制期间引起的数据完整性问题,但是ZFS可能无法自动捕获其他形式的数据损坏。
For that reason, it is important to scrub your filesystem regularly. As far as I can tell the best practice is to scrub the filesystem weekly when using consumer-grade drives like ours.
因此,定期清理文件系统非常重要。 据我所知,最佳实践是在使用像我们这样的消费级驱动器时, 每周清理一次文件系统。
Additionally, if a disk fails ZFS will limp onwards rather than falling over and dying. This means we want to check on a regular basis to find out if any of our disks have failed, so that we can intervene and replace the drive.
此外,如果磁盘发生故障,ZFS会继续瘫痪而不是掉下来而死。 这意味着我们要定期检查以发现我们的任何磁盘是否出现故障,以便我们进行干预和更换驱动器。
Doing this manually is boring, rote work. More importantly, it’s also a job that is easy to forget to do. For that reason, we’ll write up a couple of scripts to notify us instead. Also because, you know, scripts are cool.
手动执行此操作无聊,死记硬背。 更重要的是,这也是一项容易忘记做的工作。 因此,我们将编写一些脚本来通知我们。 另外,因为您知道脚本很酷。
We need two cron jobs. The first can be simply and easily added to your crontab
file. Open up /etc/crontab
and add the line:
我们需要两个cron工作。 可以轻松地将第一个添加到您的crontab
文件中。 打开/etc/crontab
并添加以下行:
For those of you that don’t speak cron
, that means that once a week, at midnight on Sunday, the command zpool scrub dpool
will run. You don’t need to use sudo
to elevate priveleges, as this command will run as root.
对于不讲cron
,这意味着每周一次,在星期日的午夜,将运行zpool scrub dpool
命令。 您不需要使用sudo
来提升特权,因为此命令将以root身份运行。
The status test needs to be scripted, because we want to be emailed. In fact, we’ll be using two scripts, because we need to be able to send an email if something goes wrong. For that reason, we’ll be using a shell script and a Python script, because Python is brilliant.
状态测试需要编写脚本,因为我们希望通过电子邮件发送。 实际上,我们将使用两个脚本,因为如果出现问题,我们需要能够发送电子邮件。 因此,我们将使用Shell脚本和Python脚本,因为Python非常出色。
Firstly, let’s write a Python script that can email you the content of a file. Here is one I whipped up in about five minutes. If you want to use this yourself, you’ll need to set your constants to be correct at the top of the file.
首先,让我们编写一个Python脚本,该脚本可以通过电子邮件将文件内容发送给您。 这是我约五分钟打过的电话。 如果要自己使用它,则需要在文件顶部将常量设置为正确。
Next, we want to write a quick bash script that will check the state of the filesystem and email you the report if something looks amiss. My first pass at this script looks like this.
接下来,我们要编写一个快速的bash脚本,它将检查文件系统的状态,并在出现问题时通过电子邮件将报告发送给您。 我对这个脚本的第一遍看起来像这样 。
When you’ve written these two scripts, put these two files into /usr/local/bin
and make sure they’re executable. Then, we add the following line to our crontab:
编写完这两个脚本后,将这两个文件放入/usr/local/bin
并确保它们是可执行的。 然后,将以下行添加到crontab中:
10 8 * * * root zfs_report
10 8 * * * root zfs_report
At ten minutes past 8 in the morning, every day, this script will run. Hopefully, this will be enough to alert you when a drive fails. Still, from time-to-time a manual test will probably help you sleep more easily at night.
每天早上8点十分,此脚本将运行。 希望这足以在驱动器发生故障时提醒您。 不过,不时进行手动测试可能会帮助您在晚上更轻松地入睡。
步骤8:完成。 (Step 8: Finalising.)
翻译自: https://www.pybloggers.com/2012/08/lets-build-a-fileserver/