AMAZON ECS(2)Disk Size on EC2

AMAZON ECS(2)Disk Size on EC2

I have a project which have a large file there, it is a known issue, we will fix that later.

I want to docker that API, I did that on EC2 instance. I create that instance long time ago, I was thinking that I only need 30G, but today. I met this problem saying not enough space for build the docker image.

Error Message:
Error response from daemon: Cannot destroy container clever_bartik: Driver devicemapper failed to remove root filesystem ef2f1e5b38fddaac78966c5e828d5a0aabd92600c3d87346f9e371bfeb5401d2: Error saving transaction metadata: Error writing metadata to /var/lib/docker/devicemapper/metadata/.tmp258670910: write /var/lib/docker/devicemapper/metadata/.tmp258670910: no space left on device

Solution:
First of all, clean the file first.

> df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 30G 30G 0 100% /
devtmpfs 7.5G 100K 7.5G 1% /dev
tmpfs 7.5G 0 7.5G 0% /dev/shm

This command will show me that all the space of disk is used.

sudo du -hsx * | sort -rh | head -10
12G home
1.2G usr
526M var
139M lib
62M boot
43M opt
23M lib64
13M sbin
7.6M etc
7.0M bin

This command will show me where did the space goes. I will then delete the files. Restart the instance.

And then I will do 2 things.

1 Add Disk Space to my Small Lonely EC2
click —> [ELASTIC BLOCK STORE] —>[Volumes] —>”Create Volume” Attach this Volume to My EC2

After create and attach the disk to that machine, that does not mean we can directly use it on linux.
List the Disk
> lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 30G 0 disk
└─xvda1 202:1 0 30G 0 part /
xvdf 202:80 0 100G 0 disk
loop0 7:0 0 100G 0 loop
└─docker-202:1-270348-pool 253:0 0 100G 0 dm
loop1 7:1 0 2G 0 loop
└─docker-202:1-270348-pool 253:0 0 100G 0 dm

I have not file system on that.
> sudo file -s /dev/xvdf
/dev/xvdf: data

Create the file system
> sudo mkfs -t ext4 /dev/xvdf
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 26214400 4k blocks and 6553600 inodes
Filesystem UUID: baca4bba-b1ea-419c-b3ba-39b0a117f6bb

Create the Directory
> sudo mkdir data

Mount the disk to the directory
> sudo mount /dev/xvdf /data

I did not try auto mount.

2 Set up and Configure Docker to use the ext storage

Check the current docker info
> sudo docker info

Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata

The problem is here. They are using up 30GB.

Stop Docker
> sudo service docker stop

Clean the old file
> sudo rm -fr /var/lib/docker/

Config the storage
> sudo cat /etc/sysconfig/docker-storage
# This file may be automatically generated by an installation program.

# By default, Docker uses a loopback-mounted sparse file in
# /var/lib/docker. The loopback makes it slower, and there are some
# restrictive defaults, such as 100GB max storage.

# If your installation did not set a custom storage for Docker, you
# may do it below.

# Example: Use a custom pair of raw logical volumes (one for metadata,
# one for data).
# DOCKER_STORAGE_OPTIONS="--storage-opt dm.metadatadev=/dev/myvg/my-docker-metadata --storage-opt dm.datadev=/dev/myvg/my-docker-data"

DOCKER_STORAGE_OPTIONS="--storage-opt dm.metadatadev=/data/my-docker-metadata --storage-opt dm.datadev=/data/my-docker-data"

Start docker there
> sudo service docker start

This does not work.

Try this
> sudo vi /etc/sysconfig/docker-storage-setup

DEVS="/dev/xvdf"

STORAGE_DRIVER=devicemapper

> /usr/bin

> sudo wget https://raw.githubusercontent.com/projectatomic/docker-storage-setup/master/docker-storage-setup.sh

> sudo chmod a+x docker-storage-setup.sh


> ./docker-storage-setup.sh

It seems not working. I do not want to spend more time on this. I will do like this.

Soft the link the file to the configuration
> sudo ln -s /data/docker /var/lib/docker

> sudo mkdir -p devicemapper/devicemapper/data

> sudo mkdir -p devicemapper/devicemapper/metadata

> sudo mkdir /var/lib/docker/graph

> sudo mkdir /var/lib/docker/tmp
> sudo mkdir /var/lib/docker/devicemapper/metadata

>sudo mkdir /var/lib/docker/containers

>sudo mkdir /var/lib/docker/init

>sudo mkdir /var/lib/docker/trust

>sudo mkdir /var/lib/docker/volumes

Error Message:
open /var/lib/docker/repositories-devicemapper: no such file or directory

Solution:
> sudo vi /var/lib/docker/repositories-devicemapper

{"Repositories":{}}

It does not work. I guess it relates to the manual deleting docker.

Remote the docker
> sudo yum remove docker

Install the docker again
> wget -qO- https://get.docker.com/ | sh

Let me try this:
> sudo mkdir /var/lib/docker

> sudo mount /dev/xvdf /var/lib/docker

It works perfect. Here is the Dockerfile
#Run a Simple REST API based on playframework

#Prepre the OS
FROM centos:7
MAINTAINER Carl Luo <luohuazju@gmail.com>

ENV DEBIAN_FRONTEND noninteractive
ENV JVM_HEAP_MAX 12g
ENV JVM_HEAP_MIN 8g
ENV HTTP_PORT 8003

#Install Java
RUN yum install -y java-1.8.0-openjdk-headless-1.8.0.60-2.b27.el7_1 lapack atlas
RUN yum install -y unzip
RUN yum install -y wget

#Install the Application
RUN mkdir /share/
WORKDIR /share/

#ADD data/classifier_models.tar.gz /share/
RUN wget https://xxxxx/classifier_models.tar.gz
RUN tar zxvf classifier_models.tar.gz
RUN rm -fr classifier_models.tar.gz

ADD target/universal/classifier-play-1.0.zip /share/
RUN unzip classifier-play-1.0.zip
RUN rm -fr classifier-play-1.0.zip

#Start the Application
EXPOSE 8003
RUN mkdir -p /app/
ADD start.sh /app/
WORKDIR /app
CMD [ "./start.sh" ]

Here is the start.sh
#!/bin/sh -ex

APPLICATION_SECRET="nosessionshere"

cd /share/classifier-play-1.0

bin/classifier-play \
-Dconfig.file=conf/application-stage.conf \
-Dhttp.port=${HTTP_PORT} -Dhttp.address=0.0.0.0 \
-J-Xms${JVM_HEAP_MIN} -J-Xmx${JVM_HEAP_MAX} -J-server \
-J-XX:+UseG1GC -J-XX:+UseStringDeduplication


Reference:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html

http://sillycat.iteye.com/blog/2256757

http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/

http://www.projectatomic.io/docs/docker-storage-recommendation/

https://coreos.com/os/docs/latest/mounting-storage.html