操作系统 ShanghaiTech CS130 | Storage Management

1.Motivation: File types also can be used to indicate the internal structure of the file. For example, the operating system requires that an executable file have a specific strcutre so that it can determine where in memory to load the file and what the location of the first instruction is. Some operating systems extend this idea into a set of system-supported file structures, with sets of special operations for manipulating files with those structures.

2.OS policies:

None - sequence of words / bytes
1. consdier each file to be a sequence of 8-bit bytes
2. no interpretation of these bits is made by the operating system.
Simple record structure
1. e.g: support only text and exe files
2. enctypted files(bit-like files) is neither of them, hard to be classified
Complex structures
1. Support multiple file structures
2. makes system large and cumbersome.

3.can simulate last two with the first method by inserting approriate control chacacters

4.decide by OS and programs

1.3 File attributes (恐龙书13.1.1)

Name/Identifier/Type/Location/Size/Protection/Time, Date, User Identification/Information about files are kept in the directory structure, maintained on disk.

Remark: Identifier is a unique tag, usually a non-human-readable number, while location is a pointer to a device and to the location of the file on that device.

Remark: Time, Date, User Identification is kept for creation, last modification and last use. These data can be useful for protection, security and usage monitoring.

1.4 File types(恐龙书13.1.3)

1.5 File operations (恐龙书13.1.2)

Create/Write/Read/Reposition within file/Delete/Truncate/Open/Close

2 Concepts of directory(恐龙书13.3)

2.1 Directory structure general ideas

1. Motivation: number of files can be extensive. Therefore, break filesystem into partitions and hold info. about files within partitions.

Both directory structure and file reside on disk.
Backup of these two structures are kept on tertiary storages(e.g: tapes)

2. Information in a device directory: File name/ File type/ Address or location / Current length / Maximum length / Date created, last accesed, last updated / Owner ID, protection information

3: operations performed on directory: Search for a file/ Create a file / Delete a file/ List a directory/ Rename a file/ Traverse the file system

4. goals:

efficency: locate a file quickly
naming: convenient to uses: have the same name for different files. THe same file can have several different names
grouping: logical grouping of files by properties

2.2 Common logical structure of a directory

1. Single level directory(13.3.1)

all files are contained in the same directory
cons: naming problems and grouping problems(duplicate name for different file)

2. Two level directory(13.3.2)

first level: each user has his own use file directory(UFD). Second level: when a user job starts or a user logs in, the system's master file directory (MFD) is searched.
need to specify path name:
- user a wants to access user b's abc.text: /userb/test.txt
- volume name: sometimes, a volume name is needed(in windows): C:\userb\test.txt
can have the same file names for different users
system files kept in separate directory.
- motivation: Programs are provided as part of the system are generally defined as files: loaders, assemblers, etc. Many command interpreters simply treat such a command as the name of a file to load and execute to invoke those programs. Naive copy is redundant. We prefer save it in a special UFD(like user 0).
- search path: when loading a file, system will first search local UFD, then search special UFD to load system files.
efficient searching(e.g: DFS)

3. Tree Structured Directories(13.3.3)

allow users to create their own subdirectories and organize their files accordingly.
A tree is the most common directory structure, which has a root directory, and every file in the system has a unique path name.
MS-DOS uses a tree structured directory

1) properties: arbitrary depth of directories, efficient searching(DFS), grouping capability.

2) Cuurent directory: each process has a current directory which is interesting to them. (e.g: a refernce to file is made by searching current directory). System call shoulb be provided to change current directory.

3) abolute or relative path name: absolute path name begins at root (which is designated by an initial "/"), while a relative path name defines a path from the current directory.

4) create a new file/sub-dir is done in current dir. (e.g: touch, mkdir <dir-name>)

5) delete a file (e.g.: rm <file-name>) / delete a sub-directory: (1) only delete if directory empty (2) recursively clear up this directory.

4. Asyclic graph directories (13.3.4)

1) Motivation: allow sharing the same subdirectory between different users.

2) Implementation by links: pointers to other files of sub-dir/ symbolic links or relative path name.

3) Implementation by shared files: def: duplicate information about shared files in both sharing directories / make original and copy indistinguishable and required consistency when a file is modified.

4) naming: file may have multiple absolute path names / two different names for the same file.

5) traversal: shared data structures traversed only once.

Remark: when we do traversal for copying, we do not want to backup the same but shared file twice.

6) deletion: may leave dangling pointers or preserve until all are deleted, which can be fulfilled y keeping a list of all refernces to a file or keeping a reference count.

Remark: Unix operating system uses this approach for nonsymbolic links (or hard links), in file information block(i node).

5. General Graph directories(13.3.5):

1) How to guarantee no cycles?: Only links to file not sub-dir/ Determine every time add a link.

2) If links to dir. are allowed: a simple graph structure/ ensure not traverse twice (e.g: search can e non-terminating like DFS)

3) File deletion: reference count can be non-zero/ need garbage collection

Remark: reference count can be non-zero means: when cycle exists, a cycle can no longer contains file but only self-referencing and is not connected to the root. GC is applied to determine whether the last reference of a file has been deleted:

first mark everything can be accessed, then delete all unmarked files.

3 Access methods(恐龙书13.2)

3.1 Sequential access（13.2.1）

Information in the file is processes in order, one record after the other.

Opeartions: Read_next/write_next/reset/no read after last write(?)

3.2 Direct access(13.2.2)

A file is made up of fixed-length logical records that allow programs to read and writes records rapidly in no particular order.

Operations:

read(n)=position to n& read_next()
write(n)=position to n&write next()
rewrite n is okay

3.3 Indexed sequential /indexed file organizatoin(13.2.3, useless in exam)

4 Protection(恐龙书13.4)

1. Type of access: read/write/execute, append/delete, list(list out all files..)

2. Problem: associate each file with an access list with username is to long

Solution: condensed version of list: (1) Mode of access: read/write/access, (2) three classes of users: owner/groups/public access

Example: In UNIX, 3 fields of length 3 bits are used: fileds are u(ser)/g(roup)/o(ther), bits are r(ead)/w(rite)/x(execuate)

It is called access-control list(ACL).

5 File system structure(恐龙书14.1)

1. File structure: logical storage unit with collection of related information

2. File system: (1) reside on secondary storage (disk). (2) To improve I/O efficiency. (3) I/O transfers are performed in blocks. (4) organized into layers

3. File control block : storage structure consiting of information about a file.

Remark: To improve I/O efficiency, I/O transfers between memory and mass storage are performed in units of blocks. Each block on the disk drive, sector size is usually 512 bytes or 4096 bytes, and the transfer methods used are similar to those used y disk drives.

Remark: file control block(FCB) is inode in UNIX file systems.

4. File system Mounting

FS must be mounted before it can be available to process on the system.

Give OS device name and mount point
OS versifies the device (if a valid FS or not)
OS notes in its dir structure

6 Allocation of Disk Space

Motivation: Disks have 2 properties: (1) A disk can be rewritten in place; it is possible to read a block from disk, modify block and write it back to the same block. (2) A disk can access directly any block of information it contains. The main problem is how to allocate space to these files in disk to guarantee that file can be accessed proficiently.

low level access methods depend upon the disk allocation scheme used to store file data:

contiguous
linked list
indexed

6.1 contiguous allocation(恐龙书14.4.1)

each file occupies a set of contiguous block on the disk

simple
suits sequential or direct access
fast and easy to recover
Problems: (1) Wasteful of space. (2) Files cannot grow. (3) Overestimate space.

Remark: For overestimate space, it means that for an ouput file, it is hard to foresee its size. We may overestimate it and resulting in considerale wasted space.

mapping from logical to physical: <Q,R>. Block to be accessed = Q(starting address). Displacement into block = R.

6.2 Linked allocation (恐龙书14.4.2)

Each file is a linked list of disk blocks. Blocks may be scattered anywhere on disk. Can be fixed size physical block or a contiguous collect of blocks (called cluster in 恐龙书). Allocate as needed. Then link together via a pointer. Pointers in list not accessible to user.

Pros:

Simple: just starting address
Free space management system(space efficient w.r.t contiguous policy)
Suited for sequential access(linked list) but not random
directory table maps files into head of list
Mapping <Q, R>: block to be accessed is the Qth block/ Displacement into block=R+1

Cons:

Slow: defines principle of locailty(need to read sequentially)
Not very reliable: system crashes can scramble files being updated.

Remark: Image such a case that a pointer is damaged. A bug in the operating-system software or a HW failure might result in picking up the wrong pointer

Solution: file allocation table(FAT). put all links in one table. One entry per physical block in disk. dir. points to first&last blocks(EOF)

Advantage: (1) advantages of linked file system. (2) can be cahed in memory. (3) searchable at CPU speeds, pseudo-random access.

Disadvantages: (1) Limited size, not suitable for every large disks. (2) FAT caches describe entire disk, not just opened files. (3) Not fast enough for large databases.

Disk de-fragmentation: re-organize blocks in disk/ so that file is contiguous (mostly) / link or FAT organization preserved. Purpose: reduce disk arm movement during sequential accesses.

6.3 Indexed allocation

1.def: brings all pointers together into the index block / need index table / support sequential, direct and indexed access.

2.dynamic access w/o external fragmentation

3.overhead of index block: (suppose max size 256k wrods/max block size 512 words/4 bytes an index/256000/512=500<512, need only 1 block for index table)

4. linked scheme: unbdouned length/ link blocks of index table (no limit on size)

5. multilevel scheme: maxfile size is bounded, but introducing more efficient

6. combined scheme: unix inode

Inode: index node/control structure/inode table on disk
- Inodes of all files in the FS
- when a file is open, bring its inode to memory
- store in a memory-resident inode table

7 Directories

def: A file that contains a list of file names plus pointers to associated inodes

(1) Implementation 1: linear list of file names with pointers to the data blocks.

notes: simple to program/time-consuming to execute/sorted lists helps

(2）Implementation 2: hash table: decrease directory search time/ collisions dealing is worth noting

8 Efficiency and performance (恐龙书 14.6)

Dependent on: (1) Disk allocation and directory algorithms. (2) Types of data. (3) Dynamic allocation of kernel structures.

Improved by: (1) on-board cache (disk controllers). (2) disk cache. (3) Virtual disk or RAM disk.

9 Recovery(恐龙书14.7)

1. consistency check: a system program such as fsck in UNIX: compares the data in the directory structure and other metadata with the state on storage and tries to fix any inconsistencies it finds.

2.backup&restore

10 Mass-Storage Structure (恐龙书 11)

terms: hard disk drives(HDD) and nonvolatile memory (NVM) devices

10.1 HDD introduction(11.1)

disk arm: that moves all head as a unit
platter each disk platter has a flat circular shape, like a CD
tracks: the surface of a platter is logically divided into circular tracks, which can be subdivide into sectors
cylinder: the set of tracks at a given arm position make up a cylinder

RPM: rotations per minute

transfer rate: the rate at which data flow between the drive and the computer

positioning time: consist of two parts: the time necessary to move disk arm to the desired cylinder, called seek time, and the time necessary for the desired sector to rotate to the disk head, called rotational latency.

Performance:

Transfer rate - theoretical - 6GB/sec
Effective Transfer Rate - real - 1Gb/sec
Seek time from 3ms to 12ms - 9ms common for desktop drives
average seek time measured or calculated based on 1/3 of tracks
latency based on spindle speed: 1/(RPM * 60)
average latency = 1/2 lentency

Access Latency = Average access time = average seek time + average latency

Average I/O time = average access time + (amount to transfer / trasfer rate) + controller overhead

10.1 HDD scheduling(11.2)

(1) FCFS(first come first service)

(2) SPTF/SSTF (shortest positioning/ seek time first)

(3) "Elevator" SCAN: sweep across disk, service all requests passed（从左往右再回来不断扫，扫到就爽到）

下面都十分trivial, 考试直接抄上吧

10.2 RAID

def: RAID: redundant array of indexpensive disks: multiple disk drives for reliability(via redundancy)

increase the MTTF/ frequently combined with NVRAM for perfomance

data striping: use a group disks as one storage unit.

6 different levels of RAID: (1) Improve performance (RAID 0). (2) Imporve reliability (RAID 1) (3) Balance (RAID 4, 5, 6)

RAID alone does not prevent or detect data corruption or other errors, just disk failures.

10.3 ZFS

ZFS adds checksum of all data and meta: (1) Keep with pointer to object. (2) detect if object is the right one. (3) detect if it chagned. (4) can detect and correct data and meta corruption. (5) ZFS also removes volumes, partitions. Disks allocated in pools.

10.4 NAS

(1)network-attached storage(NAS) is storage available over a network, rather than a local connection. (2) NFS and CIFS are common protocals. (3) implemented via remote procedure calls (RPC) between hosts and storage over typical TCP or UDP on IP network. (4) iSCSI protocol uses IP network to carry the SCSI protocal(remotely attaching to devices(in blocks)

10.5 SAN

(1) storage area network(SAN) common in large storage environments. (2) Multiple hosts attached to multiple storage arrays-flexible. (3) SAN is one or more storage arrays(connected to one or more fibre channel switches). (4) hosts also attach to the switches. (5) easy to add or remove storage.

10.6 Storage array

(1) can just attach disks, or arrays of disks. (2) storage array has controller(s), provides features to attached hosts(Ports, Memory, controlling software(NVRAM, etc), a few to thousands of disks, RAID, host spares, hot swap, features found in FSs(snapshots, clones, replication, deduplication))

10.7 Solid-State Disks

NVM/Can be more reliable than HDDs/more expensive per MB/shorter life span/less capacity/much faster/busses can be too slow/no moving parts, so no seek time or rotational latency.

NAND(higher density, faster erase and write, more errors internally, need error correction)
NOR(faster reads in smaller data units, can execute code straight out of NOR flash, significantly slower erases)

Reference

1. Operating System Concepts 10th Edition. WILEY

2. ShanghaiTech CS130 Operating system, Topic 11, Storage Management. Shu-Yin.

3. Chapter 12: Mass-sotrage systems(Silberschatz, Galvin and Gagne@2009)

liubai01

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
4
评论
操作系统 ShanghaiTech CS130 | Storage Management

ShanghaiTech CS130 | Lecture Note | Storage Management说明：笔记旨在整理我校CS130课程的基本概念。由于授课及考试语言为英文，故用英文为主，中文为辅的方式整理。由于是整理，尽提供最低限度的解释，以便提供简洁快速的查阅。全部笔记索引：【传送门】 | 上一节： Virtual Memory | 下一节：么有了目录Shanghai...
复制链接

扫一扫