Overall:
- hdfs is using the same users/groups with current linux system. One file owned to one user and one group.
- If one file need to be grunted access to multiple users ot groups. Then ACl should be used. HDFS ACLs give you the ability to specify fine-grained file permissions for specific named users or named groups, not just the file’s owner and group.
How to enable HDFS ACL:
- To use ACLs, first you’ll need to enable ACLs on the NameNode by adding the following configuration property to hdfs-site.xml and restarting the NameNode.
<property>
<name>dfs.permissions.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.namenode.acls.enabled</name>
<value>true</value>
</property> - HDFS CLI: setfacl and getfacl
- Reference: http://zh.hortonworks.com/blog/hdfs-acls-fine-grained-permissions-hdfs-files-hadoop/
hdfs user permission usecase:
Users/Files
|
File Name
| |||||
Groups
|
Users
|
System logs
|
Original data
|
Middle Result
|
Final Result
|
Critical Data(Ready data)
|
TechMg
|
manager
|
r--
|
Rwx
|
Rwx
|
Rwx
|
Rwx
|
dataCollector
|
rw-
|
Rw-
|
r--
|
r--
|
r--
| |
plateformDev
|
r--
|
r--
|
r--
|
r--
|
r--
| |
DataProcessor
|
r--
|
Rw-
|
Rwx
|
Rwx
|
r--
| |
DataAnalytics
|
r--
|
r--
|
r--
|
r--
|
r--
| |
business
|
business
|
---
|
---
|
---
|
r--
|
---
|
appDev
|
appDev
|
rwx
|
Rwx
|
---
|
---
|
---
|
Key ACL command: acl_SystemLogs.sh
hdfs dfs -setfacl -m user:appDev:rwx /fftest/SystemLogs
hdfs dfs -setfacl -m group:appDev:rwx /fftest/SystemLogs
hdfs dfs -setfacl -m user:business:--- /fftest/SystemLogs
hdfs dfs -setfacl -m group:business:--- /fftest/SystemLogs
hdfs dfs -setfacl -m user:manager:r-- /fftest/SystemLogs
hdfs dfs -setfacl -m user:dataCollector:rw- /fftest/SystemLogs
hdfs dfs -setfacl -m user:plateformDev:r-- /fftest/SystemLogs
hdfs dfs -setfacl -m user:DataProcessor:r-- /fftest/SystemLogs
hdfs dfs -setfacl -m user:DataAnalytics:r-- /fftest/SystemLogs
hdfs dfs -setfacl -m group:appDev:rwx /fftest/SystemLogs
hdfs dfs -setfacl -m user:business:--- /fftest/SystemLogs
hdfs dfs -setfacl -m group:business:--- /fftest/SystemLogs
hdfs dfs -setfacl -m user:manager:r-- /fftest/SystemLogs
hdfs dfs -setfacl -m user:dataCollector:rw- /fftest/SystemLogs
hdfs dfs -setfacl -m user:plateformDev:r-- /fftest/SystemLogs
hdfs dfs -setfacl -m user:DataProcessor:r-- /fftest/SystemLogs
hdfs dfs -setfacl -m user:DataAnalytics:r-- /fftest/SystemLogs
ACL example:
drwxrwxr-x+ - hadoop ff 0 2015-05-20 13:58 /fftest/CriticalData
drwxrwxr-x+ - hadoop ff 0 2015-05-20 13:58 /fftest/FinalResult
drwxrwxr-x+ - hadoop ff 0 2015-05-20 13:57 /fftest/MiddleResult
drwxrwxr-x+ - hadoop ff 0 2015-05-20 13:57 /fftest/OriginalData
drwxrwxr-x+ - hadoop ff 0 2015-05-20 13:56 /fftest/SystemLogs
drwxrwxr-x+ - hadoop ff 0 2015-05-20 13:58 /fftest/FinalResult
drwxrwxr-x+ - hadoop ff 0 2015-05-20 13:57 /fftest/MiddleResult
drwxrwxr-x+ - hadoop ff 0 2015-05-20 13:57 /fftest/OriginalData
drwxrwxr-x+ - hadoop ff 0 2015-05-20 13:56 /fftest/SystemLogs
[hadoop@node1 tmp]$ hdfs dfs -getfacl /fftest/SystemLogs
15/05/20 16:35:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... u sing builtin-java classes where applicable
# file: /fftest/SystemLogs
# owner: hadoop
# group: ff
user::rwx
user:DataAnalytics:r--
user:DataProcessor:r--
user:appDev:rwx
user:business:---
user:dataCollector:rw-
user:manager:r--
user:plateformDev:r--
group::r-x
group:TechMg:r--
group:appDev:rwx
group:business:---
mask::rwx
other::r-x
15/05/20 16:35:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... u sing builtin-java classes where applicable
# file: /fftest/SystemLogs
# owner: hadoop
# group: ff
user::rwx
user:DataAnalytics:r--
user:DataProcessor:r--
user:appDev:rwx
user:business:---
user:dataCollector:rw-
user:manager:r--
user:plateformDev:r--
group::r-x
group:TechMg:r--
group:appDev:rwx
group:business:---
mask::rwx
other::r-x
[hadoop@node1 tmp]$ hdfs dfs -getfacl /fftest/OriginalData
15/05/20 16:46:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
# file: /fftest/OriginalData
# owner: hadoop
# group: ff
user::rwx
user:DataAnalytics:r--
user:DataProcessor:rw-
user:appDev:rw-
user:business:---
user:dataCollector:rw-
user:manager:rwx
user:plateformDev:r--
group::r-x
group:appDev:rwx
group:business:---
mask::rwx
other::r-x
15/05/20 16:46:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
# file: /fftest/OriginalData
# owner: hadoop
# group: ff
user::rwx
user:DataAnalytics:r--
user:DataProcessor:rw-
user:appDev:rw-
user:business:---
user:dataCollector:rw-
user:manager:rwx
user:plateformDev:r--
group::r-x
group:appDev:rwx
group:business:---
mask::rwx
other::r-x
Result: business user could not access criticalData, but manager user could
[manager@node1 ~]$ hadoop fs -cat /fftest/CriticalData/test
15/05/20 17:05:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
a
s
d
g
hg
15/05/20 17:05:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
a
s
d
g
hg
[business@node1 root]$ hadoop fs -cat /fftest/CriticalData/test
15/05/20 17:06:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
cat: Permission denied: user=business, access=EXECUTE, inode="/fftest/CriticalData":hadoop:ff:drwxrwxr-x:user:DataAnalytics:r--,user:DataProcessor:r--,user:appDev:---,user:business:---,user:dataCollector:r--,user:manager:rwx,user:plateformDev:r--,group::r-x,group:appDev:---,group:business:---
15/05/20 17:06:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
cat: Permission denied: user=business, access=EXECUTE, inode="/fftest/CriticalData":hadoop:ff:drwxrwxr-x:user:DataAnalytics:r--,user:DataProcessor:r--,user:appDev:---,user:business:---,user:dataCollector:r--,user:manager:rwx,user:plateformDev:r--,group::r-x,group:appDev:---,group:business:---