Python 代码片段

山形依旧枕寒流

已于 2022-03-07 09:35:03 修改

阅读量164

点赞数

分类专栏： Python 文章标签： aws 云计算

于 2022-03-03 09:13:19 首次发布

本文链接：https://blog.csdn.net/liu14116/article/details/123204301

版权

Python 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

List item

Python 代码片段

Python 代码片段

Python 代码片段

boto3

参考文章
https://www.stackvidhya.com/download-files-from-s3-using-boto3/

Boto3 is an AWS SDK for Python. It allows users to create, and manage AWS services such as EC2 and S3. It provides object-oriented API services and low-level services to the AWS services.

create session in Boto3 [Python]

import boto3
# created a generic session 
session = boto3.Session(
    aws_access_key_id=settings.AWS_SERVER_PUBLIC_KEY,
    aws_secret_access_key=settings.AWS_SERVER_SECRET_KEY
)
# Use the below command to access S3 as a resource using the session.
s3 = session.resource('s3')

import boto3
# If you do not want to create a session and access the resource, you can create an s3 client directly by using the following command.
s3_client = boto3.client('s3',
aws_access_key_id=settings.AWS_SERVER_PUBLIC_KEY,
aws_secret_access_key=settings.AWS_SERVER_SECRET_KEY,
region_name=REGION_NAME
)

Download A Single File From S3 Using Boto3

import boto3
session = boto3.Session(
    aws_access_key_id=<Access Key ID>,
    aws_secret_access_key=<Secret Access Key>,
)
s3 = session.resource('s3')
s3.Bucket('BUCKET_NAME').download_file('OBJECT_NAME', 'FILE_NAME')
print('success')

import boto3
s3_client = boto3.client('s3',
aws_access_key_id=<Access Key ID>,
aws_secret_access_key=<Secret Access Key>,
region_name='ap-south-1'
)
s3_client.download_file('BUCKET_NAME', 'OBJECT_NAME', 'FILE_NAME')
print('success')

Download all from S3 Bucket using Boto3 [Python]

import os
import boto3
#Create Session
session = boto3.Session(
aws_access_key_id=settings.AWS_SERVER_PUBLIC_KEY,
aws_secret_access_key=settings.AWS_SERVER_SECRET_KEY,
)
#Initiate S3 Resource
s3 = session.resource('s3')
# Select Your S3 Bucket
your_bucket = s3.Bucket('your_bucket_name')
# Iterate All Objects in Your S3 Bucket Over the for Loop
for s3_object in your_bucket.objects.all():
#Use this statement if your files are available directly in your bucket. 
your_bucket.download_file(s3_object.key, filename_with_extension)
#use below three line ONLY if you have sub directories available in S3 Bucket
#Split the Object key and the file name.
#parent directories will be stored in path and Filename will be stored in the filename
path, filename = os.path.split(s3_object.key)
#Create sub directories if its not existing
os.makedirs(path)
#Download the file in the sub directories or directory if its available. 
your_bucket.download_file(s3_object.key, path/filename)

读取S3文件并下载

读取S3文件并下载到本地(创建本地路径)方法一

import boto3
import os
from pathlib import Path

s3 = boto3.resource('s3')
bucket = s3.Bucket('bucketname')
key = 'product/myproject/2021-02-15/'
objs = list(bucket.objects.filter(Prefix=key))

for obj in objs:
    # print(obj.key)
    # remove the file name from the object key
    obj_path = os.path.dirname(obj.key)
    # create nested directory structure
    Path(obj_path).mkdir(parents=True, exist_ok=True)
    # save file with full path locally
    bucket.download_file(obj.key, obj.key)

读取S3文件并下载到本地方法二

import boto3
import os

s3_client = boto3.client("s3")

def getFilesByMonth(yearStr, monthStr):
    Prefix = 'staging/2022/02/'
    files_dict = s3_client.list_objects(
        Bucket=BUCKET_NAME, Prefix=PrefixStr)

    for file_dict in files_dict['Contents']:
        print(file_dict['Key'].split('/')[-1])
        downloadfilefromS3(file_dict['Key'], file_dict['Key'].split('/')[-1])
        
def downloadfilefromS3(prefexStr, fileName):
    KEY = prefexStr
    s3.Bucket(BUCKET_NAME).download_file(
        KEY, fileName)

时间相关操作

月份加一

第三方模块：python-dateutil
安装方式：pip install python-dateutil

import datetime
from dateutil.relativedelta import relativedelta
 
datetime_now = datetime.datetime.now()
datetime_three_month_ago = datetime_now - relativedelta(months=3)
print datetime_three_month_ago

解压

解压TAR文件至指定文件夹的实例

######### Extract all files from src_dir to des_dir
def extract_tar_files(src_dir,des_dir):
  files = os.listdir(src_dir)
  for file in files:
    dir_tmp = os.path.join(src_dir, file)
    print dir_tmp
    if not os.path.isdir(dir_tmp): ##是文件，非文件夹
      #解压特定文件
      if dir_tmp.endswith("gz") and (dir_tmp.find(cs.Port_week_perfer_name_start) != -1):
        #f = zipfile.ZipFile(dir_tmp, mode="r")
        f = tarfile.open(dir_tmp)
        names = f.getnames()
        for name in names:
          f.extract(name, path=des_dir)
    else:
      extract_tar_files(dir_tmp,des_dir)
  return 0

Path相关操作

字符串相关

#coding=utf-8
str="http://www.runoob.com/python/att-string-split.html"
print("0:%s"%str.split("/")[-1])
print("1:%s"%str.split("/")[-2])
print("2:%s"%str.split("/")[-3])
print("3:%s"%str.split("/")[-4])
print("4:%s"%str.split("/")[-5])

print("5:%s"%str.split("/",-1))
print("6:%s"%str.split("/",0))
print("7:%s"%str.split("/",1))
print("8:%s"%str.split("/",2))
print("9:%s"%str.split("/",3))
print("10:%s"%str.split("/",4))
print("11:%s"%str.split("/",5))
结果是：
0:att-string-split.html
1:python
2:www.runoob.com
3:
4:http:
5:['http:', '', 'www.runoob.com', 'python', 'att-string-split.html']
6:['http://www.runoob.com/python/att-string-split.html']
7:['http:', '/www.runoob.com/python/att-string-split.html']
8:['http:', '', 'www.runoob.com/python/att-string-split.html']
9:['http:', '', 'www.runoob.com', 'python/att-string-split.html']
10:['http:', '', 'www.runoob.com', 'python', 'att-string-split.html']
11:['http:', '', 'www.runoob.com', 'python', 'att-string-split.html']
即： -1  ：全切  0 ：不切  1：切一刀  （每一块都保留）
[-1]:将最后一块切割出来    [-2]：将倒数第二块切割出来  （只保留切出来的一块）

str = '0123456789'
print str[0:3] #截取第一位到第三位的字符
print str[:] #截取字符串的全部字符
print str[6:] #截取第七个字符到结尾
print str[:-3] #截取从头开始到倒数第三个字符之前
print str[2] #截取第三个字符
print str[-1] #截取倒数第一个字符
print str[::-1] #创造一个与原字符串顺序相反的字符串
print str[-3:-1] #截取倒数第三位与倒数第一位之前的字符
print str[-3:] #截取倒数第三位到结尾
print str[:-5:-3] #逆序截取

输出结果如下：

012
0123456789
6789
0123456
2
9
9876543210
78
789
96

功能快捷键

撤销：Ctrl/Command + Z
重做：Ctrl/Command + Y
加粗：Ctrl/Command + B
斜体：Ctrl/Command + I
标题：Ctrl/Command + Shift + H
无序列表：Ctrl/Command + Shift + U
有序列表：Ctrl/Command + Shift + O
检查列表：Ctrl/Command + Shift + C
插入代码：Ctrl/Command + Shift + K
插入链接：Ctrl/Command + Shift + L
插入图片：Ctrl/Command + Shift + G
查找：Ctrl/Command + F
替换：Ctrl/Command + G