基于图像链接的批量下载

1. 获取图像路径

1.1 给定图像链接,这是一张图像

image_url = “https://univs-news-1256833609.cos.ap-beijing.myqcloud.com/123/upload/resources/image/7467914.jpg”

1.1通过网站规则得到其想要的图像链接

image_urls = [
f"https://univs-news-1256833609.cos.ap-beijing.myqcloud.com/123/upload/resources/image/{7467914 + i*2}.jpg" for i in range(50)
]

1.2 获取传入的参数,如果传入的图像路径参数是空,则判断是否使用默认的配置(非指定想要的图像链接)

if opt.image_urls:
    image_urls = opt.image_urls
else:
    while True:
        input_id = int(input("The following options:\n [0]: stop running. \n [1]: use default config.\nPlease enter the correct option: "))

        if input_id == 0:
            sys.exit()  # 退出程序
            # break
        elif input_id == 1:
            image_urls = [
                f"https://univs-news-1256833609.cos.ap-beijing.myqcloud.com/123/upload/resources/image/{7467914 + i * 2}.jpg"
                for i in range(50)
            ]
            break  # 输入正确,跳出循环
        else:
            print("input number is wrong,please input correct.\n")

2. 通过图像路径,获取图像,并下载到本地

# 遍历下载图像
    id = 1
    for i, image_url in enumerate(image_urls):
        # 发起HTTP请求获取图像数据
        response = requests.get(image_url)
        if response.status_code == 200:
            # 将图像数据转换为PIL Image对象
            image_data = BytesIO(response.content)
            image = PILImage.open(image_data)
            # 可以将 image 用于展示、保存或进一步处理
            image.save(f"./{cache_path}/{id}_{image_url.rsplit('/', 1)[1][:-4]}.png")  # 保存图像
            id += 1
        else:
            # 请求不到的数据进行错误返回
            print(f"Failed to download image: {i}_{image_url}")

3. 完整的代码

import os
import shutil
import sys
import argparse
import requests
from PIL import Image as PILImage
from io import BytesIO


def get_opt():
    """参数设置"""
    parse = argparse.ArgumentParser()
    parse.add_argument("--cache_path", type=str, default="./tmp/data", help="缓存文件路径")
    parse.add_argument("--image_urls", type=list, default=[], help="请求的图像链接")
    opt = parse.parse_args()
    return opt


def main():
    # 获取配置信息
    opt = get_opt()
    # 本地缓存路径
    cache_path = opt.cache_path

    # 清除现有的缓存文件夹(如果有的话)
    try:
        shutil.rmtree(cache_path)
    except:
        pass
    # 创建缓存文件夹
    os.makedirs(cache_path)

    # 获取图像链接
    if opt.image_urls:
        image_urls = opt.image_urls
    else:
        while True:
            input_id = int(input("The following options:\n [0]: stop running. \n [1]: use default config.\nPlease enter the correct option: "))

            if input_id == 0:
                sys.exit()  # 退出程序
                # break
            elif input_id == 1:
                image_urls = [
                    f"https://univs-news-1256833609.cos.ap-beijing.myqcloud.com/123/upload/resources/image/{7467914 + i * 2}.jpg"
                    for i in range(50)
                ]
                break  # 输入正确,跳出循环
            else:
                print("input number is wrong,please input correct.\n")

    # 遍历下载图像
    id = 1
    for i, image_url in enumerate(image_urls):
        # 发起HTTP请求获取图像数据
        response = requests.get(image_url)
        if response.status_code == 200:
            # 将图像数据转换为PIL Image对象
            image_data = BytesIO(response.content)
            image = PILImage.open(image_data)
            # 可以将 image 用于展示、保存或进一步处理
            image.save(f"./{cache_path}/{id}_{image_url.rsplit('/', 1)[1][:-4]}.png")  # 保存图像
            id += 1
        else:
            # 请求不到的数据进行错误返回
            print(f"Failed to download image: {i}_{image_url}")


if __name__ == '__main__':
    main()
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
运行环境 .NET Framework2.0 开发工具 Microsoft Visual Studio 2005 二. 部分代码说明(主要讲解异步分析和下载): 异步分析下载采取的策略是同时分析同时下载,即未等待数据全部分析完毕就开始把已经分析出来的图片链接开始下载下载成功的均在List框链接前面划上了√ ,未能下载图片有可能是分析错误或者是下载异常。 1. 异步分析部分代码 /// /// 异步分析下载 /// private void AsyncAnalyzeAndDownload(string url, string savePath) { this.uriString = url; this.savePath = savePath; #region 分析计时开始 count = 0; count1 = 0; freq = 0; result = 0; QueryPerformanceFrequency(ref freq); QueryPerformanceCounter(ref count); #endregion using (WebClient wClient = new WebClient()) { AutoResetEvent waiter = new AutoResetEvent(false); wClient.Credentials = CredentialCache.DefaultCredentials; wClient.DownloadDataCompleted += new DownloadDataCompletedEventHandler(AsyncURIAnalyze); wClient.DownloadDataAsync(new Uri(uriString), waiter); //waiter.WaitOne(); //阻止当前线程,直到收到信号 } } /// /// 异步分析 /// protected void AsyncURIAnalyze(Object sender, DownloadDataCompletedEventArgs e) { AutoResetEvent waiter = (AutoResetEvent)e.UserState; try { if (!e.Cancelled && e.Error == null) { string dnDir = string.Empty; string domainName = string.Empty; string uri = uriString; //获得域名 http://www.sina.com/ Match match = Regex.Match(uri, @"((http(s)?://)?)+[\w-.]+[^/]");//, RegexOptions.IgnoreCase domainName = match.Value; //获得域名最深层目录 http://www.sina.com/mail/ if (domainName.Equals(uri
批量标准化(Batch Normalization)和 Dropout 是深度学习中常用的正则化方法,可以有效地防止神经网络过拟合。 下面是一个基于 PyTorch 的水果图像识别模型,我们将在这个模型中添加批量标准化和 Dropout。 首先,我们需要导入 PyTorch 和相关的包: ```python import torch import torch.nn as nn import torch.nn.functional as F import torch.optim as optim from torchvision import datasets, transforms ``` 然后,我们定义一个水果图像识别模型 FruitNet,该模型包含两个卷积层、两个池化层和三个全连接层。 ```python class FruitNet(nn.Module): def __init__(self): super(FruitNet, self).__init__() self.conv1 = nn.Conv2d(3, 32, kernel_size=3, stride=1, padding=1) self.conv2 = nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1) self.pool = nn.MaxPool2d(2, 2) self.fc1 = nn.Linear(64 * 8 * 8, 128) self.fc2 = nn.Linear(128, 64) self.fc3 = nn.Linear(64, 5) def forward(self, x): x = self.pool(F.relu(self.conv1(x))) x = self.pool(F.relu(self.conv2(x))) x = x.view(-1, 64 * 8 * 8) x = F.relu(self.fc1(x)) x = F.relu(self.fc2(x)) x = self.fc3(x) return x ``` 接下来,我们定义一个函数 train_model,该函数用于训练模型,并添加批量标准化和 Dropout。 ```python def train_model(model, train_loader, optimizer, epoch): model.train() for batch_idx, (data, target) in enumerate(train_loader): optimizer.zero_grad() data = data.to(device) target = target.to(device) # 添加批量标准化 data = nn.BatchNorm2d(3)(data) output = model(data) # 添加 Dropout output = nn.Dropout(p=0.5)(output) loss = F.cross_entropy(output, target) loss.backward() optimizer.step() if batch_idx % 20 == 0: print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format( epoch, batch_idx * len(data), len(train_loader.dataset), 100. * batch_idx / len(train_loader), loss.item())) ``` 在训练模型的过程中,我们将输入数据进行批量标准化和 Dropout 处理,以防止过拟合。 最后,我们定义一个函数 test_model,该函数用于测试模型,并计算测试准确率。 ```python def test_model(model, test_loader): model.eval() test_loss = 0 correct = 0 with torch.no_grad(): for data, target in test_loader: data = data.to(device) target = target.to(device) # 添加批量标准化 data = nn.BatchNorm2d(3)(data) output = model(data) # 添加 Dropout output = nn.Dropout(p=0.5)(output) test_loss += F.cross_entropy(output, target, reduction='sum').item() pred = output.argmax(dim=1, keepdim=True) correct += pred.eq(target.view_as(pred)).sum().item() test_loss /= len(test_loader.dataset) print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format( test_loss, correct, len(test_loader.dataset), 100. * correct / len(test_loader.dataset))) ``` 现在,我们可以使用上述函数来训练和测试我们的水果图像识别模型了。 ```python # 设置设备 device = torch.device("cuda" if torch.cuda.is_available() else "cpu") # 加载数据 train_dataset = datasets.ImageFolder('fruits-360/Training', transform=transforms.Compose([ transforms.Resize((32, 32)), transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)) ])) test_dataset = datasets.ImageFolder('fruits-360/Test', transform=transforms.Compose([ transforms.Resize((32, 32)), transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)) ])) train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=64, shuffle=True) test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=64, shuffle=True) # 初始化模型和优化器 model = FruitNet().to(device) optimizer = optim.Adam(model.parameters(), lr=0.001) # 训练模型 for epoch in range(1, 11): train_model(model, train_loader, optimizer, epoch) test_model(model, test_loader) ``` 在训练过程中,我们会看到输出的训练损失和测试准确率。您可以根据需要调整批量标准化和 Dropout 的参数来优化模型的性能。

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

Kungs8

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值