Tuning and optimizer

最新推荐文章于 2024-03-09 11:39:09 发布
最新推荐文章于 2024-03-09 11:39:09 发布
阅读量272 收藏
点赞数
文章标签： statistics performance deprecated application attributes database

Operating system level tuning. 

 

System-level tuning involves the following steps:

  • Monitoring the operating system counters using a tool such as top, gtop, and GKrellM or the VTune analyzer’s counter monitor data collector for applications running on Windows.
  • Interpreting the counter data to locate system-level performance bottlenecks and opportunities for improving the way your application interacts with the system.
SQL-level tuning.
  • Tuning disk and network I/O subsystem to optimize the I/O time, network packet size and dispatching frequency is called the server kernel optimization.
  • Distribution of data can be studied by the optimizer by collecting and storing optimizer statistics. This enables intelligent execution plans.
  • Choice of db_block_size, db_cache_size, and OS parameters (db_file_multiblock_read_count, cpu_count, &c), can influence SQL performance.
  • Tuning SQL Access workload with physical indexes and materialized views.
Database design level tuning

The steps involved in database design level tuning are:

  • Determination of the data needed by an application (what relations are important, their attributes and structuring the data to best meet the performance goals)
  • Analysis of data followed by normalization to eliminate data redundancy.
  • Avoiding data contention.
  • Localizing access to the data to the partition, process and instance levels.
  • Using synchronization points in Oracle Parallel Server.
  • Implementation of 8i enhancements that can help avoid contention are:
    Consideration on partitioning the data
    Consideration over using local or global indexes.
Explain rule-based optimizer and cost-based optimizer.

 

Oracle decides how to retrieve the necessary data whenever a valid SQL statement is processed.

This decision can be made using one of two methods:

Rule Based Optimizer

If the server has no internal statistics relating to the objects referenced by the statement then the RBO method is used.
This method will be deprecated in the future releases of oracle.

Cost Based Optimizer

The CBO method is used if internal statistics are present.
The CBO checks several possible execution plans and selects the one with the lowest cost based on the system resources.

 

 

Dev2010Daily
关注
微调 (Finetuning)
程序员光剑
08-09 635
Fine-tuning: A Comprehensive Guide to Optimizing Machine Learning Models 1. Bac
第三讲：调谐分析（Tuning）和优化设计（Optimization）
qq_36033382的博客
07-09 1495
课程内容概述： 1、通过调谐分析：用户可以实时查看设计中的某个参数对整个电路性能的影响。 2、优化设计：软件可以自动找出满足设计要求的电路参数 3、 扩展功能：扩展应用功能Data Display窗口显示多个Dataset数据、设计模板（Template）的使用 4、实作讲解...
【TensorFlow 入门】2、优化器函数 Optimizer
呆呆的猫的博客
02-02 1万+
因为大多数机器学习任务就是最小化损失，在损失定义的情况下，后面的工作就交给了优化器。因为深度学习常见的是对于梯度的优化，也就是说，优化器最后其实就是各种对于梯度下降算法的优化。 常用的optimizer类 Optimizer GradientDescentOptimizer AdagradOptimizer AdagradDAOptimizer MomentumOptimi
使用SQL tuning advisor(STA)自动优化SQL
热门推荐
乐沙弥的世界
05-30 1万+
Oracle 10g之后的优化器支持两种模式，一个是normal模式，一个是tuning模式。在大多数情况下，优化器处于normal模式。基于CBO的normal模式只考虑很小部分的执行计划集合用于选择哪个执行计划，因为它需要在尽可能短的时间，通常是几秒或毫秒级来对当前的SQL语句进行解析并生成执行计划。因此并不能保证SQL语句每次都是使用最佳的执行计划。而tuning模式则将高负载的SQL语句直
mysql sql tuning_SQL tuning
weixin_28718769的博客
02-17 267
SQL是的全称是Structured Query Language(结构化查询语言)。SQL是一个在80年代中期被使用的工业标准数据库查询语言。不要把SQL语言与商业化产品如Microsoft SQL server或开源产品MySQL相混淆。所有的使用SQL缩略词的这些都是SQL标准的一部分。一、SQL tuning之前的调整下面这个粗略的方法能够节省数千小时乏味的SQL tuning，因为一旦调...
SQL调优指南笔记17：Importing and Exporting Optimizer Statistics
In-Memory Computing Technology
06-07 171
本文为SQL Tuning Guide第17章“Importing and Exporting Optimizer Statistics”的笔记。
Performance Tuning-The Query Optimizer
FightFourEggs的专栏
05-02 686
SQL优化最重要的就是优化器的理解，下面笔者对官方文档自己做了一些翻译和简化，希望对大家有帮助。版本11.2。 一．      优化器环境参数 CURSOR_SHARING：EXACT|FORCE 将sql语句中的文字字符转换为绑定变量。 DB_FILE_MULTIBLOCK_READ_COUNT： sequential 扫描时一次可以读取的块数。全表扫描所需要的IO依赖于表的大小，一次
SQL Tuning Guide
weixin_38061610的博客
06-09 152
【代码】SQL Tuning Guide。
MySQL OPTIMIZER TRACE 详解
Eddie'Blog
05-13 1097
文章目录目录OPTIMIZER TRACE（优化器跟踪）OPTIMIZER TRACE 是什么？OPTIMIZER TRACE 参数OPTIMIZER TRACE 使用OPTIMIZER_TRACE 结果分析join_preparationjoin_optimizationcondition_processingsubstitute_generated_columnstable_dependenciesref_optimizer_key_usesrows_estimationconsidered_execu
Sybase Performance and Tuning：Optimizer and Abstract Plans
09-25
为了确保数据库能够高效稳定地运行，Sybase ASE 提供了一系列性能调优工具和技术，其中最重要的是查询优化器（Optimizer）和抽象执行计划（Abstract Plans）。本文将详细介绍这两个核心概念，并探讨它们在提高Sybase...
Performance Tuning and Optimization Strategies in DBeaver
# 1. Introduction 1.1 What is DBeaver DBeaver is an open-source cross-platform database ... It offers robust database connectivity features, SQL editor, query building and execution, data export and i
Optimization Problems in MATLAB Control Systems: Parameter Tuning and Algorithm Implementation
In today's industrial automation, aerospace, and intelligent transportation systems, the performance of control systems is directly related to the overall efficiency and safety of the system....
【ChatGPT模型精调训练】AI 大模型精调 Fine-Tuning （微调）训练图文代码实战详解
程序员光剑
03-09 796
选择预训练模型：选择一个在类似任务上已经训练好的模型作为起点。数据准备：准备并预处理你的数据集，使其适合模型的输入格式。微调：在你的特定数据集上继续训练模型，调整模型的权重。评估：评估微调后模型的性能。应用：将微调后的模型部署到实际应用中。Fine-Tuning（精调）是指在预训练模型的基础上，对模型进行微调，使其适应特定任务。预训练模型通常在大量无标签数据上进行训练，以学习通用的语言表示。通过 Fine-Tuning，我们可以在较小的标签数据集上训练模型，以便模型能够更好地解决特定任务。
常用的文件系统、存储类型小整理
Hehuyi_In的博客
02-07 3422
最近接触到了五花八门的文件系统、存储类型，名词听得头大，趁假期整理学习一番~
【CSS Tricks】像素风字体、图片
09-17
包含像素风中英文字体，鼠标手势普通状态、点击状态和禁用状态，仅用作技术分享学习研究，不可用于其他用途。
卡瓦牙椅E50life中文使用说明书第一部分.pdf
最新发布
09-17
卡瓦牙椅E50life中文使用说明书第一部分.pdf
ChromiumSetup.exe
09-17
ChromiumSetup.exe
Chain of thought 链式思考赋能Transformer模型解决串行计算难题
09-17
Chain of thought（CoT）使Transformer能够执行串行计算，扩展了它们解决问题的能力，超越了仅限并行的局限性。 增强Transformer的表达能力，特别是对于本质上是顺序问题。 原始问题: 大型语言模型（LLMs）在生成最终答案之前生成中间步骤（连续思考，CoT）时，展现出异常的推理能力。CoT有效性背后的机制尚不清楚，尤其是在zero shot和错误推理场景中。   本文的关键见解: • 在没有CoT的情况下，具有有限精度和多项式(n)嵌入大小的恒定深度Transformer只能解决AC0中的问题 • 通过T步骤的CoT，使用恒定位精度和O(log n)嵌入大小的恒定深度Transformer可以解决任何由大小为T的布尔电路可解决的问题 • CoT大幅提高了低深度Transformer在本质上是串行问题上的准确性   本文的解决方案: • 为通过如下特性解决问题的恒定深度Transformer定义了新的复杂性类CoT[T(n), d(n), s(n), e(n)]： T(n) CoT步骤 d(n) 嵌入大小 s(n) 精度位 e(n) 指数位
vits-fast-fine-tuning google colab
05-20
Sure! Here are the steps to fine-tune ViT-S on a custom dataset using Google Colab: 1. Open a new Google Colab notebook and select a GPU runtime environment. 2. Install the necessary libraries: ``` !pip install torch torchvision !pip install timm ``` 3. Download and prepare the custom dataset. You can use any dataset of your choice. Make sure to split it into training and validation sets. 4. Define the data loaders: ``` import torch import torchvision.transforms as transforms from torch.utils.data import DataLoader from torchvision.datasets import ImageFolder # Define the transformations transform_train = transforms.Compose([ transforms.RandomResizedCrop(224), transforms.RandomHorizontalFlip(), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) ]) transform_val = transforms.Compose([ transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) ]) # Define the data loaders train_dataset = ImageFolder('path_to_train_data', transform=transform_train) train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True, num_workers=4) val_dataset = ImageFolder('path_to_val_data', transform=transform_val) val_loader = DataLoader(val_dataset, batch_size=32, shuffle=False, num_workers=4) ``` Replace 'path_to_train_data' and 'path_to_val_data' with the paths to your training and validation data folders, respectively. 5. Load the pre-trained ViT-S model: ``` import timm model = timm.create_model('vit_small_patch16_224', pretrained=True) ``` 6. Modify the last layer of the model to fit your custom dataset: ``` import torch.nn as nn num_classes = len(train_dataset.classes) model.head = nn.Sequential( nn.LayerNorm((768,)), nn.Linear(768, num_classes) ) ``` Replace '768' with the hidden size of the model you are using. For ViT-S, it is 768. 7. Define the optimizer and criterion: ``` import torch.optim as optim optimizer = optim.Adam(model.parameters(), lr=1e-4) criterion = nn.CrossEntropyLoss() ``` 8. Fine-tune the model: ``` device = torch.device("cuda" if torch.cuda.is_available() else "cpu") model.to(device) num_epochs = 10 for epoch in range(num_epochs): train_loss = 0.0 val_loss = 0.0 correct = 0 total = 0 # Train the model model.train() for inputs, labels in train_loader: inputs, labels = inputs.to(device), labels.to(device) optimizer.zero_grad() outputs = model(inputs) loss = criterion(outputs, labels) loss.backward() optimizer.step() train_loss += loss.item() * inputs.size(0) # Evaluate the model on validation set model.eval() with torch.no_grad(): for inputs, labels in val_loader: inputs, labels = inputs.to(device), labels.to(device) outputs = model(inputs) loss = criterion(outputs, labels) val_loss += loss.item() * inputs.size(0) _, predicted = torch.max(outputs.data, 1) total += labels.size(0) correct += (predicted == labels).sum().item() train_loss = train_loss / len(train_loader.dataset) val_loss = val_loss / len(val_loader.dataset) accuracy = 100 * correct / total print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f} \tAccuracy: {:.2f}'.format( epoch+1, train_loss, val_loss, accuracy)) ``` 9. Save the model: ``` torch.save(model.state_dict(), 'path_to_save_model') ``` Replace 'path_to_save_model' with the path where you want to save the model. That's it! You have fine-tuned ViT-S on your custom dataset using Google Colab.
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值