用katalon开发爬虫程序

katalon整合了selenium和groovy，是一款强大的web测试工具。

实际上katalon不仅可以写测试代码，还可以写简单的爬虫。katalon最大的优点就是整合了selenium的各种webdriver，不用去安装环境，开包即用非常方便。

比如说有这样一个需求，判断批量的网页是不是对应了手机的小屏幕显示（也就是响应式网页），我们下面的kotalon测试用例来实现

import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import static com.kms.katalon.core.testobject.ObjectRepository.findWindowsObject
import com.kms.katalon.core.checkpoint.Checkpoint as Checkpoint
import com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords as CucumberKW
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.model.FailureHandling as FailureHandling
import com.kms.katalon.core.testcase.TestCase as TestCase
import com.kms.katalon.core.testdata.TestData as TestData
import com.kms.katalon.core.testng.keyword.TestNGBuiltinKeywords as TestNGKW
import com.kms.katalon.core.testobject.TestObject as TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import com.kms.katalon.core.windows.keyword.WindowsBuiltinKeywords as Windows
import internal.GlobalVariable as GlobalVariable
import org.openqa.selenium.Keys as Keys
import org.openqa.selenium.WebDriver as WebDriver
import com.kms.katalon.core.webui.driver.DriverFactory as DriverFactory
import org.openqa.selenium.By as By

WebUI.openBrowser('')
WebDriver driver = DriverFactory.getWebDriver()

BufferedReader reader = new BufferedReader(new FileReader("d:\\url.txt"));

BufferedWriter writer = new BufferedWriter(new FileWriter("d:\\url_out.txt"));
	
//reader.readLine();//第一行信息，为标题信息，不用,如果需要，注释掉
String line = null;
while((line=reader.readLine())!=null){
//String item[] = line.split(",");//CSV格式文件为逗号分隔符文件，这里根据逗号切分

System.out.println(line);
	WebUI.navigateToUrl(line.trim())
	
	def metaList = driver.findElements(By.xpath('/html/head/meta[@name=\'viewport\']'))
	if(metaList.size()==0) {
		writer.println(line+',pc')
	}else {
		writer.println(line+',sp')
	}
}
writer.flush();

writer.close();


WebUI.closeBrowser()

程序说明

1.输入的url一览在url.txt中，输出的url一览在url_out.txt中，带pc的不是响应式网页，带sp的是响应式网页

2.判断是不是响应式网页的条件：是否包含viewport的meta tag，这样做只能判断页面是不是响应式网页，因为不带viewport也可以做到对应手机

很简单吧，下载一个katalon，不用配置环境，代码分分钟写完，这就是katalon的魅力。