COVID-19研究助理

最新推荐文章于 2024-09-17 23:15:58 发布

weixin_26746401

最新推荐文章于 2024-09-17 23:15:58 发布

阅读量346

点赞数

文章标签： python

原文链接：https://medium.com/swlh/covid-19-research-assistant-using-ai-watson-discovery-to-analyze-open-research-dataset-by-kaggle-9807cf467626

版权

These days scientists, researchers, doctors, and medical professionals face challenges to develop answers to their high priority scientific questions.

如今，科学家，研究人员，医生和医学专家面临着挑战，无法为其高度优先的科学问题找到答案。

The rapid acceleration in new coronavirus literature makes it difficult for the medical research community to Keep Up. Therefore there’s a growing urgency for approaches in Natural Language Processing and AI to help medical professionals generate new insights in support of the ongoing fight against this infectious disease.

新的冠状病毒文献的Swift发展使医学研究界难以跟上。 因此，越来越需要采用自然语言处理和AI的方法来帮助医学专业人士产生新见解，以支持正在进行的抵抗这种传染病的斗争。

Objective:

目的：

We aim to assist medical professionals to accelerate their work to help fight COVID19. This will help reduce search time for the medical professional time by accessing a wider range of research resources. All the resources they need in one place.

我们旨在协助医疗专业人员加快工作速度，以对抗COVID19。通过访问更广泛的研究资源，这将有助于减少医学专业人士的搜索时间。他们需要的所有资源都集中在一处。

Datasets challenge:

数据集挑战：

Kaggle has prepared free accessible datasets related to COVID-19 Open Research Dataset (CORD-19).

Kaggle已准备了与COVID-19开放研究数据集(CORD-19)相关的免费的可访问数据集。

Image for post — Open Research Dataset Challenge (CORD-19)

The Cord-19 resource offers more than 158,000 scholarly articles, including over 75,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses.

Cord-19资源提供了超过158,000篇学术文章，其中包括超过75,000篇全文，涉及COVID-19，SARS-CoV-2和相关冠状病毒。

We found these datasets useful to apply the Watson Discovery AI Search Engine on those articles.

我们发现这些数据集对于将Watson Discovery AI搜索引擎应用于这些文章非常有用。

Watson Discovery is a search tool powered by machine learning which can continue to learn and improve over time.

Watson Discovery是一种基于机器学习的搜索工具，可以随着时间的推移不断学习和改进。

With this provided datasets 158,000 scholarly articles, we have only prepared “comm_use_subset” which it has 9,120 articles to feed inside Watson Discovery.

借助此提供的数据集，有158,000篇学术文章 ， 我们仅准备了“ comm_use_subset ”，其中有 9,120 篇文章可以在Watson Discovery中提供。

Solution:

解：

We are looking into building an assistant smart AI conversational chatbot to answer the user’s high priority scientific questions.

我们正在研究建立一个辅助智能AI对话聊天机器人，以回答用户的高优先级科学问题。

Step 1: Data analysis: clean the data from JSON files based on text-only:

步骤1：数据分析：基于纯文本清除JSON文件中的数据：

we are extracting articles “full-text article” from JSON files and save the results in the form of Txts \ Html.

我们 将从JSON文件 中提取文章 “全文文章” ，并将结果保存为Txts \ Html的形式。

Due to Watson Discovery limit with 50k characters for every single document, the datasets are provided in “JSON files” which all the file has more than 50k characters because of JSON codes. Therefore, We have applied this simple py script below to extract “full-text article” from JSON files and save the results in the form of TXT \ HTML.

由于每个文档的Watson Discovery限制为50k个字符，因此在“ JSON文件”中提供了数据集，由于JSON代码，所有文件都超过50k个字符。 因此，我们在下面应用了这个简单的py脚本，以 从JSON文件中 提取 “全文文章” ，并将结果保存为TXT \ HTML格式。

In our case, we have saved the results in HTML format, because WD doesn’t support Txt format. WD only supports document formats like pdf, word, excel, PowerPoint, Html, png, jpeg, and JSON.

在本例中，我们将结果保存为HTML格式，因为WD不支持Txt格式。 WD仅支持pdf，word，excel，PowerPoint，Html，png，jpeg和JSON等文档格式。

The script below does; Each executed file formatted like the following order:

下面的脚本可以；每个执行的文件的格式如下：

title
标题
abstract
抽象
full-text article
全文

This task helped us to have clear data formatted as a text document, it will be easy to manage the data capacity for the numbers of characters in each files.

此任务帮助我们将清晰的数据格式化为文本文档，可以轻松管理每个文件中字符数的数据容量。

Py Script:

Py脚本：

import json
import os, glob#directory for atricle json files:
articles_dir = 'D:/DEV/Kaggle/CORD-19-research-challenge/comm_use_subset/comm_use_subset'
#output directory for processed files:
output_dir = 'D:/DEV/Kaggle/CORD-19-research-challenge/output'os.chdir(articles_dir)
for file in glob.glob('*.json'):
    print('Processing file: ',file)
    with open(file, 'r', encoding = 'utf8') as article_file:
        article = json.load(article_file)
        title = article['metadata']['title']
        abstract_sections = []
        abstract_texts = dict()
        
        body_sections = []
        body_texts = dict()#reading abstracts
        for abst in article['abstract']:
            abst_section = abst['section']
            abst_text = abst['text']
            if abst_section not in abstract_sections:
                abstract_sections.append(abst_section)
                abstract_texts[abst_section] = abst_text
            else:
                abstract_texts[abst_section] = abstract_texts[abst_section] + '\n' + abst_text
        
        #reading body
        for body in article['body_text']:
            body_section = body['section']
            body_text = body['text']
            if body_section not in body_sections:
                body_sections.append(body_section)
                body_texts[body_section] = body_text
            else:
                body_texts[body_section] = body_texts[body_section] + '\n' + body_text
        
        with open(output_dir+'/clean.'+file.replace('.json','.html') , 'w', encoding = 'utf8') as out_file:
            out_file.writelines(title)
            out_file.writelines('\n\n')
            #print abstracts
            for a_section in abstract_sections:
                out_file.writelines('\n\n')
                out_file.writelines(a_section)
                out_file.writelines('\n')
                out_file.writelines(abstract_texts[a_section])
            #print body
            for b_section in body_sections:
                out_file.writelines('\n\n')
                out_file.writelines(b_section)
                out_file.writelines('\n')
                out_file.writelines(body_texts[b_section])
            out_file.writelines('\n')

Step 2: Feed Watson Discovery:

步骤2：输入 Watson Discovery：

Create your IBM free Cloud account: https://ibm.biz/BdqbAU

创建您的IBM免费云帐户： https : //ibm.biz/BdqbAU

With Watson Discovery smart AI search engine, we have fed and trained our queries and rated the results with WD Machine learning.

借助Watson Discovery智能AI搜索引擎，我们已经喂饱并训练了我们的查询，并通过WD Machine learning对结果进行了评分。

Rate the best relevant article for an example question that will be asking by a researcher.

为最相关的文章评分，以解决研究人员将要提出的示例问题。

This task required a lot of reading and understanding the academic and scientific articles, we have built around 100 queries so far.

此任务需要大量阅读和理解学术和科学文章，到目前为止，我们已经建立了约100个查询。

Expected questions from the user with best relevant answers.

用户期望的问题以及最佳的相关答案。

Step4: Integrate Watson Assistant with Watson Discovery:

步骤4：将Watson Assistant与Watson Discovery集成在一起：

Watson Assistant is a conversation AI platform that helps you provide customers fast, straightforward, and accurate answers to their questions, across any application, device, or channel.

Watson Assistant是一个对话式AI平台，可帮助您在任何应用程序，设备或渠道上为客户提供快速，直接，准确的问题解答。

Calling Watson Assistant from Java Script for the server connection:

从Java Script调用Watson Assistant进行服务器连接：

const AssistantV1 = require('ibm-watson/assistant/v1');
const { getAuthenticatorFromEnvironment } = require('ibm-watson/auth');// need to manually set url and disableSslVerification to get around
// current Cloud Pak for Data SDK issue IF user uses
// `CONVERSATION_` prefix in run-time environment.
let auth;
let url;
let disableSSL = false;try {
  // ASSISTANT should be used
  auth = getAuthenticatorFromEnvironment('ASSISTANT');
  url = process.env.ASSISTANT_URL;
  if (process.env.ASSISTANT_DISABLE_SSL == 'true') {
    disableSSL = true;
  }
} catch (e) {
  // but handle if alternate CONVERSATION is used
  auth = getAuthenticatorFromEnvironment('CONVERSATION');
  url = process.env.CONVERSATION_URL;
  if (process.env.CONVERSATION_DISABLE_SSL == 'true') {
    disableSSL = true;
  }
}
console.log('Assistant auth:',JSON.stringify(auth, null, 2));const assistant = new AssistantV1({
  version: '2020-03-01',
  authenticator: auth,
  url: url,
  disableSslVerification: disableSSL
});// SDK uses workspaceID, but Assistant tooling refers to the this value as the SKILL ID.
assistant.workspaceId = process.env.ASSISTANT_SKILL_ID;module.exports = assistant;

Step 5: Test the app

步骤5 ：测试应用

The methodology is defined as: - The user interacts with Watson Assistant.- Watson Assistant Invokes Watson Discovery.- Watson Discovery finds the optimal results regarding the queries and responds to the Assistant.- Watson Assistant displays the results to the User.

该方法定义为： -用户与Watson Assistant交互。-Watson Assistant调用Watson Discovery。-Watson Discovery找到有关查询的最佳结果并响应Assistant。-Watson Assistant将结果显示给用户。

Finally, we have Integrated Watson Assistant with Watson Discovery, then configured the front-end app with Watson Assistant, then deployed on the IBM cloud. The app is live running, we are going to keep it alive for a while

最后，我们将Watson Assistant与Watson Discovery集成在一起，然后使用Watson Assistant配置了前端应用程序，然后将其部署在IBM云上。该应用程序正在运行，我们将使其保持一段时间

live Demo: https://covid19assistantcfc.mybluemix.net/

现场演示： https : //covid19assistantcfc.mybluemix.net/

We are still generating real crisis questions from the abstracts and the articles, we will be able to keep training the Discovery, and rates the best answers for the bot.

我们仍在从摘要和文章中产生真正的危机问题，我们将能够继续培训Discovery，并为该机器人评价最佳答案。

Project Demo:

项目演示：

演示地址

Conclusion:

结论：

To conclude, this conversational AI chatbot in the research community can be beneficial to help scientists and doctors reducing time and accelerating their work to fight back COVID-19.

总而言之，研究社区中的这种对话式AI聊天机器人可以帮助科学家和医生减少时间并加快反击COVID-19的工作。