人工智能+智能运维解决方案_委员会再发布三个问题，向AI解决方案供应商提问，称他们使用人工方法...-CSDN博客

人工智能+智能运维解决方案

AI has become the latest bandwagon onto which everyone wants to jump. Organizations need to figure out which solutions (and vendors) are credible and which are merely looking to benefit from the hype.

人工智能已经成为每个人都想跳入的最新潮流。组织需要确定哪些解决方案(和供应商)是可信的，哪些只是希望从炒作中受益。

In my earlier post on this subject, I proposed three questions to ask a vendor about their AI capabilities. Here I’ll build on those earlier comments with three additional questions. Taken together, these can give you a solid test of who really knows AI and who doesn’t.

在我先前关于该主题的帖子中，我提出了三个问题，向供应商询问其AI功能。在这里，我将基于先前的评论以及三个附加问题。综上所述，这些可以为您提供真正的知识，哪些人真正知道AI，哪些人不知道。

While you can ask vendors these questions, you can also ask your security or technology team the same questions to determine how they respond to issues. It’s just as important for them to understand the capabilities and limitations they face, as it will help guide their review of any AI-based solutions. I suggest using these questions to determine if your team’s responses match that of the AI solution to assure there is business value alignment.

您可以向供应商询问这些问题，也可以向安全或技术团队询问相同的问题，以确定他们对问题的响应方式。对于他们来说，了解他们面临的功能和局限性同样重要，因为它将有助于指导他们对任何基于AI的解决方案进行审查。我建议使用这些问题来确定您的团队的响应是否与AI解决方案的响应相匹配，以确保实现业务价值一致。

With that, here are the three additional questions that I recommend you ask AI solution vendors and your own team.

因此，我建议您向AI解决方案供应商和您自己的团队询问另外三个问题。

Question 4: What are the goals of each algorithm?

问题4：每种算法的目标是什么？

Weak answer: I’d watch out for anyone who provides an overly generic answer, such as, “It’s to detect threats and cyber attacks.” The role of a particular algorithm is usually much more granular than this.

答案很弱：我会提防任何提供过于笼统答案的人，例如“用于检测威胁和网络攻击”。特定算法的作用通常比此要细得多。

Strong answer: The solution provider states that they’re using many algorithms because each data set requires its own technique. Different algorithms are best suited to different types of data to achieve different objectives — just as a chef uses different knives for different tasks or different types of food. One size does not fit all. If the solution is robust, the tool should use various algorithms for clustering, classification, outliers, cluster samples based on behavior and so forth.

强有力的答案：解决方案提供商指出他们正在使用许多算法，因为每个数据集都需要使用自己的技术。不同的算法最适合于不同类型的数据，以实现不同的目标-就像厨师为不同任务或不同类型的食物使用不同的刀具一样。一种尺寸并不适合所有尺寸。如果解决方案是可靠的，则该工具应使用各种算法进行聚类，分类，离群值，基于行为的聚类样本等。

Question 5: How often are your ML models updated? How well do they perform against drift?

问题5：您的ML模型多久更新一次？ 他们对漂移的表现如何？

Weak answer: The vendor states that they don’t know. Alternatively, they might say never, which often makes the ML models slow to respond to new issues, or they might say daily, which makes it little better than a signature-based system in that it’s dependent upon very frequent updates. If the AI is trained appropriately, it learns as it goes. So while it does need updating, it certainly doesn’t need to be updated either daily or never.

答案很微弱：供应商指出他们不知道。或者，他们可能会说“永不”，这通常会使ML模型对新问题的响应变慢，或者他们可能每天都会说，这使其与基于签名的系统有所不同，因为它依赖于非常频繁的更新。如果对AI进行了适当的培训，它就会随心所欲地学习。因此，尽管确实需要更新，但是肯定不需要每天或永远不需要更新。

Strong answer: The solution provider demonstrates that they have some kind of plan or mechanism in place. The exact update process or what the schedule consists of is less important than the fact that they have one. It could be updating the models themselves, implementing “patches” between model updates or using signatures for something problematic that the system didn’t previously detect. Also, listen for signs that they’re familiar with the idea of “concept drift,” which is when the data set the system is modeling evolves over time. For example, in cybersecurity — my company’s industry — if you’re analyzing employees’ remote access to corporate servers, over time more employees may do this more often, which could look like an anomaly and generate an alert (a false alarm). Alternatively, they can adjust the model to accommodate the changing nature of legitimate behavior (drift).

强有力的答案：解决方案提供商证明他们已经制定了某种计划或机制。确切的更新过程或计划表所包含的内容并不重要。可能是在更新模型本身，在模型更新之间实施“补丁”，或者对系统先前未检测到的问题使用签名。另外，请听取迹象表明他们熟悉“概念漂移”的概念，这是系统建模的数据集随时间演变的时候。例如，在网络安全(我公司的行业)中，如果您要分析员工对公司服务器的远程访问，则随着时间的流逝，更多的员工可能会更频繁地执行此操作，这看起来像是一种异常并会生成警报(错误警报)。或者，他们可以调整模型以适应合法行为(漂移)的变化性质。

Question 6: What happens when your ML makes a bad decision?

问题6：当您的ML做出错误的决定时会发生什么？

Weak answer: The vendor says it never happens or has the user contact them for an update or wait for a signature update.

错误的回答：供应商说它永远不会发生，或者让用户联系他们进行更新或等待签名更新。

Strong answer: The solutions provider acknowledges that this happens and that when it does, action needs to be taken. Specifically, the vendor should incorporate findings into training, such as how to adapt to false positives and false negatives. These findings can then be used to reset thresholds and scoring or adjust models to accommodate the unforeseen situation that caused the error.

强有力的答案：解决方案提供商承认这种情况会发生，并且确实需要采取措施。具体来说，供应商应将调查结果纳入培训中，例如如何适应误报和误报。这些发现随后可用于重置阈值和评分或调整模型，以适应导致错误的不可预见的情况。

Another way to address “bad” decisions is to use multiple techniques simultaneously to decrease the risk of a mistake — what some call a layered approach. For example, some network traffic analysis (NTA) solutions will flag activity as anomalous, and while the activity might not be bad (i.e., malicious) it is still anomalous, so the detection is not wrong. Having other systems that understand what malicious behavior looks like, understand IP address reputation and analyze email activity could put that “bad” anomaly into a broader context that could make it clear either that it’s benign or that it’s malicious. Either way, complementing the initial detection with additional context and added layers can reduce the number of bad decisions.

解决“不良”决策的另一种方法是同时使用多种技术来降低错误的风险-有些人称之为分层方法。例如，某些网络流量分析(NTA)解决方案会将活动标记为异常，并且该活动可能还不错(即恶意)，但仍然是异常的，因此检测没有错。让其他系统了解恶意行为的外观，了解IP地址信誉并分析电子邮件活动，可能会将“不良”异常置于更广泛的范围内，从而可以清楚地表明它是良性的还是恶意的。无论哪种方式，都可以通过添加其他上下文和添加的层来补充初始检测，从而减少错误决策的数量。

Looking For The Best Solution

寻找最佳解决方案

The questions above reveal a lot about an AI solution. Sometimes vendors give strong answers on some questions and weak answers on others. Just keep in mind what you’re trying to achieve: You’re trying to make a determination about the vendor and their technology. Are they just claiming they use AI but don’t have the necessary skills, staff or expertise to use it effectively, or are they staffed with data scientists who have been working with AI for decades and can, therefore, put it to the best possible use toward creating a truly effective product?

上面的问题揭示了很多关于AI解决方案的问题。有时，供应商在某些问题上给出有力的答案，而在另一些问题上给出不强的答案。只需记住您要实现的目标：您就试图确定供应商及其技术。他们是只是声称自己在使用AI，但没有有效使用AI的必要技能，人员或专业知识，又或者是与从事AI已有数十年历史的数据科学家一起工作，因此可以将AI发挥到最大用于创造真正有效的产品？

Originally published at https://www.forbes.com.

最初在https://www.forbes.com上发布。