Summary

In any business, Microsoft Word documents are commonly used. They contain information in the form of raw text, tables, and images. And all of the documents contain facts important to that business. This code pattern addresses the problem of extracting knowledge out of text and tables in domain-specific word documents. We build a knowledge graph on the knowledge extracted, which makes the knowledge queryable. This gives you the best of both worlds – training and a rules-based approach to extract knowledge out of documents.

Description

One of the biggest challenges in the industry today is how to make machines understand data in documents just like humans understand the context and intent of the document by reading it. The first step towards this goal is to convert the unstructured information (free-floating text and tables text) to a semi-structured format and then process it further. That's where graphs play a major role – giving shape and structure to the unstructured information present in the documents. This code pattern looks at the problem of extracting knowledge out of text and tables in domain-specific Word documents. A domain-specific knowledge graph is built on the knowledge extracted, and this makes the knowledge queryable. You can use this code pattern to to shape your analysis and use the data for further processing to get better insights.

The code pattern demonstrates a way to derive insights from a document containing raw text and information in tables using IBM Cloud, IBM Watson services, the Python package Mammoth, the Python NLTK, and IBM Watson Studio.

With this code pattern, you get:

• The ability to process the tables in .docx files along with the free-floating text
• A strategy for combining the results of a real-time analysis by Watson NLU along with the results from the rules defined by a subject matter expert or domain expert.

Flow

1. The unstructured text data from the .docx files (HTML tables and free-floating text) that needs to be analyzed and correlated is extracted from the documents using custom Python code.
2. The text is classified using NLU and tagged using the Extend Watson text classification code pattern.
3. The text is correlated with other text using the Correlate documents code pattern.
4. The results are filtered using custom Python code.
5. The knowledge graph is constructed.

Instructions

Find the detailed steps for this pattern in the README. Those steps will show you how to:

1. Create IBM Cloud services.
2. Run using a Jupyter Notebook in IBM Watson Studio.
3. Analyze the results.