Creating multimedia annotators

You can create custom annotators to analyze multimedia content such as audio, video, and image files. For example, you might want to create a face recognition annotator that can analyze image files.

About this task

In UIMA, the common analysis structure (CAS) supports analysis of multiple views of a document. For example, one view of a video stream might be the video frames and another view might be the closed-caption text. Multiple CAS views are useful when different versions of a document are needed at different stages of the analysis. Each view of the document is associated with a different subject of analysis (Sofa). Watson Explorer Content Analytics supports multiple CAS views to allow you to access the following Sofas:

Custom text analysis annotators can analyze the document text that is extracted by the parsers.
Multimedia annotators can analyze the original document that is crawled by crawlers.

Procedure

To develop and deploy a multimedia annotator:

Develop a custom annotator to process multimedia content.
Use the org.apache.uima.CAS#getSofaDataStream() method to retrieve the original document that was crawled. This method returns null if the crawled document consists only of metadata. To get the MIME type of the document that was detected by the collection parser, use the org.apache.uima.CAS#getSofaMimeType() method.
To configure the multimedia annotator to use the CAS view that contains the original document content, use one of the following approaches:
- Specify the CAS view name explicitly in the multimedia annotator by using the org.apache.uima.CAS#getView() method.
- Specify the CAS view name in the UIMA descriptor XML file under the <sofaMappings> element. For example, the following code specifies that the annotator named jpgannotator uses the multi-view CAS view.
```
<sofaMappings>
  <sofaMapping>
    <componentKey>jpgannotator</componentKey>
    <aggregateSofaName>multi-view</aggregateSofaName>
  </sofaMapping>
</sofaMappings>
```
Use the administration console to deploy the annotator.
1. Add the text analysis engine to the system.
2. Associate the text analysis engine with one or more collections.
  In the CAS view for original document content field, specify the name of the CAS view that contains the original document content. Ensure that this view name matches the view name that you specified in the annotator or UIMA descriptor XML file.
  Tip: Ensure that the crawler for the associated collections is configured to include multimedia files with the appropriate extensions.
3. For each collection, upload and select the common analysis structure to index mapping file.

What to do next

After you develop and deploy the multimedia annotator, configure the Watson Explorer Content Analytics parsers to support the types of multimedia files that you want to process. To configure the parsers to process these file formats, open the ES_NODE_ROOT/master_config/collection_ID.indexservice/parser_config.xml file in a text editor and add the following parser definition.

<Parser>
  <ParserName>empty</ParserName>
  <ParserFactoryClass>com.ibm.ilel.parser.EmptyParserFactory</ParserFactoryClass>
</Parser>

You must then move the <Mimetype> element for the particular multimedia file type from the <ParserMapping> element for the terminator parser to the <ParserMapping> element for the empty parser. For example, move the <Mimetype>image/jpeg</Mimetype> element to configure the parsers to process JPEG image files.

 <ParserMapping>
  <ParserName>empty</ParserName>
  <Mimetype>image/jpeg</Mimetype>
</ParserMapping>

Parsing and indexing only image metadata: If you want to configure the parser to parse metadata information from image files but you do not need to index the binary content, complete only the instructions that follow step 2, as illustrated in the following example.

Add the following lines after the last <Parser> element in the ES_NODE_ROOT/master_config/collection_ID.indexservice/parser_config.xml file:
```
<Parser>
 <ParserName>empty</ParserName>
 <ParserFactoryClass>com.ibm.ilel.parser.EmptyParserFactory</ParserFactoryClass>
</Parser>
```
Move the <Mimetype>element for the particular multimedia file type from the <ParserMapping>element for the terminator parser to the <ParserMapping>element for the empty parser. For example:ES_NODE_ROOT/master_config/collection_ID.indexservice/parser_config.xml file:
```
<ParserMapping>
  <ParserName>empty</ParserName>
  <Mimetype>image/bmp</Mimetype>
  <Mimetype>image/gif</Mimetype>
  <Mimetype>image/jpeg</Mimetype>
  <Mimetype>image/tiff</Mimetype>
  <Mimetype>image/png</Mimetype>
</ParserMapping>
```
After you make these changes, open the administration console and restart the collection parser and indexer, and then recrawl the image files. When processing is complete, the image metadata is indexed.