Syntax

At a glance

The Syntax model performs fundamental NLP tasks on the input text:

• Sentence detection
• Tokenization: can't -> ca + n't
• Part-of-Speech tagging: I thought -> I/PRON, thought/VERB
• Lemmatization: I thought -> I/I, thought/think
• Dependency parsing: I -> nsubj -> thought -> root

For language support, see Supported languages.

This model offers two implementations:

• Izumo: Provides high accuracy and throughput at moderate computational cost. The model is built using curated human knowledge (dictionaries, complementary rules) with machine learning algorithms (Logistic Regression, Conditional Random Fields). The implementation has been tested over many years in IBM products.

• Transformer-based syntax: Provides state-of-the-art accuracy by leveraging the cutting edge technology of IBM's fine-tuned Slate model. This consumes higher computational cost; especially that the runtime throughput is lower in a CPU environment without GPU.

Pretrained models

Model names are listed below.

The Syntax models use pre-defined algorithms and models suitable for each language. The output schema for part of speech and dependency parsing follows the Universal Part-of-Speech v2 and Universal Dependency Relations v2 standard for all languages. This means that the Syntax models output the same set of parts of speech tags and dependency relations for all supported languages. For details about the Universal Part-of-Speech and Universal Dependency v2 standards, see Grammatical properties.

Both Izumo models and transformer-based syntax models have been trained with Universal Dependency corpora with commercial license, as well as additional data generated via a novel silver data generation process invented by IBM Research Yorktown and Tokyo labs. Models are continuously improved with feedback from users.

For Izumo models, the specific algorithms used in training the Syntax models for each language have been chosen to provide a good trade-off between accuracy and runtime performance. The algorithms differ across different language groups: as a general rule, simpler and fast algorithms are used for simpler languages such as English (Group A languages), and increasingly more sophisticated (and slower) algorithms are used for complex languages such as Arabic (Group B languages), and Chinese, Japanese and Korean (Group C languages).

The transformer-based syntax is purely based on machine learning approaches, thus the algorithm is common for all supported languages. Currently, only the English stock model is avaiable.

Running models

The Syntax model request accepts the following fields:

and returns the following responses:

Example requests

curl -s \
  "http://localhost:8080/v1/watson.runtime.nlp.v1/NlpService/SyntaxPredict" \
  -H "accept: application/json" \
  -H "content-type: application/json" \
  -H "Grpc-Metadata-mm-model-id: syntax_izumo_lang_en_stock" \
  -d '{ "raw_document": { "text": "This is a test sentence." }, "parsers": ["token","sentence","lemma","part_of_speech","dependency"] }'

{"text":"This is a test sentence.", "producerId":{"name":"Izumo Text Processing", "version":"0.0.1"},
 "tokens":[
  {"span":{"begin":0, "end":4, "text":"This"}, "lemma":"this", "partOfSpeech":"POS_PRON", "dependency":{"relation":"DEP_NSUBJ", "identifier":1, "head":2}, "features":[]},
  {"span":{"begin":5, "end":7, "text":"is"}, "lemma":"be", "partOfSpeech":"POS_AUX", "dependency":{"relation":"DEP_COP", "identifier":3, "head":2}, "features":[]},
  {"span":{"begin":8, "end":9, "text":"a"}, "lemma":"a", "partOfSpeech":"POS_DET", "dependency":{"relation":"DEP_DET", "identifier":4, "head":2}, "features":[]},
  {"span":{"begin":10, "end":14, "text":"test"}, "lemma":"test", "partOfSpeech":"POS_NOUN", "dependency":{"relation":"DEP_COMPOUND", "identifier":5, "head":2}, "features":[]},
  {"span":{"begin":15, "end":23, "text":"sentence"}, "lemma":"sentence", "partOfSpeech":"POS_NOUN", "dependency":{"relation":"DEP_ROOT", "identifier":2, "head":0}, "features":[]},
  {"span":{"begin":23, "end":24, "text":"."}, "lemma":"", "partOfSpeech":"POS_PUNCT", "dependency":{"relation":"DEP_PUNCT", "identifier":6, "head":2}, "features":[]}
  ],
 "sentences":[
  {"span":{"begin":0, "end":24, "text":"This is a test sentence."}}
  ],
 "paragraphs":[
  {"span":{"begin":0, "end":24, "text":"This is a test sentence."}}
  ]
}

import grpc

from watson_nlp_runtime_client import common_service_pb2, common_service_pb2_grpc

client = common_service_pb2_grpc.NlpServiceStub(grpc.insecure_channel("localhost:8085"))

response = client.SyntaxPredict(
  common_service_pb2.SyntaxRequest(
    raw_document={"text": "This is a test sentence."},
    parsers=('token', 'sentence', 'lemma', 'part_of_speech', 'dependency')
  ), 
  metadata=[("mm-model-id", "syntax_izumo_lang_en_stock")],
)

print(response)

text: "This is a test sentence."
producer_id {
  name: "Izumo Text Processing"
  version: "0.0.1"
}
tokens {
  span {
    end: 4
    text: "This"
  }
  lemma: "this"
  part_of_speech: POS_PRON
  dependency {
    relation: DEP_NSUBJ
    identifier: 1
    head: 2
  }
}
tokens {
  span {
    begin: 5
    end: 7
    text: "is"
  }
  lemma: "be"
  part_of_speech: POS_AUX
  dependency {
    relation: DEP_COP
    identifier: 3
    head: 2
  }
}
tokens {
  span {
    begin: 8
    end: 9
    text: "a"
  }
  lemma: "a"
  part_of_speech: POS_DET
  dependency {
    relation: DEP_DET
    identifier: 4
    head: 2
  }
}
tokens {
  span {
    begin: 10
    end: 14
    text: "test"
  }
  lemma: "test"
  part_of_speech: POS_NOUN
  dependency {
    relation: DEP_COMPOUND
    identifier: 5
    head: 2
  }
}
tokens {
  span {
    begin: 15
    end: 23
    text: "sentence"
  }
  lemma: "sentence"
  part_of_speech: POS_NOUN
  dependency {
    relation: DEP_ROOT
    identifier: 2
  }
}
tokens {
  span {
    begin: 23
    end: 24
    text: "."
  }
  part_of_speech: POS_PUNCT
  dependency {
    relation: DEP_PUNCT
    identifier: 6
    head: 2
  }
}
sentences {
  span {
    end: 24
    text: "This is a test sentence."
  }
}
paragraphs {
  span {
    end: 24
    text: "This is a test sentence."
  }
}