jvntokenizer
Class PennTokenizer

java.lang.Object
  jvntokenizer.PennTokenizer

public final class PennTokenizer
extends java.lang.Object

Tokenizer that conforms to the Penn Treebank conventions for tokenization.

Author:

Jimmy Lin

Constructor Summary

PennTokenizer()

Method Summary

static java.lang.String

tokenize(java.lang.String str) Tokenizes according to the Penn Treebank conventions.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PennTokenizer

public PennTokenizer()

Method Detail

tokenize

public static java.lang.String tokenize(java.lang.String str)

Tokenizes according to the Penn Treebank conventions.

Parameters:

str - the str

Returns:

the string