Skip to content

HubModuleTokenizer.md

Latest commit

 

History

History
213 lines (158 loc) · 5.92 KB

HubModuleTokenizer.md

File metadata and controls

213 lines (158 loc) · 5.92 KB
Raw

description: Tokenizer that uses a Hub module.

text.HubModuleTokenizer

View source

Tokenizer that uses a Hub module.

Inherits From: TokenizerWithOffsets, Tokenizer, SplitterWithOffsets, Splitter

text.HubModuleTokenizer(
    hub_module_handle
)

This class is just a wrapper around an internal HubModuleSplitter. It offers the same functionality, but with 'token'-based method names: e.g., one can use tokenize() instead of the more general and less informatively named split().

Example:

>>> HUB_MODULE = "https://tfhub.dev/google/zh_segmentation/1"
>>> segmenter = HubModuleTokenizer(hub.resolve(HUB_MODULE))
>>> segmenter.tokenize(["新华社北京"])
<tf.RaggedTensor [[b'\xe6\x96\xb0\xe5\x8d\x8e\xe7\xa4\xbe',
                   b'\xe5\x8c\x97\xe4\xba\xac']]>

You can also use this tokenizer to return the split strings and their offsets:

>>> HUB_MODULE = "https://tfhub.dev/google/zh_segmentation/1"
>>> segmenter = HubModuleTokenizer(hub.resolve(HUB_MODULE))
>>> pieces, starts, ends = segmenter.tokenize_with_offsets(["新华社北京"])
>>> print("pieces: %s starts: %s ends: %s" % (pieces, starts, ends))
pieces: <tf.RaggedTensor [[b'\xe6\x96\xb0\xe5\x8d\x8e\xe7\xa4\xbe',
                           b'\xe5\x8c\x97\xe4\xba\xac']]>
starts: <tf.RaggedTensor [[0, 9]]>
ends: <tf.RaggedTensor [[9, 15]]>

Args

`hub_module_handle` A string handle accepted by hub.load(). Supported cases include (1) a local path to a directory containing a module, and (2) a handle to a module uploaded to e.g., https://tfhub.dev

Methods

split

View source

split(
    input
)

Alias for Tokenizer.tokenize.

split_with_offsets

View source

split_with_offsets(
    input
)

Alias for TokenizerWithOffsets.tokenize_with_offsets.

tokenize

View source

tokenize(
    input_strs
)

Tokenizes a tensor of UTF-8 strings into words.

Args
`input_strs` An N-dimensional `Tensor` or `RaggedTensor` of UTF-8 strings.
Returns
A `RaggedTensor` of segmented text. The returned shape is the shape of the input tensor with an added ragged dimension for tokens of each string.

tokenize_with_offsets

View source

tokenize_with_offsets(
    input_strs
)

Tokenizes a tensor of UTF-8 strings into words with [start,end) offsets.

Args
`input_strs` An N-dimensional `Tensor` or `RaggedTensor` of UTF-8 strings.
Returns
A tuple `(tokens, start_offsets, end_offsets)` where: * `tokens` is a `RaggedTensor` of strings where `tokens[i1...iN, j]` is the string content of the `j-th` token in `input_strs[i1...iN]` * `start_offsets` is a `RaggedTensor` of int64s where `start_offsets[i1...iN, j]` is the byte offset for the start of the `j-th` token in `input_strs[i1...iN]`. * `end_offsets` is a `RaggedTensor` of int64s where `end_offsets[i1...iN, j]` is the byte offset immediately after the end of the `j-th` token in `input_strs[i...iN]`.