description: Splits a string tensor into bytes.
Splits a string tensor into bytes.
Inherits From: SplitterWithOffsets,
Splitter
text.ByteSplitter()
split(
input
)
Splits a string tensor into bytes.
The strings are split bytes. Thus, some unicode characters may be split into multiple bytes.
>>> ByteSplitter().split("hello")
<tf.Tensor: shape=(5,), dtype=uint8, numpy=array([104, 101, 108, 108, 111],
dtype=uint8)>
| Args | |
|---|---|
| `input` | A `RaggedTensor` or `Tensor` of strings with any shape. |
| Returns | |
|---|---|
| A `RaggedTensor` of bytes. The returned shape is the shape of the input tensor with an added ragged dimension for the bytes that make up each string. |
split_by_offsets(
input, start_offsets, end_offsets
)
Splits a string tensor into sub-strings.
The strings are split based upon the provided byte offsets.
>>> splitter = ByteSplitter()
>>> substrings = splitter.split_by_offsets("hello", [0, 4], [4, 5])
>>> print(substrings.numpy())
[b'hell' b'o']
| Args | |
|---|---|
| `input` | `Tensor` or `RaggedTensor` of strings of any shape to split. |
| `start_offsets` | `Tensor` or `RaggedTensor` of byte offsets to start splits on (inclusive). This should be one more than the rank of `input`. |
| `end_offsets` | `Tensor` or `RaggedTensor` of byte offsets to end splits on (exclusive). This should be one more than the rank of `input`. |
| Returns | |
|---|---|
| A `RaggedTensor` or `Tensor` of substrings. The returned shape is the shape of the offsets. |
split_with_offsets(
input
)
Splits a string tensor into bytes.
The strings are split bytes. Thus, some unicode characters may be split into multiple bytes.
>>> splitter = ByteSplitter()
>>> bytes, starts, ends = splitter.split_with_offsets("hello")
>>> print(bytes.numpy(), starts.numpy(), ends.numpy())
[104 101 108 108 111] [0 1 2 3 4] [1 2 3 4 5]
| Args | |
|---|---|
| `input` | A `RaggedTensor` or `Tensor` of strings with any shape. |
| Returns | |
|---|---|
| A `RaggedTensor` of bytes. The returned shape is the shape of the input tensor with an added ragged dimension for the bytes that make up each string. |
| Returns | |
|---|---|
A tuple `(bytes, offsets)` where:
|