Skip to content
This repository was archived by the owner on Sep 9, 2026. It is now read-only.

Commit 8bd0443

Browse files
Charlotte Gerhahersamsja
andauthored
feat(v2): add tensorflow embedding, audio, video (#1098)
* feat: add tensorflow tensor Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: wip add tf comp backend Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: comp backend working of TensorFlowTensor, not tf tensor Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: remove redundant print statements Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add comp backend retrieval Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: extract methods that overlap for np and tf backend Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: revert poetry lock change Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: introduce norm callables to transform tftensor Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: retrieval and add docstring Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add cosine sim for tf backend matrics Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: euclidean dist Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add typevar to register proto Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add tft to inits Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: add tests for tensorflow tensor Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: mypy checks Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: docarray from native Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: add documentatino and clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: stacked array with tf tensor Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: stack with tftensor Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: fix get item test Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: access by slice for tftensor Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add proto for tf Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: introduce pytest tensorflow marker Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: typo in ci.yml Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: try tf import Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: mypy Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: ndarray import Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: tf import Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: add tf markers Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: fix unit tests Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: fix unit tests Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: tf in array stacked Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: tf Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * chore: pytest proto marker call with -m Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: instance check use instance shape Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: tf tests Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: test Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add print statement to debug Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: tf test Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: only tf Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: remove tests for debugging Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: add all tests back to ci yml Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: fix import Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: ci debugging Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: change pytest marker for tf Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: change python version back Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: revert Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: debugging Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: test Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: tests Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: ignore paths Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: tests Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: tests Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * refactor: rename norm left and norm right Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: tft docstring Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: add comment to array stacked tf Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply suggestion from code review Co-authored-by: samsja <55492238+samsja@users.noreply.github.com> Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> * fix: apply suggestions from code review Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: fix black formatting Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: implement getitem setitem iter for tftensor Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: readme Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: update readme.md Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: remove n dim from abstract method instead use comp be Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: remove proto mark, because only test for proto 3 here Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: tf set item and add tests Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: update tf section in readme.md Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add tensorflow video audio embedding Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: predefined docs add tf Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * tests: integrations for tf Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: unit tests Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * refactor: use is_tf_available Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: audio tensor Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: audio tensor Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: tf import utils misc Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * chore: update ruff Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * chore: update lock file Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: add missing tensorflow pytest marker Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: fix tf test and clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: fix video test tf Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: remove copy paste errors for torch available Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: merge poetry lock Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> --------- Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> Co-authored-by: samsja <55492238+samsja@users.noreply.github.com>
1 parent d260d2b commit 8bd0443

54 files changed

Lines changed: 2045 additions & 1353 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.pre-commit-config.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,6 @@ repos:
2121
exclude: ^(docarray/proto/pb/docarray_pb2.py|docarray/proto/pb/docarray_pb2.py|docs/|docarray/resources/)
2222

2323
- repo: https://github.com/charliermarsh/ruff-pre-commit
24-
rev: v0.0.165
24+
rev: v0.0.243
2525
hooks:
2626
- id: ruff

docarray/array/array_stacked.py

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -21,27 +21,27 @@
2121
from docarray.typing import NdArray
2222
from docarray.typing.tensor.abstract_tensor import AbstractTensor
2323
from docarray.utils._typing import is_tensor_union
24+
from docarray.utils.misc import is_tf_available, is_torch_available
2425

2526
if TYPE_CHECKING:
2627
from pydantic import BaseConfig
2728
from pydantic.fields import ModelField
2829

2930
from docarray.proto import DocumentArrayStackedProto
3031

31-
try:
32+
torch_available = is_torch_available()
33+
if torch_available:
3234
from docarray.typing import TorchTensor
33-
except ImportError:
35+
else:
3436
TorchTensor = None # type: ignore
3537

36-
try:
38+
tf_available = is_tf_available()
39+
if tf_available:
3740
import tensorflow as tf # type: ignore
3841

3942
from docarray.typing import TensorFlowTensor
40-
41-
tf_available = True
42-
except (ImportError, TypeError):
43+
else:
4344
TensorFlowTensor = None # type: ignore
44-
tf_available = False
4545

4646
T = TypeVar('T', bound='DocumentArrayStacked')
4747
IndexIterType = Union[slice, Iterable[int], Iterable[bool], None]

docarray/documents/audio.py

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,12 +7,17 @@
77
from docarray.typing.bytes.audio_bytes import AudioBytes
88
from docarray.typing.tensor.abstract_tensor import AbstractTensor
99
from docarray.typing.tensor.audio.audio_tensor import AudioTensor
10-
from docarray.utils.misc import is_torch_available
10+
from docarray.utils.misc import is_tf_available, is_torch_available
1111

1212
torch_available = is_torch_available()
1313
if torch_available:
1414
import torch
1515

16+
tf_available = is_tf_available()
17+
if tf_available:
18+
import tensorflow as tf # type: ignore
19+
20+
1621
T = TypeVar('T', bound='Audio')
1722

1823

@@ -102,7 +107,9 @@ def validate(
102107
if isinstance(value, str):
103108
value = cls(url=value)
104109
elif isinstance(value, (AbstractTensor, np.ndarray)) or (
105-
torch_available and isinstance(value, torch.Tensor)
110+
torch_available
111+
and isinstance(value, torch.Tensor)
112+
or (tf_available and isinstance(value, tf.Tensor))
106113
):
107114
value = cls(tensor=value)
108115

docarray/documents/image.py

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,14 +6,18 @@
66
from docarray.typing import AnyEmbedding, ImageBytes, ImageUrl
77
from docarray.typing.tensor.abstract_tensor import AbstractTensor
88
from docarray.typing.tensor.image.image_tensor import ImageTensor
9-
from docarray.utils.misc import is_torch_available
9+
from docarray.utils.misc import is_tf_available, is_torch_available
1010

1111
T = TypeVar('T', bound='Image')
1212

1313
torch_available = is_torch_available()
1414
if torch_available:
1515
import torch
1616

17+
tf_available = is_tf_available()
18+
if tf_available:
19+
import tensorflow as tf # type: ignore
20+
1721

1822
class Image(BaseDocument):
1923
"""
@@ -91,8 +95,10 @@ def validate(
9195
) -> T:
9296
if isinstance(value, str):
9397
value = cls(url=value)
94-
elif isinstance(value, (AbstractTensor, np.ndarray)) or (
95-
torch_available and isinstance(value, torch.Tensor)
98+
elif (
99+
isinstance(value, (AbstractTensor, np.ndarray))
100+
or (torch_available and isinstance(value, torch.Tensor))
101+
or (tf_available and isinstance(value, tf.Tensor))
96102
):
97103
value = cls(tensor=value)
98104
elif isinstance(value, bytes):

docarray/documents/point_cloud.py

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,16 @@
55
from docarray.base_document import BaseDocument
66
from docarray.typing import AnyEmbedding, AnyTensor, PointCloud3DUrl
77
from docarray.typing.tensor.abstract_tensor import AbstractTensor
8-
from docarray.utils.misc import is_torch_available
8+
from docarray.utils.misc import is_tf_available, is_torch_available
99

1010
torch_available = is_torch_available()
1111
if torch_available:
1212
import torch
1313

14+
tf_available = is_tf_available()
15+
if tf_available:
16+
import tensorflow as tf # type: ignore
17+
1418
T = TypeVar('T', bound='PointCloud3D')
1519

1620

@@ -100,7 +104,9 @@ def validate(
100104
if isinstance(value, str):
101105
value = cls(url=value)
102106
elif isinstance(value, (AbstractTensor, np.ndarray)) or (
103-
torch_available and isinstance(value, torch.Tensor)
107+
torch_available
108+
and isinstance(value, torch.Tensor)
109+
or (tf_available and isinstance(value, tf.Tensor))
104110
):
105111
value = cls(tensor=value)
106112

docarray/documents/text.py

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,7 @@ class MultiModalDoc(BaseDocument):
7474
)
7575
mmdoc.text_doc.text = mmdoc.text_doc.url.load()
7676
77-
#or
77+
# or
7878
7979
mmdoc.text_doc.bytes = mmdoc.text_doc.url.load_bytes()
8080
@@ -87,13 +87,13 @@ class MultiModalDoc(BaseDocument):
8787
8888
.. code-block:: python
8989
90-
from docarray.documents Text
90+
from docarray.documents import Text
9191
9292
doc = Text(text='This is the main text', url='exampleurl.com')
9393
doc2 = Text(text='This is the main text', url='exampleurl.com')
9494
95-
doc == 'This is the main text' # True
96-
doc == doc2 # False, their ids are not equivalent
95+
doc == 'This is the main text' # True
96+
doc == doc2 # False, their ids are not equivalent
9797
"""
9898

9999
text: Optional[str] = None
@@ -126,7 +126,7 @@ def __contains__(self, item: str) -> bool:
126126
"""
127127
This method makes `Text` behave the same as an `str`.
128128
129-
.. code-block:: python
129+
.. code-block:: python
130130
131131
from docarray.documents import Text
132132

docarray/documents/video.py

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,18 @@
88
from docarray.typing.tensor.abstract_tensor import AbstractTensor
99
from docarray.typing.tensor.video.video_tensor import VideoTensor
1010
from docarray.typing.url.video_url import VideoUrl
11-
from docarray.utils.misc import is_torch_available
11+
from docarray.utils.misc import is_tf_available, is_torch_available
1212

1313
torch_available = is_torch_available()
1414
if torch_available:
1515
import torch
1616

17+
18+
tf_available = is_tf_available()
19+
if tf_available:
20+
import tensorflow as tf # type: ignore
21+
22+
1723
T = TypeVar('T', bound='Video')
1824

1925

@@ -106,7 +112,9 @@ def validate(
106112
if isinstance(value, str):
107113
value = cls(url=value)
108114
elif isinstance(value, (AbstractTensor, np.ndarray)) or (
109-
torch_available and isinstance(value, torch.Tensor)
115+
torch_available
116+
and isinstance(value, torch.Tensor)
117+
or (tf_available and isinstance(value, tf.Tensor))
110118
):
111119
value = cls(tensor=value)
112120

docarray/typing/__init__.py

Lines changed: 20 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -38,11 +38,10 @@
3838
'ImageNdArray',
3939
]
4040

41-
try:
42-
import torch # noqa: F401
43-
except ImportError:
44-
pass
45-
else:
41+
from docarray.utils.misc import is_tf_available, is_torch_available
42+
43+
torch_available = is_torch_available()
44+
if torch_available:
4645
from docarray.typing.tensor import TorchEmbedding, TorchTensor # noqa: F401
4746
from docarray.typing.tensor.audio.audio_torch_tensor import AudioTorchTensor # noqa
4847
from docarray.typing.tensor.image import ImageTorchTensor # noqa: F401
@@ -58,11 +57,20 @@
5857
]
5958
)
6059

61-
try:
62-
import tensorflow as tf # type: ignore # noqa: F401
63-
except (ImportError, TypeError):
64-
pass
65-
else:
66-
from docarray.typing.tensor import TensorFlowTensor # noqa: F401
60+
tf_available = is_tf_available()
61+
if tf_available:
62+
from docarray.typing.tensor import TensorFlowTensor
63+
from docarray.typing.tensor.audio import AudioTensorFlowTensor # noqa: F401
64+
from docarray.typing.tensor.embedding import TensorFlowEmbedding # noqa: F401
65+
from docarray.typing.tensor.image import ImageTensorFlowTensor # noqa: F401
66+
from docarray.typing.tensor.video import VideoTensorFlowTensor # noqa
6767

68-
__all__.extend(['TensorFlowTensor'])
68+
__all__.extend(
69+
[
70+
'TensorFlowTensor',
71+
'TensorFlowEmbedding',
72+
'AudioTensorFlowTensor',
73+
'ImageTensorFlowTensor',
74+
'VideoTensorFlowTensor',
75+
]
76+
)

docarray/typing/tensor/__init__.py

Lines changed: 12 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -13,22 +13,23 @@
1313
'TensorFlowTensor',
1414
]
1515

16-
try:
17-
import torch # noqa: F401
18-
except ImportError:
19-
pass
20-
else:
16+
from docarray.utils.misc import is_tf_available, is_torch_available
17+
18+
torch_available = is_torch_available()
19+
if torch_available:
2120
from docarray.typing.tensor.embedding import TorchEmbedding # noqa: F401
2221
from docarray.typing.tensor.image import ImageTorchTensor # noqa: F401
2322
from docarray.typing.tensor.torch_tensor import TorchTensor # noqa: F401
2423

2524
__all__.extend(['TorchEmbedding', 'TorchTensor', 'ImageTorchTensor'])
2625

27-
try:
28-
import tensorflow as tf # type: ignore # noqa: F401
29-
except (ImportError, TypeError):
30-
pass
31-
else:
26+
torch_available = is_torch_available()
27+
28+
29+
tf_available = is_tf_available()
30+
if tf_available:
31+
from docarray.typing.tensor.embedding import TensorFlowEmbedding # noqa: F401
32+
from docarray.typing.tensor.image import ImageTensorFlowTensor # noqa: F401
3233
from docarray.typing.tensor.tensorflow_tensor import TensorFlowTensor # noqa: F401
3334

34-
__all__.extend(['TensorFlowTensor'])
35+
__all__.extend(['TensorFlowEmbedding', 'TensorFlowTensor', 'ImageTensorFlowTensor'])

docarray/typing/tensor/audio/__init__.py

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,19 @@
22

33
__all__ = ['AudioNdArray']
44

5-
try:
6-
import torch # noqa: F401
7-
except ImportError:
8-
pass
9-
else:
5+
from docarray.utils.misc import is_tf_available, is_torch_available
6+
7+
torch_available = is_torch_available()
8+
if torch_available:
109
from docarray.typing.tensor.audio.audio_torch_tensor import AudioTorchTensor # noqa
1110

1211
__all__.extend(['AudioTorchTensor'])
12+
13+
14+
tf_available = is_tf_available()
15+
if tf_available:
16+
from docarray.typing.tensor.audio.audio_tensorflow_tensor import ( # noqa
17+
AudioTensorFlowTensor,
18+
)
19+
20+
__all__.extend(['AudioTensorFlowTensor'])

0 commit comments

Comments
 (0)