luxonis · SzabolcsGergely · Jul 5, 2021 · May 26, 2021 · May 27, 2021 · May 28, 2021
diff --git a/depthai-core b/depthai-core
diff --git a/docs/source/components/nodes/stereo_depth.rst b/docs/source/components/nodes/stereo_depth.rst
@@ -62,7 +62,7 @@ as:
 (this confidence score is kind-of inverted, if say comparing with NN)
 
 For the final disparity map, a filtering is applied based on the confidence threshold value: the pixels that have their confidence score larger than
-the threshold get invalidated, i.e. their disparity value is set to zero. You can set the confidence threshold with :code:`stereo.setConfidenceThreshold()`.
+the threshold get invalidated, i.e. their disparity value is set to zero. You can set the confidence threshold with :code:`stereo.initialConfig.setConfidenceThreshold()`.
 
 Current limitations
 ###################

diff --git a/docs/source/samples/edge_detector.rst b/docs/source/samples/edge_detector.rst
@@ -0,0 +1,38 @@
+Edge detector
+=============
+
+This example performs edge detection on 3 different inputs: left, right and RGB camera.
+HW accelerated sobel filter 3x3 is used.
+Sobel filter parameters can be changed by keys 1 and 2.
+
+Demo
+####
+
+
+Setup
+#####
+
+.. include::  /includes/install_from_pypi.rst
+
+Source code
+###########
+
+.. tabs::
+
+    .. tab:: Python
+
+        Also `available on GitHub <https://github.com/luxonis/depthai-python/blob/main/examples/edge_detector.py>`__
+
+        .. literalinclude:: ../../../examples/edge_detector.py
+           :language: python
+           :linenos:
+
+    .. tab:: C++
+
+        Also `available on GitHub <https://github.com/luxonis/depthai-core/blob/main/examples/src/edge_detector.cpp>`__
+
+        .. literalinclude:: ../../../depthai-core/examples/src/edge_detector.cpp
+           :language: cpp
+           :linenos:
+
+.. include::  /includes/footer-short.rst
diff --git a/docs/source/tutorials/code_samples.rst b/docs/source/tutorials/code_samples.rst
@@ -29,6 +29,7 @@ Code samples are used for automated testing. They are also a great starting poin
 - :ref:`Video & MobilenetSSD` - Runs MobileNetSSD on the video from the host
 - :ref:`IMU Accelerometer & Gyroscope` - Accelerometer and gyroscope at 500hz rate
 - :ref:`IMU Rotation Vector` - Rotation vector at 400 hz rate
+- :ref:`Edge detector` - Edge detection on input frame
 
 .. rubric:: Complex
 

diff --git a/docs/source/tutorials/hello_world.rst b/docs/source/tutorials/hello_world.rst
@@ -70,9 +70,10 @@ Let's verify we're able to load all of our dependencies. Open the :code:`hello_w
 
 .. code-block:: python
 
-  import numpy as np # numpy - manipulate the packet data returned by depthai
-  import cv2 # opencv - display the video stream
-  import depthai # access the camera and its data packets
+  import numpy as np  # numpy - manipulate the packet data returned by depthai
+  import cv2  # opencv - display the video stream
+  import depthai  # depthai - access the camera and its data packets
+  import blobconverter  # blobconverter - compile and download MyriadX neural network blobs
 
 Try running the script and ensure it executes without error:
 
@@ -111,13 +112,16 @@ Now, first node we will add is a :class:`ColorCamera`. We will use the :code:`pr
   cam_rgb.setPreviewSize(300, 300)
   cam_rgb.setInterleaved(False)
 
-Up next, let's define a :class:`NeuralNetwork` node with `mobilenet-ssd network <https://docs.openvinotoolkit.org/latest/omz_models_public_mobilenet_ssd_moblenet_ssd.html>`__.
-The blob file for this example can be found `here <https://github.com/luxonis/depthai-tutorials/raw/e37989e07a36a57ffef624b7aa8cf20ab493fa07/1-hello-world/mobilenet-ssd/mobilenet-ssd.blob>`__
+Up next, let's define a :class:`MobileNetDetectionNetwork` node with `mobilenet-ssd network <https://docs.openvinotoolkit.org/latest/omz_models_public_mobilenet_ssd_moblenet_ssd.html>`__.
+The blob file for this example will be compiled automatically using `blobconverter tool <https://pypi.org/project/blobconverter/>`__, we'll be provided with a ready-to-use blob path.
+With this node, the output from nn will be parsed on device side and we'll receive a ready to use detection objects. For this to work properly, we need also to set the confidence threshold
+to filter out the incorrect results
 
 .. code-block:: python
 
-  detection_nn = pipeline.createNeuralNetwork()
-  detection_nn.setBlobPath("/path/to/mobilenet-ssd.blob")
+  detection_nn = pipeline.createMobileNetDetectionNetwork()
+  detection_nn.setBlobPath(str(blobconverter.from_zoo(name='mobilenet-ssd', shaves=6)))
+  detection_nn.setConfidenceThreshold(0.5)
 
 And now, let's connect a color camera :code:`preview` output to neural network input
 
@@ -143,12 +147,11 @@ and in our case, since we want to receive data from device to host, we will use
 Initialize the DepthAI Device
 #############################
 
-Having the pipeline defined, we can now initialize a device and start it
+Having the pipeline defined, we can now initialize a device with pipeline and start it
 
 .. code-block:: python
 
-  device = depthai.Device(pipeline)
-  device.startPipeline()
+  with depthai.Device(pipeline) as device:
 
 .. note::
 
@@ -158,7 +161,7 @@ Having the pipeline defined, we can now initialize a device and start it
 
   .. code-block:: python
 
-    device = depthai.Device(pipeline, True)
+    device = depthai.Device(pipeline, usb2mode=True)
 
 
 
@@ -181,19 +184,22 @@ for rgb frame and one for nn results
 .. code-block:: python
 
   frame = None
-  bboxes = []
+  detections = []
 
 Also, due to neural network implementation details, bounding box coordinates in inference results are represented
 as floats from <0..1> range - so relative to frame width/height (e.g. if image has 200px width and nn returned x_min
 coordinate equal to 0.2, this means the actual (normalised) x_min coordinate is 40px).
 
-That's why we need to define a helper function, :code:`frame_form`, that will convert these <0..1> values into actual
+That's why we need to define a helper function, :code:`frameNorm`, that will convert these <0..1> values into actual
 pixel positions
 
 .. code-block:: python
 
-  def frame_norm(frame, bbox):
-      return (np.array(bbox) * np.array([*frame.shape[:2], *frame.shape[:2]])[::-1]).astype(int)
+    def frameNorm(frame, bbox):
+        normVals = np.full(len(bbox), frame.shape[0])
+        normVals[::2] = frame.shape[1]
+        return (np.clip(np.array(bbox), 0, 1) * normVals).astype(int)
+
 
 Consuming the results
 #####################
@@ -215,65 +221,22 @@ Now, inside this loop, first thing to do is fetching latest results from both nn
 The :code:`tryGet` method returns either the latest result or :code:`None` if the queue is empty.
 
 Results, both from rgb camera or neural network, will be delivered as 1D arrays, so both of them will require transformations
-to be useful for display (we have already defined one of the transformations needed - the :code:`frame_norm` function)
+to be useful for display (we have already defined one of the transformations needed - the :code:`frameNorm` function)
 
-First up, if we receive a frame from rgb camera, we need to convert it from 1D array into HWC form (HWC stands for
-Height Width Channels, so 3D array, with first dimension being width, second height, and third the color channel)
+First up, if we receive a frame from rgb camera using the :code:`getCvFrame` command
 
 .. code-block:: python
 
   if in_rgb is not None:
-      shape = (3, in_rgb.getHeight(), in_rgb.getWidth())
-      frame = in_rgb.getData().reshape(shape).transpose(1, 2, 0).astype(np.uint8)
-      frame = np.ascontiguousarray(frame)
+      frame = in_rgb.getCvFrame()
 
-Second, the neural network results will also need transformations. These are also returned as a 1D array, but this time
-the array has a fixed size (constant, no matter how many results the neural network has actually produced).
-Actual results in array are followed with :code:`-1` and then filled to meet the fixed size with :code:`0`.
-One results has 7 fields, each being respectively :code:`image_id, label, confidence, x_min, y_min, x_max, y_max`.
-We will want only the last four values (being the bounding box), but we'll also filter out the ones which :code:`confidence`
-is below a certain threshold - it can be anywhere between <0..1>, and for this example we will use :code:`0.8` threshold
+Second, we will receive the neural network results. Default MobileNetSSD result has 7 fields, each being respectively :code:`image_id, label, confidence, x_min, y_min, x_max, y_max`,
+and by accessing the :code:`detections` array, we receive the detection objects that allow us to access these fields
 
 .. code-block:: python
 
   if in_nn is not None:
-      bboxes = np.array(in_nn.getFirstLayerFp16())
-      bboxes = bboxes[:np.where(bboxes == -1)[0][0]]
-      bboxes = bboxes.reshape((bboxes.size // 7, 7))
-      bboxes = bboxes[bboxes[:, 2] > 0.8][:, 3:7]
-
-To better understand this flow, let's take an example. Let's assume the :code:`np.array(in_nn.getFirstLayerFp16())` returns the following array
-
-.. code-block:: python
-
-  [0, 15, 0.99023438, 0.45556641, 0.34399414  0.88037109, 0.9921875, 0, 15, 0.98828125, 0.03076172, 0.23388672, 0.60205078, 1.0078125, -1, 0, 0, 0, ...]
-
-First operation, :code:`bboxes[:np.where(bboxes == -1)[0][0]]`, removes the trailing zeros from the array, so now the bbox array will look like this
-
-.. code-block:: python
-
-  [0, 15, 0.99023438, 0.45556641, 0.34399414  0.88037109, 0.9921875, 0, 15, 0.98828125, 0.03076172, 0.23388672, 0.60205078, 1.0078125]
-
-Second one - :code:`bboxes.reshape((bboxes.size // 7, 7))`, reshapes the 1D array into 2D array - where each row is a separate result
-
-.. code-block:: python
-
-  [
-    [0, 15, 0.99023438, 0.45556641, 0.34399414  0.88037109, 0.9921875],
-    [0, 15, 0.98828125, 0.03076172, 0.23388672, 0.60205078, 1.0078125]
-  ]
-
-Last one - :code:`bboxes = bboxes[bboxes[:, 2] > 0.8][:, 3:7]` - will filter the results based on the confidence column (3rd one, with index :code:`2`)
-to be above a defined threshold (:code:`0.8`) - and from these results, it will only take the last 4 columns being the bounding boxes.
-Since both our results have a very high confidence (:code:`0.99023438` and :code:`0.98828125` respectively), they won't be filtered, and the final
-array will look like this
-
-.. code-block:: python
-
-  [
-    [0.45556641, 0.34399414  0.88037109, 0.9921875],
-    [0.03076172, 0.23388672, 0.60205078, 1.0078125]
-  ]
+      detections = in_nn.detections
 
 Display the results
 ###################
@@ -283,12 +246,12 @@ Up to this point, we have all our results consumed from the DepthaI device, and
 .. code-block:: python
 
   if frame is not None:
-      for raw_bbox in bboxes:
-          bbox = frame_norm(frame, raw_bbox)
+      for detection in detections:
+          bbox = frameNorm(frame, (detection.xmin, detection.ymin, detection.xmax, detection.ymax))
           cv2.rectangle(frame, (bbox[0], bbox[1]), (bbox[2], bbox[3]), (255, 0, 0), 2)
       cv2.imshow("preview", frame)
 
-You can see here the usage of :code:`frame_norm` we defined earlier for bounding box coordinates normalization.
+You can see here the usage of :code:`frameNorm` we defined earlier for bounding box coordinates normalization.
 By using :code:`cv2.rectangle` we draw a rectangle on the rgb frame as an indicator where the face position is, and then
 we display the frame using :code:`cv2.imshow`
 

diff --git a/docs/source/tutorials/local_convert_openvino.rst b/docs/source/tutorials/local_convert_openvino.rst
@@ -275,8 +275,8 @@ In particular, let's change the :code:`setBlobPath` invocation to load our model
 
 .. code-block:: diff
 
-  - detection_nn.setBlobPath("/path/to/mobilenet-ssd.blob")
-  - detection_nn.setBlobPath("/path/to/face-detection-retail-0004.blob")
+  - detection_nn.setBlobPath(str(blobconverter.from_zoo(name='mobilenet-ssd', shaves=6)))
+  + detection_nn.setBlobPath("/path/to/face-detection-retail-0004.blob")
 
 And that's all!
 

diff --git a/docs/source/tutorials/simple_samples.rst b/docs/source/tutorials/simple_samples.rst
@@ -21,7 +21,7 @@ Simple
    ../samples/video_mobilenet.rst
    ../samples/imu_accelerometer_gyroscope.rst
    ../samples/imu_rotation_vector.rst
-
+   ../samples/edge_detector.rst
 
 These samples are great starting point for the gen2 API.
 
@@ -38,3 +38,4 @@ These samples are great starting point for the gen2 API.
 - :ref:`RGB & MobileNetSSD @ 4K` - Runs MobileNetSSD on RGB frames and displays detections on both preview and 4k frames
 - :ref:`Mono & MobilenetSSD` - Runs MobileNetSSD on mono frames and displays detections on the frame
 - :ref:`Video & MobilenetSSD` - Runs MobileNetSSD on the video from the host
+- :ref:`Edge detector` - Edge detection on input frame
diff --git a/examples/CMakeLists.txt b/examples/CMakeLists.txt
@@ -124,7 +124,5 @@ add_python_example(stereo_depth_from_host stereo_depth_from_host.py)
 add_python_example(stereo_depth_video stereo_depth_video.py)
 add_python_example(imu_gyroscope_accelerometer imu_gyroscope_accelerometer.py)
 add_python_example(imu_rotation_vector imu_rotation_vector.py)
-add_python_example(calibration_flash_v5 calibration_flash_v5.py)
-add_python_example(calibration_flash calibration_flash.py)
-add_python_example(calibration_load calibration_load.py)
-add_python_example(calibration_reader calibration_reader.py)
+add_python_example(rgb_depth_aligned rgb_depth_aligned.py)
+add_python_example(edge_detector edge_detector.py)
diff --git a/examples/depth_crop_control.py b/examples/depth_crop_control.py
@@ -39,7 +39,7 @@
 
 manip.initialConfig.setCropRect(topLeft.x, topLeft.y, bottomRight.x, bottomRight.y)
 manip.setMaxOutputFrameSize(monoRight.getResolutionHeight()*monoRight.getResolutionWidth()*3)
-stereo.setConfidenceThreshold(200)
+stereo.initialConfig.setConfidenceThreshold(200)
 
 # Linking
 configIn.out.link(manip.inputConfig)

diff --git a/examples/depth_preview.py b/examples/depth_preview.py
@@ -29,9 +29,9 @@
 monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT)
 
 # Create a node that will produce the depth map (using disparity output as it's easier to visualize depth this way)
-depth.setConfidenceThreshold(200)
+depth.initialConfig.setConfidenceThreshold(200)
 # Options: MEDIAN_OFF, KERNEL_3x3, KERNEL_5x5, KERNEL_7x7 (default)
-depth.setMedianFilter(dai.StereoDepthProperties.MedianFilter.KERNEL_7x7)
+depth.initialConfig.setMedianFilter(dai.MedianFilter.KERNEL_7x7)
 depth.setLeftRightCheck(lr_check)
 depth.setExtendedDisparity(extended_disparity)
 depth.setSubpixel(subpixel)

diff --git a/examples/edge_detector.py b/examples/edge_detector.py
@@ -0,0 +1,102 @@
+#!/usr/bin/env python3
+
+import cv2
+import depthai as dai
+import numpy as np
+
+# Create pipeline
+pipeline = dai.Pipeline()
+
+# Define sources and outputs
+camRgb = pipeline.createColorCamera()
+monoLeft = pipeline.createMonoCamera()
+monoRight = pipeline.createMonoCamera()
+
+edgeDetectorLeft = pipeline.createEdgeDetector()
+edgeDetectorRight = pipeline.createEdgeDetector()
+edgeDetectorRgb = pipeline.createEdgeDetector()
+
+xoutEdgeLeft = pipeline.createXLinkOut()
+xoutEdgeRight = pipeline.createXLinkOut()
+xoutEdgeRgb = pipeline.createXLinkOut()
+xinEdgeCfg = pipeline.createXLinkIn()
+
+edgeLeftStr = "edge left"
+edgeRightStr = "edge right"
+edgeRgbStr = "edge rgb"
+edgeCfgStr = "edge cfg"
+
+xoutEdgeLeft.setStreamName(edgeLeftStr)
+xoutEdgeRight.setStreamName(edgeRightStr)
+xoutEdgeRgb.setStreamName(edgeRgbStr)
+xinEdgeCfg.setStreamName(edgeCfgStr)
+
+# Properties
+camRgb.setBoardSocket(dai.CameraBoardSocket.RGB)
+camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
+
+monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
+monoLeft.setBoardSocket(dai.CameraBoardSocket.LEFT)
+monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
+monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT)
+
+edgeDetectorRgb.setMaxOutputFrameSize(camRgb.getVideoWidth() * camRgb.getVideoHeight())
+
+# Linking
+monoLeft.out.link(edgeDetectorLeft.inputImage)
+monoRight.out.link(edgeDetectorRight.inputImage)
+camRgb.video.link(edgeDetectorRgb.inputImage)
+
+edgeDetectorLeft.outputImage.link(xoutEdgeLeft.input)
+edgeDetectorRight.outputImage.link(xoutEdgeRight.input)
+edgeDetectorRgb.outputImage.link(xoutEdgeRgb.input)
+
+xinEdgeCfg.out.link(edgeDetectorLeft.inputConfig)
+xinEdgeCfg.out.link(edgeDetectorRight.inputConfig)
+xinEdgeCfg.out.link(edgeDetectorRgb.inputConfig)
+
+# Connect to device and start pipeline
+with dai.Device(pipeline) as device:
+
+    # Output/input queues
+    edgeLeftQueue = device.getOutputQueue(edgeLeftStr, 8, False)
+    edgeRightQueue = device.getOutputQueue(edgeRightStr, 8, False)
+    edgeRgbQueue = device.getOutputQueue(edgeRgbStr, 8, False)
+    edgeCfgQueue = device.getInputQueue(edgeCfgStr)
+
+    print("Switch between sobel filter kernels using keys '1' and '2'")
+
+    while(True):
+        edgeLeft = edgeLeftQueue.get()
+        edgeRight = edgeRightQueue.get()
+        edgeRgb = edgeRgbQueue.get()
+
+        edgeLeftFrame = edgeLeft.getFrame()
+        edgeRightFrame = edgeRight.getFrame()
+        edgeRgbFrame = edgeRgb.getFrame()
+
+        # Show the frame
+        cv2.imshow(edgeLeftStr, edgeLeftFrame)
+        cv2.imshow(edgeRightStr, edgeRightFrame)
+        cv2.imshow(edgeRgbStr, edgeRgbFrame)
+
+        key = cv2.waitKey(1)
+        if key == ord('q'):
+            break
+
+        if key == ord('1'):
+            print("Switching sobel filter kernel.")
+            cfg = dai.EdgeDetectorConfig()
+            sobelHorizontalKernel = [[1, 0, -1], [2, 0, -2], [1, 0, -1]]
+            sobelVerticalKernel = [[1, 2, 1], [0, 0, 0], [-1, -2, -1]]
+            cfg.setSobelFilterKernels(sobelHorizontalKernel, sobelVerticalKernel)
+            edgeCfgQueue.send(cfg)
+
+        if key == ord('2'):
+            print("Switching sobel filter kernel.")
+            cfg = dai.EdgeDetectorConfig()
+            sobelHorizontalKernel = [[3, 0, -3], [10, 0, -10], [3, 0, -3]]
+            sobelVerticalKernel = [[3, 10, 3], [0, 0, 0], [-3, -10, -3]]
+            cfg.setSobelFilterKernels(sobelHorizontalKernel, sobelVerticalKernel)
+            edgeCfgQueue.send(cfg)
+
+12 −3		CMakeLists.txt
+1 −1		README.md
+1 −1		cmake/Depthai/DepthaiDeviceSideConfig.cmake
+4 −4		cmake/Flags.cmake
+32 −32		cmake/Hunter/config.cmake
+2 −2		cmake/depthaiDependencies.cmake
+5 −3		examples/CMakeLists.txt
+1 −1		examples/src/depth_crop_control.cpp
+2 −2		examples/src/depth_preview.cpp
+114 −0		examples/src/edge_detector.cpp
+1 −1		examples/src/mono_depth_mobilenetssd.cpp
+9 −3		examples/src/rgb_depth_aligned.cpp
+1 −1		examples/src/rgb_encoding_mono_mobilenet_depth.cpp
+1 −1		examples/src/spatial_location_calculator.cpp
+1 −1		examples/src/spatial_mobilenet.cpp
+1 −1		examples/src/spatial_mobilenet_mono.cpp
+1 −1		examples/src/spatial_object_tracker.cpp
+1 −1		examples/src/spatial_tiny_yolo.cpp
+2 −2		examples/src/stereo_depth_video.cpp
+2 −0		include/depthai/common/CameraBoardSocket.hpp
+2 −0		include/depthai/common/UsbSpeed.hpp
+2 −26		include/depthai/depthai.hpp
+2 −2		include/depthai/device/DataQueue.hpp
+5 −4		include/depthai/pipeline/Node.hpp
+40 −0		include/depthai/pipeline/datatype/EdgeDetectorConfig.hpp
+66 −0		include/depthai/pipeline/datatype/StereoDepthConfig.hpp
+1 −0		include/depthai/pipeline/datatype/Tracklets.hpp
+16 −0		include/depthai/pipeline/datatypes.hpp
+78 −0		include/depthai/pipeline/node/EdgeDetector.hpp
+60 −3		include/depthai/pipeline/node/StereoDepth.hpp
+20 −0		include/depthai/pipeline/nodes.hpp
+1 −1		shared/depthai-shared
+13 −8		src/device/CalibrationHandler.cpp
+81 −89		src/device/DataQueue.cpp
+14 −15		src/device/Device.cpp
+1 −1		src/device/DeviceBootloader.cpp
+22 −0		src/pipeline/datatype/EdgeDetectorConfig.cpp
+7 −5		src/pipeline/datatype/NNData.cpp
+45 −0		src/pipeline/datatype/StereoDepthConfig.cpp
+21 −1		src/pipeline/datatype/StreamPacketParser.cpp
+48 −0		src/pipeline/node/EdgeDetector.cpp
+3 −3		src/pipeline/node/NeuralNetwork.cpp
+64 −8		src/pipeline/node/StereoDepth.cpp
+4 −7		src/xlink/XLinkStream.cpp
+3 −6		tests/CMakeLists.txt