Making a simple model overlay application

In the first instance, let’s just get a 3D rendering of a model, overlaid on live video from your webcam, something like this …

00 - Simple overlay application

Using your favourite text editor or Python development environment (e.g., `pycharm`_, `vscode`_, etc), create a new file called vtkoverlay_app.py or similar under a new directory applications or your preferred name.

Start with some import statements

import sys
from PySide2.QtWidgets import QApplication
from sksurgeryutils.common_overlay_apps import OverlayBaseWidget

scikit-surgery provides an OverlayBaseWidget module that creates a qtwidget showing a live stream from a video source, overlaid with a rendered surface model. scikit-surgery leaves the update method unimplemented so that the user can implement their own version in an child class.

#create an OverlayApp class, that inherits from OverlayBaseWidget
class OverlayApp(OverlayBaseWidget):

and implement a minimal update method

def update_view(self):

    #read a new image from the video source
    _, image = self.video_source.read()

    #copy the image to the overlay window
    self.vtk_overlay_window.set_video_image(image)

    #and render
    self.vtk_overlay_window.Render()

Now we build the application itself.

You’ll need a surface model (stl, vtk, vtp), which you can put in a directory named “models”. You can download the model used in the video above from the project repository, or use a model of your own.

if __name__ == '__main__':
  #first we create an application
  app = QApplication([])

  #then an instance of OverlayApp. The video source
  #is set when we create the instance. This is an index
  #starting at 0. If you have more than one webcam, you can
  #try using different numbered sources
  video_source = 0
  viewer = OverlayApp(video_source)

  #Set a model directory containing the models you wish
  #to render and optionally a colours.txt defining the
  #colours to render in.
  model_dir = '../models'
  viewer.add_vtk_models_from_dir(model_dir)

  #start the viewer
  viewer.show()
  viewer.start()

  #start the application
  sys.exit(app.exec_())

Now run the application with

python vtkoverlay_app.py

or similar. If successful you should see a live video stream overlaid with a rendered surface model, something like the video at the top of the page. Congratulations. If not you can download a finished example and compare. Play around with it, see what happens if you delete some line or change part of the update method.

Next we will add some code to the update loop to move the rendered model for each frame update.