When we work with Deep Neural Networks on limited HW-resources we must get an overview over CPU- and VRAM-consumption, training progress, change of metrical variables of our network models, etc. Most of us will probably want to see the development of our system- and model-related variables in a graphical way. All of this requires dynamic plots, which are updated periodically and thus display monitored data live.
As non-professionals we probably use Python code in standalone Jupyter notebooks or (multiple) Python notebooks in a common Jupyterlab environment. All within a browser.
The Jupyterlab interface resembles a typical IDE. Its structure and code are more complicated than those of pure Jupyter Notebooks. Jupyterlab comes with more configuration options, but also with more SW-problems. But Jupyterlab has some notable advantages. Once you turned to it you probably won’t go back to isolated Jupyter notebooks.
In this post series I want to discuss how you can create, update and organize multiple dynamic plots with Jupyterlab 4 (4.0.8 in a Python 3.9 environment), Python 3 and Matplotlib. There are various options and – depending on which way you want to go – one must also overcome some obstacles. I will describe options only for a Linux KDE environment. But things should work on Gnome and other Linux GUIs, too. I do not care about Microbesoft.
In this first post I show you how to get an overview over Matplotlib’s relevant graphics backends. Further posts will then describe whether and how the backends “Qt5Agg”, “TkAgg”, “Gtk3Agg”, “WebAgg” and “ipympl” work. Only two of these backends automatically and completely update plot figures after each of a series of multiple plot commands in a notebook cell. The Gtk3-backend poses a specific problem – but this gives me a welcome opportunity to discuss a method to trigger periodic plot updates with the help of user-controlled asynchronous background tasks.
Addendum and major changes, 11/25/23 and 11/28/23: This post was updated and partially rewritten according to new insights. I have also added an objective regarding plot updates from background jobs.
Objectives – updates of multiple plot frames with live data
We need dynamic, i.e. live plots, whose frames are once created in various Jupyter cells – and which later can be updated based on changed and extended data values.
- We will organize multiple plot figures at some central place on our desktop screen – within Jupyterlab or outside Jupyterlab on the Linux desktop GUI.
- We will perform updates of multiple plots with changed data – first one after the other with code in separate and independent Jupyter cells (and with the help of respective functions).
- We will perform live updates of multiple plots in parallel and continuously by one common loop that gathers new data and updates related figures.
- We will perform continuous updates of our plot figures from background jobs.
Please regard the following the following scope and limitations:
- Regarding the 1st point: The way of preparing dynamic plot frames is a matter of both code and work organization. We can use independent windows on our Linux desktop-GUI or sub-areas of the Jupyterlab interface. While Qt5-, Tk- and Gk3 windows give us a lot of freedom here, we have to be more careful regarding a standard web-interface (WebAgg) supported by Matplotlib. WebAgg forces us to visualize all of our figures at a certain common point in time if we do not want to clatter our browser with many tabs and multiple superfluous instances of a web-page. Jupyterlab with its ipympl-backend offers us the option to show the output of cells in separate sub-windows of the Jupyterlab-interface.
- Regarding the 2nd point: We will pick up some new data in a notebook cell and just individually update plot figures which we have created before. I will discuss what we have to do when the update of the figures’ canvasses is not done automatically.
- Regarding the 3rd point: We focus on a situation where we periodically receive new data for all plots and update them in parallel. We will control both data gathering and plot updates by one central loop. (This indirectly covers the case of Keras based ANN-models where we would use callbacks provided to the fit()-function of the training loop. These callbacks would pick up new data and trigger the redrawing of matplotlib-figures and axes-subplot frames.) Note that such an approach blocks the use of further cells in our notebooks until the loop has finished.
- Regarding the 4th point: In the present post series we will not care much about details of how the changing data are produced or gathered from multiple and different resources. We will just produce artificial new data on the fly. But in real world ML-scenarios we may want to follow data creation of programs that were started independently of the plot figures. And we would like to work with other cells in our notebooks while the plots are updated. To cover such a non-blocking situation I will discuss both data and plot updates from Python threads running in the background of Jupyterlab.
Dynamic plots in Jupyterlab – interactive mode
All of the points named above require that we can produce dynamic live plots at all with Jupyterlab. What does dynamic mean? It basically means that we create a plot figure (with sub-plots) once and then only update the contents of the canvas region (and related tick-marks on the axes). We would expect that Matplotlib supports an automatic update of the figures’ contents. If this were true we would only have to execute plot commands and re-draw the canvas.
So, is there a mode for notebooks and Matplotlib which provides an automatic update of plot figures when we use plotting commands for already defined figures? Yes, this is Matplotlib’s interactive mode.
From my experience with Jupyter Notebook’s plot backends I had expected that starting interactive plot mode via the pyplot.ion() would be sufficient in Jupyterlab to guarantee automatic figure updates as soon as we re-draw the figure’s canvasses. This only works directly and well for 2 backends. Other backends require additional commands.
Addendum 11/28/2023: I am tempted meanwhile to accept this as a feature. Wait for a discussion in the next post.
In addition, at least on my test system (Leap 15.4, KDE 5.90, Qt 5.15, Python 3.9.6, Matplotlib 3.8.2, Jupyterlab 4.0.8), Gtk3 caused further problems which require an additional trick. I will show for the problematic cases how we can take over control by our own periodic update loops in Jupyterlab’s background.
Note that there is also another meaning of “dynamic” and “interactive”: We may also mean the ability to use control buttons for a plot frame which enable us to interactively move, zoom or save plot contents. We will see in the context of a Gtk3-related graphics backend that this kind of interactivity is closely related to the canvas update problem.
Theoretical properties of the interactive mode [ triggered by ion() ]
According to matplotlib’s documentation we would expect 3 things in interactive mode; I quote:
- newly created figures will be shown immediately;
- figures will automatically redraw on change;
- pyplot.show() will not block by default.
The second point leaves some room for interpretation. A hard interpretation would be that any plotting action would automatically be displayed on figures and their canvasses under the control of a Matplotlib-backend as soon as we request a re-draw of the figure’s canvas. A softer interpretation would say: Well, produce real output only when you explicitly provide your basic render results to the output engine of your GUI-backend. And by thinking a bit more about the consequences for the nterplay between a command UI and e.g. a desktop GUI we may find that the “right” way of doing it is a matter of implementation reasoning and even efficiency. We will dig a bit deeper in thenext post.
To make a long test story short for the time being:
In Jupyterlab the so called Jupyterlab specific “inline”-backend does not display changes automatically. The backend “notebook” which often is used in stand-alone Jupyter Notebooks does not work at all with Jupyterlab. Most of so called “interactive backends” do not automatically update the canvas regions in their (external) plot-windows when we call the draw( )-command for rendering. In forthcoming posts we will see that there are two sides to this. And Gtk3 – at least on my system – is not working as expected.
Overview over Matplotlib backends
Matplotlib works together with a variety of backends. A present list of supported standard backends is
‘GTK3Agg’, ‘GTK3Cairo’, ‘GTK4Agg’, ‘GTK4Cairo’, ‘MacOSX’, ‘nbAgg’, ‘QtAgg’, ‘QtCairo’, ‘Qt5Agg’, ‘Qt5Cairo’, ‘TkAgg’, ‘TkCairo’, ‘WebAgg’, ‘WX’, ‘WXAgg’, ‘WXCairo’, ‘agg’, ‘cairo’, ‘pdf’, ‘pgf’, ‘ps’, ‘svg’, ‘template’
See below for Python code cell taken from here to get such a list. You find the list also in the Matplotlib-documentation. Which of these standard backends really are available depends on your system, its installed Linux packages and on installed Python modules. On my system
[‘agg’, ‘gtk3cairo’, ‘gtk3agg’, ‘svg’, ‘tkcairo’, ‘qt5agg’, ‘template’, ‘cairo’, ‘ps’, ‘nbagg’, ‘pgf’, ‘qtagg’, ‘pdf’, ‘tkagg’, ‘webagg’, ‘qtcairo’, ‘qt5cairo’]
were found to be valid backends. Of these only some are so called “interactive backends“, which are required to perform dynamic updates.
Standard backends
- Static standard backends: The backends {‘pdf’, ‘pgf’, ‘ps’, ‘svg, ‘template’, ‘cairo’} are static ones and not interactive. They do not work in our dynamic context.
- Interactive standard backends: All of the other backends are theoretically interactive ones. Note that the Cairo-variants of the other backends work similar to their basic counterparts. On my system “GTK4” and “WX” are not fully installed. So I could not test them.
Other Jupyter– and Jupyterlab-related backends
There are some IPython/Jupyter-specific backends. These backends are implemented in Jupyter or Jupyterlab:
- inline – standard plot-backend in Jupyterlab. It does not support interactive mode and cannot be used in our context.
- ipympl – the only Ipython-specific plot-backend which supports matplotlib’s interactive mode within Jupyterlab notebooks,
- notebook [nb, equivalent to nbAgg] – does not work in Jupyterlab at all (error message).
On standalone Jupyter Notebooks the “notebook”-backend does not only work; there it also supports ion(). However, on Jupyterlab “notebook” will lead to a warning like:
notebook backend: Javascript Error: IPython is not defined.
How to activate a backend
You can select and set a specific standard backend to be used in a notebook via the commands:
- matplotlib.use(‘TkAgg’) or pyplot.switch_backend(‘TkAgg’).
The backend strings will be evaluated case-insensitive.
Note: If you want to change a backend within a running notebook you will almost always get an error messages in Jupyterlab. To test a new backend you normally have to restart the notebook’s kernel.
The Jupyter specific backends can be invoked in Jupyter notebooks by so called magic cell commands , e.g.
- Jupyterlab: %matplotlib inline, %matplotlib ipympl (or equivalently %matplotlib widget) .
This should, of course, be done before performing figure definitions.
Python code to get lists of supported matplotlib backends
Code to get a list of basically supported standard backends of Matplotlib
from __future__ import print_function, division, absolute_import
from pylab import *
import time
import matplotlib.backends
import matplotlib.pyplot as plt
import os.path
def is_backend_module(fname):
"""Identifies if a filename is a matplotlib backend module"""
return fname.startswith('backend_') and fname.endswith('.py')
def backend_fname_formatter(fname):
"""Removes the extension of the given filename, then takes away the leading 'backend_'."""
return os.path.splitext(fname)[0][8:]
# get the directory where the backends live
backends_dir = os.path.dirname(matplotlib.backends.__file__)
# filter all files in that directory to identify all files which provide a backend
backend_fnames = filter(is_backend_module, os.listdir(backends_dir))
backends = [backend_fname_formatter(fname) for fname in backend_fnames]
print("Supported standard backends: \n" + str(backends))
print()
Code to get a list of valid standard Matplotlib backends on your Linux system
# validate backends
backends_valid = []
for b in backends:
try:
plt.switch_backend(b)
backends_valid += [b]
except:
continue
print("Valid standard backends on my system: \n" + str(backends_valid))
print()
Backends working directly with interactive mode in Jupyterlab Python notebooks
Presently the only Matplotlib-backends which truly respect ion() in Jupyterlab (in the sense of our hard interpretation; see above) are
- “WebAgg” (browser-related-backend based on the tornado-server).
- “ipympl” (Jupyter specific-backend)
Both react directly to figure.canvas.draw()-requests.
Backends that need flushing events in Jupyterlab to work dynamical
The following backends do not support multiple automatic updates of figure canvasses when these updates are requested within a loop executed in the same notebook cell:
- “Qt5Agg” (supports Qt5 windows on the desktop outside Juypterlab)
- “TkAgg” (supports Qt5 windows on the desktop outside Juypterlab)
- “Gtk3Agg” (supports Qt5 windows on the desktop outside Juypterlab. But requires a workaround regarding an in initial error)
The canvas of a figure will only be updated on the GUI when the last Python code command of a cell has finished. All these backends require a so called “flushing” of canvas related draw()-requests and respective events to the GUI. I will explain in the next post in more detail how we enforce the output production of accumulated render-results which typically are gathered in a queue for a figure’s canvas at any time.
We will see that automatic periodic updates can be performed by asynchronous processes in the background of Jupyterlab. This will lead to the construction of a workaround for all of the named “problematic” backends – if and when we really want a plot window to directly react to draw()-requests in Jupyterlab notebooks.
Workaround for Gtk3-backend error in Jupyterlab?
The Gtk3Agg-backend not only does not work with ion() in Jupyterlab. It creates an additional error both on KDE and Gnome – if ion() is not started ahead of setting the backend to be used. However, when we start ion() ahead even the interactive control elements (buttons) do not work afterward. But we will see that the workaround scheme which we will build for problematic backends doe ssolve this problem, too.
Conclusion
There are several Matplotlib backends which we can potentially use to dynamically update the canvas contents of already created figures with new data. Unfortunately (or by design?), not all backends support direct or continuous automatic canvas updates requested by multiple draw-commands in one and the same notebook cell. If we want to display the results of certain certain plot commands directly and dynamically we either need to issue special commands or develop a specific background job in a Jupyterlab notebook to cover such a situation for critical backends like Qt5Agg.
In addition, for real world scenarios we do not want to our plot production to behave blocking on the level of notebook cells: We want to be able to perform such updates from any cell in our notebook, from a common loop (with and without blocking the respective notebook cell) and in particular from background jobs. To avoid cell blocking we will necessarily have to include background execution into our workaround.
The development of a workaround solutions will be the topic of the next post. I will demonstrate different approaches first for the example of the Qt5-backend and figure windows external to Jupyterlab. In further posts I will discuss the use of Tk– and Gtk3-windows. Afterward, I will demonstrate the usage of the Web-backend of Matplotlib. A final post will cover IPython’s ipympl-backend for dynamic plot figures within the Jupyterlab interface.