Python is one of the most popular programming languages especially when it comes to data wrangling and analytics. If you have to work with data analysis on a daily basis — you might be very familiar with Jupyter notebooks.
Jupyter notebooks are an interactive web-based tool that is used for quick hypothesis testing, rapid prototyping, or consistent storytelling with your data. Since Jupyter notebooks are based on IPython kernel, they have integrated Magics which provide additional functionality to your IPython notebooks.
In this article, we will try to address the most popular magic commands that are being used in daily work by many data scientists/analysts.
Before we start: it is important to note that a single
% applies your chosen command on a given line ONLY and
%% applies the command on the entire cell.
One of the most useful commands when optimizing your Python code for the performance is
%time . It allows us to monitor the execution time of a Python statement or expression. In fact, there are two quite different commands, namely:
%timeit . Let’s find out what is the difference between the two.
%time command provides very basic timing functionality: the CPU and wall clock times are printed (one run), and the value of the expression (if any) is returned. Use the
%timeit magic for more control over the measurement (runs your statement multiple times therefore provides mean and standard deviation metrics of execution time).
Example benchmark of iterating and applying a function on the same data in
Once again, this function can be used both as a line and cell magic:
- Inline mode you can time a single-line statement (though multiple ones can be chained with using semicolons).
- In-cell mode, you can time the cell body (a directly following statement raises an error).
2. HTML snippets
HTML snippets can be freely used in your notebook cells as long as you start the cell with
<img src="https://bit.ly/3djzw7h" alt="fiiire ball" />
3. System interactions
You can execute system commands directly from your cells. In fact, there are two ways to do that:
# Output: ['Sun 31 May 2020 15:40:38 BST']!date
# Output: Sat 18 Apr 2020 14:35:53 BST!ls
# Output: all files and folders in your current directory
Very handy when you are missing a package installation, and do not want to switch to your system’s terminal:
!pip install numpy
4. Write to a file/read a file
Magic commands provide an easy way to read from and write to files. Here in the example below, we write a function
shout to a new file
written_file.py and subsequently, execute it to add the function definition to the scope of our notebook which we can directly run:
%who will list all variables that exist in the global scope of your notebook. It can be used to see what data/variables are there in memory.
%who: all variables present;
%who int: lists all the integers present in the scope;
%whos: similar to
%whobut gives more information about each variable
number = 42
string = "hello world"%who
# number string%who int
# Variable Type Data/Info
# number int 42
# string str hello world
6. Share variables between notebooks
When you are doing exploratory analysis, different analyses go into separate notebook files. However, sometimes you would like to share results from your first notebook and use them directly in the new one (continue your research, without dumping the intermediate data in a database).
If you want to share data between Jupyter notebooks, you can use the
%store command. First, execute this in the notebook containing the data:
your_data = 'your object'
Execute this in your new notebook to retrieve stored data:
%store -r your_data
%env — list all the environment variables.
%maptlotlib inline — allowing the output of
matplotlib plotting command to be displayed inline i.e. in Jupyter lab UI.
%load_ext autoreload— this magic command
load_ext allows you to load another important extension —
autoreload reloads any updated referenced code so you are sure you are always using the latest functionality of your package/modules (by default, notebooks tend to load referenced code (e.g. imported functions) only once).
%lsmagic— browse the full list of available magic commands. I am sure you will find your favorites as well.
It is brilliant how many out-of-the-box useful functionalities
IPython provides us. These magic commands make our data exploratory and code development work a lot more productive, as well as, more controllable.
You can find the full documentation here.
Which commands are your favorite ones? Let me know in the comments so that the most popular ones can be addressed in another post! 👏