Scriptable Data Analysis with Jupyter Notebooks

The client organization had identified that most of the researchers working with their hardware used Microsoft Excel for data analysis and visualization. They wanted to offer their users a system that would give them more flexibility and power — while avoiding the pitfalls of using Excel for scientific purposes.

Being familiar with Python, the client was interested in exploring the possibility of offering pre-written Jupyter notebooks to researchers. They didn't want to divert already committed internal resources to this experiment and instead were looking for an outside team to lead the project. They decided to tap Six Feet Up for help following previous successful engagements.

For the project to be successful, it was important to create a system that empowered researchers to use the skills they already possessed. That meant they couldn't be expected to have a degree in computer science and know the intricacies of programming. The system had to be intuitive and require only basic coding concepts.

Six Feet Up worked closely with members of the client's team and constructed several Jupyter notebooks. A big part of the work was developing a brand-new API from scratch to interact with the client's existing laboratory systems. Without the API, the only ways to interact with the data the laboratory systems collect was through the GUI (which was not very flexible) and through SQL (which most researchers are not versed in). The API reduced complex SQL queries into simple one-line commands that were easy to learn and look up. The API also allowed researchers to publish custom statistics back to the laboratory database.

For the data analysis portion, the notebooks relied on Pandas DataFrame. Pandas is a Python data analysis library, and DataFrames are 2D data structures rendered in pandas. This saved researchers from having to learn to implement 2D matrices in Python from scratch. In order to visualize results and make graphs, the system used Altair, a visualization library based on Vega. Altair was chosen over similar options like MatPlotLib because it uses a simple syntax that's easy to learn and doesn't require intimate programming knowledge.

The Jupyter notebooks were preinstalled and preconfigured to interact with the laboratory hardware. One specific Jupyter notebook was set up to serve as documentation by providing instructions on how to run and modify the software. Examples were also included that users could run right on the document with real data. The goal was that researchers be able edit and duplicate the notebooks, as well as write their own from scratch if they so desired.