Data Visualization in Python with Pandas and Matplotlib

We recently helped my battalion compare different organizational changes by looking at how we place personnel. We wanted to optimize how we assigned personnel to specific jobs. The goal is to place the right amount of people based on the tasks we train each week. To get a feel for how to assign personnel, we asked our coworkers two questions for each training event:

1. What’s the fewest number of people you need to successfully run training?
2. If you had no constraints, how many people would you like to have to run training?

We gathered 10 responses on 17 weeks of training for 3 ranks with an upper bound and upper bound. So, the total number of data points is $10 \cdot 17 \cdot 3 \cdot 2 = 1080$ data points.

Setup

Thanks to Anaconda, a Python distribution, getting started on Windows was refreshingly straight-forward. The magical sequence of commands to get a working Jupyter notebook was:

conda create -n cadre python=3 numpy jupyter matplotlib pandas
activate cadre
jupyter notebook

The Jupyter Notebook

Jupyter notebooks are amazing for explatory programming. I initially tried to graph the data in Excel, but I couldn’t figure out how to add multiple lines to a plot with a computed standard deviation without making three copies of the data.

First, we have to import the necessary libraries. Pandas is a data analysis library, kind of like Excel in Python. Matplotlib is, as the name hints, a plotting library. The abbreviations for the imports seem to be the standard way of importing pandas and the plotting capabilities of matplotlib.

# Magic comment to display matplotlib charts in the notebook
%matplotlib inline

import matplotlib
import matplotlib.pyplot as plt
import os
import pandas as pd

ggplot is the default style used by the R language. It’s an easy way to get decent looking charts with little effort.

matplotlib.style.use('ggplot')

Now, we load the data. Pandas will load the data into a data-frame, which lets us do all sorts of neat analysis.

DATA_FILE = os.path.expanduser('~/prog/cadre/cadre.csv')
CADRE_DATA = pd.read_csv(DATA_FILE)

def cadre_estimate(series, rank, bound):
    """Given a dataframe series, return the data grouped by week of an estimate
    for rank and bound."""
    is_rank = series['Rank'] == rank
    is_bound = series['Bound'] == bound

    narrowed_data = series[is_rank & is_bound]
    est_by_week = narrowed_data.groupby('Week')['Est']
    return est_by_week

def get_std_dev_by_rank(series, rank):
    """Compute the standard deviation for rank, by using the mean of the upper
    bound and the lower bound.

    The link below said the right way to do it is to square the standard
    of each bound, divide by the number of samples and take the square
    root of the whole thing.  I'm operating on pretty much blind faith here.
    The website had that trustworthy math-professor feel to it, so I think
    we're good.

    http://mathbench.umd.edu/modules/statistical-tests_t-tests/page05.htm"""

    upper = cadre_estimate(series, rank, 'max')
    upper_std_dev = upper.std() ** 2 / upper.count()

    lower = cadre_estimate(series, rank, 'min')
    lower_std_dev = lower.std() ** 2 / lower.count()

    mean_std_dev = (upper_std_dev + lower_std_dev) ** 0.5

    return mean_std_dev

We want better colors, so we’ll borrow a snippet from a blog post data visualization that uses colors from an advanced data analysis product called Tableau. That blog post goes into much more depth on making good charts based on Tufte’s The Visual Display of Quantitative Information.

colors = [(214, 39, 40), (44, 160, 44), (31, 119, 180)]
tableau_red, tableau_green, tableau_blue = [(r / 255., g / 255., b / 255.) for r,g,b in colors]

Let’s graph the data. This took a lot of tweaking. I finally decided it was good enough.

fig, axes = plt.subplots(nrows=3, ncols=1, sharex=True)
fig.dpi = 300
fig.set_size_inches(8, 7)


plt.xlabel("IBOLC Week Number")
plt.xlim([0,18])
fig.suptitle("Cadre Estimates for Number of Personnel by Type by Week")

ranks = ['CPT', 'SFC', 'SSG']
for ax,rank in zip(axes, ranks):
    ax.set_ylim([0, 16])


    weeks = range(1,18)
    min_bound_mean = cadre_estimate(CADRE_DATA, rank=rank, bound="min").mean().values
    max_bound_mean = cadre_estimate(CADRE_DATA, rank=rank, bound="max").mean().values
    overall_mean = (min_bound_mean + max_bound_mean) / 2


    # Upper bound
    ax.plot(weeks,
            max_bound_mean,
            marker=".",
            linestyle="None",
            label="{} upper bound".format(rank),
            color=tableau_red)

    # Mean of lower and upper bound
    ax.errorbar(weeks,
                overall_mean,
                marker='.',
                linestyle='dotted',
                yerr=get_std_dev_by_rank(CADRE_DATA, rank).values,
                label="mean",
                color=tableau_green)

    # Lower bound
    ax.plot(weeks,
            min_bound_mean,
            marker='.',
            linestyle="None",
            label="{} lower bound".format(rank),
            color=tableau_blue)


    ax.set_title("Number of {}s Estimate".format(rank),
                 fontdict={'fontsize': 10, 'verticalalignment': 'top'},
                 y=0.95)

    ax.set_ylabel("Number of {}s".format(rank))


    ax.yaxis.set_ticks_position('left')
    ax.xaxis.set_ticks_position('bottom')



plt.savefig(os.path.expanduser("~/prog/cadre/cadre_out.pdf"), dpi=300)
plt.show()