Athans Blog

For understanding core datatypes in pandas, including pandas.array, pandas.Series, pandas.DataFrame, pandas.Catgorical; resampling and masking

Python structure domains and why Numpy is memory efficient

Python systems have two programmatic domains:

• program structuring
• computational

In the program structuring domain, we may use lots of tools like the Python built-in types (e.g. list, dict, set) as well as Python metaphors (e.g. context manager for sequenced operations, decorators for wrapped behaviors, generators for lazy computations or for explicit lexical linearization of implicit state-graphs)

# this is not a computational type,
# otherwise these operations would be interpreted computaitonally
xs = [1,2,3,4]
print(f'{xs * 2 = }') # "structure"-level operation -> repetition

ys = [5,6,7,8]
print(f'{xs + ys =}') # "structure"-level operation -> concatenation

The contents of a Python built-in type are references to box types, as a consequence, operations within these sturctures are hard to optimizsed and thus slow.

from codetiming import Timer
from random import randint

dot = lambda xs, ys: sum(x*y for x, y in zip(xs, ys))

with Timer(text='creating "list" with comprehension syntax: {:0.4f} seconds'):
  xs = [randint(-100, 100) for _ in range(10000000)]
  ys = [randint(-100, 100) for _ in range(10000000)]
  
with Timer(text='"list" dot product of xs and ys: {:0.4f} seconds'):
  dot(xs, ys)

In comparison, numpy provides us with computatioinal type. This is type is a “manager class” that fully controls its contents. thus, it is free to store its contents in an optimal fashion and it is able to enfornce constraints on its contents.

from codetiming import Timer
from numpy.random import randint

with Timer(text='creating "numpy.ndarray": {:0.4f} seconds'):
	xs = randint(-100, 100, size=10000000)
	ys = randint(-100, 100, size=10000000)
  
with Timer(text='"numpy.ndarray": dot product of xs and ys: {:0.4f} seconds'):
  xs.dot(ys)

Additionaly, the numpy.ndarray is clearly a computational type. The contents are machine-typed (i.e. fixed bit-width int64). The numpy.ndarray array is actually just a view of some memory region.

from numpy import array

xs = array([1,2,3,4])
print(f'{xs * 2 = }') # elementwise operation

ys = array([5,6,7,8])
print(f'{xs + ys =}') # elementwise operation

print(f'{xs.__array_interface__["data"][0] = :#_x}') # memory location
print(f'{xs.dtype}') # intepreted type
print(f'{xs.shape}') # intepreted shape
print(f'{xs.strides}') # intepreted strides

(ys := xs.copy()).shape = 2, 2
print(f'{ys = }')

from numpy.lib.stride_tricks import as_strided
ys = as_strided(xs, strides=(16, 8), shape=(2, 2))
print(f'{ys = }')

from numpy import set_printoptions
set_printoptions(linewidth=float('inf'), threshold=24)

# cast VS convert
xs = array([1,2,3,4]) * 100
(ys := xs.copy()).dtype = 'int8'
print('.dtype = ...'.center(20, '-'))
print(f'{ys = }')

ys = xs.copy().astype('int8')
print('.astype(...)'.center(20, '-'))
print(f'{ys = }')

Therefore, analysing memory usage of numpy code is very easy.

from numpy import shares_memory
from numpy.random import normal

xs = normal(size=1000000)
print(f'{xs.nbytes = :,}') # total number of bytes

xs = normal(size=1000000).astype('float32')
print(f'{xs.nbytes = :,}') # total number of bytes

Numpy's limitations and Pandas design

However, numpy as a pure computational type has limits:

• its ergonomics aren’t great (we may prefer a ’named tensor’ like xarray.DataArray)
• it lacks business-level information, such as missing data
• it lacks business-level information, such as querying/filtering mechanisms

xs = array([1,2,3,None])
print(f'{xs.dtype = }')
xs = array([1,2,3,float('nan')])
print(f'{xs.dtype = }')

Pandas patches useful meta data on top of the numpy objects, try to cater to the realistic data's characteristics and facilitate querying capabilities within these computational objects. Moreover, Pandas supports wider statistical operations and time-series operations.

from pandas import array, Categorical

xs = array([1,2,3,None])
print(f'{xs.dtype = }')
print(f'{xs._data = }')
print(f'{xs._mask = }')

from numpy import shares_memory
from numpy.random import choice
from string import ascii_lowercase

ws = choice([*ascii_lowercase], size=(10, 4)).view('<U4').ravel()
print(f'{ws.nbytes = }')

xs = array(ws)
ys = array(choice([*ascii_lowercase], size=(100000, 4)).view('<U4').ravel())

print(f'{xs.memory_usage() = :,}')
print(f'{ys.memory_usage() = :,}')

print(f'{shares_memory(xs._ndarray, ws) = }')

zs = Categorical(ys)
print(f'{zs.memory_usage() = :,}')

from pandas import Series
from numpy.random import normal

s = Series(xs := normal(size=100000), name='nums')
print(f'{shares_memory(s.array._ndarray, xs) = }')

print(len(s[s > 2]) / len(s))
print(s[(s >= 2) | (s <= -2)])
print(xs[(xs >= 2) | (xs <= -2)])
print(s.sum())
print(s.var())
print(s.std())

print(s[lambda s: s >= 2])
print(xs[lambda xs: xs >= 2]) # NOT implemented
print(s.shift())
print(s.diff())

Pandas has alternative lookup modalities, pandas.Series provides a structure with two lookup models:

1. by positional index
2. by labelled index

from pandas import Series, MultiIndex, to_datetime
from numpy import arange, array
from itertools import islice, cycle
from datetime import datetime, timedelta
from random import randrange
from numpy.random import permutation

s = Series(arange(size := 6), index=[*ascii_lowercase[:size]], name='nums')
print(s.iloc[0])
print(s.loc['a'])
print(s.iloc[0:2]) # use half-open interval
print(s.loc["a":"c"]) # use open interva

index0 = 'ab'
assert size % len(index0) == 0
index1 = lambda off: islice(cycle(ascii_lowercase), off, off + (size // len(index0)))

s.index = MultiIndex.from_tuples((x, y) for x in index0 for y in index1(ord(x)))
print(s)
print(s.loc["a"])
print(s.loc["a","t"])

s = Series(arange(size := 6), name='nums')
s.index = to_datetime(array([timedelta(seconds=randrange(1,60)) for _ in range(size)]).cumsum() + datetime.now().replace(hour=9, minute=0, second=0, microsecond=0))
print(s)
print(s.loc["2021-04"])
print(s.loc["2021-04-29 09:00"])

print(f'{s.index.is_monotonic = }')
s.index = permutation(s.index)
print(f'{s.index.is_monotonic = }')

A pandas.Series is a one-dimentional structure with an index, looking like:

A pandas.DataFrame is a two-dimentional structure with a major and minor index, looking like:

A quick guide to pandas.DataFrame operations:

• df[x] -> look-up, using “column” index, labelled

• df[df.columns[x]] -> look-up, using “column” index, positional

• df.loc[x] -> look-up, using “row” index, labelled

• df.iloc[x] -> look-up, using “row” index, positional

• df.groupby(..., axis=0) -> aggregate data along rows using “row” index

• df.groupby(..., axis=1) -> aggregate data along columns using “column” index

• df.resample(...) -> perform N:M mapping of data along rows using “row” index

• df.stack -> turn “column” index into a “row” index

• df.unstack -> turn “row” index into a “index” index

• df.melt -> take data along rows with corresponding “column” index values and turn into new column

• df.pivot -> pivot data along rows into new columns

• df.pivot_table -> perform .groupby and .unstack with finger control

from pandas import DataFrame, to_datetime

df = DataFrame({
  'a': arange(size := 6),
  'b': arange(10, 10+size),
  'c': arange(100, 100+size),
})

df.index = to_datetime(array([timedelta(seconds=randrange(1,60)) for _ in range(size)]).cumsum() + datetime.now().replace(hour=9, minute=0, second=0, microsecond=0))

print(df)
df = df.transpose()
print(df)