Zhizhong John DIng

This is a cheatsheet and tutorial for the Numpy library.

• 1. Advantage of Numpy over python list
• 2. Basics
  • Data type conversion
• 3. Array creation
  • 3.1. From known value
    • Create from Python list
    • Create using function
  • 3.2 Initializing with placeholder
  • 3.3 Import
    • 3.3.1 Import from csv
      • Method 1. use csv reader
      • Method 2. load csv with Numpy function
      • Method 3. load csv with Pandas
• 4. Basic operations

NumPy is more compact, convenient than Python Lists. The vector and matrix operations in NumPy is well and efficiently implemented. NumPy is much faster and it also have more functionality such as FFTs, convolutions, fast searching, basic statistics, linear algebra, histograms, etc.

• In Numpy, dimension are called axes
• Numpy's class type is called ndarray

import numpy as py
a = np.array([[1,2,3],[4,5,6],[7,8,9],[10,11,12]])

ndarray.astype(type) is a great tool to convert data to int, float or str.

a = np.array([])

Create from python list, python tuple or python list of tuples, or python list of list. Results are same no matter from list of list or from tuple of list. Recommend using List to avoid confusion

Note: You can also use tuple to create numpy Array

a = np.array((1,2,3,4))
a = np.array([(1,2),(3,4),(5,6)])
a = np.array(((1,2),(3,4),(5,6)))

But personally I don't recommend using tuple. Because using list is consistent with python list for either array and matrix, easy to remember.

def f(x,y):
    return 10*x+y    
a = np.fromfunction(f,(5,4))
print(a)

[[ 0. 1. 2. 3.]
[10. 11. 12. 13.]
[20. 21. 22. 23.]
[30. 31. 32. 33.]
[40. 41. 42. 43.]]

Original data: test_import.csv

Latitude,Longitude,Elevation
48.89016000,2.689270000,71.0
48.89000000,2.689730000,72.0
48.88987000,2.689810000,72.0
48.88924000,2.689570000,67.0
48.88934000,2.690050000,67.0
48.88949000,2.691400000,65.0

import csv
import numpy as np
data_path = 'test_import.csv'
with open(data_path, 'r') as f:
    reader = csv.reader(f, delimiter=',')
    # get header from first row
    headers = next(reader)
    # get all the rows as a list
    data = list(reader)
    # transform data into numpy array
    data = np.array(data).astype(float)

output looks like:

print(headers)
print(data.shape)
print(data[:3])

['Latitude', 'Longitude', 'Elevation']
(6, 3)
[[48.89016 2.68927 71. ]
[48.89 2.68973 72. ]
[48.88987 2.68981 72. ]]

Comment:

1. This method could be slow, if the data file is huge.
2. next() returns the next item from the iterator

Two functions:

1. numpy.loadtxt
2. numpy.genfromtxt

numpy.genfromtxt is recommended over the other because np.genfromtxt can read CSV files with missing data and gives you options like the parameters missing_values and filling_values that help with missing values in the CSV.

Original data: test_import.csv

X,Y,Name,small,large,circular,mini_hoop,total_rack
982903.56993819773,205129.99858243763,1 7 AV S,5,0,0,0,5
987330.41607135534,191302.73030526936,1 BOERUM PL,1,0,0,0,1
983210.95318169892,199016.51343409717,1 CENTRE ST,10,0,0,0,10
985897.83954019845,207157.88527469337,1 E 13 ST,1,0,0,0,1
1010993.9694659412,252137.33960694075,1 E 183 ST,0,0,2,0,2
987774.37089210749,210586.44665901363,1 E 28 ST,1,0,0,0,1

import numpy as np
data_path = "test_import2.csv"
types = ['f8', 'f8', 'U50', 'i4', 'i4', 'i4', 'i4', 'i4']
data = np.genfromtxt(data_path, dtype=types, delimiter=',',names=True)
a = data['X']

output looks like:

print(data)
print(data['X'])

[( 982903.5699382 , 205129.99858244, '1 7 AV S', 5, 0, 0, 0, 5)
( 987330.41607136, 191302.73030527, '1 BOERUM PL', 1, 0, 0, 0, 1)
( 983210.9531817 , 199016.5134341 , '1 CENTRE ST', 10, 0, 0, 0, 10)
( 985897.8395402 , 207157.88527469, '1 E 13 ST', 1, 0, 0, 0, 1)
(1010993.96946594, 252137.33960694, '1 E 183 ST', 0, 0, 2, 0, 2)
( 987774.37089211, 210586.44665901, '1 E 28 ST', 1, 0, 0, 0, 1)]

[ 982903.5699382 987330.41607136 983210.9531817 985897.8395402 1010993.96946594 987774.37089211]

Comment:

1. This method returns a tuple list
2. names = True to access the header and use it as column name to return a specific column

Note: if the data is generated in excel file, should export the data as csv file.

1. get array from data

import pandas as pd
path = "test_import2.csv"
df=pd.read_csv(path, delimiter = ',', header = 0)
a = df['X'].values

output looks like:

print(a)
print(type(a))

[ 982903.5699382 987330.41607136 983210.9531817 985897.8395402 1010993.96946594 987774.37089211]
<class 'numpy.ndarray'>

2. get matrix from data

import pandas as pd
path = "test_import2.csv"
df=pd.read_csv(path, delimiter = ',', header = 0)
a = df.iloc[:,:2].values

output looks like:

print(a)
print(type(a))

[[ 982903.5699382 205129.99858244]
[ 987330.41607136 191302.73030527]
[ 983210.9531817 199016.5134341 ]
[ 985897.8395402 207157.88527469]
[1010993.96946594 252137.33960694]
[ 987774.37089211 210586.44665901]]
<class 'numpy.ndarray'>

Rule: all operations for array (vector) are element wise.

a = np.array([20,30,40,50])
b = np.arange(4) # b = np.array([0,1,2,3])

Rule: except matrix product, all operations are element wise.

a = np.array([[1,1],[0,1]])
[[1 1]
[0 1]]
b = np.array([[2,0],[3,4]])
[[2 0]
[3 4]]

cumsum min exp etc needs update

under construction

Comment:

1. A key difference between flatten() and ravel() is that flatten() is a method of an ndarray object and hence can only be called for true numpy arrays. In contrast ravel() is a library-level function and hence can be called on any object that can successfully be parsed.
2. Turn 1D array into 2D array:

import numpy as np
A = np.arange(8)
A = A.reshape(1,8)
#or
A = A.reshape(8,1)

def f(x,y):
    return 10*x+y
a = np.fromfunction(f,(3,4))

print(a)

[[ 0. 1. 2. 3.]
[10. 11. 12. 13.]
[20. 21. 22. 23.]]

b = a.reshape(2,6)
print(b)
print(a)

[[ 0. 1. 2. 3. 10. 11.]
[12. 13. 20. 21. 22. 23.]]

[[ 0. 1. 2. 3.]
[10. 11. 12. 13.]
[20. 21. 22. 23.]]

 b = a.resize(2,6)
 print(b)
 print(a)

None

Comment: resize works on the array itself, it returns None

[[ 0. 1. 2. 3. 10. 11.]
[12. 13. 20. 21. 22. 23.]]

b = np.ravel(a)
 print(b)
 print(a)

[ 0. 1. 2. 3. 10. 11. 12. 13. 20. 21. 22. 23.]

[[ 0. 1. 2. 3.]
[10. 11. 12. 13.]
[20. 21. 22. 23.]]

b = a.flatten()
print(b)
print(a)

[ 0. 1. 2. 3. 10. 11. 12. 13. 20. 21. 22. 23.]

[[ 0. 1. 2. 3.]
[10. 11. 12. 13.]
[20. 21. 22. 23.]]

Comment:

1. I don't recommend using concatenate. It's basically same as operators mentioned above by changing axis. And it could be confusing.
2. For np.append, when axis is specified, values must have the correct dimension(2D and 2D or 1D and 1D)

import numpy as np
A = np.arange(2,6)
B = np.arange(1,5)*2
print(A,'\n',B)

[2 3 4 5]
[2 4 6 8]

print(np.hstack((A,B)))

[2 3 4 5 2 4 6 8]

print(np.column_stack((A,B)))

[[2 2]
[3 4]
[4 6]
[5 8]]

print(np.vstack((A,B)))

[[2 3 4 5]
[2 4 6 8]]

np.append([[1, 2, 3], [4, 5, 6]], [[7, 8, 9]], axis=0)

[[1, 2, 3]
[4, 5, 6]
[7, 8, 9]]

np.append([1, 2, 3], [[4, 5, 6], [7, 8, 9]])

[1, 2, 3, ..., 7, 8, 9]

1. add a 1D vector to 2D array (stack horizontally)**

A_2d = np.array([[2, 3],
                 [4, 5],
                 [6, 7],
                 [8, 9]])
B_1d = np.array([1, 2, 3, 4])
B_1d_reshape = B_1d.reshape(len(B_1d),1)
new_matrix = np.hstack((A_2d,B_1d_reshape))
print(new_matrix)

[[2 3 1]
[4 5 2]
[6 7 3]
[8 9 4]]

2. Put together multiple arrays into a matrix

# matrix: t, x, y, v
t = np.array([0,1,2,3,4])
x = np.array([-2,-1,0,1,2])
y = np.array([-3,-1.5,0,1.5,3])
v = np.array([10,20,30,40,50])
new_matrix = np.hstack((t.reshape(len(t),1),
                        x.reshape(len(x),1),
                        y.reshape(len(y),1),
                        v.reshape(len(v),1)))
print(new_matrix)

[[ 0. -2. -3. 10. ]
[ 1. -1. -1.5 20. ]
[ 2. 0. 0. 30. ]
[ 3. 1. 1.5 40. ]
[ 4. 2. 3. 50. ]]

Same as python list.

import numpy as np
a = np.arange(10)**2
print(a)

[ 0 1 4 9 16 25 36 49 64 81]

print(a[2])

4

print(a[2:5])

[ 4 9 16]

print(a[::-1])

[81 64 49 36 25 16 9 4 1 0]

Same as python 2D list

def f(x,y):
    return 10*x+y
a = np.fromfunction(f,(5,4))

print(a[2,3])

23

print(a[1:4,2])

[12. 22. 32.]

print(a[1:3,:])

[[10. 11. 12. 13.]
[20. 21. 22. 23.]]

print(a[-1])

[40. 41. 42. 43.]

a.flat is a 1D iterator over the array

def f(x,y):
    return 10*x+y
a = np.fromfunction(f,(3,4))
print(a)
print(a.flat[6])

[[ 0. 1. 2. 3.]
[10. 11. 12. 13.]
[20. 21. 22. 23.]]
12.0