Blog
This is one place where I keep track of solving problems related to data, transportation, and design.
Installing Vue+Vuex+Vuetify
This short guide will hopefully clear up how node, vue, @vue-cli, and vue plugins all work together. Background These are the technologies needed to get a Vue single page application (SPA) going: Node.js is a JavaScript runtime built on Chrome’s V8 JavaScript engine. Think about it as the Python shell, you use the Python shell to run .py files. Node is used to run .js files npm is the default package manager for Node.
Toy deep neural network implementation with PyTorch
Linear Algebra and Learning from Data by Gilbert Strang has the clearest explanation of deep neural networks I have seen, so far. The main reason why I say this is because it describes a neural network as a composition of Continuous Piecewise Linear (CPL) functions using linear algebra notation. In addition, the key rule to compute the parameters of these CPL functions is stochastic gradient descent using backpropagation to execute the chain rule that shows up when calculating the gradient.
Conditionals on sequential data using numpy
It is well known that vectorized operations using numpy run much faster than equivalent loop based operations. Sometimes though, a problem is easier to think about in terms of loop based operations rather than vectorized operations. The problem Suppose you need to check for a particular sequence in a list. import pandas as pd import numpy as np data = pd.DataFrame({ 'acceleration': [ 1, -3, -2, 7, -10, -3, -2, 8, 2, -1, 5, 10, 7, -3, -4, -6, -7, -9, 10, -4, -2, 3, 6, 8, 9, 12, -11, -9, -7, -3 ] }) The most straightforward way to think about it is to check for the condition using a for loop.
Tip on running Flask applications on Apache/2.4.46 (FreeBSD) using CGI
After receiving several Internal Server Error from Apache and making changes to httpd-custom.conf when trying to run a Flask app using CGI, I realized that setting environment variables in your .profile does not guarantee your .cgi or .py scripts in the cgi-bin directory will have access to them. Knowing the behavior of crontab and Python scripts, I should have figured this out earlier. For example, if script.py looks something like
Tip on using Environment Variables inside Python scripts
After receiving several error emails from cron when trying to run the ETL pipeline I describe here, I realized that setting environment variables in your .profile does not guarantee your .sh scripts will have access to them. For example, if script.py looks like import os sqlusr = os.getenv('SQLUSR') sqlpass = os.getenv('SQLCRD') and these variables are required for something else in the script, the following cron job will not work: */10 * * * * /usr/bin/python script.
Simple ETL pipeline using Python, MySQL, and crontab
Here’s a simple ETL (extract, transform, load) process that can be setup quickly using Python, MySQL, and crontab. For more complicated processes, it’s better to substitute crontab with a more sophisticated tool, like Airflow. This setup is a ‘naive’ setup and I’ll probably iterate on it, but for a simple project it seems to be working. Project Planning When planning the development of this project my objectives were: To only use Python, MySQL, and crontab for the ETL pipeline.
Linear Algebra using NumPy
For beginner NumPy users with a linear algebra background, it’s easy to understand that to define a matrix, $A=\begin{bmatrix}1&2\\3&4\end{bmatrix}$, we should use A = np.array([[1,2],[3,4]]) since A Out[2]: array([[1, 2], [3, 4]]) A.shape Out[3]: (2, 2) However, how should $x=\begin{bmatrix}1\\2\end{bmatrix}$ be defined? Should we use a 1-D array x1 = np.array([1,2]) x1 Out[4]: array([1, 2]) x1.shape Out[5]: (2,) or a 2-D array with 1 column? x2=np.array([[1],[2]]) x2 Out[6]: array([[1], [2]]) x2.