Getting Started¶
Requirements¶
Note: Alternatively, biweeklybudget is also distributed as a Docker container. Using the dockerized version will eliminate all of these dependencies aside from MySQL and Vault (the latter only if you choose to take advantage of the OFX downloading), both of which you can also run in containers.
- Python 2.7 or 3.4+ (currently tested with 2.7, 3.4, 3.5, 3.6 and developed with 3.6)
- Python VirtualEnv and
pip
(recommended installation method; your OS/distribution should have packages for these) - MySQL, or a compatible database (e.g. MariaDB ). biweeklybudget uses SQLAlchemy for database abstraction, but currently specifies some MySQL-specific options, and is only tested with MySQL.
- To use the automated OFX transaction downloading functionality:
- A running, reachable instance of Hashicorp Vault with your financial institution web credentials stored in it.
- If your bank does not support OFX remote access (“Direct Connect”), you will need to write a custom screen-scraper class using Selenium and a browser.
Installation¶
It’s recommended that you install into a virtual environment (virtualenv / venv). See the virtualenv usage documentation for information on how to create a venv.
This app is developed against Python 3.6, but should work back to 2.7. It does not support Python3 < 3.4.
mkdir biweeklybudget
virtualenv --python=python3.6 .
source bin/activate
pip install biweeklybudget
Important Note: Anyone who’s using this project for actual data should install
from the package on PyPI. While the master
branch of the git repository is always
in a runnable state, there is no guarantee that data will be not be harmed by upgrading
directly to master. Specifically, database migrations are only compatible between released
versions; master
is considered a pre-release/development version, and can have migrations
removed or altered in breaking ways between official releases.
Upgrading¶
Documentation for upgrades depends on how you’ve installed and run biweeklybudget:
- For non-docker installations, see Flask Application - Database Migrations
- For Docker installations, no special action is needed.
- For development installations, see Development - Alembic DB Migrations
In all cases, you should always perform a full backup of your database before an upgrade.
Configuration¶
biweeklybudget can take its configuration settings via either constants defined in a Python module or environment variables. Configuration in environment variables always overrides configuration from the settings module.
Settings Module¶
biweeklybudget.settings
imports all globals/constants from a
module defined in the SETTINGS_MODULE
environment variable. The recommended
way to configure this is to create your own separate Python package for customization
(either in a private git repository, or just in a directory on your computer)
and install this package into the same virtualenv as biweeklybudget. You then
set the SETTINGS_MODULE
environment variable to the Python module/import
path of this module (i.e. the dotted path, like packagename.modulename
).
Once you’ve created the customization package, you can install it in the virtualenv
with pip install -e <git URL>
(if it is kept in a git repository) or
pip install -e <local path>
.
This customization package can also be used for Loading Data during development, or implementing Custom OFX Downloading via Selenium. It is the recommended configuration method if you need to include more logic than simply defining static configuration settings.
Environment Variables¶
Every configuration setting can also be specified by setting an environment
variable with the same name; these will override any settings defined in
a SETTINGS_MODULE
, if specified. Note that some environment variables
require specific formatting of their values; see the
settings module documentation
for a list
of these variables and the required formats.
Running Locally¶
Setup¶
source bin/activate
export SETTINGS_MODULE=<settings module>
It’s recommended that you create an alias to do this for you. Alternatively,
instead of setting SETTINGS_MODULE
, you can export the required environment
variables (see above).
Running In Docker¶
Biweeklybudget is also distributed as a docker image, to make it easier to run without installing as many Requirements.
You can pull the latest version of the image with docker pull jantman/biweeklybudget:latest
, or
a specific release version X.Y.Z
with docker pull jantman/biweeklybudget:X.Y.Z
. It is recommended
that you run a specific version number, and that you make sure to perform a database backup before upgrading.
The only dependencies for a Docker installation are:
- MySQL, which can be run via Docker (MariaDB official image recommended) or local on the host
- Vault, if you wish to use the OFX downloading feature, which can also be run via Docker
Important Note: If you run MySQL and/or Vault in containers, please make sure that their data is backed up and will not be removed.
The image runs with the tini init wrapper and uses gunicorn under Python 3.6 to serve the web UI, exposed on port 80. Note that, while it runs with 4 worker threads, there is no HTTP proxy in front of Gunicorn and this image is intended for local network use by a single user/client. The image also automatically runs database migrations in a safe manner at start, before starting the Flask application.
For ease of running, the image defaults the SETTINGS_MODULE
environment variable to
biweeklybudget.settings_example
. This allows leveraging the environment variable
configuration overrides so that you need only
specify configuration options that you want to override from
settings_example.py.
For ease of running, it’s highly recommended that you put your configuration in a Docker-readable environment variables file.
Environment Variable File¶
In the following examples, we reference the following environment variable file.
It will override settings from settings_example.py
as needed; specifically, we need to override the database connection string,
pay period start date and reconcile begin date. In the examples below, we would
save this as biweeklybudget.env
:
DB_CONNSTRING=mysql+pymysql://USERNAME:PASSWORD@HOST:PORT/DBNAME?charset=utf8mb4
PAY_PERIOD_START_DATE=2017-03-28
RECONCILE_BEGIN_DATE=2017-02-15
Containerized MySQL Example¶
This assumes that you already have a MySQL database container running with the container name “mysql” and exposing port 3306, and that we want the biweeklybudget web UI served on host port 8080:
In our biweeklybudget.env
, we would specify the database connection string for the “mysql” container:
DB_CONNSTRING=mysql+pymysql://USERNAME:PASSWORD@mysql:3306/DBNAME?charset=utf8mb4
And then run biweeklybudget:
docker run --name biweeklybudget --env-file biweeklybudget.env \
-p 8080:80 --link mysql jantman/biweeklybudget:latest
Host-Local MySQL Example¶
It is also possible to use a MySQL server on the physical (Docker) host system. To do so,
you’ll need to know the host system’s IP address. On Linux when using the default “bridge”
Docker networking mode, this will coorespond to a docker0
interface on the host system.
The Docker documentation on adding entries to the Container’s hosts file
provides a helpful snippet for this (on my systems, this results in 172.17.0.1
):
ip -4 addr show scope global dev docker0 | grep inet | awk '{print $2}' | cut -d / -f 1
In our biweeklybudget.env
, we would specify the database connection string that uses the “dockerhost” hosts file entry, created by the --add-host
option:
# "dockerhost" is added to /etc/hosts via the `--add-host` docker run option
DB_CONNSTRING=mysql+pymysql://USERNAME:PASSWORD@dockerhost:3306/DBNAME?charset=utf8mb4
So using that, we could run biweeklybudget listening on port 8080 and using our host’s MySQL server (on port 3306):
docker run --name biweeklybudget --env-file biweeklybudget.env \
--add-host="dockerhost:$(ip -4 addr show scope global dev docker0 | grep inet | awk '{print $2}' | cut -d / -f 1)" \
-p 8080:80 jantman/biweeklybudget:latest
You may need to adjust those commands depending on your operating system, Docker networking mode, and MySQL server.
Settings Module Example¶
If you need to provide biweeklybudget with more complicated configuration, this is
still possible via a Python settings module. The easiest way to inject one into the
Docker image is to mount
a python module directly into the biweeklybudget package directory. Assuming you have
a custom settings module on your local machine at /opt/biweeklybudget-settings.py
, you would
run the container as shown below to mount the custom settings module into the container and use it.
Note that this example assumes using MySQL in another container; adjust as necessary if you are using
MySQL running on the Docker host:
docker run --name biweeklybudget -e SETTINGS_MODULE=biweeklybudget.mysettings \
-v /opt/biweeklybudget-settings.py:/app/lib/python3.6/site-packages/biweeklybudget/mysettings.py \
-p 8080:80 --link mysql jantman/biweeklybudget:latest
Note on Locales¶
biweeklybudget uses Python’s locale module
to format currency. This requires an appropriate locale installed on the system. The docker image
distributed for this package only includes the en_US.UTF-8
locale. If you need a different one,
please cut a pull request against docker_build.py
.
Running ofxgetter in Docker¶
If you wish to use the ofxgetter script inside the Docker container, some special settings are needed:
- You must mount the statement save path (
STATEMENTS_SAVE_PATH
) into the container. - You must mount the Vault token file path (
TOKEN_PATH
) into the container. - You must set either the
VAULT_ADDR
environment variable, or theVAULT_ADDR
setting.
As an example, for using ofxgetter in Docker with your statements saved to /home/myuser/statements
on your host computer and your Vault token stored in /home/myuser/.vault-token
on your host computer, you would set STATEMENTS_SAVE_PATH
in your settings file to /statements
and TOKEN_PATH
to /.token
, and add to your docker run
command:
-v /home/myuser/statements:/statements \
-v /home/myuser/.vault-token:/.token
Assuming your container was running with --name biweeklybudget
, you could run ofxgetter (e.g. via cron) as:
docker exec biweeklybudget /bin/sh -c 'cd /statements && /app/bin/ofxgetter'
We run explicitly in the statements directory so that if ofxgetter
encounters an error
when using a ScreenScraper
class, the screenshots
and HTML output will be saved to the host filesystem.
Command Line Entrypoints and Scripts¶
biweeklybudget provides the following setuptools entrypoints (command-line
script wrappers in bin/
). First setup your environment according to the
instructions above.
bin/db_tester.py
- Skeleton of a script that connects to and inits the DB. Edit this to use for one-off DB work. To get an interactive session, usepython -i bin/db_tester.py
.loaddata
- Entrypoint for dropping all existing data and loading test fixture data, or your base data. This is an awful, manual hack right now.ofxbackfiller
- Entrypoint to backfill OFX Statements to DB from disk.ofxgetter
- Entrypoint to download OFX Statements for one or all accounts, save to disk, and load to DB. See OFX.wishlist2project
- For any projects with “Notes” fields matching an Amazon wishlist URL of a public wishlist (^https://www.amazon.com/gp/registry/wishlist/
), synchronize the wishlist items to the project. Requireswishlist==0.1.2
.