python-shell-enhancement - Add History and Tab Completion to the Default Python Shell

I decided to take the blog post I wrote earlier this month, Add History and Tab Completion to the Default Python Shell, and commit code to github under the name

python-shell-ehnancement

I updated the script with feedback I got from the blog and reddit and fixed tab completion to work correctly on a mac by default.

Double Window Sniper Bonk

This is a blatant geek post because playing Team Fortress 2 is one of favorite hobbies. Anyone who knows me (gaming-wise), knows I like playing the Double Cross map. On this map there is a underhanded tactic called window sniping where a sniper stands sideways in he window and is able to shoot into the enemy courtyard. The other day, two snipers were hanging out in the window and I managed to get this screen cap of me taunt killing them both. Double Bonk!

Introducing virtualenvwrapper-django

I recently took one of my previous posts entitled "Never Have to Type "python manage.py" with Django Again", and combined it with another bash function to auto set DJANGO_MODULE_SETTINGS by interrogating a Django project's directory. I then rolled everything together into github as

virtualenvwrapper-django

Its a little rough I admit, and I welcome any feedback for improvements.  As a result, I'm now able to run commands like the following anywhere once I have my virtual environment loaded.

(spock)jbisbee@tacquito:~/src/spock$ manage dbshell
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 95
Server version: 5.5.31-0ubuntu0.12.04.2 (Ubuntu)

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Edit: I've gotten some feedback and I realize I didn't sell the real reason behind dynamically setting the DJANGO_SETTINGS_MODULE variable. I use Django at work and we have moved from the generic one file settings.py to a settings directory which looks something like this

  settings/
    base.py
    dev.py           # imports from base.py
    test.py          # imports from base.py
    production.py    # imports from base.py
    jbisbee.py       # imports from dev.py

This way you can have cascading Django settings with overrides at multiple levels. This bash function I wrote finds the settings directory and will set DJANGO_SETTINGS_MODULE to the user's specific dev settings (or dev if the user can't be found)

DJANGO_SETTINGS_MODULE=projectname.settings.jbisbee

This way in jbisbee.py I can override my database, turn on crazy debugging, or do anything else I want and be able to commit those settings without affecting others.

Disclaimer: If you manage your Django project's settings differently let me know. I'd love to make the determine_django_module_settings more flexible and patches are more than welcome!

Django's DateTimeField auto_now* Option Caveats

I'm new to Python and Django and had a run in with DateTimeField's auto_now and auto_now_new. I'm importing data from an existing database and there currently is a column called upd_stamp which I've renamed to modified. Looking through the documentation, I was happy that I could just add auto_now to the options to get the behavior I wanted without any additional code. Well it didn't and I'll explain why as well as the solution I came up with (thanks to a bit of Goggling and Stack Overflow).

created = models.DateTimeField(auto_now_add=True)
modified = models.DateTimeField(auto_now=True)

The problem I ran into was that I wanted to keep the legacy timestamp from the old database when I loaded the record for the first time then have the auto_now kick in. The problem is that auto_now is very stupid (as it should be) and doesn't understand the concept of insert or update and whether your new record is coming in with pre-populated DateTimeField attribute. It gladly throws away any value away and always replaces the value with the current time. I didn't notice it on first import but soon realized auto_now wouldn't meet my needs.

After a quick search, I found this Stack Overflow question that had the perfect fix for my problem. It suggested implementing my own save that would allow me to pass in values for created or modified but would fall back to the current time. I changed the code a bit because I didn't need a created timestamp for my model, but this should give you an idea.

Note: you have to remove any additional values from kwargs before passing to the parent class or super.save will complain that created and modified aren't valid arguments.

import datetime

class User(models.Model):
    created  = models.DateTimeField(editable=False)
    modified = models.DateTimeField()

   def save(self, *args, **kwargs):

        # original Code: called datetime.datetime.today()
        #                multiple places.  Modified kwargs
        #
        # created = kwargs.pop('created', None)
        # if created:
        #     self.created = created
        # elif not self.id:
        #     self.created = datetime.datetime.today()
        # 
        # modified = kwargs.pop('modified', None)
        # if modified:
        #     self.modified = modified
        # else:
        #     self.modified = datetime.datetime.today()

        # Edit #1: Changed example to more efficient 
        #          version from blog comments.
        #          single today()
        # 
        # today = datetime.datetime.today()
        #
        # if not self.id
        #     self.created = kwargs.pop('created', today)
        # elif kwargs.get('created', None)
        #     self.created = kwargs.pop('created')
        #
        # self.modified = kwargs.pop('modified', today)


        # Edit #2: no longer pass arguments via save
        #          was orignially how I wanted to solve
        #          the problem and how I felt auto_now
        #          should have worked.  Thanks to
        #          indosauros from reddit. :)
        #
        #          On inital creation of the object
        #          only set today if values are not
        #          already populated.  auto_now*
        #          wipes them out regardless

        today = datetime.datetime.today()

        if not self.id:
            if not self.created:
                self.created = today
            if not self.modified:
                self.modified = today
        else:
            self.modified = today

        super(User, self).save(*args, **kwargs)

If you see any glaring python or style related issues, please let me know. I'm still finding my way pythonwise and would love any best practices feedback.

Set UTF-8 as the Default Mysql Encoding

Ran into a small issue today with Mysql and encodings.  I'm running Ubuntu 12.04 and the default mysql database encoding is lantin1_sweedish_ci. To change the default to utf8, add the following to your my.cnf file

Edit: Turns out utf8 and utf8_general_ci is not the full utf8 implmentation in mysql. See this blog post for an in-depth explaination on how they're different (has to do with 4 byte utf8 characters being thrown out by mysql)

Edit #2: From reddit, credomane stated that character-set-handshake caused him troubles. With existing applications. I'm going to do a bit more research, but here is his response to sharing this blog entry on r/mysql

You might want to consider removing character-set-client-handshake or at least mention it could cause troubles with some programs that use mysql. Changing it to false or skipping the handshake makes MySQL behave like MySQL 4.0 according to the mysql docs. When I was looking over your blog post yesterday and checking the mysql docs that seemed like a bad idea to deny the client the ability to change to a character set it prefers/designed for.

Story time:
I went through the whole process of taking mysql offline, backing up my databases, altering/converting the tables, updating the mysql config and bringing mysql back online. At first everything seemed fine then I started to notice a lot of 500 errors appearing in my apache2 access logs and some things not working. Mainly applications sitting behind apache2's mod_wsgi. After spending several hours restoring databases and redoing everything database by database I eventually found I could convert all my databases just fine. Know I know the problem isn't in the database conversions but with my config changes.

  character-set-server = utf8mb4
  collation-server = utf8mb4_unicode_ci

Caused no problems what so ever but as soon as I added

  character-set-client-handshake = FALSE

I started having problems again. Investigating the problem applications show that they desire utf8_unicode_ci but since the mysql server says utf8mb4_unicode_ci only, they abort the mysql connection when they can't get the charset they wanted to prevent any possible corruption.

[mysqld]
collation-server = utf8_general_ci
init-connect='SET NAMES utf8'
character-set-server = utf8
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci

Great Stack Overflow post on the topic. Be sure to scroll down for Mysql 5.5 and above

Mysql reference: character_set_server, character-set-client-handshake, collation-server, init-connect

Special thanks to johns-appendix from my reddit post to r/programming for pointing out that I had the wrong encoding. I welcome the down votes as I learned something. :)

The Nest Thermostat Helped Me Save $360 So Far and Paid for Itself After 5 Months

About 8 months ago I went to Amazon.com to rent a streaming movie and saw an ad for the The Nest Learning Thermostat, saying it was one of the hottest selling items on Amazon. I was a bit skeptical because I figured a thermostat is a thermostat right, and asked myself, "why is this thing so popular?"

Well it turns out that the magic is in the merging of technologies. The Nest combines

• A Thermostat
• A motion sensor
• Wi-Fi
• Data from the internet
• Great interface (like the interface on an iPod)

The real magic is in the motion sensor. The biggest problem with optimizing a thermostat is you have to sit down and program the schedule you want it to follow. The Nest takes care of that for you. About a week after I installed it, I was walking down my hallway and the Nest lit up and stated it had, "learned my schedule" and would optimize my cooling. I was a bit taken aback but it makes perfect sense. The best part is that the default behavior is after it goes into "away" mode it doesn't turn back on until it senses motion again.

The Nest also does something else very subtly, it motivates you slowly turn up your target temperature to earn a "green leaf". I live in Florida and used to run my thermostat at a steady 77 degrees. Now with the Nest, I've bumped that up to 80 and supplemented with fans and surprisingly haven't noticed much of a difference in temperature. What I have noticed is a drastic change in my power bill.

I took my power bills from FP&L (power company in Florida) from the last 8 months and plotted the kilowatts per day usage and amazingly it has dropped over 30%. To put it in perspective, I've saved over $360 so far from December 2012 through July 2013 and Nest paid for itself after just 5 months. Best of all it was extremely simple to setup and install. It comes down to just plugging in just few wires and they have excellent phone support that can help walk you through the process step by step.

Disclaimer: I work during the day and have no one at home that would set off the motion sensor to turn the air back on. If you do have someone at home all day or you're one of the awesome people who actually program their thermostat, then most likely you won't see savings as large as mine. Also I live in south Florida were air conditioning is king and we only run the heat one or two times a year :)

Never Have to Type "python manage.py" with Django Again

I got very quickly got tired making sure I was in the right directory and typing python manage.py. I found django-shortcuts but didn't like the fact that it only mapped a handful of the commands and I wanted access to everything manage.py could do.

I then briefly considered creating a bash function that would determine manage.py's location on every invocation but figured that was silly and decided I might as leverage virtualenv's virtual environment and as Ron Popiel says, "You set it and forget it!"

It only took a minute of poking to find the postactivate hook file within my ~/.virtualenvs directory. Below is the fruit of my labor.

#!/bin/bash
# ~/.virtualenvs/postactive
# This hook is run after every virtualenv is activated.
VIRTUAL_ENV_NAME=$(basename $VIRTUAL_ENV)
SRC_DIR="$HOME/src"
if [ -d "$SRC_DIR/$VIRTUAL_ENV_NAME" ]
then
    MANAGE_PY=$(find "$SRC_DIR/$VIRTUAL_ENV_NAME" -name "manage.py" -type f)
    if [ -e "$MANAGE_PY" ]
    then
        alias django="python $MANAGE_PY"
    else
        unalias django
    fi
else
    unalias django
fi

The end result is that I have a new django alias that will work anywhere and act as though I typed python manage.py

jbisbee@beni:~$ workon django-tutorial
(django-tutorial)jbisbee@beni:~$ alias | grep django
alias django='python /Users/jbisbee/src/django-tutorial/mysite/manage.py'
(django-tutorial)jbisbee@beni:~$

Disclaimer: I'm making a big assumption here that the virtualenv name you're using is identical to the name of the project you're working on. It's a pretty big assumption so I apologize if it does't work right out of the box for you.

Team Fortress 2 High Five Taunt

So the latest thing in the office is to do the TF2 high five taunt while saying, "Aye!" They key is to do the cheesy demoman smile while you do it. You have to sell it by remaining motionless for a couple of seconds. If you get a high five, do the random taunt of your choice.

+Marlon Bailey , looking at you buddy!

Proper Way to Start Programming Python

So after quite a few missteps I've fallen into the correct pattern for programming python. There are two main tools for installing python modules: pip and setuptools' easy_install script. On a virgin system you'll want to the following packages from the OS distribution. I've repeated this process on my macbook air, windows machine under cgywin, and my Ubuntu virtual machine.

1. Install python and python-setuptools

You'll need to install python and python-setuptools before you begin. Macs come with these packages pre-installed so you can skip this step if you'd like.

python
python-setuptools

2. easy_install virtualenvwrapper

Then you'll want to install virtualenvwrapper via setuptool's easy_install. It should be the only package you should ever install with easy_install.

jbisbee@benny:~$ easy_install virtualenvwrapper
Searching for virtualenvwrapper
Reading http://pypi.python.org/simple/virtualenvwrapper/
Best match: virtualenvwrapper 4.0
...
Finished processing dependencies for virtualenvwrapper

3. Find locaiton of virtualenvwrapper.sh

Once you have virtualenvwrapper installed you'll want to see where it was installed in your path. Please note that it is a shell script and does end with the .sh file extension.

jbisbee@benny:~$ which virtualenvwrapper.sh
/usr/bin/virtualenvwrapper.sh

4. Add virtualenvwrapper.sh to Your Shell Startup

# your ~/.bashrc or ~/.profile file
export WORKON_HOME=~/.virtualenvs
# virtualenvwrapper.sh may be installed in 
#     /usr/local/bin on your system
source /usr/bin/virtualenvwrapper.sh  

5. Reload Your Shell

jbisbee@benny:~$ source ~/.bashrc
# source .profile

6. Create a New virtualenv Environment

Now you're in business to start creating virtual workspaces and installing libraries within them with pip. Using django as an example :)

jbisbee@benny:~$ mkvirtualenv tutorial
New python executable in tutorial/bin/python2.7
Also creating executable in tutorial/bin/python
Installing setuptools............done.
Installing pip...............done.
(tutorial)jbisbee@benny:~$

7. Install Python Modules with pip

Now install python modules with pip within your virtualenv sandbox:

(tutorial)jbisbee@benny:~$ pip install django
Downloading/unpacking django
  Downloading Django-1.5.1.tar.gz (8.0MB): 8.0MB downloaded
  Running setup.py egg_info for package django
...
Successfully installed django
Cleaning up...
(tutorial)jbisbee@benny:~$

Have fun and let me know if you run into any issues as I wrote this post pretty quickly and may have left something out.

Edit: This blog post has been translated to Spanish by +Apokalyptica Painkiller. Thank you!

Add History and Tab Completion to the Default Python Shell

So I've in the python shell quite a bit lately and figured it would be nice to be able to have readline history between sessions as well as tab completion. Yes I know things like iPython exist as well as others but wanted to keep it simple. It turns out its not that hard to enable and here is how you do it.

First you'll need to add a line to your .bashrc or .profile setting your PYTHONSTARTUP environment variable

# ~/.bashrc
export PYTHONSTARTUP=$HOME/.pythonstartup.py

Then copy this scripts contents to file your new environment variable points to

# ~/.pythonstartup.py
try:
    import readline
    import rlcompleter
    import atexit
    import os
except ImportError:
    print "Python shell enhancement modules not available."
else:
    histfile = os.path.join(os.environ["HOME"], ".pythonhistory")
    import rlcompleter
    readline.parse_and_bind("tab: complete")
    if os.path.isfile(histfile):
        readline.read_history_file(histfile)
    atexit.register(readline.write_history_file, histfile)
    del os, histfile, readline, rlcompleter, atexit
    print "Python shell history and tab completion are enabled."

That's it. Now anytime you pop into the python shell you'll have history from previous sessions as well as tab completion. Enjoy!

Edit: If you're not running the latest version of OSX you may need to change the readline.pasre_and_bind line to the following

#readline.parse_and_bind("tab: complete")
readline.parse_and_bind("bind ^I rl_complete")

Cache Pip Downloads

Working with virtualenvwrapper makes python development so clean. And I found you can speed up installing common libraries in your different projects by telling pip to cache the downloads. Easy peasy lemon sqeezey!

export PIP_DOWNLOAD_CACHE=$HOME/.pip-download-cache

Now anytime I pip install anything the package gets cached locally so I won't have to fetch it again.

Stupid Mysql

I really really don't like Mysql but it seems like a necessary evil sometimes.  There are so many bugs and edge cases but so many people us it that its hard to get away from it.  I can never remember the syntax for creating users and giving them access to db's so I'm going to just leave this here for myself...

CREATE DATABASE database_name;
CREATE USER 'who'@'localhost' IDENTIFIED BY 'password';
GRANT ALL ON database_name.* TO 'who'@'localhost';

Now this is really basic stuff and I realize it, but the point is that I hate looking it up. The sad thing is that after writing this blog post I'll probably never forget it.

Here We Go...

I recently took the job as AlertSite's Architect and I'll be switching gears and beginning to code python. I plan on keeping this blog pretty tech related, I plan on slipping in few life hacks as well has some light Team Fortress 2 banter as its one of my guilty pleasures. I've tried blogging in the past with minimal success, but for some reason I think this time it's going to stick... We'll see!