There are a number of ways to do this, but for me setting up an ssh config file was not practical or desirable. I wanted to be able to make a fabric script that becomes a user (that does not have a shell) and does a git pull.
This approach works very well, on the command line, or in a fabric script:
ssh-agent bash -c 'ssh-add /path/to/private_key; git pull'