Software Carpentry: GCB Academy - Introduction to Scientific Computing for Genomics

General Information

Computing has become an integral and indispensable part of genomic biology. This course teaches basic skills in scientific computing, with a focus on applications for genomic science, aimed at making you more productive, your computational work more reliable, and your research easier to reproduce and extend, including by your future self. The course includes introductions to

using Unix shell commands to efficiently find, organize, and stage data for analysis;
basic data types, control flows, functions, and 3rd party packages for the Python programming language commonly encountered in scientific computing;
using version control to manage with confidence the numerous directions research code takes from inception to publication; and
techniques for optimizing how your computational analyses run on a high-performance computing cluster.

The format of the course is inspired by the acclaimed Software Carpentry-style bootcamps. Hence, this is a fully hands-on workshop, and students are expected to bring a laptop.

Who: The course is aimed at graduate students, postdocs, early career faculty, and other researchers.

Where: Perkins Library, Room 218. Get directions with OpenStreetMap or Google Maps.

Requirements: Participants must bring a laptop with a few specific software packages installed (listed below). They are also required to abide by Software Carpentry's Code of Conduct.

Contact: Please mail gcb-help@duke.edu for more information.

Schedule

Day 1, 1-4.30pm

13:00	Automating tasks with the Unix shell
14:45	Break
16:30	Wrap-up

Day 2, 1-4.30pm

13:00	Building programs with Python
14:45	Break
16:30	Wrap-up

Day 3, 1-4.30pm

13:00	Version control with Git
14:45	Break
16:30	Wrap-up

Day 4, 1-4.30pm

13:00	Running programs on an HPC cluster
14:45	Break
16:30	Wrap-up

Etherpad: http://pad.software-carpentry.org/SciComp-Nov-2017.
We will use this Etherpad for chatting, taking notes, and sharing URLs and bits of code.

Syllabus

Automating tasks with the Unix shell

Introducing the shell
Navigating files and directories
Working with files and directories
Pipes and redirection
Writing scripts
Project organization
Reference...

Building programs with Python

Data types and variables
Loops and conditionals
Creating and using functions
Working with files
Scripts and command-line
Libraries and packages

Version Control with Git

Creating a repository
Recording changes to files: add, commit, ...
Viewing changes: status, diff, ...
Ignoring files
Working on the web: clone, pull, push, ...
Resolving conflicts
Open licenses
Where to host work, and why
Reference...

Running programs on an HPC cluster

Why you need a cluster
How a cluster works
Running a job: srun
Running a batch job: sbatch
Monitoring your jobs: squeue, sacct, scancel
Moving files in/out: scp
Finding/getting software
Helpful Resources

Materials

Bash shell

Download dc_sample_data.zip

Programming w/ Python

Download python-fasta.zip

Git version control

Running programs on an HPC cluster

Setup

To participate in this workshop, you will need access to the software described below. In addition, you will need an up-to-date web browser. Installation instructions will become available here closer to the date of the course.

We maintain a list of common issues that occur during installation as a reference for instructors and learners on the Configuration Problems and Solutions wiki page.

The Bash Shell

Bash is a commonly-used shell that gives you the power to do simple tasks more quickly.

Windows

Download the Git for Windows installer. Run the installer. Important: on the 6th page of the installation wizard (the page titled `Configuring the terminal emulator...`) select `Use Windows' default console window`. If you forgot to do this programs that you need for the workshop will not work properly. If this happens rerun the installer and select the appropriate option. On the dialog titled "Adjusting your PATH environment", we recommend you choose the option labeled "Use Git from the Windows Command Prompt". This will allow RStudio to work with Git out of the box. We teach Python, not R, in this course, but R is frequently used in computational genomics as well, including for RNAseq analysis.

This will provide you with both Git and Bash in the Git Bash program.

Mac OS X

The default shell in all versions of Mac OS X is bash, so no need to install anything. You access bash from the Terminal (found in /Applications/Utilities). You may want to keep Terminal in your dock for this workshop.

Linux

The default shell is usually Bash, but if your machine is set up differently you can run it by opening a terminal and typing bash. There is no need to install anything.

Git

Git is a version control system that lets you track who made changes to what when and has options for easily updating a shared or public version of your code on github.com. You will need a supported web browser (current versions of Chrome, Firefox or Safari, or Internet Explorer version 9 or above).

Windows

Git should be installed on your computer as part of your Bash install (described above).

Mac OS X

For OS X 10.9 and higher, install Git for Mac by downloading and running the most recent "mavericks" installer from this list. After installing Git, there will not be anything in your /Applications folder, as Git is a command line program. For older versions of OS X (10.5-10.8) use the most recent available installer labelled "snow-leopard" available here.

Linux

If Git is not already available on your machine you can try to install it via your distro's package manager. For Debian/Ubuntu run sudo apt-get install git and for Fedora run sudo yum install git.

Python

Python is a popular language for scientific computing, and great for general-purpose programming as well. Installing all of its scientific packages individually can be a bit difficult, so we recommend Anaconda, an all-in-one installer.

Regardless of how you choose to install it, please make sure you install Python version 2.x (e.g., 2.7 is fine).

We will teach Python using the IPython notebook, a programming environment that runs in a web browser. For this to work you will need a reasonably up-to-date browser. The current versions of the Chrome, Safari and Firefox browsers are all supported (some older browsers, including Internet Explorer version 9 and below, are not).

Windows

Open https://www.anaconda.com/download with your web browser.
Click on the Windows logo.
Click on the Python 2.7 version download.
Install Python 2 using all of the defaults for installation except make sure to check Make Anaconda the default Python.

Mac OS X

Open https://www.anaconda.com/download with your web browser.
Click on the macOS logo.
Click on the Python 2.7 version download.
Install Python 2 using all of the defaults for installation.

Linux

Open https://www.anaconda.com/download with your web browser.
Click on Jump to UNIX.
Click on the Python 2.7 version download.
Install Python 2 using all of the defaults for installation (Installation requires using the shell. If you aren't comfortable doing the installation yourself stop here and request help at the workshop.)
Open a terminal window.
Type
```
bash Anaconda-
```
and then press tab. The name of the file you just downloaded should appear.
Press enter. You will follow the text-only prompts. When there is a colon at the bottom of the screen press the down arrow to move down through the text. Type yes and press enter to approve the license. Press enter to approve the default location for the files. Type yes and press enter to prepend Anaconda to your PATH (this makes the Anaconda distribution the default Python).

Once you are done installing the software listed above, please go to this page, which has instructions on how to test that everything was installed correctly.