This post explains how to install nginx from sources, compiling it (on Linux). You might want to use apt, zif, yum or ports, but I prefer building from sources. So, to build from sources, make sure you have all development dependencies (C headers, including the PCRE library headers, nginx rewrite module uses it). If you want to build nginx with SSL support, keep in mind that you will need the libssl headers too.

Build nginx from source is a straightforward process: all you need to do is download it from the official site and build with some simple options. In our setup, we’re going to install nginx under /opt/nginx, and use it with the nginx system user. So, let’s download and extract the latest stable version (1.0.9) from nginx website:

Once you have extracted it, just configure, compile and install:

As you can see, we provided the /opt/nginx to configure, make sure the /opt directory exists. Also, make sure that there is a user and a group called nginx, if they don’t exist, add them:

After that, you can start nginx using the command line above:

Linode provides an init script that uses start-stop-daemon, you might want to use it.

nginx configuration

nginx comes with a default nginx.conf file, let’s change it to reflect the following configuration requirements:

• nginx should start workers with the nginx user
• nginx should have two worker processes
• the PID should be stored in the /opt/nginx/log/nginx.pid file
• nginx must have an access log in /opt/nginx/logs/access.log
• the configuration for the Django project we’re going to develop should be versioned with the entire code, so it must be included in the nginx.conf file (assume that the library project is in the directory /opt/projects).

So here is the nginx.conf for the requirements above:

Now we just need to write the configuration for our Django project. I’m using an old sample project written while I was working at Giran: the name is lojas giranianas, a nonsense portuguese joke with a famous brazilian store. It’s an unfinished showcase of products, it’s like an e-commerce project, but it can’t sell, so it’s just a product catalog. The code is available at Github. The nginx.conf file for the repository is here:

The server listens on port 80, responds for the localhost hostname (read more about the Host header). The location /static directive says that nginx will serve the static files of the project. It also includes an expires directive for caching control. The location / directive makes a proxy_pass, forwarding all requisitions to an upstream server listening on port 8000, this server is the subject of the next post of the series: the Green Unicorn (gunicorn) server.

Not only the HTTP request itself is forwarded to the gunicorn server, but also some headers, that helps to properly deal with the request:

• X-Real-IP: forwards the remote address to the upstream server, so it can know the real IP of the user. When nginx forwards the request to gunicorn, without this header, all gunicorn will know is that there is a request coming from localhost (or wherever the nginx server is), the remote address is always the IP address of the machine where nginx is running (who actually make the request to gunicorn)
• Host: the Host header is forwarded so gunicorn can treat different requests for different hosts. Without this header, it will be impossible to Gunicorn to have these constraints
• X-Forwarded-For: also known as XFF, this header provide more precise information about the real IP who makes the request. Imagine there are 10 proxies between the user machine and your webserver, the XFF header will all these proxies comma separated. In order to not turn a proxy into an anonymizer, it’s a good practice to always forward this header.

So that is it, in the next post we’re going to install and run gunicorn. In other posts, we’ll see how to make automated deploys using Fabric, and some tricks on caching (using the proxy_cache directive and integrating Django, nginx and memcached).

See you in the next posts :)

Here is the lesson: don’t rely on caching. Cache doesn’t exist to make applications work, it exists to make them faster and/or more scalable. Use a cache system as permanent storage is such a bad idea, except when you’re not using it for caching (what is not the case here). Cache is a shortcut for a path that might be painful to follow, but you still should not rely on cache. You must be able to traverse the path without the cache.

Some international dudes are also attending to the conference: Wesley Chun is going to talk about Python 3 and Google App Engine; Alan Runyan will talk about free and open source software, and Steve Holden will be talking about the issues involved in trying to build a global Python user group.

There is also Maciej Fijalkowski, PyPy core developer, talking about little things PyPy makes possible.

As I pointed before, I’m going to talk about scalability, based in some experiences aquired scaling Django applications at Globo.com, like G1, the greatest news portal in the Latin America.

What is a splinter sprint?

Basically, a splinter sprint is an excuse for people to focus their undivided attention, for a set time frame, on improving splinter. It’s a focused, scheduled effort to fix bugs, add new features and improve documentation.

Anybody, anywhere around the world, can participate and contribute. If you’ve never contributed to splinter before, this is the perfect chance for you to chip in.

How to contribute

1. Choose an issue
2. Create a fork
3. Send a pull request

Remember: all new features should be well tested and documented. An issue can’t be closed if there isn’t docs for the solution code.

Preparing for the sprint

Get a IRC client, so that you can join us in the channel #cobrateam on Freenode.

See all you there :)

In this post, I will show some basic usage of Splinter for simple web application tests. Splinter is a tool useful on tests of any web application. You can even test a Java web application using Splinter. This post example is a “test” of a Facebook feature, just because I want to focus on how to use Splinter, not on how to write a web application. The feature to be tested is the creation of an event (the Splinter sprint), following all the flow: first the user will login on Facebook, then click on “Events” menu item, then click on “Create an Event button”, enter all event informations and click on “Create event” button. So, let’s do it…

First step is create a Browser instance, which will provide method for interactions with browser (where the browser is: Firefox, Chrome, etc.). The code we need for it is very simple:

Browser is a class and its constructor receives the driver to be used with that instance. Nowadays, there are three drivers for Splinter: firefox.webdriver, chrome.webdriver and zope.testbrowser. We are using Firefox, and you can easily use Chrome by simply changing the driver from webdriver.firefox to webdriver.chrome. It’s also very simple to add another driver to Splinter, and I plan to cover how to do that on another blog post here.

A new browser session is started when we got the browser object, and this is the object used for Firefox interactions. Let’s start a new event on Facebook, the Splinter Sprint. First of all, we need to visit the Facebook homepage. There is a visit method on Browser class, so we can use it:

visit method is blocking: it waits for page to load, then we can navigate, click on links, fill forms, etc. Now we have Facebook homepage opened on browser, and you probably know that we need to login on Facebook page, but what if we are already logged in? So, let’s create a method that login on Facebook with provided authentication data only the user is not logged in (imagine we are on a TestCase class):

What was made here? First of all, the method checks if there is an element present on the page, using a CSS selector. It checks for the div that contains the username and password fields. If that div is present, we tell the browser object to fill those fields, then find the submit button and click on it. The last line is an assert to guarantee that the login was successful and the current page is the Facebook homepage (by checking the presence of “Account” li).

We could also find elements by its texts, labels or whatever appears on screen, but remember: Facebook is an internationalized web application, and we can’t test it using only a specific language.

Okay, now we know how to visit a webpage, check if an element is present, fill a form and click on a button. We’re also logged in on Facebook and can finally go ahead create the Splinter sprint event. So, here is the event creation flow, for a user:

1. On Facebook homepage, click on “Events” link, of left menu
2. The “Events” page will load, so click on “Create an Event” button
3. The user see a page with a form to create an event
4. Fill the date and chose the time
5. Define what is the name of the event, where it will happen and write a short description for it
6. Invite some guests
7. Upload a picture for the event
8. Click on “Create Event” button

We are going to do all these steps, except the 6th, because the Splinter Sprint will just be a public event and we don’t need to invite anybody :) There are some boring AJAX requests on Facebook that we need to deal, so there is not only Splinter code for those steps above. First step is click on “Events” link. All we need to do is find the link and click on it:

The find_by_css method takes a CSS selector and returns a ElementList. So, we get the first element of the list (even when the selector returns only an element, the return type is still a list) and click on it. Like visit method, clicking in a link is a blocking action: the driver will only listen for new actions when the request is finished (the page is loaded).

We’re finally on “new event” page, and there is a form on screen waiting for data of the Splinter Sprint. Let’s fill the form. Here is the code for it:

That is it: the event is going to happen on May 21th 2011, at 8:00 in the morning (480 minutes). As we know, the event name is Splinter sprint, and we are going to join some guys down here in Brazil. We filled out the form using fill and select methods.

The fill method is used to fill a field (a textarea, an input, or whatever you can fill). It receives two strings: the first is the name of the field to fill and the second is the value that will fill the field. The select method is used to select an option in a select element (a “combo box”). It also receives two string parameters: the first is the name of the select element, and the second is the value of the option being selected.

Imagine you have the following select element:

To select “Male”, you would call the select method this way:

The last action before click on “Create Event” button is upload a picture for the event. On new event page, Facebook loads the file field for picture uploading inside a iframe, so we need to switch to this frame and interact with the form present inside the frame. To show the frame, we need to click on “Add Event Photo” button and then switch to it, we already know how click on a link:

When we click this link, Facebook makes a asynchronous request, which means the driver doesn’t stay blocked waiting the request ends, so if we try to interact with the frame BEFORE it appears, we will get a ElementDoesNotExist exception. Splinter provides a is_element_present method that receives an argument called wait_time, which is the time Splinter will be waiting for the element to appear on the screen. If the element doesn’t appear on screen, we can’t go on, so we can assume the test failed (remember we are testing a Facebook feature):

The is_element_present_by_css method takes a CSS selector and tries to find an element using it. It also receives a wait_time parameter that indicates a timeout for the search of the element. So, if the iframe element with ID=”upload_pic_frame” is not present or doesn’t appears on screen in 10 seconds, the method returns False, otherwise it returns True.

Important: The fail method is a pseudocode sample and doesn’t exist (if you’re using unittest library, you can invoke self.fail in a TestCase, exactly what I did in complete snippet for this example, available at Github).

Now we see the iframe element on screen and we can finally upload the picture. Imagine we have a variable that contains the path of the picture (and not a file object, StringIO, or something like this), and this variable name is picture_path, this is the code we need:

Splinter provides the get_iframe method that change the context and returns another objet to interact with the content of the frame. So we call the attach_file method, who also receives two strings: the first is the name of the input element and the second is the absolute path of the file being sent. Facebook also uploads the picture asynchronously, but there’s no way to wait some element to appear on screen, so I just put Python to sleep 10 seconds on last line.

After follow all these steps, we can finally click on “Create Event” button and asserts that Facebook created it:

After create an event, Facebook redirects the browser to the event page, so we can check if it really happened by asserting the header of the page. That’s what the code above does: in new event page, it click on submit button, and after the redirect, get the text of a span element and asserts that this text equals to “Splinter sprint”.

That is it! This post was an overview on Splinter API. Check out the complete snippet, written as a test case and also check out Splinter repository at Github.

Google App Engine is a powerful tool for web developers. I am sure that it is useful and every developer should taste it =) Python was the first programming language supported by App Engine, and is a programming language with a lot of web frameworks. So, you can use some of these frameworks on Google App Engine. In a series of three blog posts, I will show how to use three Python web frameworks on App Engine: Django, Flask and web2py (not necessarily in this sequence).

The first framework is Django, the most famous of all Python frameworks and maybe is used the most.

Django models is the strongest Django feature. It is a high level database abstraction layer with a powerful object-relational mapper, it supports a lot of relational database management systems, but App Engine doesn’t use a relational database. The database behind App Engine is called BigTable, which is a distributed storage system for managing structured data, designed to scale to a very large size (Reference: Bigtable: A Distributed Storage System for Structured Data). It is not based on schemas, tables, keys or columns, it is like a big map indexed by a row key, column key and a timestamp. We can not use native version of Django models with Bigtable, because the Django models framework was not designed for non relational databases.

So, what can we do? There is a Django fork, the django-nonrel project, which aims to bring the power of the Django model layer to non-relational databases. I will use the djangoappengine sub-project to build the sample application of this post, that will be deployed on Google App Engine :)

The sample application is the default: a blog. A very simple blog, with only a form protected by login (using Django built-in authentication system instead of Google Accounts API) and a public page listing all blog posts. It is very easy and simple to do, so let’s do it.

First, we have to setup our environment. According the djangoappengine project documentation, we need to download 4 zip files and put it together. First, I downloaded the django-testapp file, extract its contents and renamed the project directory from django-testapp to blog_gae. After this step, I downloaded the other files and put it inside the blog_gae directory. Here is the final project structure:

“django” directory is from the django-nonrel zip file, “djangoappengine” directory is from djangoappengine zip file and “djangotoolbox” directory is from djangotoolbox zip file. Look that is provided an app.yaml file, ready to be customized. I just changed the application id inside this file. The final code of the file is the following:

I will use one version for each part of the series, so it is the first version because it is the first part =D In settings.py, we just uncomment the app django.contrib.auth line inside the INSTALLED_APPS tuple, because we want to use the built-in auth application instead of the Google Accounts API provided by App Engine.

All settings are ok now, it is time to create the core application. In the Django project, we will use the core application to manage models and serve some views. We just start it using the following command:

It is a famous Django command, that creates the application structure, which is a Python package containing 3 Python modules: models, tests and views. Now we have to create the Post model. Here is the code of models.py file:

Now we just need to “install” the core application putting it on INSTALLED_APPS tuple in settings.py file and Django will be ready to play with BigTable. :) We will use the django.contrib.auth app, so let’s run a manage command to create a superuser:

After create the superuser, we need to setup login and logout URLs, and make two templates. So, in urls.py file, put two mappings to login and logout views. The file will look like this:

Here is the registration/login.html template:

And registration/logged_out.html template:

See the two added lines in highlight. In settings.py file, add three lines:

And we are ready to code =) Let’s create the login protected view, where we will write and save a new post. To do that, first we need to create a Django Form, to deal with the data. There are two fields in this form: title and content, when the form is submitted, the user property is filled with the current logged user and the date property is filled with the current time. So, here is the code of the ModelForm:

Here is the views.py file, with the two views (one “mocked up”, with a simple redirect):

There is only two steps to do to finally save posts on BigTable: map a URL for the views above and create the new_post.html template. Here is the mapping code:

And here is the template code:

Now, we can run on terminal ./manage.py runserver and access the URL http://localhost:8000/posts/new on the browser, see the form, fill it and save the post :D The last one step is list all posts in http://localhost:8000/posts/. The list_posts view is already mapped to the URL /posts/, so we just need to create the code of the view and a template to show the list of posts. Here is the view code:

And the list_posts.html template code:

Finished? Not yet :) The application now is ready to deploy. How do we deploy it? Just one command:

Done! Now, to use everything that we have just created on App Engine remote server, just create a super user in that server and enjoy:

You can check this application flying on Google App Engine: http://1.latest.gaeseries.appspot.com (use demo for username and password in login page).

You can check this application code out in Github: http://github.com/fsouza/gaeseries/tree/django.

Enjoy :)

Selenium drives a browser, so you can have true test results on navigation, but Selenium is not needed to test a page title or a response status code, for example. Selenium tests are powerful, but slow, so you can preserve them to use when you really need, and use more simple and fast tools to more simple and fast tests.

Django has a powerful tool for web tests: the test client. Using this module, you can make requests and test the response (its content, status code, URL and everything inside a response).

Imagine that you have a Django application that manage all books of the world, and you want to test if the title of the /books page is “All the books of the world”, you can use Selenium, but IMHO, you should not use Selenium.

Suppose that there is the following Lettuce feature file:

If we use Selenium to define the scenario “Listing all books in the world”, we should have a terrain.py file with the following content:

And define the steps definition with the following code:

There is nothing difficult here, the steps definition code is very simple and the test itself too, but how many time Lettuce spends running this test? Let’s check:

About 7 seconds to run a really really really simple and small test. We can earn time testing Django applications using the Django test client. Let’s rewrite the test above using Django test client, we just need to change the terrain.py:

And the step definitions:

The only notice here is that is we have a little much code to test the title of the page. How many time Lettuce spends testing it?

About 3 seconds :) We didn’t need to open Firefox for visit the page and get it’s title, we just use the Django test client that knows how to deal with Django requests, and lxml to deal with the response content. Think about how many seconds you can earn just stopping to open a browser for the simplest tests of the world =P

There is a lot of other stuffs that you can do using the Django Test Client, and I plan to write more about it, but now, just check the official docs and have fun.

P.S.: Another great tool for Django testing is Django sane testing.

Happy testing :)

Important: If you don’t know Django, this article is not recommended for you.

Nose is plugin based, so to use Django under Nose, we need to install a plugin called NoseDjango. To work isolated from the database, we need a mock framework. Let’s use Mock, wich is a simple and powerful mock framework for Python. We can install Django, Freshen, Mock, Nose and NoseDjango with a simple command:

It works in any operating system with Python and PIP.

After installing everything we need, we just start a Django Project, called “library” (just supposing that we are developing a software for the library on the next corner):

Looking quickly on Django settings, in the settings.py file, only database and templates settings:

After start and setup the project, let’s start our Django application. Let’s start an application called “loans”. It app will manage the loans of our library:

After start the application, let’s define our Book model and install the application. You can see below the code of models module inside the loans app:

Now, we need to describe our acceptance tests, or as Freshen and Cucumber call: our system features. Let’s describe the loans application’s features inside the loans application, creating a directory called “features” inside the application.

The first feature of the application will be a feature that describes the access to a list containg all the books on the path “/loans/books”:

Let’s understand the feature: A Freshen feature is a description of a system feature! :) On the list of books, we have a scenario where we list all books. That scenario is described using some steps: given, when and then. You can learn more about Given, When and Then looking for some resources about BDD.

After describe a feature and it’s scenarios, we need to define the steps of each scenario. So, we put these definitions in a Python module called steps.py in our features directory. The scenario List all books has three steps: “Given there are 20 books in the system”; “When I navigate to the books list page” and “Then I should see the title of the 20 books”. First, let’s define the first step: Given there are 20 books in the system. Each step definition is a Python function decorated with the keyword that defines that step.

So, to define “Given” step, Freshen provides the @Given decorator. Let’s define a function that creates and saves 20 books in the database. But we will not really save it in the database, because we are working isolated from the database. We will just store the book list in a context object.

We use a regular expression to determine a pattern to receive a parameter in the function. The function create_books receives a parameter wich indicates the number of books that will be created. All parameters obtained from step specification is an instance of str and need to be converted properly if you want to use it as a number.

The second line of the create_books function defines an empty list called books inside an object called scc. Freshen provides three objects for context storage: glc, the global context, wich is never cleaned; ftc, the feature context, cleaned at the start of each feature and scc, the scenario context, wich is cleaned at the start of each scenario. Once we put the books object list in the scenario context, we can use this list in all steps of that scenario.

Now, let’s define the second step: When I navigate to the books list page. Now we will use the @When decorator to decorate a function called navigate_to_books_list. Here is the code:

This function just makes a GET request on the URL /loans/books and saves the response in a object called response, stored in scenario context. The Book model was mocked here, using the patch decorator from Mock. We said to the mock: “when Book.objects.all() is called, return my list of books, stored in scenario context”. The last line of the function asserts if the method is really called in our GET request.

Now we can run the nosetests and see what happens. To execute Nose with Django and Freshen, we need to run the following command:

The output at this point is:

The test fails with the following exception: TemplateDoesNotExist: 404.html. So, let’s create that template and see what happens.

After add a 404.html template to the templates directory in the root of the project, just run nosetests again, using the verbose mode:

Step fails again, why? On function “navigate_to_books_list” the last line defines that the method Book.objects.all() should be called with no parameters in the GET request (lines above), but this method was not called anywhere, once that GET request returned a 404 response, because there is not an URL mapping to a view that responds by the action ‘/loans/books’, and there is not a view wich makes the expected method call. So, let’s make the view and map it to the “/loans/books” path. Here is the view code:

Look that the view has only two lines of code and returns a HttpResponse object with blank content, it’s the enough code to make the Step definition works. To map the view to a URL path, create a urls.py module inside the application directory, with the following code:

And include it on the main urls.py of project, that will contain the code bellow:

Run nosetests again and see what happens:

Freshen said: I found an undefined step (“I should see the title of the 20 books”). Let’s define that step, using the @Then decorator to decorate our function called check_presence_of_the_title_of_books:

The function checks if the books titles is included in a li HTML element. Let’s call nosetests again and see the test failing:

The content returned by the GET request (an empty string) is not the content expected by the defined step, so we see an AssertionError. Now, we need to make this test pass. To do it, we need only to refactor the view:

And we run nosetests with Django and Freshen, in verbose mode again, and see that everything works fine =D

That is all, folks! :) You can download this Django Project at Github.