This post explains how to install nginx from sources, compiling it (on Linux). You might want to use apt, zif, yum or ports, but I prefer building from sources. So, to build from sources, make sure you have all development dependencies (C headers, including the PCRE library headers, nginx rewrite module uses it). If you want to build nginx with SSL support, keep in mind that you will need the libssl headers too.

Build nginx from source is a straightforward process: all you need to do is download it from the official site and build with some simple options. In our setup, we’re going to install nginx under /opt/nginx, and use it with the nginx system user. So, let’s download and extract the latest stable version (1.0.9) from nginx website:

Once you have extracted it, just configure, compile and install:

As you can see, we provided the /opt/nginx to configure, make sure the /opt directory exists. Also, make sure that there is a user and a group called nginx, if they don’t exist, add them:

After that, you can start nginx using the command line above:

Linode provides an init script that uses start-stop-daemon, you might want to use it.

nginx configuration

nginx comes with a default nginx.conf file, let’s change it to reflect the following configuration requirements:

• nginx should start workers with the nginx user
• nginx should have two worker processes
• the PID should be stored in the /opt/nginx/log/nginx.pid file
• nginx must have an access log in /opt/nginx/logs/access.log
• the configuration for the Django project we’re going to develop should be versioned with the entire code, so it must be included in the nginx.conf file (assume that the library project is in the directory /opt/projects).

So here is the nginx.conf for the requirements above:

Now we just need to write the configuration for our Django project. I’m using an old sample project written while I was working at Giran: the name is lojas giranianas, a nonsense portuguese joke with a famous brazilian store. It’s an unfinished showcase of products, it’s like an e-commerce project, but it can’t sell, so it’s just a product catalog. The code is available at Github. The nginx.conf file for the repository is here:

The server listens on port 80, responds for the localhost hostname (read more about the Host header). The location /static directive says that nginx will serve the static files of the project. It also includes an expires directive for caching control. The location / directive makes a proxy_pass, forwarding all requisitions to an upstream server listening on port 8000, this server is the subject of the next post of the series: the Green Unicorn (gunicorn) server.

Not only the HTTP request itself is forwarded to the gunicorn server, but also some headers, that helps to properly deal with the request:

• X-Real-IP: forwards the remote address to the upstream server, so it can know the real IP of the user. When nginx forwards the request to gunicorn, without this header, all gunicorn will know is that there is a request coming from localhost (or wherever the nginx server is), the remote address is always the IP address of the machine where nginx is running (who actually make the request to gunicorn)
• Host: the Host header is forwarded so gunicorn can treat different requests for different hosts. Without this header, it will be impossible to Gunicorn to have these constraints
• X-Forwarded-For: also known as XFF, this header provide more precise information about the real IP who makes the request. Imagine there are 10 proxies between the user machine and your webserver, the XFF header will all these proxies comma separated. In order to not turn a proxy into an anonymizer, it’s a good practice to always forward this header.

So that is it, in the next post we’re going to install and run gunicorn. In other posts, we’ll see how to make automated deploys using Fabric, and some tricks on caching (using the proxy_cache directive and integrating Django, nginx and memcached).

See you in the next posts :)

The Dining Philosophers Problem was proposed by Dijkstra in 1965, when dinosaurs ruled the earth. It appears in a number of variations, but the standard features are a table with five plates, five forks (or chopsticks) and a big bowl of spaghetti.

There are some constraints:

• Only one philosopher can hold a fork at a time
• It must be impossible for a deadlock to occur
• It must be impossible for a philosopher to starve waiting for a fork
• It must be possible for more than one philosopher to eat at the same time

No more talk, here is my solution for the problem:

Any feedback is very welcome :)

Here is the lesson: don’t rely on caching. Cache doesn’t exist to make applications work, it exists to make them faster and/or more scalable. Use a cache system as permanent storage is such a bad idea, except when you’re not using it for caching (what is not the case here). Cache is a shortcut for a path that might be painful to follow, but you still should not rely on cache. You must be able to traverse the path without the cache.

Some international dudes are also attending to the conference: Wesley Chun is going to talk about Python 3 and Google App Engine; Alan Runyan will talk about free and open source software, and Steve Holden will be talking about the issues involved in trying to build a global Python user group.

There is also Maciej Fijalkowski, PyPy core developer, talking about little things PyPy makes possible.

As I pointed before, I’m going to talk about scalability, based in some experiences aquired scaling Django applications at Globo.com, like G1, the greatest news portal in the Latin America.

It’s based in a famous slide presentation. It’s a simple script that generates HTML from a source file, which can be formatted using reStructuredText, Textile or Markdown.

Let’s make a very simple presentation as a proof of concept: we’re going to create a “Python flow control” presentation, showing some basic structures of the language: if, for and while. We need a cover, a slide for each structure (with some topics and code examples) and the last slide for questions and answers. Here is the RST code for it:

As you can see it’s very simple. If you’re familiar with RST syntax, you can guess what landslide does: it converts the entire content to HTML and then split it by <hr /> tag. Each slide will contain two sections: a header and a body. The header contains only an <h1></h1> element and the body contains everything.

We can generate the HTML output by calling the landslide command in the terminal:

To use landslide command, you need to install it. I suggest you do this via pip:

landslide supports theming, so you can customize it by creating your own theme. Your theme should contain two CSS files: screen.css (for the HTML version of slides) and print.css (for the PDF version of the slides). You might also customize the HTML (base.html) and JS files (slides.js), but you have to customize the CSS files in your theme. You specify the theme using the --theme directive. You might want to check all options available in the command line utility using --help:

It’s quite easy to extend landslide changing its theme or adding new macros. Check the official repository at Github. This example, and a markdown version for the same example are available in a repository in my github profile.

You can also see the slides live!

What is a splinter sprint?

Basically, a splinter sprint is an excuse for people to focus their undivided attention, for a set time frame, on improving splinter. It’s a focused, scheduled effort to fix bugs, add new features and improve documentation.

Anybody, anywhere around the world, can participate and contribute. If you’ve never contributed to splinter before, this is the perfect chance for you to chip in.

How to contribute

1. Choose an issue
2. Create a fork
3. Send a pull request

Remember: all new features should be well tested and documented. An issue can’t be closed if there isn’t docs for the solution code.

Preparing for the sprint

Get a IRC client, so that you can join us in the channel #cobrateam on Freenode.

See all you there :)

First step is download the standalone version of Jasmine, then extract it and edit the runner. The runner is a simple HTML file, that loads Jasmine and all JavaScript files you want to test. But, wait… why not test using node.js or something like this? Do I really need the browser on this test? You don’t *need*, but I think it is important to test a plugin that works with the DOM using a real browser. Let’s delete some files and lines from SpecRunner.html file, so we adapt it for our plugin. This is how the structure is going to look like:

.
├── SpecRunner.html
├── lib
│   ├── jasmine-1.0.2
│   │   ├── MIT.LICENSE
│   │   ├── jasmine-html.js
│   │   ├── jasmine.css
│   │   └── jasmine.js
│   └── jquery-1.6.1.min.js
├── spec
│   └── ZebrafySpec.js
└── src
    └── jquery.zebrafy.js

You can create the files jquery.zebrafy.js and ZebrafySpec.js, but remember: it is BDD, we need to describe first the behavior we want, then write the code. So let’s start to write the specs in ZebrafySpec.js file using Jasmine. If you are familiar with RSpec syntax, it’s easy to understand how to write spec withs Jasmine, if you aren’t, here is the clue: Jasmine is a lib with some functions used for writing tests in an easier way. I’m going to explain each function “on demmand”, when we need something, we learn how to use it! ;)

First of all, we need to start a new test suite. Jasmine provides the describe function for that, this function receives a string and another function. The string describes the test suite and the function is a callback that delimites the scope of the test suite. Here is the Zebrafysuite:

Let’s start describing the behavior we want to get from the plugin. The most basic is: we want different CSS classes for odd an even lines in a table. Jasmine provides the it function for writing the tests. It also receives a string and a callback: the string is a description for the test and the callback is the function executed as test. Here is the test:

Okay, here we go: in the first line of the callback, we are using jQuery to select a table using the #zebra-table selector, which will look up for a table with the ID attribute equals to “zebra-table”, but we don’t have this table in the DOM. What about add a new table to the DOM in a hook executed before the test run and remove the table in another hook that runs after the the test? Jasmine provide two functions: beforeEach and afterEach. Both functions receive a callback function to be executed and, as the names suggest, the beforeEach callback is called before each test run, and the afterEach callback is called after the test run. And here are the hooks:

The beforeEach callback uses jQuery to create a table with 10 rows and 3 columns and add it to the DOM. In afterEach callback, we just remove that table using jQuery again. Okay, now the table exists, let’s go back to the test:

In second line, we call our plugin, that is not ready yet, so let’s forward to the next line, where we used the expect function. Jasmine provides this function, that receives an object and executes a matcher against it, there are a lot of built-in matchers on Jasmine, but toBeZebrafyied is not a built-in matcher. Here is where we know another Jasmine feature: the capability to write custom matchers, but how to do this? You can call the beforeEach again, and use the addMatcher method of Jasmine object:

The method addMatchers receives an object where each property is a matcher. Your matcher can receive arguments if you want. The object being matched can be accessed using this.actual, so here is what the method above does: it takes all odd <tr> elements of the table (this.actual) and check if them have the CSS class zebrafy-odd and don’t have the CSS class zebrafy-even, then do the same checking with even <tr> lines.

Now we have wrote the test, it’s time to write the plugin. Here some jQuery code:

I’m not going to explain how to implement a jQuery plugin neither what are those brackets on function, this post aims to show how to use Jasmine to test jQuery plugins.

By convention, jQuery plugins are “chainable”, so let’s make sure the zebrafy plugin is chainable using a spec:

As you can see, we used the built-in matcher toBeTruthy, which asserts that an object or expression is true. All we need to do ir return the jQuery object in the plugin and the test will pass:

So, the plugin is tested and ready to release! :) You can check the entire code and test with more spec in a Github repository.

In this post, I will show some basic usage of Splinter for simple web application tests. Splinter is a tool useful on tests of any web application. You can even test a Java web application using Splinter. This post example is a “test” of a Facebook feature, just because I want to focus on how to use Splinter, not on how to write a web application. The feature to be tested is the creation of an event (the Splinter sprint), following all the flow: first the user will login on Facebook, then click on “Events” menu item, then click on “Create an Event button”, enter all event informations and click on “Create event” button. So, let’s do it…

First step is create a Browser instance, which will provide method for interactions with browser (where the browser is: Firefox, Chrome, etc.). The code we need for it is very simple:

Browser is a class and its constructor receives the driver to be used with that instance. Nowadays, there are three drivers for Splinter: firefox.webdriver, chrome.webdriver and zope.testbrowser. We are using Firefox, and you can easily use Chrome by simply changing the driver from webdriver.firefox to webdriver.chrome. It’s also very simple to add another driver to Splinter, and I plan to cover how to do that on another blog post here.

A new browser session is started when we got the browser object, and this is the object used for Firefox interactions. Let’s start a new event on Facebook, the Splinter Sprint. First of all, we need to visit the Facebook homepage. There is a visit method on Browser class, so we can use it:

visit method is blocking: it waits for page to load, then we can navigate, click on links, fill forms, etc. Now we have Facebook homepage opened on browser, and you probably know that we need to login on Facebook page, but what if we are already logged in? So, let’s create a method that login on Facebook with provided authentication data only the user is not logged in (imagine we are on a TestCase class):

What was made here? First of all, the method checks if there is an element present on the page, using a CSS selector. It checks for the div that contains the username and password fields. If that div is present, we tell the browser object to fill those fields, then find the submit button and click on it. The last line is an assert to guarantee that the login was successful and the current page is the Facebook homepage (by checking the presence of “Account” li).

We could also find elements by its texts, labels or whatever appears on screen, but remember: Facebook is an internationalized web application, and we can’t test it using only a specific language.

Okay, now we know how to visit a webpage, check if an element is present, fill a form and click on a button. We’re also logged in on Facebook and can finally go ahead create the Splinter sprint event. So, here is the event creation flow, for a user:

1. On Facebook homepage, click on “Events” link, of left menu
2. The “Events” page will load, so click on “Create an Event” button
3. The user see a page with a form to create an event
4. Fill the date and chose the time
5. Define what is the name of the event, where it will happen and write a short description for it
6. Invite some guests
7. Upload a picture for the event
8. Click on “Create Event” button

We are going to do all these steps, except the 6th, because the Splinter Sprint will just be a public event and we don’t need to invite anybody :) There are some boring AJAX requests on Facebook that we need to deal, so there is not only Splinter code for those steps above. First step is click on “Events” link. All we need to do is find the link and click on it:

The find_by_css method takes a CSS selector and returns a ElementList. So, we get the first element of the list (even when the selector returns only an element, the return type is still a list) and click on it. Like visit method, clicking in a link is a blocking action: the driver will only listen for new actions when the request is finished (the page is loaded).

We’re finally on “new event” page, and there is a form on screen waiting for data of the Splinter Sprint. Let’s fill the form. Here is the code for it:

That is it: the event is going to happen on May 21th 2011, at 8:00 in the morning (480 minutes). As we know, the event name is Splinter sprint, and we are going to join some guys down here in Brazil. We filled out the form using fill and select methods.

The fill method is used to fill a field (a textarea, an input, or whatever you can fill). It receives two strings: the first is the name of the field to fill and the second is the value that will fill the field. The select method is used to select an option in a select element (a “combo box”). It also receives two string parameters: the first is the name of the select element, and the second is the value of the option being selected.

Imagine you have the following select element:

To select “Male”, you would call the select method this way:

The last action before click on “Create Event” button is upload a picture for the event. On new event page, Facebook loads the file field for picture uploading inside a iframe, so we need to switch to this frame and interact with the form present inside the frame. To show the frame, we need to click on “Add Event Photo” button and then switch to it, we already know how click on a link:

When we click this link, Facebook makes a asynchronous request, which means the driver doesn’t stay blocked waiting the request ends, so if we try to interact with the frame BEFORE it appears, we will get a ElementDoesNotExist exception. Splinter provides a is_element_present method that receives an argument called wait_time, which is the time Splinter will be waiting for the element to appear on the screen. If the element doesn’t appear on screen, we can’t go on, so we can assume the test failed (remember we are testing a Facebook feature):

The is_element_present_by_css method takes a CSS selector and tries to find an element using it. It also receives a wait_time parameter that indicates a timeout for the search of the element. So, if the iframe element with ID=”upload_pic_frame” is not present or doesn’t appears on screen in 10 seconds, the method returns False, otherwise it returns True.

Important: The fail method is a pseudocode sample and doesn’t exist (if you’re using unittest library, you can invoke self.fail in a TestCase, exactly what I did in complete snippet for this example, available at Github).

Now we see the iframe element on screen and we can finally upload the picture. Imagine we have a variable that contains the path of the picture (and not a file object, StringIO, or something like this), and this variable name is picture_path, this is the code we need:

Splinter provides the get_iframe method that change the context and returns another objet to interact with the content of the frame. So we call the attach_file method, who also receives two strings: the first is the name of the input element and the second is the absolute path of the file being sent. Facebook also uploads the picture asynchronously, but there’s no way to wait some element to appear on screen, so I just put Python to sleep 10 seconds on last line.

After follow all these steps, we can finally click on “Create Event” button and asserts that Facebook created it:

After create an event, Facebook redirects the browser to the event page, so we can check if it really happened by asserting the header of the page. That’s what the code above does: in new event page, it click on submit button, and after the redirect, get the text of a span element and asserts that this text equals to “Splinter sprint”.

That is it! This post was an overview on Splinter API. Check out the complete snippet, written as a test case and also check out Splinter repository at Github.

As you can see, when a client do a request to the nginx server, it first checks on memcached if the response is already cached. If the response was not found on cache server, then nginx forward the request to Tomcat, which process the request, cache the response on memcached and returns it to nginx. Tomcat works only for the first client, and all other clients requesting the same resource will get the cached response on RAM. My objective with this post is to show how we built this architecture.

nginx

nginx was compiled following Linode instructions for nginx installation from source. The only difference is that we added the nginx memcached module. So, first I downloaded the memc_module source from Github and then built nginx with it. Here is the commands for compiling nginx with memcached module:

After install nginx and create an init script for it, we can work on its settings for integration with Tomcat. Just for working with separate settings, we changed the nginx.conf file (located in /opt/nginx/conf directory), and it now looks like this:

See the last line on http directive: this line tells nginx to include all settings present in the /opt/nginx/sites-enabled directory. So, now, let’s create a default file in this directory, with this content:

Some stuffs must be explained here: the default_type directive is necessary for proper serving of cached responses (if you are cache other content types like application/json or application/xml, you should take a look at nginx documentation and deal conditionally with content types). The location / scope defines some settings for proxy, like IP and host. We just do it because we need to pass the right information to our backend (Tomcat or memcached). See more about proxy_set_header at nginx documentation. After that, there is a simple verification on the request method. We don’t want to cache POST requests.

Now we get the magic: first we set the $memcached_key and then we use the memcached_pass directive, the $memcached_key is the URI. memcached_pass directive is very similar to proxy_pass, nginx “proxies” memcached to get the response. So we can get some HTTP status code, like 200, 404 or 502. We define two error handlers for two status codes:

• 404: memcached module returns a 404 error when the key is not on memcached server
• 502: memcached module returns a 502 error when it can’t found memcached server (it is a bad gateway error, the same you get if you start nginx withou start Tomcat ;D)

So, when nginx gets any of those errors, it should forward the request to Tomcat, creating another proxy. We configured it out on fallback, an internal location that builds a proxy between nginx and Tomcat (listening on port 8080). Everything is set up with nginx. As you can see in the picture or in the nginx configuration file, nginx doesn’t put anything on cache, it only gets cached items. The application should put everything on cache. Let’s do it :)

Java application

Now is the time to write some code :) I chose an application written by a friend. It’s a very simple CRUD of users, written by Washington Botelho with the goal of introducing VRaptor, a powerful and fast development focused web framework. Washington also wrote a blog post explaining the application, if you don’t know VRaptor or want to know how the application was built, check the blog post “Getting started with VRaptor 3″. I forked the application, made some minor changes and added a magic filter for caching. All Java code that I want to show here is the filter code:

First, the dependency: for memcached communication, we used spymemcached client. It is a simple and easy to use memcached library. I won’t explain all the code, line by line, but I can tell the idea behind the code: first, call doFilter method on FilterChain, because we want to get the response and work with that. Look the MemcachedHttpServletResponseWrapper object, it encapsulates the response and makes easier to play with response content.

We get the content, write it on response writer and put it in cache using the MemcachedClient provided by spymemcached. The request URI is the key and timeout is 5 seconds.

web.xml

Last step is to add the filter on web.xml file of the project, map it before the VRaptor filter is very important for proper working:

That is it! Now you can just run Tomcat on port 8080 and nginx on port 80, and access http://localhost on your browser. Try some it: raise up the cache timeout, navigate on application and turn off Tomcat. You will still be able to navigate on some pages that use GET request method (users list, home and users form).

Check the entire code out on Github: https://github.com/fsouza/starting-with-vraptor-3. If you have any questions, troubles or comments, please let me know! ;)

You can just pass a parameter profile when getting the driver, so you will use the created profile. Here is a Python code example:

Look that I also wrote a stupid title assertion :) Selenium will search for the provided profile on Firefox profile.ini file. If you don’t know how to create a good Firefox profile for Selenium, check this link out: http://girliemangalo.wordpress.com/2009/02/05/creating-firefox-profile-for-your-selenium-rc-tests/.

That is a very simple tip, and I hope it could help someone. I also hope that I can back to write here :)