For extracting specific parts from some text, we tend to think about regular expressions first. For example, the regular expression for extracting links might look like:

/<a.*?href=".*?">/im

This regular expression is too simple and it can easily cause problems. The quotes may be single or double or they may be totally omitted. There can be spaces on both sides of equal sign. Thinking about all the “special” conditions is difficult.

Of course the better method should be getting a DOM tree from the HTML document, like we do in JavaScript. It’s so easy to get all links in JavaScript – document.getElementsByTagName("a"). For more complex situations, there are so many handy frameworks like jQuery, prototype, etc. Though we normally don’t use JavaScript outside a browser, there are also libraries for parsing HTML documents in other languages (If everyone writes HTML as well-formed XML, we could just use XML parsers ).

I recently did a search for such parsers in Ruby, and found two: Hpricot and Nokogiri. Sadly I didn’t get Hpricot to work correctly on my Mac, but Nokogiri did work. I have been using jQuery extensively in web development, so I love the CSS selectors in Nokogiri very much. They also provide searching methods based on XPath.

Of course if you just want to extract the title of the HTML document, using regular expressions is convenient enough and often faster than parsing the entire HTML document into a tree. But in more complex situations, you should consider a HTML parser like Nokogiri because they’re more robust and safer.

And it’s interesting to see that Nokogiri and Hpricot are competing on performance…

The process was not very smooth for me, as we always expect when trying out new things. The lighttpd server failed to start and by looking into the error log I found these lines:


2009-12-15 19:48:04: (server.c.1503) server stopped by UID = 0 PID = 25128 2009-12-15 19:48:30: (log.c.166) server started
2009-12-15 19:48:30: (mod_fastcgi.c.1104) the fastcgi-backend /var/www/code.py failed to start:
2009-12-15 19:48:30: (mod_fastcgi.c.1108) child exited with status 1 /var/www/code.py
2009-12-15 19:48:30: (mod_fastcgi.c.1111) If you're trying to run your app as a FastCGI backend, make sure you're using the FastCGI-enabled version.If this is PHP on Gentoo, add 'fastcgi' to the USE flags.
2009-12-15 19:48:30: (mod_fastcgi.c.1399) [ERROR]: spawning fcgi failed. 2009-12-15 19:48:30: (server.c.931) Configuration of plugins failed. Going down.


Finally I got it to work and here I’ll provide a check list in case someone else encounter the same problem.

Is code.py executable?

If not:

chmod 755 code.py

Also you need something like this on top of file:

#!/usr/bin/env python

Does helloword example work?

Use the example on web.py’s homepage (remember to add the line of python executable to the top of the file):

#!/usr/bin/env python
import web

urls = (
    '/(.*)', 'hello'
)
app = web.application(urls, globals())

class hello:
    def GET(self, name):
        if not name:
            name = 'world'
        return 'Hello, ' + name + '!'

if __name__ == "__main__":
    app.run()

Chances are this will work. Because this app is so simple and it doesn’t require many libraries.

check egg cache dir permission

Some libraries need to extract files to a directory (python egg cache). I used MySQL-python in my program and it might fail on that.
/sbin/.python-eggs/
Now change the GET method of hello class to:

try:
    import MySQLdb
except Exception, e:
    return str(e);
return 'hello'

If egg cache permission is the cause, you’ll see something like this when visiting the server:

Can’t extract file(s) to egg cache The following error occurred while trying to extract file(s) to the Python egg cache: [Errno 13] Permission denied: ‘/sbin/.python-eggs’ The Python egg cache directory is currently set to: /sbin/.python-eggs Perhaps your account does not have write access to this directory? You can change the cache directory by setting the PYTHON_EGG_CACHE environment variable to point to an accessible directory.

There are many ways to solve this problem, but I just changed the owner of this directory to the user who runs lighttpd (daemon in my case):

chown -R daemon.daemon /sbin/.python-eggs

Is lighttpd using the expected version of python?

My VPS is running CentOS 5.4 and I installed python 2.6 on it (CentOS ships with old 2.4 version). I have two python executables (/usr/local/bin/python and /usr/bin/python) on my $PATH, but python 2.6 has precedence on mine (root). At last it turned out that lighttpd was using the old python 2.4

So I changed the first line of code.py to be:

#!/usr/local/bin/python

Now lighttpd can start happily.

I’m not quite familiar with PHP, but with version 5 on my local dev machine, I was able to do this very quickly. Just use file_get_contents to get the whole page as a string, and then use preg_match_all to search for the parts I want.

Unexpected things happened after I uploaded the script to the server. It said function file_get_contents was not defined. Then I realized that I was on a machine with Red Hat 9, the PHP I was using was version 4.2.2 bundled with RH9. OK. I rewrote the code to use fopen/fread directly. This time, it complained that it couldn’t handle the scheme (I don’t remember the error report string clearly).

I don’t know if it was because of my configuration, or version 4.2.2 doesn’t support the wrappers. It made me crazy. I don’t want to do an upgrade because all the packages are old. It takes time and may cause more problems. I even couldn’t find the apxs binary to compile PHP from source.

Finally, I got a workaround. First use exec to call wget to download the url to a file in /tmp, and then use fopen/fread to read this temp file. It really works.

Another problem was that preg_match_all doesn’t accept the last $offset parameter in PHP 4.2.2, but it’s simple to fix, I think.

This took me some time, but made me realize that how the development of software/language tools eased our daily work.

As I noticed there are some problems with the widget colorer, I checked MyBlogLog and found that they’ve changed the widget theme.

Well, the new theme is polished and cool but lacks flexibility due to the gradient background.

So I must mark the project as “obsolete”. But people still can play around with it, and the “obsolete” code can still be used on websites.

• Firefox
• Firebug (extension for Firefox)

Super large numbers as LatLng’s

Open this page from Google’s example. In firebug’s console, execute the following code:

map.setCenter(new GLatLng(1e100, 1e100));

Setting super large numbers as latitude and longitude like this always makes my Firefox 2.0.0.11 in Windows XP hang.

I don’t know why Google didn’t restrict the range of the numbers, and what computation it did to consume so much CPU time. It should ignore such numbers after a simple judgment.

Strings as zoom parameters

In the same page, execute this piece of code:

var zoom = '16';
map.setZoom(zoom);

As you see, nothing happens. Google must be ignoring the parameter when expression typeof(zoom) == ‘number’ is false. In my situation, I got the value of variable zoom from the url, and didn’t noticed it was the type mismatch which prevented the code to work, since I’m used to JavaScript’s dynamic type conversion feature:

a = 2 * '3'; // a = 6

It will be converted to number when needed. Of course, it’s good to perform strict type checking. But it’ll be better if the API threw an exception instead of being silent.

This can be even worse (Digg front page):

OK. This is fake . Digg did a perfect drop-down menu, though they’re ignoring something. I removed some CSS rules using Firebug and got this effect. It’s ugly, isn’t it?

The Cause

When a link is clicked, it gets the focus. Firefox adds dotted borders to links that have focus to remind the users.

The Solution

To avoid the border, add this CSS rule to your page:

a {
outline: none;
}

This piece of code is similar to the code I removed from Digg to get the screen shot above.

In fact this is a common problem asked by many developers. For more information, search for “firefox link border“. Many authors wrote good articles on this topic.

Update – This can be seen on every WordPress blog login page (click to enlarge):

This screen shot was taken from a page on StumbleUpon which received 35 reviews. StumbleUpon automatically finds out who’s living in the same city with you and places them in the beginning of the list. To make it more clear, StumbleUpon adds an “city” icon () in front of the user’s name.

Yes. This icon caused the mess of the list. Using Firebug to inspect the elements, I found the dimensions of the first two list items with city icon to be 145×89, whereas the other items in the list have dimensions of 145×88. This 1px difference, according to the CSS rule, made left-floating elements to be positioned on the right of them rather than under them.

In fact, just one more CSS rule will fix this issue:

div.listPeople dl {
line-height: 16px;
}

Another solution is to have “clear:both” style on each element starting a row.

This page looks OK in all other major browsers including IE 6, Opera 9.24 and Safari 3.0.4 as I tested. However market share of Firefox is growing steadily, so websites should pay enough attention on it. Besides, with the great power of Firebug, it’s much easier to develop a site for Firefox compared to the ancient browser – IE.

Today lots of sites are using readonly text inputs to carry out some urls for propagation or some code snippets for embedding in your own pages. For example, every Youtube video has such a input field just on the right of it. When you click on it, the text gets selected and you can press Ctrl+C to copy it to the clipboard.

The code is like this:

<input type="text" readonly="readonly" value="some read only text..." onclick="this.select()" />

and the rendered output:

In all other major browsers it looks like a normal text input. But in Firefox, the background color is made gray (#D6D5D9, actually), maybe emphasizing the difference. There are always cases when we want to make it white. But when added “background-color:#FFFFFF;”, it showed no difference. At that time I thought I couldn’t change the background color.

But soon I found the “white” text input when watching this video (I love the puppy!). After inspecting the code with Firebug I found it was “border:1px solid #D2D2D0″ who did the trick. Adding it to the example code above:

Then I found that the background color is ok with any color except #FFFFFF. For example, setting it to “#FEFEFE” (Almost white. And note the default border style is “outset”):

Weird, isn’t it? I don’t know if it’s a bug or “feature”.