Bite Sized Python Tips

Python decorators finally demystified

2013-12-04T01:44:00-05:00

Hi there guys! I hope all of you are fine and doing well. Recently I was hanging out on a python related IRC where I got a request from someone to write an article on decorators in Python. It is perhaps one of the most difficult concept to grasp. So as usual without wasting anytime let get on with it.

- Everything in python is an object (Functions too!):

First of all let's understand functions in python:

def hi(name="yasoob"):
    return "hi "+name

print hi()
#output: 'hi yasoob'

#We can even assign a function to a variable like
greet = hi
#We are not using parentheses here because we are not calling the function hi
#instead we are just putting it into the greet variable. Let's try to run this

print greet()
#output: 'hi yasoob'

#lets see what happens if we delete the old hi function!
del hi
print hi()
#outputs: NameError

print greet()
#outputs: 'hi yasoob'

- Defining functions within functions:

So those are the basics when it comes to functions. Lets take your knowledge one step further! In Python we can define functions inside other functions. You might be wondering what sorcery is this! Let me explain it with an example.

def hi(name="yasoob"):
    print "now you are inside the hi() function"

    def greet():
        return "now you are in the greet() function"

    def welcome():
        return "now you are in the welcome() function"

    print greet()
    print welcome()
    print "now you are back in the hi() function"

hi()
#output:now you are inside the hi() function
#       now you are in the greet() function
#       now you are in the welcome() function
#       now you are back in the hi() function

# This shows that whenever you call hi(), greet() and welcome()
# are also called. However the greet() and welcome() functions
# are not available outsite the hi() function e.g:

greet()
#outputs: NameError: name 'greet' is not defined

So now we know that we can define functions in other functions. In simpler words we can make nested functions. Now you need to learn one more thing that functions can return functions too.

- Returning functions from within functions:

It is not necessary to execute a function within another function, we can return it as an output as well. Let's take a look at it with an example!

def hi(name="yasoob"):
    def greet():
        return "now you are in the greet() function"

    def welcome():
        return "now you are in the welcome() function"

    if name == "yasoob":
        return greet
    else:
        return welcome

a = hi()
print a
#outputs: <function greet at 0x7f2143c01500>

#This clearly shows that `a` now points to the greet() function in hi()
#Now try this

print a()
#outputs: now you are in the greet() function

Just take a look at the code again! In the if/else clause we are returning greet and welcome, not greet() and welcome(). Why is that? It is so because when you put parentheses around it the function gets executed whereas if you don't put parenthesis around it then it can be passed around and can be assigned to other variables without executing it. Did you get it ? Let me explain it a little bit in more detail. When we write a = hi() hi() gets executed and because the name is yasoob by default, the function greet is returned. If we change the statement to a = hi(name = "ali") then the welcome function will be returned. We can also do print hi()() which outputs now you are in the greet() function. I hope you have not fainted by now. Was it difficult ? No? I guess you are in the mood to finally learn about decorators! Lets continue our talk and move forward.

- Giving a function as an argument to another function:

Lets take a look at an example:

def hi():
    return "hi yasoob!"

def doSomethingBeforeHi(func):
    print "I am doing some  boring work before executing hi()"
    print func()

doSomethingBeforeHi(hi)
#outputs:I am doing some  boring work before executing hi()
#        hi yasoob!

Congratulations! You have all of the knowledge to entitle you as a decorator wrangler! Oh wait, I still haven't told you what decorators really are. Here is a short definition decorators let you execute code before and after the function they decorate

- Writing your first decorator:

You have already written your first decorator! Do you know when? In the last example we actually made a decorator! Lets modify the previous decorator and make a little bit more usable program:

def a_new_decorator(a_func):

    def wrapTheFunction():
        print "I am doing some  boring work before executing a_func()"

        a_func()

        print "I am doing some boring work after executing a_func()"

    return wrapTheFunction

def a_function_requiring_decoration():
    print "I am the function which needs some decoration to remove my foul smell"

a_function_requiring_decoration()
#outputs: "I am the function which needs some decoration to remove my foul smell"

a_function_requiring_decoration = a_new_decorator(a_function_requiring_decoration)
#now a_function_requiring_decoration is wrapped by wrapTheFunction()

a_function_requiring_decoration()
#outputs:I am doing some  boring work before executing a_function_requiring_decoration()
#        I am the function which needs some decoration to remove my foul smell
#        I am doing some boring work after executing a_function_requiring_decoration()

Did you get it? We just applied the previously learned principles. This is exactly what the decorators do in python! They wrap a function and modify its behaviour in one way or the another. Now you might be wondering that we did not use the @ anywhere in our code? That is just a short way of making up a decorated function. Here is how we could have run the previous code sample using @.

@a_new_decorator
def a_function_requiring_decoration():
    print "I am the function which needs some decoration to remove my foul smell"

a_function_requiring_decoration()
#outputs: I am doing some  boring work before executing a_function_requiring_decoration()
#         I am the function which needs some decoration to remove my foul smell
#         I am doing some boring work after executing a_function_requiring_decoration()

#the @a_new_decorator is just a short way of saying:
a_function_requiring_decoration = a_new_decorator(a_function_requiring_decoration)

- Decorators Demystified:

I hope you now have a basic understanding of how decorators work in Python. They are not something to be afraid of at all. In addition, we can chain two or more than two decorators! For example:

def bread(func):
    def wrapper():
        print "</''''''\>"
        func()
        print "<\______/>"
    return wrapper

def ingredients(func):
    def wrapper():
        print "#tomatoes#"
        func()
        print "~salad~"
    return wrapper

def sandwich(food="--ham--"):
    print food

sandwich()
#outputs: --ham--
sandwich = bread(ingredients(sandwich))
sandwich()
#outputs:
#</''''''\>
# #tomatoes#
# --ham--
# ~salad~
#<\______/>

In the following example e-satis really does a great job demonstrating how decorators are used. We can run the previous example as so:

@bread
@ingredients
def sandwich(food="--ham--"):
    print food

sandwich()
#outputs:
#</''''''\>
# #tomatoes#
# --ham--
# ~salad~
#<\______/>

The order in which you use decorators matters and can change the whole behaviour of your decorated function if they are not executed in the intended order. Suppose you write this:

@ingredients
@bread
def sandwich(food="--ham--"):
    print food

sandwich()
#outputs:
##tomatoes#
#</''''''\>
# --ham--
#<\______/>
# ~salad~

Nobody wants a sandwich with tomato on the top and salad on the bottom. This demonstrates why you should not mix up the arrangement of the decorators; otherwise you'll pay the price. In our case we paid the price by ruining our beautiful sandwich.

I hope you learned something useful from this post! If there's any demand I will cover some more gotchas and features of decorators in the future. Please share your views in the comments below. If you have any suggestions, then please do tell me about them. You can also follow me on facebook, twitter, or you can send me an email. The links to my social media accounts are at the bottom of the post. Farewell, and don't forget to subscribe to my blog, tweet, and share this post on facebook.

Kivy 101: How to Use BoxLayouts

2013-12-02T22:18:00-05:00

Recently I've started learning about Kivy, a Python Natural User Interface (NUI) toolkit. As I understand it, Kivy is kind of a spiritual successor to pyMT, which you can read more about here. In this article, we will be learning how Kivy handles layout management. While you can position widgets using x/y coordinates, in every GUI toolkit I've used, it's almost always better to use some kind of layout management that the toolkit provides. This allows the widgets to resize and move appropriately as the user changes the window's size. In Kivy, these things Layouts. If you've used wxPython, they are analogous to wxPython's sizers.

I should also note that Kivy can do layouts in two different ways. The first way is to do Layouts with Python code only. The second way is to use a mixture of Python and Kv language. This is to promote the model-view-controller way of doing things. It looks kind of like CSS and reminds me a little of wxPython and XRC. We will look at how to use both methods in this article. While Kivy supports multiple types of Layouts, this article will be focusing only on the BoxLayout. We will show how to nest BoxLayouts.

Kivy, Python and BoxLayout

Creating a BoxLayout in Kivy using Python is actually pretty easy and quite intuitive. We'll start out with a code example and then follow the code with an explanation. Let's get started!

import kivy
import random

from kivy.app import App
from kivy.uix.button import Button
from kivy.uix.boxlayout import BoxLayout

red = [1,0,0,1]
green = [0,1,0,1]
blue =  [0,0,1,1]
purple = [1,0,1,1]

########################################################################
class HBoxLayoutExample(App):
    """
    Horizontally oriented BoxLayout example class
    """

    #----------------------------------------------------------------------
    def build(self):
        """
        Horizontal BoxLayout example
        """
        layout = BoxLayout(padding=10)
        colors = [red, green, blue, purple]

        for i in range(5):
            btn = Button(text="Button #%s" % (i+1),
                         background_color=random.choice(colors)
                         )

            layout.add_widget(btn)
        return layout

########################################################################
class VBoxLayoutExample(App):
    """
    Vertical oriented BoxLayout example class
    """

    #----------------------------------------------------------------------
    def setOrientation(self, orient):
        """"""
        self.orient = orient

    #----------------------------------------------------------------------
    def build(self):
        """"""
        layout = BoxLayout(padding=10, orientation=self.orient)

        for i in range(5):
            btn = Button(text="Button #%s" % (i+1) )
            layout.add_widget(btn)
        return layout

#----------------------------------------------------------------------
if __name__ == "__main__":
    app = HBoxLayoutExample()
    #app = VBoxLayoutExample()
    #app.setOrientation(orient="vertical")
    app.run()

Here we have created a vertically oriented BoxLayout class and a horizontally oriented BoxLayout class. Each class contains 5 buttons with random background colors. The colors follow RGBA, but can have individual values that are between zero and one. Oddly enough, if you use numbers greater than one, the color becomes brighter. I happened to use 255 instead of 1 when I created the screenshot above, so if you happen to run this code and see a more muted set of colors, that's why.

To make the examples extremely simple, we only import Kivy's App, Button and BoxLayout classes. The BoxLayout class accepts several arguments, but we'll focus on the following 3: orientation, padding and spacing. Because BoxLayout is a sub-class of Layout and Widget, it inherits many other methods and keyword arguments that are not covered here. But back to the arguments we currently care about. The padding argument tells Kivy how much space there should be between the Layout and its children, whereas the spacing arguments tells it how much spacing there should be between the children.

To create the buttons, we use a simple loop that loops over a small range of numbers. Each iteration creates a button with a random background color and adds that button to the Layout instance. Then we return the layout at the end.

The vertical BoxLayout example in the VBoxLayoutExample class is slightly different in that I thought it would be fun to be able to set the orientation programmatically. The code is pretty much the same except that I added a setOrientation method. Note that if you call setOrientation again, it will have no effect. As one of my commenters so kindly pointed out, you would need to bind the orientation to the App orient property or use the Kv language to achieve this.

If you comment out the call to HBoxLayoutExample at the end of the script and un-comment out the other two lines, then you should end up seeing something like this:

Notice that when you don't set a background color, Kivy defaults to a dark grey. Kivy does not try to look like a native application. This may or may not be a big deal to you depending on what sort of program you're trying to achieve, but it should be noted. Now we're ready to learn about nesting!

Nesting BoxLayouts

Nesting BoxLayouts inside of each other is pretty easy with Kivy too. Whenever you go to create an application with a complicated interface that will need nested sizers, you should take some time to sketch the layout out with pencil and paper. Then you can draw boxes around the widgets in different ways to help you visualize which Layouts you'll need and how to nest them in each other. I have found this quite helpful with wxPython and I think it applies to any other GUI toolkit that doesn't have a WYSIWYG editor. By the way, BoxLayouts are very powerful. If you know what you're doing, you can make just about any interface with them alone just be using clever nesting.

Enough talk, let's look at some code!

import kivy
import random

from kivy.app import App
from kivy.uix.button import Button
from kivy.uix.boxlayout import BoxLayout

red = [1,0,0,1]
green = [0,1,0,1]
blue =  [0,0,1,1]
purple = [1,0,1,1]

########################################################################
class NestedLayoutExample(App):
    """
    An example of nesting three horizontally oriented BoxLayouts inside
    of one vertically oriented BoxLayout
    """

    #----------------------------------------------------------------------
    def build(self):
        """
        Horizontal BoxLayout example
        """
        main_layout = BoxLayout(padding=10, orientation="vertical")

        colors = [red, green, blue, purple]

        for i in range(3):
            h_layout = BoxLayout(padding=10)
            for i in range(5):
                btn = Button(text="Button #%s" % (i+1),
                             background_color=random.choice(colors)
                             )

                h_layout.add_widget(btn)
            main_layout.add_widget(h_layout)

        return main_layout

#----------------------------------------------------------------------
if __name__ == "__main__":
    app = NestedLayoutExample()
    app.run()

This example is very similar to the last one. The devil is in the details though. Here we have a nested for loop that creates 3 BoxLayouts that contain 5 buttons a piece. Each Layout is then inserted into the top level Layout at the end of each iteration in the outside loop. In case you missed it, scroll back up to see how it turned out. The trick is to create one top-level or main Layout and add other Layouts to it. Now let's turn our attention to learning how to do these things with the Kv language.

Kv+Python and BoxLayout

It's almost always a little painful to learn a new language. Fortunately, the Kv language actually follows Python pretty closely, including Python's requirement of using indentation levels to denote when a section of code begins and ends. You may want to spend a few minutes reading about the Kv language on the Kivy website. Whenever you're ready, we can continue. First we'll start off with the Python code:

# kvboxlayout.py

from kivy.app import App
from kivy.uix.boxlayout import BoxLayout

########################################################################
class KVMyHBoxLayout(BoxLayout):
    pass

########################################################################
class KVBoxLayoutApp(App):
    """"""

    #----------------------------------------------------------------------
    def build(self):
        """"""
        return KVMyHBoxLayout()

#----------------------------------------------------------------------
if __name__ == "__main__":
    app = KVBoxLayoutApp()
    app.run()

This code is much simpler than our previous examples, but it's also rather mysterious. First of all, we create an empty sub-class of BoxLayout. Then we create our App class which has a build method that just returns an instance of the empty BoxLayout class. What's going on here? Well we have to look at the Kv file to find out!

<MyButton@Button>:
    color: .8,.9,0,1
    font_size: 32

<MyButton@Button>:
    orientation: 'horizontal'
    MyButton:
        text: "Btn1"
        background_color: 1,0,0,1
    MyButton:
        text: "Btn2"
        background_color: 0,1,0,1
    MyButton:
        text: "Btn3"
        background_color: 0,0,1,1
    MyButton:
        text: "Btn4"
        background_color: 1,0,1,1
    MyButton:
        text: "Btn5"
        background_color: 1,0,0,1

When you save the code above, you'll have to name it to be the same as the App class, but with a .kv instead of a .py and in lowercase. That means the name of this Kv file needs to be kvboxlayout.kv. You will note that you also need to strip off the App part of the class name such that KVBoxLayoutApp becomes kvboxlayout. Yes, it's a little confusing. If you don't follow the naming conventions correctly, the file will run but you will have an empty black window.

Anyway, first off in the Kv file, we have a section that starts with :. This tells Kivy that we are sub-classing the Button class and calling our sub-class MyButton. Then we indent the required four spaces and set the button's label color and font size. Next we create a BoxLayout section. Notice that we didn't create a sub-class this time. Then we tell it what orientation it should be and add 5 MyButton instances, each one having its own individual label and color.

One of the core Kivy developers pointed out that by creating the BoxLayout in this manner, I am redefining the BoxLayout for all usages. This is not a good thing, even if it does make the example simpler. Thus in the next example, we'll stop doing that and do it the right way instead!

Nesting BoxLayouts with Kv

Nesting BoxLayouts in Kv is a little confusing at first, but once you get the hang of it, you'll find that it's really quite easy. We'll start out with the Python code, take a look at how it works and then look at the Kv code.

from kivy.app import App
from kivy.uix.boxlayout import BoxLayout
from kivy.uix.widget import Widget

########################################################################
class HBoxWidget(Widget):
    pass

########################################################################
class VBoxWidget(Widget):
    pass

########################################################################
class KVNestedBoxLayoutApp(App):
    """"""

    #----------------------------------------------------------------------
    def build(self):
        """"""
        return VBoxWidget()

#----------------------------------------------------------------------
if __name__ == "__main__":
    app = KVNestedBoxLayoutApp()
    app.run()

This time around, we need to create two generic Widget classes: HBoxWidget and VBoxWidget. These are actually dummy classes that become BoxLayouts in the Kv code. Speaking of which, let's take a look at that now. Note that you'll need to name the Kv file kvnestedboxlayout.kv, which you'll note, is a lowercase version of KVNestedBoxLayoutApp.

:
    color: .8,.9,0,1
    font_size: 32

:
    BoxLayout:
        size: root.size
        pos: root.pos
        orientation: 'horizontal'
        MyButton:
            text: "Btn1"
            background_color: 1,1,1,1
        MyButton:
            text: "Btn2"
            background_color: 0,1,0,1
        MyButton:
            text: "Btn3"
            background_color: 0,0,1,1
        MyButton:
            text: "Btn4"
            background_color: 1,0,1,1
        MyButton:
            text: "Btn2"
            background_color: 1,0,0,1

:
    BoxLayout:
        size: root.size
        pos: root.pos
        id: foo_bar
        orientation: 'vertical'
        HBoxWidget:
        HBoxWidget:

The button code is the same as before. Next we have the HBoxWidget which we define as a horizontal BoxLayout with 5 buttons in it. Then we create an instance of VBoxWidget that is a vertical BoxLayout, but this Layout contains two instances of the HBoxWidget. You'll note that in the Python code's build method, we are returning the VBoxWidget, so that's where the action is. If you remove those two HBoxWidget calls, the result will be an empty black window.

There is another way to use Kv files in Kivy. It is through the kivy.lang.Builder.load_file (or load_string) API, which gives you the ability to load Kv files without needing to remember to name the Kv file in some special way. You can read about the API on their website and see an example of it in action in the Kivy examples on github. The only caveat to using this method is that you need to be careful not to load the same file twice or your UI may get messed up.

Wrapping Up

This just scratches the surface of Kivy's Layout system. There are 6 other Layout types available. However, I think you'll find that the examples in this article will get you started down the road of successfully creating cool Kivy applications of your very own. If you need help learning Kivy, there's a pretty good set of documentation on their website. They also have a Google Group and a #kivy channel on freenode.

Download the Source

Note: This is an official cross-posting of an article from the Mouse Vs Python blog. You can read the original here.

Reverse engineering myspace

2013-11-23T01:44:00-05:00

Hi guys! In todays post we will be looking into myspace.com and check whether it is possible to download songs with python or not. If it is possible then we will make a python script which will assist us in downloading the songs. I guess you are as eager to start as I am. So lets get started. However let me make it clear in the beginning that this is only for educational purposes.

Tools :

python - I am using py2k
browser - I am using chrome
rtmpdump & rtmpsrv
internet connection

Lets start :

So lets start. Lets go to myspace:

We don't need this page. Lets go to a song page for instance I am going to open this page. Now lets check out the source of the page to see whether we can find any mp3 link. What I usually do is that I right click on the name of the song and click on "Inspect this". Usually the name is in the song link on most websites. So just do this and see if you get anything. Here is what I got:

Can you see that data stream url ? Oh yes the one which I have highlighted. That is an rtmp url. RTMP is a streaming protocol. Now the problem is that we can not directly download the rtmp links so what should we do now ? We are very lucky because there is already a software which can help us in downloading this and it is called

rtmpdump

However rtmpdump requires some additional parameters for downloading the song. They are the play path, stream url, file name and etc. So how do we get those ? So again we are very lucky because there is a software for this as well and it is developed by the same people who have developed rtmpdump. This one is called

rtmpsrv

Just search for it on google and download it. This is like a proxy. It scans all of your data and searches for rtmp streams. When it has found a stream it just outputs a single command which can be used to download a file with rtmpdump. However first we have to route all the tcp traffic to port 1935. Thats the port where rtmpsrv runs. In order to do so just type this in the terminal:

sudo iptables -t nat -A OUTPUT -p tcp --dport 1935 -j REDIRECT

This will add a firewall rule. Now go to myspace and play a song. After you hit play and wait for some seconds then you will see that a couple of lines are outputted by rtmpsrv. Check this out:

Now if you copy this command and paste it in the terminal you will see that rtmpdump will start to download the file. Now close rtmpsrv and revert the firewall rule by using this command :

sudo iptables -t nat -D OUTPUT -p tcp --dport 1935 -j REDIRECT

Now comes the main part. It is not convenient to go through all of these steps again and again just to download a single song. Now we need to automate this and for that we will use python. After doing a little bit of tinkering I saw that only the -y and -p parameters are changed for every song. Further down the lane we can see that the -p parameter is just the url of the song page and the -y parameter is taken from the rtmp link which we found on the song page.

Writing Python script :

So lets first make a script which will get the rtmp link from the page. Here is my solution:

import requests as r
import re
html = r.get(song_page_url)
rtmp_link = re.findall('data-stream-url="(.+?)"',
            html.text)[0] 
print rtmp_link

So now we have a script which gets the rtmp link. Now we need to modify this script so that it outputs the required rtmpdump command. Again here is my solution :

import requests as r
import re
html = r.get(song_page_url)
rtmp_link = re.findall('data-stream-url="(.+?)"',
            html.text)[0] 
y = rtmp_link.split(';')[1]
name = ' '.join(song_page_url.split('/')[-1].split('-')[:-2])
rtmp_command = 'rtmpdump -r "rtmpte://fms.ec-music.myspacecdn.com/" \
-a "" -f "LNX 11,9,900,152" -o "{}.flv" \
-W "http://lads.myspacecdn.com/music/sdkwrapper/SDKWrapper.2.2.16.swf" \
-p "http://www.myspace.com" -y "{}"'.format(name,y)

So now we just need to open rtmpdump with the parameters. For that I will be using the subprocess module. Here is the final code:

import re
import sys
import requests
import subprocess

song_page_url = sys.argv[1]
html = requests.get(song_page_url).text
match = re.search('data-stream-url="(rtmpe://[^;]+);([^"]+)"', html)
rtmpurl, playpath = match.group(1, 2)
filename = ' '.join(song_page_url.split('/')[-1].split('-')[:-2]) + '.flv'

print 'downloading', filename
subprocess.call(['rtmpdump', '-o', filename, '-r', rtmpurl, '-y', playpath],
                    stderr=subprocess.DEVNULL)

Just run this file like this :

$ python myspace.py https://myspace.com/brunomars/music/song/somewhere-in-brooklyn-69059954-74825216

Thats it! We now have a simple script that will download myspace songs for us. I know you are thinking about more features in this script. Let me give you some ideas. This script downloads just songs and does not have any input validation in place. Maybe you can make a similar script for videos ? When you make something do let me know. That was todays tutorial. I will see you later and don't forget to follow this blog, like it on facebook, follow me on twitter and reply below :) This is the best way to repay me. Lastly stay tuned for the next post.

Disclaimer: Some of the above code can be written more compactly. I wrote it this way so that the beginners could understand what is going on.

Introduction to unittest

2013-11-15T23:04:00-05:00

Hi there folks. I recently thought that I have not written even a single post about testing in python. Testing is one of the most important part of any language. In this post I am going to share some information about unittest with you. So what exactly is unittest ? You might have heard about it on SO or some other forum. It is a testing framework for python just like Junit for Java. It comes pre-installed with python from 2.1. It is easy to use. However there are a lot of other testing frameworks for python out there as well but I will be focusing on unittest today as it is the default testing framework for python. So without wasting any time lets get started.

The standard workflow while using unittest is:

derive your own class from unittest.TestCase
write your tests in functions which start with "test_"
finally write unittest.main() at the end of your file to run the tests

I was also used to be scared by testing. Testing seemed to me as something from outer space but now I have learned its importance and it is essential for every programmer to learn it.

A simple script

So lets write a simple script which we can later test. This script is going to do some math functions for us. So here is the script:

# save it as math.py
def multiply(n1, n2):
    return n1 * n2

def add(n1, n2):
    return n1 + n2

So thats our little script. Now lets move forward and write our first test.

Our first test

So as I told you before that every unittest file contains a custom class derived from unittest.TestCase so lets create that:

import unittest
from math import multiply, add

class TestingMath(unittest.TestCase):
    pass

if __name__ == "__main__":
    unittest.main()

So that was the first part. Now we need to define our tests. In unittest there is a setUp() and tearDown() function. The setUp() function is used to set up the test environment and tearDown() is used to clean up after a test. We do not need them as they are usually used when a lot of tests are written for a larger software. The default implementation of setUp() and tearDown() does nothing. So now lets write our first test by editing our previous script.

import unittest
from math import multiply, add

class TestingMath(unittest.TestCase):

    def setUp(self):
        pass
    def test_multiply(self):
        self.assertEqual( multiply(3,5), 15)

    def test_add(self):
        self.assertEqual( add(3,5), 8)
if __name__ == "__main__":
    unittest.main()

Now save this file as test.py and run it from the command line and you will see some output like this:

yasoob@yasoob:~/Desktop$ python test.py
..
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK

Congratulations! You have written your very first fully working test suite in python. Wait! what are those assertEqual() statements there ? Let me explain them. These assert funtions are the backbones of testing in unittest. They check whether the result is correct or not. There are a lot of assert functions in unittest. In our case we used assertEqual() which checks whether two values are equal or not. We gave 2 parameters to assertEqual() now the job of assertEqual() is to check whether both parameters are equal or not. Just for reference some other assert functions are:

assertTrue(x)
assertFalse(x)
assertNotEqual(a, b)
assertIs(a, b)
assertIsNot(a, b)
assertIsNone(x)

Test Discovery

So just think about it for a minute. You have a lot of test files and want to run all of them. The only method which comes to mind is to manually run all of those files separately. This is possible if you have 10 files but what if you have a 100 files ? That is were automatic test discovery comes to rescue. So if I have all of my tests in app/tests directory I would simply run:

python -m unittest discover app/tests

So this command would gather all of the test files in the tests directory and will run them. I hope that was helpful.

Furthur

So now you have got some idea of how testing works in python. Remember that this was just an introduction and there is a whole lot to unittest than this. If you want to further increase your knowledge then check out the original python docs about unittest. I will cover other testing frameworks as well in the future. I hope you enjoyed this article and do remember this one tip that the best time to write tests is while developing. If you write tests bit by bit while developing then they will not become a burden and your application will be rock solid. Do share your views in the comments below or feel free to email me or tweet me. Last but not the least follow this blog and stay tuned for the next post.

New Python books for GUI development

2013-11-13T23:35:00-05:00

Hi folks welcome back to yet another article. This one is going to be short. In this post I will be sharing two new books which have recently come out. These books are regarding GUI development in Python. In past several months we have not seen a lot of books focused towards GUI development in Python. Hopefully these two books will fill the gap. So without wasting any time lets look at them.

* Tkinter GUI Application Development Hotshot :

Its been a long time since we saw a new Tkinter book out there. This book has received some good critics by the professional python programmers. According to the overview given on the website you will :

Develop 10 real GUI applications using procedural and object-oriented styles
Write network programs, database-driven programs, multi-threaded programs and more with Tkinter
A practical guide to discover the best practices involved in writing GUI applications

* Test-Driven Development with Python :

Test-Driven Development with Python focuses on web development, with some coverage of JavaScript. This book uses a concrete example—the development of a website, from scratch—to teach the TDD metholology. This book is not completed as yet however most of it has been completed. You can read this book online for free or buy it from O'Reilly.

So now we come to an end of yet another useful post. Do follow me on twitter, follow this blog and comment below to share your views and to give me an idea about a future post. I will be waiting for your comments.

Some Python projects worth looking at

2013-11-13T23:30:00-05:00

Hi there guys. So a lot has happened while I was away. It has been only a few weeks and a whole ton of great python project have emerged. In this post I will share a list of some of the excellent projects I saw in the past few days. So lets take a look at them without wasting any more time.

Lans.py

This project is interesting mostly for those who are into network security. This is a single script providing a set of useful commands. It is a packet parsing/injecting arp spoofer. Check out the github page for more information

CherryMusic

CherryMusic is not a new project but it recently got a new release. It is basically a music streaming server written in python. It allows you to stream your own music collection to all your devices! I have personally tested it and really recommend it.

Python Wheels

Wheels are the new standard of python distribution and are intended to replace eggs. Among other advantages, they offer faster installation and allow secure digital signing. This project does not support a lot of python packages at the moment but they are trying to support as many projects as possible.

Dataset

dataset provides two key functions that make using SQL databases in Python a breeze. A simple abstraction layer removes most direct SQL statements without the necessity for a full ORM model and Database contents can be exported (frozen) using a sophisticated plain file generatorwith JSON and CSV support.

Openpyxl

OpenPyxl is a Python library to read/write Excel 2007 xlsx/xlsm files. It was born from lack of existing library to read/write natively from Python the new Open Office XML format. Do check this project if you deal a lot with Excel files. You never know when you find something useful

If you are still reading then I guess you are hungry for more projects. Never mind I will do more posts about such things. For the time being do follow me on twitter, follow this blog and comment below to share your views and to give me an idea about a future post.

PyDev 3.0 Released

2013-11-13T22:18:00-05:00

Hi there folks. Its been a long time since I posted here. That was because my studies were giving me some problems. So no worrys because the problems are all resolved and I am back with yet another news. So what is PyDev ? If you have been coding python scripts with eclipse (it's an IDE) then the chances are that you know what PyDev is. So for those who are not familiar with pydev read on. PyDev is a third-party plug-in for Eclipse. It is an Integrated Development Environment used for programming in Python supporting code refactoring, graphical debugging, code analysis and many other features. Recently it's 3.0 version was released after a long break. Some believed that the development for PyDev had stopped but this release has given hope to eclipse lovers. According to the official pydev website PyDev comes with many goodies such as:

Django integration
Code completion
Code completion with auto import
Type hinting
Code analysis
Go to definition
Refactoring
Debugger
Remote debugger
Tokens browser
Interactive console
Unittest integration
Code coverage
and many others:

I believe it is worth to check out this project. The link to the official website is http://pydev.org/ and here is the link to this release announcement . Do share your views in the comments below and stay tuned for the next post. If you know someone who can benefit from my post then do make sure that you tell them about this blog.

Python 3.4 to be equipped with pip by default

2013-10-22T21:59:00-04:00

Hi there Pythonistas. A news has arrived thatpip will be available by default with python 3.4. PEP 453 which was "Explicit bootstrapping of pip in Python installations" has been accepted. Python 3.4 which has entered into the beta phase after the release of final alpha will have pip by default. If you are new to python then you might be wondering what pip is ? Pip is a tool for installing and managing Python packages. Previously easy_install has been the most used package manager. Easy_Install is a python module (easy_install) bundled with setuptools that lets you automatically download, build, install, and manage Python packages but has a lot of weak points in front of pip. In order to use pip you first have to install it by using easy_install. Why pip is better:

The command is simpler, shorter
First-class support of virtualenv
More commands than just install, including uninstall. pip bundle, pip freeze and pip search are pretty nice as well.
Can install from a VCS (via -e), or from source
Requirements files are easy to use, clearly describes what happens in its process
If it can not download all the dependencies, it will not install anything (though I don't think it rolls stuff back if an installation fails

I hope you are as much happy as I am with the acceptance of this PEP. Do share your views in the comments below and don't forget to share this post and follow this blog in order to get the latest news relating with the python world.

PyCharm 3.0 Community Edition

2013-10-22T21:43:00-04:00

The makers of the hugely popular Python IDE PyCharm recently announced a community edition of their Python IDE. JetBrains announced that they will be offering a community Edition of PyCharm however it will have some stripped down features for example it will not have builtin support for pyramid etc. However the news for a Community version of PyCharm became hot in the python community recently because PyCharm is currently the best IDE available for Python. This community edition has been out for quite some time now and has already been downloaded by a lot of developers. Today the Source code of this IDE was also open sourced by JetBrains. The source code is available on github . This IDE has a lot of cool features. Here are a few of the features of the community version:

Intelligent Editor, with code completion, on-the-fly error highlighting, auto-fixes, etc.
Automated code refactorings and rich navigation capabilities
Integrated debugger and unit testing support
Native VCS integrations
Customizable UI and key-bindings, with VIM
emulation available
And much more, all available under the Apache 2 license

I hope you are as much excited as I am. I have already taken this IDE for a test drive and have been fascinated by it's power. It is a one stop solution for every python developer whether he is a system engineer or a web developer. I hope you will take this IDE for a test drive. Do share your experience in the comments below so that others can know about your experience and don't forget to follow this blog if you want to get latest news, tips, tricks and tutorials relating to python.

Python News 13 Oct 2013

2013-10-14T01:59:00-04:00

Hi folks in this post i am going to share with you news from the python world. This is like a roundup of monthly news.

Latest News
1. NHS drops Oracle for Python, Riak, Redis, RabbitMQ, Tornado, Flask and more
The UK government's quest to get public services to use more open source technologies seems to be taking hold, judging by the revamp of the NHS's very large Spine service.

2. Microsoft donates $100,000 to the IPython team
Microsoft donates $100,000 to the IPython team for the continued development of IPython.

3. PyCharm 3.0 Community Edition
Jet Brains recently surprised the Python community by launching a community version of it's extremely popular Python IDE. It's a must have for a Python programmer.

4. PyMongo 2.6.3 Released
MongoDB is one of the leading NoSQL Database. The PyMongo distribution contains tools for interacting with MongoDB database from Python.

Python Conferences:
1. PyCon 2014 Keynotes announced
The keynote speakers for the largest Python Conference have been announced. This year the keynote speakers include Van Lindberg, Jessica McKellar, Fernando Perez, Guido Van Rossum and John Perry Barlow.

2. PyTennessee Call for Papers!
If you are in the Tennessee area and know python then make sure that you do not miss this great chance to attend the PyTennnesse. Submit your papers now. The deadline is 1st November.

Articles:
1. Breaking Out of Secured Python Shells
Ever thought of hacking secure python shells ? I hope not but this guy definitely thought about that and even accomplished. Hop over to his website to read about his discovery. It is definitely worth a read.

2. Cubr!
A really cool project. This one involves the use of a mixed up Rubik's Cube. Just bring the mixed layout of a Rubik's Cube in front of your webcam and let this nifty application do it's trick.

3. Mapping data in Python using Pandas and Vincent
This Blogpost clearly shows the power of Vincent. Do check it out if you are into data mining and data representation. This Blogpost will surely help you.

4. Training Random Forests in Python using the GPU
Random Forests have emerged as a very popular learning algorithm for tackling complex prediction problems. These guys have released GPU Random Forest library for Python called CudaTree. Check this out if you want to learn more.

5. Introduction to Hardware Hacking with Arduino
There are not a lot of post out there which explain clearly how to hack an Arduino using Python and turning it into a useful creature. This post is for Hardware Hackers who are a bit familiar with Python.

6. Why are there so many Pythons ?
This post starts from scratch and takes you step by step into the world of different Python implementations. Read it if you have time.

7. Programming a Perceptron in Python
A perceptron classifier is a simple model of a neuron. In this post the author programs a Perceptron in Python.

8. Read and Write Video Frames in Python Using FFMPEG
This article shows how easy it is to read or write video frames with Python, by calling the external software FFMPEG through pipes. Worth the read if you are interested in video manipulation.

Interesting Projects:
1. csscompressor -- A python port of YUI CSS Compressor
Port of YUI CSS Compressor to Python

2. CIntruder
Captcha Intruder is an automatic pentesting tool to bypass captchas.

3. Arrow 0.4
Better dates & times for Python, now with ISO-8601 parsing and week support!

4. Lifelines
Survival analysis in Python.

5. Pymultitor
With this algorithm you can use multiple tor threads to make multiple requests with multiple IP addresses.

6. Flask-Sockets: Easy WebSockets in Flask, by Kenneth Reitz
This guy is super nice and always contribute useful libraries. This one is no less. Check this out if you are working with websockets.

7. Whoosh - a fast, pure Python search engine library.
Really a wonderful project. Whoosh is a fast, pure Python search engine library. Whoosh is not really a search engine, it’s a programmer library for creating a search engine

8. Centrifuge - real-time messaging in web applications
Simple real-time messaging in web applications.The main goal of Centrifuge is the same as of Pusher or Pubnub services. The main difference is that Centrifuge is open-source and requires installation.

I hope you liked this roundup. Make sure that you share this post on Facebook and Twitter. Stay tuned for the next post! Also do follow my blog so that you do not miss a single post and finally comment below if I missed something and if you think that I should do similar posts in the future as well.

All About Decorators in Python

2013-10-10T21:18:00-04:00

Hi there fellas. This is an answer posted on stackoverflow by e-satis. The original link to the answer is given at the end. No credit goes to me. All of the credit goes to the original author. This answer is posted just because most of us are unaware of how decorators work in python and this answer solves that problem beautifully.

Python's functions are objects

To understand decorators, you must first understand that functions are objects in Python.
This has important consequences. Let's see why with a simple example :

def shout(word="yes"):
    return word.capitalize()+"!"

print shout()
# outputs : 'Yes!'

# As an object, you can assign the function to a variable like any
# other object

scream = shout

# Notice we don't use parentheses: we are not calling the function, we are
# putting the function "shout" into the variable "scream". 
# It means you can then call "shout" from "scream":

print scream()
# outputs : 'Yes!'

# More than that, it means you can remove the old name 'shout', and
# the function will still be accessible from 'scream'

del shout
try:
    print shout()
except NameError, e:
    print e
    #outputs: "name 'shout' is not defined"

print scream()
# outputs: 'Yes!'

OK, keep that in mind, we are going back to it soon. Another interesting property of Python functions is they can be defined... inside another function!

def talk():

    # You can define a function on the fly in "talk" ...
    def whisper(word="yes"):
        return word.lower()+"..."

    # ... and use it right away!

    print whisper()

# You call "talk", that defines "whisper" EVERY TIME you call it, then
# "whisper" is called in "talk". 
talk()
# outputs: 
# "yes..."

# But "whisper" DOES NOT EXIST outside "talk":

try:
    print whisper()
except NameError, e:
    print e
    #outputs : "name 'whisper' is not defined"*

Functions references

OK, still here? Now the fun part, you've seen that functions are objects and therefore:

can be assigned to a variable;
can be defined in another function.

Well, that means that a function can return another function :-) Have a look:

def getTalk(type="shout"):

    # We define functions on the fly
    def shout(word="yes"):
        return word.capitalize()+"!"

    def whisper(word="yes") :
        return word.lower()+"...";

    # Then we return one of them
    if type == "shout":
        # We don't use "()", we are not calling the function,
        # we are returning the function object
        return shout  
    else:
        return whisper

# How do you use this strange beast?

# Get the function and assign it to a variable
talk = getTalk()

# You can see that "talk" is here a function object:
print talk
#outputs : <function shout at 0xb7ea817c>

# The object is the one returned by the function:
print talk()
#outputs : Yes!

# And you can even use it directly if you feel wild:
print getTalk("whisper")()
#outputs : yes...

But wait, there is more. If you can return a function, then you can pass one as a parameter:

def doSomethingBefore(func): 
    print "I do something before then I call the function you gave me"
    print func()

doSomethingBefore(scream)
#outputs: 
#I do something before then I call the function you gave me
#Yes!

Well, you just have everything needed to understand decorators. You see, decorators are wrappers which means that they let you execute code before and after the function they decorate without the need to modify the function itself.

Handcrafted decorators

How you would do it manually:

# A decorator is a function that expects ANOTHER function as parameter
def my_shiny_new_decorator(a_function_to_decorate):

    # Inside, the decorator defines a function on the fly: the wrapper.
    # This function is going to be wrapped around the original function
    # so it can execute code before and after it.
    def the_wrapper_around_the_original_function():

        # Put here the code you want to be executed BEFORE the original 
        # function is called
        print "Before the function runs"

        # Call the function here (using parentheses)
        a_function_to_decorate()

        # Put here the code you want to be executed AFTER the original 
        # function is called
        print "After the function runs"

    # At this point, "a_function_to_decorate" HAS NEVER BEEN EXECUTED.
    # We return the wrapper function we have just created.
    # The wrapper contains the function and the code to execute before
    # and after. It's ready to use!
    return the_wrapper_around_the_original_function

# Now imagine you create a function you don't want to ever touch again.
def a_stand_alone_function():
    print "I am a stand alone function, don't you dare modify me"

a_stand_alone_function() 
#outputs: I am a stand alone function, don't you dare modify me

# Well, you can decorate it to extend its behavior.
# Just pass it to the decorator, it will wrap it dynamically in 
# any code you want and return you a new function ready to be used:

a_stand_alone_function_decorated = my_shiny_new_decorator(a_stand_alone_function)
a_stand_alone_function_decorated()
#outputs:
#Before the function runs
#I am a stand alone function, don't you dare modify me
#After the function runs

Now, you probably want that every time you call a_stand_alone_function, a_stand_alone_function_decorated is called instead. That's easy, just overwrite a_stand_alone_function with the function returned by my_shiny_new_decorator:

a_stand_alone_function = my_shiny_new_decorator(a_stand_alone_function)
a_stand_alone_function()
#outputs:
#Before the function runs
#I am a stand alone function, don't you dare modify me
#After the function runs

# And guess what? That's EXACTLY what decorators do!

Decorators demystified

The previous example, using the decorator syntax:

@my_shiny_new_decorator
def another_stand_alone_function():
    print "Leave me alone"

another_stand_alone_function()  
#outputs:  
#Before the function runs
#Leave me alone
#After the function runs

Yes, that's all, it's that simple. @decorator is just a shortcut to:

another_stand_alone_function = my_shiny_new_decorator(another_stand_alone_function)

Decorators are just a pythonic variant of the decorator design pattern. There are several classic design patterns embedded in Python to ease development, like iterators.

Of course, you can cumulate decorators:

def bread(func):
    def wrapper():
        print "</''''''\>"
        func()
        print "<\______/>"
    return wrapper

def ingredients(func):
    def wrapper():
        print "#tomatoes#"
        func()
        print "~salad~"
    return wrapper

def sandwich(food="--ham--"):
    print food

sandwich()
#outputs: --ham--
sandwich = bread(ingredients(sandwich))
sandwich()
#outputs:
#</''''''\>
# #tomatoes#
# --ham--
# ~salad~
#<\______/>

Using the Python decorator syntax:

@bread
@ingredients
def sandwich(food="--ham--"):
    print food

sandwich()
#outputs:
#</''''''\>
# #tomatoes#
# --ham--
# ~salad~
#<\______/>

The order you set the decorators MATTERS:

@ingredients
@bread
def strange_sandwich(food="--ham--"):
    print food

strange_sandwich()
#outputs:
##tomatoes#
#</''''''\>
# --ham--
#<\______/>
# ~salad~

Eventually answering the question

As a conclusion, you can easily see how to answer the question:

# The decorator to make it bold
def makebold(fn):
    # The new function the decorator returns
    def wrapper():
        # Insertion of some code before and after
        return "<b>" + fn() + "</b>"
    return wrapper

# The decorator to make it italic
def makeitalic(fn):
    # The new function the decorator returns
    def wrapper():
        # Insertion of some code before and after
        return "<i>" + fn() + "</i>"
    return wrapper

@makebold
@makeitalic
def say():
    return "hello"

print say() 
#outputs: <b><i>hello</i></b>

# This is the exact equivalent to 
def say():
    return "hello"
say = makebold(makeitalic(say))

print say() 
#outputs: <b><i>hello</i></b>

You can now just leave happy, or burn your brain a little bit more and see advanced uses of decorators.

Passing arguments to the decorated function

# It's not black magic, you just have to let the wrapper 
# pass the argument:

def a_decorator_passing_arguments(function_to_decorate):
    def a_wrapper_accepting_arguments(arg1, arg2):
        print "I got args! Look:", arg1, arg2
        function_to_decorate(arg1, arg2)
    return a_wrapper_accepting_arguments

# Since when you are calling the function returned by the decorator, you are
# calling the wrapper, passing arguments to the wrapper will let it pass them to 
# the decorated function

@a_decorator_passing_arguments
def print_full_name(first_name, last_name):
    print "My name is", first_name, last_name

print_full_name("Peter", "Venkman")
# outputs:
#I got args! Look: Peter Venkman
#My name is Peter Venkman

Decorating methods

What's great with Python is that methods and functions are really the same, except methods expect their first parameter to be a reference to the current object (self). It means you can build a decorator for methods the same way, just remember to take self in consideration:

def method_friendly_decorator(method_to_decorate):
    def wrapper(self, lie):
        lie = lie - 3 # very friendly, decrease age even more :-)
        return method_to_decorate(self, lie)
    return wrapper


class Lucy(object):

    def __init__(self):
        self.age = 32

    @method_friendly_decorator
    def sayYourAge(self, lie):
        print "I am %s, what did you think?" % (self.age + lie)

l = Lucy()
l.sayYourAge(-3)
#outputs: I am 26, what did you think?

Of course, if you make a very general decorator and want to apply it to any function or method, no matter its arguments, then just use *args, **kwargs:

def a_decorator_passing_arbitrary_arguments(function_to_decorate):
    # The wrapper accepts any arguments
    def a_wrapper_accepting_arbitrary_arguments(*args, **kwargs):
        print "Do I have args?:"
        print args
        print kwargs
        # Then you unpack the arguments, here *args, **kwargs
        # If you are not familiar with unpacking, check:
        # http://www.saltycrane.com/blog/2008/01/how-to-use-args-and-kwargs-in-python/
        function_to_decorate(*args, **kwargs)
    return a_wrapper_accepting_arbitrary_arguments

@a_decorator_passing_arbitrary_arguments
def function_with_no_argument():
    print "Python is cool, no argument here."

function_with_no_argument()
#outputs
#Do I have args?:
#()
#{}
#Python is cool, no argument here.

@a_decorator_passing_arbitrary_arguments
def function_with_arguments(a, b, c):
    print a, b, c

function_with_arguments(1,2,3)
#outputs
#Do I have args?:
#(1, 2, 3)
#{}
#1 2 3

@a_decorator_passing_arbitrary_arguments
def function_with_named_arguments(a, b, c, platypus="Why not ?"):
    print "Do %s, %s and %s like platypus? %s" %  
   (a, b, c, platypus)

function_with_named_arguments("Bill", "Linus", "Steve", platypus="Indeed!")
#outputs
#Do I have args ? :
#('Bill', 'Linus', 'Steve')
#{'platypus': 'Indeed!'}
#Do Bill, Linus and Steve like platypus? Indeed!

class Mary(object):

    def __init__(self):
        self.age = 31

    @a_decorator_passing_arbitrary_arguments
    def sayYourAge(self, lie=-3): # You can now add a default value
        print "I am %s, what did you think ?" % (self.age + lie)

m = Mary()
m.sayYourAge()
#outputs
# Do I have args?:
#(<__main__.Mary object at 0xb7d303ac>,)
#{}
#I am 28, what did you think?

Passing arguments to the decorator

Great, now what would you say about passing arguments to the decorator itself? Well this is a bit twisted because a decorator must accept a function as an argument and therefore, you cannot pass the decorated function arguments directly to the decorator.

Before rushing to the solution, let's write a little reminder:

# Decorators are ORDINARY functions
def my_decorator(func):
    print "I am a ordinary function"
    def wrapper():
        print "I am function returned by the decorator"
        func()
    return wrapper

# Therefore, you can call it without any "@"

def lazy_function():
    print "zzzzzzzz"

decorated_function = my_decorator(lazy_function)
#outputs: I am a ordinary function

# It outputs "I am a ordinary function", because that's just what you do:
# calling a function. Nothing magic.

@my_decorator
def lazy_function():
    print "zzzzzzzz"

#outputs: I am a ordinary function

It's exactly the same. "my_decorator" is called. So when you @my_decorator, you are telling Python to call the function 'labeled by the variable "my_decorator"'. It's important, because the label you give can point directly to the decorator... or not! Let's start to be evil!

def decorator_maker():

    print "I make decorators! I am executed only once: "+  
         "when you make me create a decorator."

    def my_decorator(func):

        print "I am a decorator! I am executed only when you decorate a function."

        def wrapped():
            print ("I am the wrapper around the decorated function. "
                  "I am called when you call the decorated function. "
                  "As the wrapper, I return the RESULT of the decorated function.")
            return func()

        print "As the decorator, I return the wrapped function."

        return wrapped

    print "As a decorator maker, I return a decorator"
    return my_decorator

# Let's create a decorator. It's just a new function after all.
new_decorator = decorator_maker()       
#outputs:
#I make decorators! I am executed only once: when you make me create a decorator.
#As a decorator maker, I return a decorator

# Then we decorate the function

def decorated_function():
    print "I am the decorated function."

decorated_function = new_decorator(decorated_function)
#outputs:
#I am a decorator! I am executed only when you decorate a function.
#As the decorator, I return the wrapped function

# Let's call the function:
decorated_function()
#outputs:
#I am the wrapper around the decorated function. I am called when you call the decorated function.
#As the wrapper, I return the RESULT of the decorated function.
#I am the decorated function.

No surprise here. Let's do EXACTLY the same thing, but skipping intermediate variables:

def decorated_function():
    print "I am the decorated function."
decorated_function = decorator_maker()(decorated_function)
#outputs:
#I make decorators! I am executed only once: when you make me create a decorator.
#As a decorator maker, I return a decorator
#I am a decorator! I am executed only when you decorate a function.
#As the decorator, I return the wrapped function.

# Finally:
decorated_function()    
#outputs:
#I am the wrapper around the decorated function. I am called when you call the decorated function.
#As the wrapper, I return the RESULT of the decorated function.
#I am the decorated function.

Let's make it AGAIN, even shorter:

@decorator_maker()
def decorated_function():
    print "I am the decorated function."
#outputs:
#I make decorators! I am executed only once: when you make me create a decorator.
#As a decorator maker, I return a decorator
#I am a decorator! I am executed only when you decorate a function.
#As the decorator, I return the wrapped function.

#Eventually: 
decorated_function()    
#outputs:
#I am the wrapper around the decorated function. I am called when you call the decorated function.
#As the wrapper, I return the RESULT of the decorated function.
#I am the decorated function.

Hey, did you see that? We used a function call with the "@" syntax :-)

So back to decorators with arguments. If we can use functions to generate the decorator on the fly, we can pass arguments to that function, right?

def decorator_maker_with_arguments(decorator_arg1, decorator_arg2):

    print "I make decorators! And I accept arguments:", decorator_arg1, decorator_arg2

    def my_decorator(func):
        # The ability to pass arguments here is a gift from closures.
        # If you are not comfortable with closures, you can assume it's ok,
        # or read: http://stackoverflow.com/questions/13857/can-you-explain-closures-as-they-relate-to-python
        print "I am the decorator. Somehow you passed me arguments:", decorator_arg1, decorator_arg2

        # Don't confuse decorator arguments and function arguments!
        def wrapped(function_arg1, function_arg2) :
            print ("I am the wrapper around the decorated function.\n"
                  "I can access all the variables\n"
                  "\t- from the decorator: {0} {1}\n"
                  "\t- from the function call: {2} {3}\n"
                  "Then I can pass them to the decorated function"
                  .format(decorator_arg1, decorator_arg2,
                          function_arg1, function_arg2))
            return func(function_arg1, function_arg2)

        return wrapped

    return my_decorator

@decorator_maker_with_arguments("Leonard", "Sheldon")
def decorated_function_with_arguments(function_arg1, function_arg2):
    print ("I am the decorated function and only knows about my arguments: {0}"
           " {1}".format(function_arg1, function_arg2))

decorated_function_with_arguments("Rajesh", "Howard")
#outputs:
#I make decorators! And I accept arguments: Leonard Sheldon
#I am the decorator. Somehow you passed me arguments: Leonard Sheldon
#I am the wrapper around the decorated function. 
#I can access all the variables 
#    - from the decorator: Leonard Sheldon 
#    - from the function call: Rajesh Howard 
#Then I can pass them to the decorated function
#I am the decorated function and only knows about my arguments: Rajesh Howard

Here it is, a decorator with arguments. Arguments can be set as variable:

c1 = "Penny"
c2 = "Leslie"

@decorator_maker_with_arguments("Leonard", c1)
def decorated_function_with_arguments(function_arg1, function_arg2):
    print ("I am the decorated function and only knows about my arguments:"
           " {0} {1}".format(function_arg1, function_arg2))

decorated_function_with_arguments(c2, "Howard")
#outputs:
#I make decorators! And I accept arguments: Leonard Penny
#I am the decorator. Somehow you passed me arguments: Leonard Penny
#I am the wrapper around the decorated function. 
#I can access all the variables 
#    - from the decorator: Leonard Penny 
#    - from the function call: Leslie Howard 
#Then I can pass them to the decorated function
#I am the decorated function and only knows about my arguments: Leslie Howard

As you can see, you can pass arguments to the decorator like any function using this trick. You can even use *args, **kwargs if you wish. But remember decorators are called only once. Just when Python imports the script. You can't dynamically set the arguments afterwards. When you do "import x", the function is already decorated, so you can't
change anything.

Let's practice: a decorator to decorate a decorator

OK, as a bonus, I'll give you a snippet to make any decorator accept generically any argument. After all, in order to accept arguments, we created our decorator using another function.
We wrapped the decorator.
Anything else we saw recently that wrapped function?
Oh yes, decorators!
Let's have some fun and write a decorator for the decorators:

def decorator_with_args(decorator_to_enhance):
    """ 
    This function is supposed to be used as a decorator.
    It must decorate an other function, that is intended to be used as a decorator.
    Take a cup of coffee.
    It will allow any decorator to accept an arbitrary number of arguments,
    saving you the headache to remember how to do that every time.
    """

    # We use the same trick we did to pass arguments
    def decorator_maker(*args, **kwargs):

        # We create on the fly a decorator that accepts only a function
        # but keeps the passed arguments from the maker.
        def decorator_wrapper(func):

            # We return the result of the original decorator, which, after all, 
            # IS JUST AN ORDINARY FUNCTION (which returns a function).
            # Only pitfall: the decorator must have this specific signature or it won't work:
            return decorator_to_enhance(func, *args, **kwargs)

        return decorator_wrapper

    return decorator_maker

It can be used as follows:

# You create the function you will use as a decorator. And stick a decorator on it :-)
# Don't forget, the signature is "decorator(func, *args, **kwargs)"
@decorator_with_args 
def decorated_decorator(func, *args, **kwargs): 
    def wrapper(function_arg1, function_arg2):
        print "Decorated with", args, kwargs
        return func(function_arg1, function_arg2)
    return wrapper

# Then you decorate the functions you wish with your brand new decorated decorator.

@decorated_decorator(42, 404, 1024)
def decorated_function(function_arg1, function_arg2):
    print "Hello", function_arg1, function_arg2

decorated_function("Universe and", "everything")
#outputs:
#Decorated with (42, 404, 1024) {}
#Hello Universe and everything

# Whoooot!

I know, the last time you had this feeling, it was after listening a guy saying: "before understanding recursion, you must first understand recursion". But now, don't you feel good about mastering this?

Best practices while using decorators

They are new as of Python 2.4, so be sure that's what your code is running on.
Decorators slow down the function call. Keep that in mind.
You can not un-decorate a function. There are hacks to create decorators that can be removed but nobody uses them. So once a function is decorated, it's done. For all the code.
Decorators wrap functions, which can make them hard to debug.

Python 2.5 solves this last issue by providing the functools module including functools.wraps that copies the name, module and docstring of any wrapped function to it's wrapper. Fun fact, functools.wraps is a decorator :-)

# For debugging, the stacktrace prints you the function __name__
def foo():
    print "foo"

print foo.__name__
#outputs: foo

# With a decorator, it gets messy    
def bar(func):
    def wrapper():
        print "bar"
        return func()
    return wrapper

@bar
def foo():
    print "foo"

print foo.__name__
#outputs: wrapper

# "functools" can help for that

import functools

def bar(func):
    # We say that "wrapper", is wrapping "func"
    # and the magic begins
    @functools.wraps(func)
    def wrapper():
        print "bar"
        return func()
    return wrapper

@bar
def foo():
    print "foo"

print foo.__name__
#outputs: foo

How can the decorators be useful?

Now the big question: what can I use decorators for? Seem cool and powerful, but a practical example would be great. Well, there are 1000 possibilities. Classic uses are extending a function behavior from an external lib (you can't modify it) or for a debug purpose (you don't want to modify it because it's temporary). You can use them to extends several functions with the same code without rewriting it every time, for DRY's sake. E.g.:

def benchmark(func):
    """
    A decorator that prints the time a function takes
    to execute.
    """
    import time
    def wrapper(*args, **kwargs):
        t = time.clock()
        res = func(*args, **kwargs)
        print func.__name__, time.clock()-t
        return res
    return wrapper


def logging(func):
    """
    A decorator that logs the activity of the script.
    (it actually just prints it, but it could be logging!)
    """
    def wrapper(*args, **kwargs):
        res = func(*args, **kwargs)
        print func.__name__, args, kwargs
        return res
    return wrapper


def counter(func):
    """
    A decorator that counts and prints the number of times a function has been executed
    """
    def wrapper(*args, **kwargs):
        wrapper.count = wrapper.count + 1
        res = func(*args, **kwargs)
        print "{0} has been used: {1}x".format(func.__name__, wrapper.count)
        return res
    wrapper.count = 0
    return wrapper

@counter
@benchmark
@logging
def reverse_string(string):
    return str(reversed(string))

print reverse_string("Able was I ere I saw Elba")
print reverse_string("A man, a plan, a canoe, pasta, heros, rajahs, a coloratura, maps, snipe, percale, macaroni, a gag, a banana bag, a tan, a tag, a banana bag again (or a camel), a crepe, pins, Spam, a rut, a Rolo, cash, a jar, sore hats, a peon, a canal: Panama!")

#outputs:
#reverse_string ('Able was I ere I saw Elba',) {}
#wrapper 0.0
#wrapper has been used: 1x 
#ablE was I ere I saw elbA
#reverse_string ('A man, a plan, a canoe, pasta, heros, rajahs, a coloratura, maps, snipe, percale, macaroni, a gag, a banana bag, a tan, a tag, a banana bag again (or a camel), a crepe, pins, Spam, a rut, a Rolo, cash, a jar, sore hats, a peon, a canal: Panama!',) {}
#wrapper 0.0
#wrapper has been used: 2x
#!amanaP :lanac a ,noep a ,stah eros ,raj a ,hsac ,oloR a ,tur a ,mapS ,snip ,eperc a ,)lemac a ro( niaga gab ananab a ,gat a ,nat a ,gab ananab a ,gag a ,inoracam ,elacrep ,epins ,spam ,arutaroloc a ,shajar ,soreh ,atsap ,eonac a ,nalp a ,nam A

Of course the good thing with decorators is that you can use them right away on almost anything without rewriting. DRY, I said:

@counter
@benchmark
@logging
def get_random_futurama_quote():
    import httplib
    conn = httplib.HTTPConnection("slashdot.org:80")
    conn.request("HEAD", "/index.html")
    for key, value in conn.getresponse().getheaders():
        if key.startswith("x-b") or key.startswith("x-f"):
            return value
    return "No, I'm ... doesn't!"

print get_random_futurama_quote()
print get_random_futurama_quote()

#outputs:
#get_random_futurama_quote () {}
#wrapper 0.02
#wrapper has been used: 1x
#The laws of science be a harsh mistress.
#get_random_futurama_quote () {}
#wrapper 0.01
#wrapper has been used: 2x
#Curse you, merciful Poseidon!

Python itself provides several decorators: property, staticmethod, etc. Django use decorators to manage caching and view permissions. Twisted to fake inlining asynchronous functions calls. This really is a large playground.

Source: Stackoverflow

The Python yield keyword explained

2013-09-29T21:34:00-04:00

Hi there folks. Again welcome to yet another useful tutorial. This is again a stackoverflow answer. This one is related to the Python yield keyword. It explains you what yield, generators and iterables are. So without wasting any time lets continue with the answer.

To understand what yield does, you must understand what generators are. And before generators come iterables.

Iterables

When you create a list, you can read its items one by one, and it's called iteration:

>>> mylist = [1, 2, 3]
>>> for i in mylist:
...    print(i)
1
2
3

Mylist is an iterable. When you use a list comprehension, you create a list, and so an iterable:

>>> mylist = [x*x for x in range(3)]
>>> for i in mylist:
...    print(i)
0
1
4

Everything you can use "for... in..." on is an iterable: lists, strings, files...
These iterables are handy because you can read them as much as you wish, but you store all the values in memory and it's not always what you want when you have a lot of values.

Generators

Generators are iterators, but you can only iterate over them once. It's because they do not store all the values in memory, they generate the values on the fly:

>>> mygenerator = (x*x for x in range(3))
>>> for i in mygenerator:
...    print(i)
0
1
4

It is just the same except you used () instead of []. BUT, you can not perform for i in mygenerator a second time since generators can only be used once: they calculate 0, then forget about it and calculate 1, and end calculating 4, one by one.

Yield

Yield is a keyword that is used like return, except the function will return a generator.

>>> def createGenerator():
...    mylist = range(3)
...    for i in mylist:
...        yield i*i
...
>>> mygenerator = createGenerator() # create a generator
>>> print(mygenerator) # mygenerator is an object!
<generator object createGenerator at 0xb7555c34>
>>> for i in mygenerator:
...     print(i)
0
1
4

Here it's a useless example, but it's handy when you know your function will return a huge set of values that you will only need to read once.

To master yield, you must understand that when you call the function, the code you have written in the function body does not run. The function only returns the generator object, this is a bit tricky :-)

Then, your code will be run each time the for uses the generator.

Now the hard part:

The first time the for calls the generator object created from your function, it will run the code in your function from the beginning until it hits yield, then it'll return the first value of the loop. Then, each other call will run the loop you have written in the function one more time, and return the next value, until there is no value to return.

The generator is considered empty once the function runs but does not hit yield anymore. It can be because the loop had come to an end, or because you do not satisfy a "if/else" anymore.

Your code explained

Generator:

# Here you create the method of the node object that will return the generator
def node._get_child_candidates(self, distance, min_dist, max_dist):

  # Here is the code that will be called each time you use the generator object:

  # If there is still a child of the node object on its left
  # AND if distance is ok, return the next child
  if self._leftchild and distance - max_dist < self._median:
                yield self._leftchild

  # If there is still a child of the node object on its right
  # AND if distance is ok, return the next child
  if self._rightchild and distance + max_dist >= self._median:
                yield self._rightchild

  # If the function arrives here, the generator will be considered empty
  # there is no more than two values: the left and the right children

Caller:

# Create an empty list and a list with the current object reference
result, candidates = list(), [self]

# Loop on candidates (they contain only one element at the beginning)
while candidates:

    # Get the last candidate and remove it from the list
    node = candidates.pop()

    # Get the distance between obj and the candidate
    distance = node._get_dist(obj)

    # If distance is ok, then you can fill the result
    if distance <= max_dist and distance >= min_dist:
        result.extend(node._values)

    # Add the children of the candidate in the candidates list
    # so the loop will keep running until it will have looked
    # at all the children of the children of the children, etc. of the candidate
    candidates.extend(node._get_child_candidates(distance, min_dist, max_dist))

return result

This code contains several smart parts:

The loop iterates on a list but the list expands while the loop is being iterated :-) It's a concise way to go through all these nested data even if it's a bit dangerous since you can end up with an infinite loop. In this case, candidates.extend(node._get_child_candidates(distance, min_dist, max_dist)) exhausts all the values of the generator, but while keeps creating new generator objects which will produce different values from the previous ones since it's not applied on the same node.
The extend() method is a list object method that expects an iterable and adds its values to the list.

Usually we pass a list to it:

>>> a = [1, 2]
>>> b = [3, 4]
>>> a.extend(b)
>>> print(a)
[1, 2, 3, 4]

But in your code it gets a generator, which is good because:

You don't need to read the values twice.
You can have a lot of children and you don't want them all stored in memory.

And it works because Python does not care if the argument of a method is a list or not. Python expects iterables so it will work with strings, lists, tuples and generators! This is called duck typing and is one of the reason why Python is so cool. But this is another story, for another question...

You can stop here, or read a little bit to see a advanced use of generator:

Controlling a generator exhaustion

>>> class Bank(): # let's create a bank, building ATMs
...    crisis = False
...    def create_atm(self):
...        while not self.crisis:
...            yield "$100"
>>> hsbc = Bank() # when everything's ok the ATM gives you as much as you want
>>> corner_street_atm = hsbc.create_atm()
>>> print(corner_street_atm.next())
$100
>>> print(corner_street_atm.next())
$100
>>> print([corner_street_atm.next() for cash in range(5)])
['$100', '$100', '$100', '$100', '$100']
>>> hsbc.crisis = True # crisis is coming, no more money!
>>> print(corner_street_atm.next())
<type 'exceptions.StopIteration'>
>>> wall_street_atm = hsbc.create_atm() # it's even true for new ATMs
>>> print(wall_street_atm.next())
<type 'exceptions.StopIteration'>
>>> hsbc.crisis = False # trouble is, even post-crisis the ATM remains empty
>>> print(corner_street_atm.next())
<type 'exceptions.StopIteration'>
>>> brand_new_atm = hsbc.create_atm() # build a new one to get back in business
>>> for cash in brand_new_atm:
...    print cash
$100
$100
$100
$100
$100
$100
$100
$100
$100
...

It can be useful for various things like controlling access to a resource.

Itertools, your best friend

The itertools module contains special functions to manipulate iterables. Ever wish to duplicate a generator?
Chain two generators? Group values in a nested list with a one liner? Map / Zip without creating another list?

Then just import itertools.

An example? Let's see the possible orders of arrival for a 4 horse race:

>>> horses = [1, 2, 3, 4]
>>> races = itertools.permutations(horses)
>>> print(races)
<itertools.permutations object at 0xb754f1dc>
>>> print(list(itertools.permutations(horses)))
[(1, 2, 3, 4),
 (1, 2, 4, 3),
 (1, 3, 2, 4),
 (1, 3, 4, 2),
 (1, 4, 2, 3),
 (1, 4, 3, 2),
 (2, 1, 3, 4),
 (2, 1, 4, 3),
 (2, 3, 1, 4),
 (2, 3, 4, 1),
 (2, 4, 1, 3),
 (2, 4, 3, 1),
 (3, 1, 2, 4),
 (3, 1, 4, 2),
 (3, 2, 1, 4),
 (3, 2, 4, 1),
 (3, 4, 1, 2),
 (3, 4, 2, 1),
 (4, 1, 2, 3),
 (4, 1, 3, 2),
 (4, 2, 1, 3),
 (4, 2, 3, 1),
 (4, 3, 1, 2),
 (4, 3, 2, 1)]

Understanding the inner mechanisms of iteration

Iteration is a process implying iterables (implementing the __iter__() method) and iterators (implementing the __next__() method).
Iterables are any objects you can get an iterator from. Iterators are objects that let you iterate on iterables.

More about it in this article about how does the for loop work.

source = Stackoverflow question

Python Quick Reference v2.7

2013-09-26T19:24:00-04:00

Hi there everyone. We all have experienced that time when we say "what was that function called ?", "was it called get_attr() or post_attr() ?" Recently I came across a wonderful Python Quick Reference for version 2.7 . It contains some useful information. I hope you will find it very useful just like me. So here is the link. Go forward and get it. You will not regret it. Do tell me in the comments that whether you like it or not or do you want something to be added to it. I would really love to hear your response. Let me make it clear as well that I am not the author of this great reference guide.

Download Python Quick Reference v2.7

source: Python Quick Ref

All about the metaclasses in Python!

2013-09-20T19:41:00-04:00

Hi there pythonistas. Recently I was searching about metaclasses in Python and came across a very good explanation about metaclasses on stackoverflow. I found the answer really helpful so I think you should read it as well. Who knows when you might find something useful in there.

Disclaimer: very long post.

Classes as objects

Before understanding metaclasses, you need to master classes in Python. And Python has a very peculiar idea of what classes are, borrowed from the Smalltalk language.

In most languages, classes are just pieces of code that describe how to produce an object. That's kinda true in Python too:

  >>> class ObjectCreator(object):
  ...       pass
  ...

  >>> my_object = ObjectCreator()
  >>> print(my_object)
  <__main__.ObjectCreator object at 0x8974f2c>

But classes are more than that in Python. Classes are objects too.

Yes, objects.

As soon as you use the keyword class, Python executes it and creates an OBJECT. The instruction

  >>> class ObjectCreator(object):
  ...       pass
  ...

creates in memory an object with the name ObjectCreator.

This object (the class) is itself capable of creating objects (the instances), and this is why it's a class.

But still, it's an object, and therefore:

you can assign it to a variable
you can copy it
you can add attributes to it
you can pass it as a function parameter

e.g.:

  >>> print(ObjectCreator) # you can print a class because it's an object
  <class '__main__.ObjectCreator'>
  >>> def echo(o):
  ...       print(o)
  ... 
  >>> echo(ObjectCreator) # you can pass a class as a parameter
  <class '__main__.ObjectCreator'>
  >>> print(hasattr(ObjectCreator, 'new_attribute'))
  False
  >>> ObjectCreator.new_attribute = 'foo' # you can add attributes to a class
  >>> print(hasattr(ObjectCreator, 'new_attribute'))
  True
  >>> print(ObjectCreator.new_attribute)
  foo
  >>> ObjectCreatorMirror = ObjectCreator # you can assign a class to a variable
  >>> print(ObjectCreatorMirror.new_attribute)
  foo
  >>> print(ObjectCreatorMirror())
  <__main__.ObjectCreator object at 0x8997b4c>

Creating classes dynamically

Since classes are objects, you can create them on the fly, like any object.

First, you can create a class in a function using class:

  >>> def choose_class(name):
  ...     if name == 'foo':
  ...         class Foo(object):
  ...             pass
  ...         return Foo # return the class, not an instance
  ...     else:
  ...         class Bar(object):
  ...             pass
  ...         return Bar
  ...     
  >>> MyClass = choose_class('foo') 
  >>> print(MyClass) # the function returns a class, not an instance
  <class '__main__.Foo'>
  >>> print(MyClass()) # you can create an object from this class
  <__main__.Foo object at 0x89c6d4c>

But it's not so dynamic, since you still have to write the whole class yourself.

Since classes are objects, they must be generated by something.

When you use the class keyword, Python creates this object automatically. But as with most things in Python, it gives you a way to do it manually.

Remember the function type? The good old function that lets you know what type an object is:

>>> print(type(1))
<type 'int'>
>>> print(type("1"))
<type 'str'>
>>> print(type(ObjectCreator))
<type 'type'>
>>> print(type(ObjectCreator()))
<class '__main__.ObjectCreator'>

Well, type has a completely different ability, it can also create classes on the fly. type can take the description of a class as parameters, and return a class.

(I know, it's silly that the same function can have two completely different uses
according to the parameters you pass to it. It's an issue due to backwards compatibility in Python)

type works this way:

  type(name of the class, 
       tuple of the parent class (for inheritance, can be empty), 
       dictionary containing attributes names and values)

e.g.:

>>> class MyShinyClass(object):
...       pass

can be created manually this way:

  >>> MyShinyClass = type('MyShinyClass', (), {}) # returns a class object
  >>> print(MyShinyClass)
  <class '__main__.MyShinyClass'>
  >>> print(MyShinyClass()) # create an instance with the class
  <__main__.MyShinyClass object at 0x8997cec>

You'll notice that we use "MyShinyClass" as the name of the class and as the variable to hold the class reference. They can be different, but there is no reason to complicate things.

type accepts a dictionary to define the attributes of the class. So:

>>> class Foo(object):
...       bar = True

Can be translated to:

  >>> Foo = type('Foo', (), {'bar':True})

And used as a normal class:

  >>> print(Foo)
  <class '__main__.Foo'>
  >>> print(Foo.bar)
  True
  >>> f = Foo()
  >>> print(f)
  <__main__.Foo object at 0x8a9b84c>
  >>> print(f.bar)
  True

And of course, you can inherit from it, so:

  >>>   class FooChild(Foo):
  ...         pass

would be:

  >>> FooChild = type('FooChild', (Foo,), {})
  >>> print(FooChild)
  <class '__main__.FooChild'>
  >>> print(FooChild.bar) # bar is inherited from Foo
  True

Eventually you'll want to add methods to your class. Just define a function
with the proper signature and assign it as an attribute.

>>> def echo_bar(self):
...       print(self.bar)
... 
>>> FooChild = type('FooChild', (Foo,), {'echo_bar': echo_bar})
>>> hasattr(Foo, 'echo_bar')
False
>>> hasattr(FooChild, 'echo_bar')
True
>>> my_foo = FooChild()
>>> my_foo.echo_bar()
True

You see where we are going: in Python, classes are objects, and you can create a class on the fly, dynamically.

This is what Python does when you use the keyword class, and it does so by using a metaclass.

What are metaclasses (finally)

Metaclasses are the 'stuff' that creates classes.

You define classes in order to create objects, right?

But we learned that Python classes are objects.

Well, metaclasses are what create these objects. They are the classes' classes,
you can picture them this way:

  MyClass = MetaClass()
  MyObject = MyClass()

You've seen that type lets you do something like this:

  MyClass = type('MyClass', (), {})

It's because the function type is in fact a metaclass. type is the metaclass Python uses to create all classes behind the scenes.

Now you wonder why the heck is it written in lowercase, and not Type?

Well, I guess it's a matter of consistency with str, the class that creates strings objects, and int the class that creates integer objects. type is just the class that creates class objects.

You see that by checking the __class__ attribute.

Everything, and I mean everything, is an object in Python. That includes ints, strings, functions and classes. All of them are objects. And all of them have been created from a class:

  >>> age = 35
  >>> age.__class__
  <type 'int'>
  >>> name = 'bob'
  >>> name.__class__
  <type 'str'>
  >>> def foo(): pass
  >>> foo.__class__
  <type 'function'>
  >>> class Bar(object): pass
  >>> b = Bar()
  >>> b.__class__
  <class '__main__.Bar'>

Now, what is the __class__ of any __class__ ?

  >>> age.__class__.__class__
  <type 'type'>
  >>> name.__class__.__class__
  <type 'type'>
  >>> foo.__class__.__class__
  <type 'type'>
  >>> b.__class__.__class__
  <type 'type'>

So, a metaclass is just the stuff that creates class objects.

You can call it a 'class factory' if you wish.

type is the built-in metaclass Python uses, but of course, you can create your own metaclass.

The `metaclass` attribute

You can add a __metaclass__ attribute when you write a class:

class Foo(object):
  __metaclass__ = something...
  [...]

If you do so, Python will use the metaclass to create the class Foo.

Careful, it's tricky.

You write class Foo(object) first, but the class object Foo is not created in memory yet.

Python will look for __metaclass__ in the class definition. If it finds it, it will use it to create the object class Foo. If it doesn't, it will use type to create the class.

Read that several times.

When you do:

class Foo(Bar):
  pass

Python does the following:

Is there a __metaclass__ attribute in Foo?

If yes, create in memory a class object (I said a class object, stay with me here), with the name Foo by using what is in __metaclass__.

If Python can't find __metaclass__, it will look for a __metaclass__ in Bar (the parent class), and try to do the same.

If Python can't find __metaclass__ in any parent, it will look for
a __metaclass__ at the MODULE level, and try to do the same.

Then if it can't find any __metaclass__ at all, it will use type to create the class object.

Now the big question is, what can you put in __metaclass__ ?

The answer is: something that can create a class.

And what can create a class? type, or anything that subclasses or uses it.

Custom metaclasses

The main purpose of a metaclass is to change the class automatically, when it's created.

You usually do this for APIs, where you want to create classes matching the current context.

Imagine a stupid example, where you decide that all classes in your module should have their attributes written in uppercase. There are several ways to do this, but one way is to set __metaclass__ at the module level.

This way, all classes of this module will be created using this metaclass, and we just have to tell the metaclass to turn all attributes to uppercase.

Luckily, __metaclass__ can actually be any callable, it doesn't need to be a formal class (I know, something with 'class' in its name doesn't need to be a class, go figure... but it's helpful).

So we will start with a simple example, by using a function.

# the metaclass will automatically get passed the same argument
# that you usually pass to `type`
def upper_attr(future_class_name, future_class_parents, future_class_attr):
  """
    Return a class object, with the list of its attribute turned 
    into uppercase.
  """

  # pick up any attribute that doesn't start with '__' and uppercase it
  uppercase_attr = {}
  for name, val in future_class_attr.items():
      if not name.startswith('__'):
          uppercase_attr[name.upper()] = val
      else:
          uppercase_attr[name] = val

  # let `type` do the class creation
  return type(future_class_name, future_class_parents, uppercase_attr)

__metaclass__ = upper_attr # this will affect all classes in the module

class Foo(): # global __metaclass__ won't work with "object" though
  # but we can define __metaclass__ here instead to affect only this class
  # and this will work with "object" children
  bar = 'bip'

print(hasattr(Foo, 'bar'))
# Out: False
print(hasattr(Foo, 'BAR'))
# Out: True

f = Foo()
print(f.BAR)
# Out: 'bip'

Now, let's do exactly the same, but using a real class for a metaclass:

# remember that `type` is actually a class like `str` and `int`
# so you can inherit from it
class UpperAttrMetaclass(type): 
    # __new__ is the method called before __init__
    # it's the method that creates the object and returns it
    # while __init__ just initializes the object passed as parameter
    # you rarely use __new__, except when you want to control how the object
    # is created.
    # here the created object is the class, and we want to customize it
    # so we override __new__
    # you can do some stuff in __init__ too if you wish
    # some advanced use involves overriding __call__ as well, but we won't
    # see this
    def __new__(upperattr_metaclass, future_class_name, 
                future_class_parents, future_class_attr):

        uppercase_attr = {}
        for name, val in future_class_attr.items():
            if not name.startswith('__'):
                uppercase_attr[name.upper()] = val
            else:
                uppercase_attr[name] = val

        return type(future_class_name, future_class_parents, uppercase_attr)

But this is not really OOP. We call type directly and we don't override call the parent __new__. Let's do it:

class UpperAttrMetaclass(type):

    def __new__(upperattr_metaclass, future_class_name, 
                future_class_parents, future_class_attr):

        uppercase_attr = {}
        for name, val in future_class_attr.items():
            if not name.startswith('__'):
                uppercase_attr[name.upper()] = val
            else:
                uppercase_attr[name] = val

        # reuse the type.__new__ method
        # this is basic OOP, nothing magic in there
        return type.__new__(upperattr_metaclass, future_class_name, 
                            future_class_parents, uppercase_attr)

You may have noticed the extra argument upperattr_metaclass. There is nothing special about it: a method always receives the current instance as first parameter. Just like you have self for ordinary methods.

Of course, the names I used here are long for the sake of clarity, but like for self, all the arguments have conventional names. So a real production metaclass would look like this:

class UpperAttrMetaclass(type):

    def __new__(cls, clsname, bases, dct):

        uppercase_attr = {}
        for name, val in dct.items():
            if not name.startswith('__'):
                uppercase_attr[name.upper()] = val
            else:
                uppercase_attr[name] = val

        return type.__new__(cls, clsname, bases, uppercase_attr)

We can make it even cleaner by using super, which will ease inheritance (because yes, you can have metaclasses, inheriting from metaclasses, inheriting from type):

class UpperAttrMetaclass(type):

    def __new__(cls, clsname, bases, dct):

        uppercase_attr = {}
        for name, val in dct.items():
            if not name.startswith('__'):
                uppercase_attr[name.upper()] = val
            else:
                uppercase_attr[name] = val

        return super(UpperAttrMetaclass, cls).__new__(cls, clsname, bases, uppercase_attr)

That's it. There is really nothing more about metaclasses.

The reason behind the complexity of the code using metaclasses is not because of metaclasses, it's because you usually use metaclasses to do twisted stuff relying on introspection, manipulating inheritance, vars such as __dict__, etc.

Indeed, metaclasses are especially useful to do black magic, and therefore complicated stuff. But by themselves, they are simple:

intercept a class creation
modify the class
return the modified class

Why would you use metaclasses classes instead of functions?

Since __metaclass__ can accept any callable, why would you use a class since it's obviously more complicated?

There are several reasons to do so:

The intention is clear. When you read UpperAttrMetaclass(type), you know
what's going to follow
You can use OOP. Metaclass can inherit from metaclass, override parent methods.
Metaclasses can even use metaclasses.
You can structure your code better. You never use metaclasses for something as
trivial as the above example. It's usually for something complicated. Having the ability to make several methods and group them in one class is very useful to make the code easier to read.
You can hook on __new__, __init__ and __call__. Which will allow you to do different stuff. Even if usually you can do it all in __new__,
some people are just more comfortable using __init__.
These are called metaclasses, damn it! It must mean something!

Why the hell would you use metaclasses?

Now the big question. Why would you use some obscure error prone feature?

Well, usually you don't:

Metaclasses are deeper magic than
99% of users should ever worry about.
If you wonder whether you need them,
you don't (the people who actually
need them know with certainty that
they need them, and don't need an
explanation about why).

Python Guru Tim Peters

The main use case for a metaclass is creating an API. A typical example of this is the Django ORM.

It allows you to define something like this:

  class Person(models.Model):
    name = models.CharField(max_length=30)
    age = models.IntegerField()

But if you do this:

  guy = Person(name='bob', age='35')
  print(guy.age)

It won't return an IntegerField object. It will return an int, and can even take it directly from the database.

This is possible because models.Model defines __metaclass__ and it uses some magic that will turn the Person you just defined with simple statements into a complex hook to a database field.

Django makes something complex look simple by exposing a simple API and using metaclasses, recreating code from this API to do the real job behind the scenes.

The last word

First, you know that classes are objects that can create instances.

Well in fact, classes are themselves instances. Of metaclasses.

  >>> class Foo(object): pass
  >>> id(Foo)
  142630324

Everything is an object in Python, and they are all either instances of classes or instances of metaclasses.

Except for type.

type is actually its own metaclass. This is not something you could
reproduce in pure Python, and is done by cheating a little bit at the implementation
level.

Secondly, metaclasses are complicated. You may not want to use them for very simple class alterations. You can change classes by using two different techniques:

monkey patching
class decorators

99% of the time you need class alteration, you are better off using these.

But 99% of the time, you don't need class alteration at all :-)

source : What is a metaclass in python (stackoverflow)

Making a Reddit + twitter Bot

2013-09-14T19:45:00-04:00

Hi there pythonistas. I hope you are all fine. In this post I am going to teach you how we can make a Reddit + Twitter bot. What this bot will do is that it will copy post titles and url from any subreddit that you want and then it will post them to twitter keeping the 140 characters length in mind. Firstly let me tell you what Reddit is. Reddit is a social link sharing site where good links are upvoted and bad links are down voted. So lets start.

Required External Libraries
1. Praw - (Reddidt API wrapper in python) pip install praw
2. Requests - (HTTP library) pip install requests
3. Tweepy - (Twitter python API) pip install tweepy

First step - Register Yourself
So first of all you will have to register an app on http://dev.twitter.com/apps and after registering copy the

access token
access token secret
consumer key
consumer secret

You will have to edit the permissions for your app under the settings tab and grant your application read and write permission. So now we are ready to move on.

Start by importing
So now lets start writing our script. First of all we will have to import the required libraries and set up some basic variables:

import praw
import json
import requests
import tweepy
import time

access_token = 'YOUR ACCESS TOKEN HERE'
access_token_secret = 'YOUR ACCESS TOKEN SECRET HERE'
consumer_key = 'YOUR CONSUMER KEY HERE'
consumer_secret = 'YOUR CONSUMER SECRET HERE'

Initiate connection with Reddit
Now we have to initiate connection with Reddit. Lets define a function to do just that.

def setup_connection_reddit(subreddit):
    print "[bot] setting up connection with Reddit"
    r = praw.Reddit('yasoob_python reddit twitter bot '
                'monitoring %s' %(subreddit)) 
    subreddit = r.get_subreddit(subreddit)
    return subreddit

This method connects with Reddit and gets the subreddit of our choice and then returns that subreddit for us to work further with.

Get posts and links from Reddit
So now we have to define a function that gets the list of posts and there urls from Reddit for us. So lets just do that as well.

def tweet_creator(subreddit_info):
    post_dict = {}
    post_ids = []
    print "[bot] Getting posts from Reddit"
    for submission in subreddit_info.get_hot(limit=20):
                # strip_title function is defined later
        post_dict[strip_title(submission.title)] = submission.url
        post_ids.append(submission.id)
    print "[bot] Generating short link using goo.gl"
    mini_post_dict = {}
    for post in post_dict:
        post_title = post
        post_link = post_dict[post]   
                # the shorten function is defined later     
        short_link = shorten(post_link)
        mini_post_dict[post_title] = short_link 
    return mini_post_dict, post_ids

First of all we are declaring a dictionary to hold the post title and link and after that we are making a list to hold the unique ids of every post that we grab. This is used to track which posts we have already grabbed. After that we are looping over the posts and appending values to the dictionary and the list. If you use twitter very frequently then you know that how disgusting long links look like so in order to tackle that we are using goo.gl to generate short links for us. That's the next thing we have done in the above function. We loop over the post dict and make a short link for every link and append it to a new dictionary which is mini_post_dict.

No one likes long links
Now lets define a function which will actually shorten the links for us. So here it is:

def shorten(url):
    headers = {'content-type': 'application/json'}
    payload = {"longUrl": url}
    url = "https://www.googleapis.com/urlshortener/v1/url"
    r = requests.post(url, data=json.dumps(payload), headers=headers)
    link = json.loads(r.text)['id']
    return link

The above function contains a header and a payload which we are going to send to google and after that google will return the short link in the form of json.

Twitter hates more than 140 characters
If you use twitter regularly then I am sure that you know that twitter does not like tweets that are more than 140 characters. So in order to tackle that lets define a function that will truncate long tweets to short ones.

def strip_title(title):
    if len(title) < 94:
        return title
    else:
        return title[:93] + "..."

In the above method we will pass a title and the above method will check that whether the title is 93 characters or more. If it is more than 93 characters then it will truncate it and append three dots at its end.

Dealing with duplicate posts
So now we have started to shape our final script. There is one thing that we have to keep in mind. No one likes duplicate posts so we have to make sure that we do not post same tweets over and over again. In order to tackle this issue we are going to make a file with the name of posted_posts.txt . When ever we grab a post from Reddit we will add it's ID to this file and when posting to twitter we will check whether the post with this ID has already been posted or not. Lets define two more functions. The first one will write the IDs to file and the second one will check whether the post is already posted or not.

def add_id_to_file(id):
    with open('posted_posts.txt', 'a') as file:
        file.write(str(id) + "\n")

def duplicate_check(id):
    found = 0
    with open('posted_posts.txt', 'r') as file:
        for line in file:
            if id in line:
                found = 1
    return found

Make a function for twitter will ya
So now lets make our one of the main function. This function is actually going to post to twitter.

def tweeter(post_dict, post_ids):
    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
    auth.set_access_token(access_token, access_token_secret)
    api = tweepy.API(auth)
    for post, post_id in zip(post_dict, post_ids):
        found = duplicate_check(post_id)
        if found == 0:
            print "[bot] Posting this link on twitter"
            print post+" "+post_dict[post]+" #Python #reddit #bot"
            api.update_status(post+" "+post_dict[post]+" #Python #reddit #bot")
            add_id_to_file(post_id)
            time.sleep(30)
        else:
            print "[bot] Already posted"

Firstly we setup connection with twitter by using the credentials we defined in the beginning. After that we loop over the post_dict and post_ids. Then we check for duplicate posting. If it is not previously posted then we post it and add the id of the post in the posted_posts.txt file. After posting we wait for 30 seconds so that we do not spam twitter with tweets.

Wheres the main function bud
So lets define our last function. This function will co ordinate with all other functions. Here is the code for that last function:

def main():
    subreddit = setup_connection_reddit('python')
    post_dict, post_ids = tweet_creator(subreddit)
    tweeter(post_dict, post_ids)

So now we are ready just add this little line at the end as well:

if __name__ == '__main__':
    main()

This checks whether the script is directly executed or is it imported. If it is directly executed only then the main() function is executed.

Complete code
Here is the complete script:

import praw
import json
import requests
import tweepy
import time

access_token = 'YOUR ACCESS TOKEN HERE'
access_token_secret = 'YOUR ACCESS TOKEN SECRET HERE'
consumer_key = 'YOUR CONSUMER KEY HERE'
consumer_secret = 'YOUR CONSUMER SECRET HERE'

def strip_title(title):
    if len(title) < 94:
        return title
    else:
        return title[:93] + "..."

def tweet_creator(subreddit_info):
    post_dict = {}
    post_ids = []
    print "[bot] Getting posts from Reddit"
    for submission in subreddit_info.get_hot(limit=20):
        post_dict[strip_title(submission.title)] = submission.url
        post_ids.append(submission.id)
    print "[bot] Generating short link using goo.gl"
    mini_post_dict = {}
    for post in post_dict:
        post_title = post
        post_link = post_dict[post]         
        short_link = shorten(post_link)
        mini_post_dict[post_title] = short_link 
    return mini_post_dict, post_ids

def setup_connection_reddit(subreddit):
    print "[bot] setting up connection with Reddit"
    r = praw.Reddit('yasoob_python reddit twitter bot '
                'monitoring %s' %(subreddit)) 
    subreddit = r.get_subreddit(subreddit)
    return subreddit

def shorten(url):
    headers = {'content-type': 'application/json'}
    payload = {"longUrl": url}
    url = "https://www.googleapis.com/urlshortener/v1/url"
    r = requests.post(url, data=json.dumps(payload), headers=headers)
    link = json.loads(r.text)['id']
    return link

def duplicate_check(id):
    found = 0
    with open('posted_posts.txt', 'r') as file:
        for line in file:
            if id in line:
                found = 1
    return found

def add_id_to_file(id):
    with open('posted_posts.txt', 'a') as file:
        file.write(str(id) + "\n")

def main():
    subreddit = setup_connection_reddit('python')
    post_dict, post_ids = tweet_creator(subreddit)
    tweeter(post_dict, post_ids)

def tweeter(post_dict, post_ids):
    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
    auth.set_access_token(access_token, access_token_secret)
    api = tweepy.API(auth)
    for post, post_id in zip(post_dict, post_ids):
        found = duplicate_check(post_id)
        if found == 0:
            print "[bot] Posting this link on twitter"
            print post+" "+post_dict[post]+" #Python #reddit #bot"
            api.update_status(post+" "+post_dict[post]+" #Python #reddit #bot")
            add_id_to_file(post_id)
            time.sleep(30)
        else:
            print "[bot] Already posted"

if __name__ == '__main__':
    main()

Save this file with the name of reddit_bot.py and make a file with the name of posted_posts.txt and then execute the python script from the terminal. Your output will look something like this:

yasoob@yasoob:~/Desktop$ python reddit_bot.py
[bot] setting up connection with Reddit
[bot] Getting posts from Reddit
[bot] Generating short link using goo.gl
[bot] Posting this link on twitter
Miloslav Trmač, -1 for Structured Logging http://goo.gl/sF8Xgm #Python #reddit #bot

And after some time your posted_posts.txt file will look something like this:

1mb4y4
1mb867
1mb4hl
1mbh3t
1mbni0
1m9bod
1mbhpt
1mbhnc
1mbcp2
1m9d2t
1maeio
1m9bi5
1m8tgr
1m86e4
1ma5r5
1m8fud
1mdh1t
1mbst4

Goodbye
I hope you enjoyed today's post as much as I enjoyed writing it. I hope to see you in future with some more tutorials. Do follow my blog to give me some support and get regular updates. Goodbye till next time.

A surprize for you

2013-09-09T19:10:00-04:00

Hi there pythonistas. You might have noticed that I haven't been active much on this blog. There's a reason for that. I was preparing a PyCon talk. Yes I was really preparing a PyCon talk. The title of the talk is web scraping in python. I have submitted the proposal and now I am waiting for it's acceptance. Do pray for it's acceptance and after that you will be able to see me in Montreal next year ;) . I will keep you updated with the status of my talk that whether it is accepted or not.

Making a twitter bot in python

2013-09-04T21:53:00-04:00

Hi there guys. I hope you are fine. In this post I am going to tell you how you can make a twitter auto favouriter or a.k.a Twitter Bot. I am not going to make a twitter follower as it will get you banned quickly if you use it a lot. This post is going to help you to increase your twitter followers organically. By organically I mean that your followers will be those who really want to listen to your content. You can either use services on the web to increase your followers inorganically or else you can use this script ;) . So what affect does favouriting has on the user who's post you favourite ? When you favourite someone's post he will get curious as who favourited his post. Then he will visit your profile and will follow you if he likes your tweets. So without wasting any time lets begin.

The basic mechanism or the way of working of our bot will be that you will give it a keyword like "python" and it will search twitter for that keyword. After searching it will start favouriting the tweets from top to bottom.

First of all I am going to use "twitter" module for this project. There are a lot of twitter related modules on PyPI but this module did the trick so I thought to use it (https://pypi.python.org/pypi/twitter/1.10.0) . Secondly in order to follow along you will have to make a dev account on twitter and register an application with any name (https://dev.twitter.com/apps). Lets begin by importing the twitter module and initiating connection with twitter using the OAUTH tokens. :

from twitter import Twitter, OAuth, TwitterHTTPError

OAUTH_TOKEN = 'your oauth token'
OAUTH_SECRET = 'your oauth secret'
CONSUMER_KEY = 'your consumer key'
CONSUMER_SECRET = 'your consumer secret'

t = Twitter(auth=OAuth(OAUTH_TOKEN, OAUTH_SECRET,
            CONSUMER_KEY, CONSUMER_SECRET))

So now we have initiated connection with twitter. This is the basic structure of our twitter auto favouriter or a.k.a twitter bot. Now lets move on and define a method which will search for tweets on twitter.

def search_tweets(q, count=100):
    return t.search.tweets(q=q, result_type='recent', count=count)

So now we have to define a method which will favourite a tweet:

def fav_tweet(tweet):
    try:
        result = t.favorites.create(_id=tweet['id'])
        print "Favorited: %s" % (result['text'])
        return result
    # when you have already favourited a tweet
    # this error is thrown
    except TwitterHTTPError as e:
        print "Error: ", e
        return None

So now we have a searching method and a favouriting method. Now tell me what is left ? Obviously the main code that will drive our whole program is left. So here you go:

def auto_fav(q, count=100):
    result = search_tweets(q, count)
    a = result['statuses'][0]['user']['screen_name']
    print a
    success = 0
    for tweet in result['statuses']:
        if fav_tweet(tweet) is not None:
            success += 1
    print "We Favorited a total of %i out of %i tweets" % (success,
          len(result['statuses']))

So thats it! We have a complete bot for twitter. This bot currently only favourites tweets based on a keyword. Save this code in a file called twitter_bot.py . Now you can use this bot by typing this in the terminal:

$ python

>>> import twitter_bot
>>> twitter_bot.auto_fav('#python','20')

The above command will search for 20 latest tweets containing the hashtag "python". After that it will favourite all of those tweets. This bot itself works but I can not gurantee that you will get followers quickly and I can not even gurantee that you will not get banned by twitter for using this a lot. You can use this once in a day. Hopefully I will write a twitter follower bot in the future as well but till then this script is your best bet.

This is the complete script:

from twitter import Twitter, OAuth, TwitterHTTPError

OAUTH_TOKEN = 'your oauth token'
OAUTH_SECRET = 'your oauth secret'
CONSUMER_KEY = 'your consumer key'
CONSUMER_SECRET = 'your consumer secret'

t = Twitter(auth=OAuth(OAUTH_TOKEN, OAUTH_SECRET,
            CONSUMER_KEY, CONSUMER_SECRET))

def search_tweets(q, count=100):
    return t.search.tweets(q=q, result_type='recent', count=count)

def fav_tweet(tweet):
    try:
        result = t.favorites.create(_id=tweet['id'])
        print "Favorited: %s" % (result['text'])
        return result
    # when you have already favourited a tweet
    # this error is thrown
    except TwitterHTTPError as e:
        print "Error: ", e
        return None

def auto_fav(q, count=100):
    result = search_tweets(q, count)
    a = result['statuses'][0]['user']['screen_name']
    print a
    success = 0
    for tweet in result['statuses']:
        if fav_tweet(tweet) is not None:
            success += 1
    print "We Favorited a total of %i out of %i tweets" % (success,
          len(result['statuses']))

This is the final result of running this sript:

yasoob@yasoob:~/Desktop$ python
Python 2.7.3 (default, Apr 10 2013, 06:20:15) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import twitter_bot as bot
>>> bot.auto_fav('django',1)
Favorited: Enjoying @freakboy3742’s discussion of cultural naming patterns vis a vis Django users at #DjangoCon
We Favorited a total of 1 out of 1 tweets
>>>

I hope you liked todays post. If you have any questions then feel free to comment below and do follow us if you want to get regular updates from our blog. If you want me to write on one specific topic then do tell it to me in the comments below. Stay tuned for my next post.

Python Resources

2013-09-01T15:40:00-04:00

Beginner's Delight

Beginner's guide to Python
The Hitchhiker's guide to Python
Learn Python the hard way
Google's Python class
Python for Beginners
Python tracks at Codecademy
Python Monk
Best way to learn python
Python Tips, tools and resources
Python Koans: Learn Python through TDD
LearnStreet Python course
Beginner projects for a python newbie
Learn X in Y minutes - Python
Python Practice book
Facts and myths about Python names and values
[args and *kwargs in python explained][]
Common newbie mistakes in Python- Part 1
Common newbie mistakes in Python- Part 2
PySchools
Some cool Python Tricks
http://blog.amir.rachum.com/post/30176371115/you-cant-handle-the-truth
Better Python APIs
Python Gotchas
Declaring dependencies in Python

Style Guide and Idioms

Dictionary

Decorators

Generators

Generator Tricks for Systems Programmers

Yield

Python yield keyword explained

Context Managers

Understanding Python's with statement

Unicode

Metaclasses

Documentation

Sphinx

Quick Sphinx documentation for Python

Debugging

Debugging Tools in Python

Testing

Profiling

A guide to analyzing Python performance

Packaging

Deployment

Fabric

Using Fabric in a development environment

Web

Python and Realtime Web

Frameworks

Flask

Scraping

Web Scraping with BeautifulSoup
Web Scraping 101 with Python
[How to automatically search and download torrents with Python and Scrapy][]

Mobile Development

Kivy

Creating an app in Kivy

Image Processing

http://www.riisen.dk/dop/pil.html

WSGI

Databases

SQLAlchemy

http://www.aosabook.org/en/sqlalchemy.html
http://lucumr.pocoo.org/2011/7/19/sqlachemy-and-you/

Data Mining

Linux system mining using Python

Data Analysis

Pandas

A 10 min intro to Pandas
http://manishamde.github.io/blog/2013/03/07/pandas-and-python-top-10/

Design Patterns

Python Best Practice Patterns by Vladimir Keleshev

Concurrency Patterns

Distributed Systems

http://architects.dzone.com/articles/gevent-zeromq

Python 2 vs 3

Python 2 or 3 ?

Porting to Python 3

http://python3porting.com/

Books

Free

Some free books on Python
Invent with Python
Think Python
Byte of Python
Snake Wrangling for Kids
[Data Structures and Algorithms with Object-Oriented Design Patterns in Python][]
Python programming on wikibooks
http://readwrite.com/2011/03/25/python-is-an-increasingly-popu#awesm=\~oalneJOKY2tp7U
Dive Into Python 3

Paid

Some list of paid books on python

Discussions

https://news.ycombinator.com/item?id=3746692
https://news.ycombinator.com/item?id=5944863
http://stackoverflow.com/questions/92230/python-beyond-the-basics
http://stackoverflow.com/questions/2573135/python-progression-path-from-apprentice-to-guru
http://stackoverflow.com/questions/228181/zen-of-python
http://stackoverflow.com/questions/101268/hidden-features-of-python

Conferences and events

Official PyCon website
http://lanyrd.com/topics/python/
http://lanyrd.com/topics/django/
http://python.meetup.com/
http://django.meetup.com/

Videos

Guido Van Rossum on the history of Python
PyVideo
http://lanyrd.com/topics/python/video/
http://showmedo.com/videotutorials/python
Talks by Brandon Rhodes

Editors and IDE's for Python programming

Miscellaneous

source: pycrumbs github

[How to automatically search and download torrents with Python and Scrapy]: http://fuzz-box.blogspot.in/2013/03/how-to-automatically-search-download-torrent-python-scrapy.html [Data Structures and Algorithms with Object-Oriented Design Patterns in Python]: http://www.brpreiss.com/books/opus7/html/book.html

Simple Sudoku solver in python

2013-09-01T01:43:00-04:00

Hi there pythonistas! We all know that Sudoku is a great game. Some of us even bet on this game but did you know that you can use python to make a Sudoku solver ? In this post I am going to share with you a Sudoku solver written in python. From now on you will win all Sudoku challenges. However let me tell you that I am not the original writer of this script. I stumbled over this script on stack-overflow. So without wasting any time let me share the script with you:

import sys

def same_row(i,j): return (i/9 == j/9)
def same_col(i,j): return (i-j) % 9 == 0
def same_block(i,j): return (i/27 == j/27 and i%9/3 == j%9/3)

def r(a):
  i = a.find('0')
  if i == -1:
    sys.exit(a)

  excluded_numbers = set()
  for j in range(81):
    if same_row(i,j) or same_col(i,j) or same_block(i,j):
      excluded_numbers.add(a[j])

  for m in '123456789':
    if m not in excluded_numbers:
      r(a[:i]+m+a[i+1:])

if __name__ == '__main__':
  if len(sys.argv) == 2 and len(sys.argv[1]) == 81:
    r(sys.argv[1])
  else:
    print 'Usage: python sudoku.py puzzle'
    print '  where puzzle is an 81 character string 
             representing the puzzle read left-to-right,
             top-to-bottom, and 0 is a blank'

Hey there wait! let me share with you a shorter obfuscated version of the same Sudoku solving script. However this short version of Sudoku solver is a lot slower but I think that I should share it with you just to show you that even in python obfuscated code can be written. So here is the shorter obfuscated version:

def r(a):i=a.find('0');~i or exit(a);[m
in[(i-j)%9*(i/9^j/9)*(i/27^j/27|i%9/3^j%9/3)or a[j]for
j in range(81)]or r(a[:i]+m+a[i+1:])for m in'%d'%5**18]
from sys import*;r(argv[1])

If you liked this post about Sudoku solver then don't forget to share this on twitter and facebook. If you want to get regular updates then don't forget to follow our blog.

Fixing error - maximum recursion depth reached

2013-08-31T00:35:00-04:00

Hi there folks. In this post I am going to teach you how to increase the recursion depth in python. Most of us at some time get this error :

RuntimeError: maximum recursion depth exceeded

If you want to fix this error just increase the default recursion depth limit but how to do it ? Just import the sys module in your script and in the beginning of the script type this :

sys.setrecursionlimit(1500)

This will increase the default limit to 1500. For the record the default limit is 1000. I hope that you found this post useful. Do share this on facebook and twitter and stay tuned for our next post.

Looking Inside Dropbox - whitepaper

2013-08-30T14:53:00-04:00

Hi there folks. We all know that dropbox is made in python. It's website is made with pyramid and it's desktop applications are made with python. The existing Python bytecode reversing techniques are not enough for reversing hardened applications like Dropbox. Recently I came accross a whitepaper written by Dhiru Kholia and Przemysław Wegrzyn. This paper presents new and generic techniques, to reverse engineer frozen Python applications, which are not limited to just the Dropbox world. It describes a method to bypass Dropbox's two factor authentication and hijack Dropbox accounts. Additionally, generic techniques to intercept SSL data using code injection techniques and monkey patching are presented. The methods presented in this whitepaper are not limited to dropbox.

You can download this whitepaper from usenix website.

Do share your views after reading this whitepaper and don't hesitate following us in order to get regular updates.

Finding the module version

2013-08-28T08:57:00-04:00

Hi there folks. I know I have not been active recently and it was because I was not in town. I was in Canada. So this post will be short. In this post i will show you how you can find the version number of any python module. Sometimes you need to find the version number in order to know whether you have the desired version of the module or not. So here is the trick.

\1. First of all launch python:

$ python

\2. Import the module which in our case is requests:

>>> import requests

\3. Lastly type this:

>>> requests.__version__
'1.2.3'

So I hope you got the trick. If you have any questions then feel free to post them below and stay tuned for the next post.

Extracting pixel values of an image in python

2013-08-28T08:38:00-04:00

This is a very nice post which teaches you some of the basics of PIL (python imaging library) . Make sure that you read it if you are interested in images.

Making a song downloader with python

2013-08-13T22:50:00-04:00

Hi there folks. Some time ago i was very much interested in how music downloaders for various websites worked. In order to learn this i went forward to make my own downloader for ex.fm . Currently ex.fm does not allow users to download songs so this was a very good chance for me to learn song downloading with python. During this process i also came forward to youtube-dl. It is a python script which allows anyone to download music and videos from numerous websites. Now let me tell you how i went forward to make this downloader.

First of all I opened ex.fm in chrome and also started chrome web inspector to intercept network calls. Then i clicked on a song for it to play. When the song started to play i saw the XHR requests made by ex.fm . There i saw a request being made to ex.fm/api . This was interesting because most websites use apis to provide info for specific songs. I opened the link and saw that the link returned json data with direct link to the song. That way i came to know the url used by ex.fm to get the direct link to the song.

After that I wanted to know how to automate song searching from python. For this i again opened ex.fm and chrome web inspector. This time instead of playing a song i clicked on search and searched for a song. Again a url got my attention. It was http://ex.fm/api/v3/song/search/{search term}?start=0&results=20 . This url also returned json response. After i became aware of these two urls i started to write a python script to automate this. I used only three libraries and all three of them come preinstalled with python (batteries included). These three libraries were urllib2 json sys I won't be explaining the code here instead i will give you the code itself. If you have any questions about the code feel free to comment below. Lastly let me make it clear that this script can be written in better ways but i just made it for learning purposes. [Here is the link][] to the github repository. [Here is the link]: https://github.com/yasoob/ex.fm-dl

Software Spotlight: Bitmessage

2013-08-13T22:15:00-04:00

One of the latest application made with python. Definitely worth checking out. Do share your views in the comments below.

Storing and Loading Data with JSON

2013-08-08T04:55:00-04:00

We've already learned about pickle, so why do we need another way to (de)serialize Python objects to(from) disk or a network connection? There are three major reasons to prefer JSON over pickle:

When you're unpickling data, you're essentially allowing your data source to execute arbitrary Python commands. If the data is trustworthy (say stored in a sufficiently protected directory), that may not be a problem, but it's often really easy to accidentally leave a file unprotected (or read something from network). In these cases, you want to load data, and not execute potentially malicious Python code!
Pickled data is not easy to read, and virtually impossible to write for humans. For example, the pickled version of {"answer": [42]} looks like this:
```
(dp0
S'answer'
p1
(lp2
I42
as.
```

In contrast, the JSON representation of {"answer": [42]} is .... {"answer": [42]}. If you can read Python, you can read JSON; since all JSON is valid Python code!

Pickle is Python-specific. In fact, by default, the bytes generated by Python 3's pickle cannot be read by a Python 2.x application! JSON can be read by virtually any programming language - just scroll down on the official homepage to see implementations in all major and some minor languages.

So how do you get the JSON representation of an object? It's simple, just call [json.dumps][]:

import json
obj = {u"answer": [42.2], u"abs": 42}
print(json.dumps(obj))
# output:  {"answer": [42.2], "abs": 42}

Often, you want to write to a file or network stream. In both Python 2.x and 3.x you can call [dump][] to do that, but in 3.x the output must be a character stream, whereas 2.x expects a byte stream.

Let's look how to load what we wrote. Fittingly, the function to load is called loads (to load from a string) / load (to load from a stream):

import json
obj_json = u'{"answer": [42.2], "abs": 42}'
obj = json.loads(obj_json)
print(repr(obj))

When the objects we load and store grow larger, we puny humans often need some hints on where a new sub-object starts. To get these, simply pass an indent size, like this:

import json
obj = {u"answer": [42.2], u"abs": 42}
print(json.dumps(obj, indent=4))

Now, the output will be a beautiful

{
    "abs": 42, 
    "answer": [
        42.2
    ]
}

I often use this indentation feature to debug complex data structures.

The price of JSON's interoperability is that we cannot store arbitrary Python objects. In fact, JSON can only store the following objects:

character strings
numbers
booleans (True/False)
None
lists
dictionaries with character string keys

Every object that's not one of these must be converted - that includes every object of a custom class. Say we have an object alice as follows:

class User(object):
    def __init__(self, name, password):
        self.name = name
        self.password = password
alice = User('Alice A. Adams', 'secret')

then converting this object to JSON will fail:

>>> import json
>>> json.dumps(alice)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.3/json/__init__.py", line 236, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.3/json/encoder.py", line 191, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.3/json/encoder.py", line 249, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.3/json/encoder.py", line 173, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <__main__.User object at 0x7f2eccc88150> is not JSON serializable

Fortunately, there is a simple hook for conversion: Simply define a default method.:

def jdefault(o):
    return o.__dict__
print(json.dumps(alice, default=jdefault))
# outputs: {"password": "secret", "name": "Alice A. Adams"}

o.__dict__ is a simple catch-all for user-defined objects, but we can also add support for other objects. For example, let's add support for sets by treating them like lists:

def jdefault(o):
    if isinstance(o, set):
        return list(o)
    return o.__dict__

pets = set([u'Tiger', u'Panther', u'Toad'])
print(json.dumps(pets, default=jdefault))
# outputs: ["Tiger", "Panther", "Toad"]

For more options and details (ensure_ascii and sort_keys may be interesting options to set), have a look at the official documentation for JSON. JSON is available by default in Python 2.6 and newer, before that you can use simplejson as a fallback.

You might also like :
*) The self variable in python explained
*) Python socket network programming
*) [*args and **kwargs in python explained][]

The self variable in python explained

2013-08-07T02:21:00-04:00

Hi there fellas. In this post I am going to teach you about the self variable in python. I have seen many beginners struggling to grasp the concept of self variable. If you are one of them then this post is for you. So lets start by making a class involving the self variable.

A simple class :

So here is our class:

class Restaurant(object):
    bankrupt = False
    def open_branch(self):
        if not self.bankrupt:
            print("branch opened")

First let me explain the above code without the technicalities. First of all we make a class Restaurant. Then we assign it a property "bankrupt" which is currently false. After that we assign it a function open_branch which can only occur if "bankrupt" is False which means that the Restaurant has some money.

Making a resturant:

Now that we have made a class for a Restaurant, lets actually make a resturant:

x = Restaurant()

Now x is a Restaurant which has a property bankrupt and a function open_branch. Now we can access the property bankrupt by typing:

x.bankrupt

The above command is same as:

Restaurant().bankrupt

Now you can see that self refers to the bound variable or object. In the first case it was x because we had assigned the Restaurant class to x whereas in the second case it referred to Restaurant(). Now if we have another Restaurant y, self will know to access the bankrupt value of y and not x. For example check this example:

>>> x = Restaurant()
>>> x.bankrupt
False

>>> y = Restaurant()
>>> y.bankrupt = True
>>> y.bankrupt
True

>>> x.bankrupt
False

The first argument of every class method, including __init__, is always a reference to the current instance of the class. By convention, this argument is always named self. In the __init__ method, self refers to the newly created object; in other class methods, it refers to the instance whose method was called. For example the below code is the same as the above code.

class Restaurant(object):
    bankrupt = False
    def open_branch(this):
        if not this.bankrupt:
            print("branch opened")

Free Tip:

However self is not a reserved keyword in python it's just a strong convention. Many people say that why do we have to write self ? Why can't we have it set automatically like in Java ? Someone also filed a PEP (improvement suggestion) in which he suggested to remove the explicit assignment of self keyword. However Guido Van Rossum (the maker of python) wrote a blogpost in which he told why explicit self has to stay.

I hope you have understood to concept of self. If you have any other questions then feel free to comment. If you liked this post then make sure that you share it on facebook, tweet it on twitter and follow our blog.

You might also like:
*) Python socket network programming
*) [*args and **kwargs in python explained][]
*) Making a url shortener in python

Python socket network programming

2013-08-06T02:06:00-04:00

Hi there fellows. In this post I am going to take you on an adventure with python sockets. They are the real backbones behind web browsing. In simpler terms there is a server and a client. We will deal with the client first. So lets first begin by importing the socket library and making a simple socket.

import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

Here we made a socket instance and passed it two parameters. The first parameter is AF_INET and the second one is SOCK_STREAM. AF_INET refers to the address family ipv4. ipv6 requires something different but we won't be focusing on it. Secondly the SOCK_STREAM means connection oriented TCP protocol. Now we will connect to a server using this socket.

Connecting to a server:
Well firstly let me tell you that if any error occurs during the creation of a socket then a socket.error is thrown and secondly we can only connect to a server by knowing it's ip. You can find the ip of the server by doing this in the command prompt or the terminal:

$ ping www.google.com

You can also find the ip using python:

import socket

ip = socket.gethostbyname('www.google.com')
print ip

So lets begin with writing our script. In this script we will connect with google. Here's the code for it:

import socket # for socket
import sys

try:
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    print "Socket successfully created"
except socket.error as err:
    print "socket creation failed with error %s" %(err)

# default port for socket
port = 80

try:
    host_ip = socket.gethostbyname('www.google.com')
except socket.gaierror:
    # this means could not resolve the host
    print "there was an error resolving the host"
    sys.exit()

# connecting to the server
s.connect((host_ip,port))

print "the socket has successfully connected to google

on port == %s" %(host_ip)

Now save this script and then run it. You will get something like this in the terminal:

Socket successfully created
the socket has successfully connected to google 
on port == 173.194.40.19

So what did we do here ? First of all we made a socket. Then we resolved google's ip and lastly we connected to google. This was not useful for us though so lets move on. So now we need to know how can we send some data through a socket. For sending data the socket library has a sendall function. This function allows you to send data to a server to which the socket is connected and server can also send data to the client using this function. Now lets make a simple server-client program to see all of this in action and hopefully it will make your concepts more clear.

Making a server :
First of all let me explain a little bit about a server. A server has a bind() method which binds it to a specific ip and port so that it can listen to incoming requests on that ip and port. Next a server has a listen() method which puts the server into listen mode. This allows the server to listen to incoming connections. And lastly a server has an accept() and close() method. The accept method initiates a connection with the client and the close method closes the connection with the client. Now lets begin making our simple server:

# first of all import the socket library
import socket

# next create a socket object
s = socket.socket()         
print "Socket successfully created"

# reserve a port on your computer in our
# case it is 12345 but it can be anything
port = 12345

# Next bind to the port
# we have not typed any ip in the ip field
# instead we have inputted an empty string
# this makes the server listen to requests 
# coming from other computers on the network
s.bind(('', port))        
print "socket binded to %s" %(port)

# put the socket into listening mode
s.listen(5)     
print "socket is listening"

# a forever loop until we interrupt it or 
# an error occurs
while True:
   # Establish connection with client.
   c, addr = s.accept()     
   print 'Got connection from', addr

   # send a thank you message to the client. 
   c.send('Thank you for connecting')
   # Close the connection with the client
   c.close()

So what are we doing here ? First of all we import socket which is necessary. Then we made a socket object and reserved a port on our pc. After that we binded our server to the specified port. Passing an empty string means that the server can listen to incoming connections from other computers as well. If we would have passed 127.0.0.1 then it would have listened to only those calls made within the local computer. After that we put the server into listen mode. What does 5 mean ? It means that 5 connections are kept waiting if the server is busy and if a 6th socket trys to connect then the connection is refused. Lastly we make a while loop and start to accept all incoming connections and close those connections after a thank you message to all connected sockets. Now we have to program a client.

Making a client :
Now we need something with which a server can interact. We could tenet to the server like this just to know that our server is working. Type these commands in the terminal:

# start the server
$ python server.py

# keep the above terminal open
# now open another terminal and type:
$ telnet localhost 12345

After typing those commands you will get the following output in your terminal:

# in the server.py terminal you will see
# this output:
Socket successfully created
socket binded to 12345
socket is listening
Got connection from ('127.0.0.1', 52617)

# In the telnet terminal you will get this:
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Thank you for connectingConnection closed by foreign host.

This output shows that our server is working. Now lets make our client:

# Import socket module
import socket

# Create a socket object
s = socket.socket()

# Define the port on which you want to connect
port = 12345

# connect to the server on local computer
s.connect(('127.0.0.1', port))

# receive data from the server
print s.recv(1024)
# close the connection
s.close()

The above script is self explanatory but still let me explain to the beginners. First of all we make a socket object. Then we connect to localhost on port 12345 (the port on which our server runs) and lastly we receive data from the server and close the connection. Was that difficult ? I hope not. Now save this file as client.py and run it from the terminal after starting the server script:

# start the server:
$ python server.py
Socket successfully created
socket binded to 12345
socket is listening
Got connection from ('127.0.0.1', 52617)

$ python client.py
Thank you for connecting

I hope this intro to sockets in python was digestible for beginners. In my server and client script I have not included exception handling as it would have boggled the beginners. Now I hope that you have a solid understanding about the working of sockets in python. However there's a lot more to it. For further study i recommend the Official python docs and this article on binary tides. If you liked this post then don't forget to share it on facebook, tweet it on twitter and follow our blog.

You might also like:
*) [*args and **kwargs in python explained][]
*) Making a url shortener in python
*) 10 inspirations for your next python project

*args and **kwargs in python explained

2013-08-04T18:14:00-04:00

Hi there folks. I have come to see that most new python programmers have a hard time figuring out the *args and **kwargs magic variables. So what are they ? First of all let me tell you that it is not necessary to write *args or **kwargs. Only the * (aesteric) is necessary. You could have also written *var and **vars. Writing *args and **kwargs is just a convention. So now lets take a look at *args first.

Usage of *args
*args and **kwargs are mostly used in function definitions. *args and **kwargs allow you to pass a variable number of arguments to a function. What does variable mean here is that you do not know before hand that how many arguments can be passed to your function by the user so in this case you use these two keywords. *args is used to send a non-keyworded variable length argument list to the function. Here's an example to help you get a clear idea:

def test_var_args(f_arg, *argv):
    print "first normal arg:", f_arg
    for arg in argv:
        print "another arg through *argv :", arg

test_var_args('yasoob','python','eggs','test')

This produces the following result:

first normal arg: yasoob
another arg through *argv : python
another arg through *argv : eggs
another arg through *argv : test

I hope this cleared away any confusion that you had. So now lets talk about **kwargs

Usage of **kwargs
**kwargs allows you to pass keyworded variable length of arguments to a function. You should use **kwargs if you want to handle named arguments in a function. Here is an example to get you going with it:

def greet_me(**kwargs):
    if kwargs is not None:
        for key, value in kwargs.iteritems():
            print "%s == %s" %(key,value)

>>> greet_me(name="yasoob")
name == yasoob

So can you see how we handled a keyworded argument list in our function. This is just the basics of **kwargs and you can see how useful it is. Now lets talk about how you can use *args and **kwargs to call a function with a list or dictionary of arguments.

Using *args and **kwargs to call a function
So here we will see how to call a function using *args and **kwargs. Just consider that you have this little function:

def test_args_kwargs(arg1, arg2, arg3):
    print "arg1:", arg1
    print "arg2:", arg2
    print "arg3:", arg3

Now you can use *args or **kwargs to pass arguments to this little function. Here's how to do it:

# first with *args
>>> args = ("two", 3,5)
>>> test_args_kwargs(*args)
arg1: two
arg2: 3
arg3: 5

# now with **kwargs:
>>> kwargs = {"arg3": 3, "arg2": "two","arg1":5}
>>> test_args_kwargs(**kwargs)
arg1: 5
arg2: two
arg3: 3

Order of using *args **kwargs and formal args
So if you want to use all three of these in functions then the order is

some_func(fargs,*args,**kwargs)

I hope you have understood the usage of *args and **kwargs. If you have got any problems or confusions with this then feel free to comment below. For further study i suggest the official python docs on defining functions and [*args and **kwargs on stackoverflow][].

You might also like :
*) Making a url shortener in python
*) 20 Python libraries you can’t live without
*) Targeting python 2 and 3 at the same time.

Making a url shortener in python

2013-08-03T09:54:00-04:00

Hi there fellas. Today in this post i am going to show you how we can use python to make bulk urls tiny. Let me clear that we are not making a url shortening SERVICE instead what we are going to do is that we are going to unofficially use the tinyurl api (Tinyurl is a url shortening service). There's not any official python api released by tinyurl. So lets begin with this:

The First Step :
So first of all we have to do some imports. We have to import 7 libraries to make this work.

from __future__ import with_statement
import contextlib
try:
    from urllib.parse import urlencode
except ImportError:
    from urllib import urlencode
try:
    from urllib.request import urlopen
except ImportError:
    from urllib2 import urlopen
import sys

We could have imported just one library to make this work but in order to make a good url shortener we have to import these seven libraries.

The Second Step :
So now we have to start making a method that will handle the url shortening. Watch the code closely because it is self explanatory but however i will explain it later.

def make_tiny(url):
    request_url = ('http://tinyurl.com/api-create.php?' + 
    urlencode({'url':url}))
    with contextlib.closing(urlopen(request_url)) as response:
        return response.read().decode('utf-8')

Did you manage to understand the code ? Let me explain it to the beginners. First of all we define a make_tiny function which takes a url as an input. After that we start defining the working of our function. The url_encode takes a url as an input and encodes it i.e escapes it. After that we append the escaped url to the end of tinyurl's api url. Then we open the request_url using urlopen. And lastly we read the response after converting it to utf-8. Why did we convert the response to utf-8 ? The reason for this is that the urlopen function returns a stream of bytes rather than a string so in order to print it and alter it we have to convert it into utf-8. Was that difficult ?

So now the next step is to get the input from the user. For this we are going to use the sys library.

The Third Step :
So lets write the main() function for our little script. Here's the code for it:

def main():
    for tinyurl in map(make_tiny, sys.argv[1:]):
    print(tinyurl)

So what are we doing here ? We are getting the user input by using sys.argv . We have not limited ourself to only one url as an input instead we are saying that give us as many urls as you want, we will crush them and make them tiny. What sys.argv[1:] does is that it leaves the first two arguments (counting starts from 0) and takes all the rest of the arguments and produces a list of those arguments. For example if you type:

$ python script.py url1 url2 url3

then sys.argv[1:] will leave python and script.py and will produce the following list:

[url1,url2,url3]

Wait! what is that map() function over there ? Most beginners will feel confused as most of them have never used map. Map() is a simple way of looping over a list and passing it to a function one by one. The map() function above there is equivalent to :

def main():
    urls = sys.argv[1:]
    for url in urls:
        tinyurl = make_tiny(url)
        print tinyurl

I hope the above code example cleared away any confusions about map() function.

The Fourth Step :
Now lets wrap up our code. The only thing left is this :

if __name__ == '__main__':
    main()

Add this to the end of your code. This tells us whether the script was executed independently from the shell or was it called by another script. This is pretty handy if you want to use this script later in any other project.

Finally :
So here's the complete script:

from __future__ import with_statement
import contextlib
try:
    from urllib.parse import urlencode
except ImportError:
    from urllib import urlencode
try:
    from urllib.request import urlopen
except ImportError:
    from urllib2 import urlopen
import sys

def make_tiny(url):
    request_url = ('http://tinyurl.com/api-create.php?' + 
    urlencode({'url':url}))
    with contextlib.closing(urlopen(request_url)) as response:
        return response.read().decode('utf-8')

def main():
    for tinyurl in map(make_tiny, sys.argv[1:]):
        print(tinyurl)

if __name__ == '__main__':
    main()

Considering that you have saved this script as url_shortener.py you have to run it like this from the shell:

$ python url_shortener.py url1 url2 url3

If you want to save these tinyurls into a txt file then issue this command:

$ python url_shortener.py url1 > urls.txt

I hope you liked today's post. This script can work on python 2 and python 3 both. It was primarily aimed at two kind of audience. Firstly those who are learning python and want to learn how to make a simple yet useful script and those who want to learn how to make a url shortener in python. Do share your view in the comments bellow and stay tuned for the next post. If you want regular dose of python tips and tutorials delivered to your doorstep then consider following my blog.

You might also like:
* 5 quality screencasts worth watching
* Packaging and distributing your python libraries
* 10 inspirations for your next python project

5 quality screencasts worth watching

2013-08-02T18:27:00-04:00

Hello there fellas. I am back with another list of useful resources. This time i am going to share with you 5 quality python screen casts worth watching. Unlike ruby there are not really much quality screen casts related to python out there. So without wasting a dime lets continue with the list of those screen casts. So here it is:

1.The first one if Encoding Specifiers and Magical Mode Lines in python. There's a lot in it for learning. Go check it out and you won't regret it.

2.The second one is by Neck Beard Republic. This one is about lambda functions in python. Neck Beard Republic have great python screen casts but most of them are paid however they have some free screen casts as well.

3.The third one is a series of screen casts by Kenneth Love. They are related to django. The title is Getting Started With Django. They are great. I have personally watched them. There's something for everyone in it.

4.The fourth one is again by Neckbeard Republic. This one is a must watch for every python programmer out there. The title of this screen cast is [__dunder__ functions in python][]. There can be alot of mystery involved in Python when it comes to the dunder (double underscore) methods. This screen cast will reveal the secret behind these function. However i have not personally watched it but it's on my todo list.

5.Ah so now comes the last one. This one is again a Neckbeard Republic production. This one is about the super() literal in python. I can see how much effort Mahdi Yusuf has put into making these screen casts.

So now we come to the end of another great post about 5 quality python screen casts worth watching. However there are some other video resources as well like Showmedo which i will cover in a future post. If you are feeling adventurous then check out 10 inspirations for your next python project or Packaging and distributing your python libraries. I hope you liked today's post. Do share your views in the comments below and stay tuned for the next post.

What is Pickle in python ?

2013-08-02T08:59:00-04:00

Hi there fellas. In this post i am going to tell you about pickle. It is used for serializing and de-serializing a Python object structure. Any object in python can be pickled so that it can be saved on disk. What pickle does is that it “serialises” the object first before writing it to file. Pickling is a way to convert a python object (list, dict, etc.) into a character stream. The idea is that this character stream contains all the information necessary to reconstruct the object in another python script.

So lets continue:

1. First of all you have to import it through this command:

import pickle

pickle has two main methods. The first one is dump, which dumps an object to a file object and the second one is load, which loads an object from a file object.

2. Prepare something to pickle:
Now you just have to get some data which you can pickle. For the sake of simplicity i will be pickling a python dictionary. So just read the below code and you will be able to figure it out yourself.

import pickle

a = ['test value','test value 2','test value 3']
a
['test value','test value 2','test value 3']

file_Name = "testfile"
# open the file for writing
fileObject = open(file_Name,'wb')

# this writes the object a to the
# file named 'testfile'
pickle.dump(a,fileObject)

# here we close the fileObject
fileObject.close()
# we open the file for reading
fileObject = open(file_Name,'r')  
# load the object from the file into var b
b = pickle.load(fileObject)  
b
['test value','test value 2','test value 3']
a==b
True

The above code is self explanatory. It shows you how to import the pickled object and assign it to a variable. So now we need to know where we should use pickling. It is very useful when you want to dump some object while coding in the python shell. So after dumping whenever you restart the python shell you can import the pickled object and deserialize it. But there are other use cases as well which i found on stackoverflow. Let me list them below.

1) saving a program's state data to disk so that it can carry on where it left off when restarted (persistence)

\2) sending python data over a TCP connection in a multi-core or distributed system (marshalling)

\3) storing python objects in a database

\4) converting an arbitrary python object to a string so that it can be used as a dictionary key (e.g. for caching & memoization).

One thing to note is that there is a brother of pickle as well with the name of cpickle. As the name suggests it is written in c which makes it 1000 times more faster than pickle. So why should we ever use pickle instead of cpickle ? Here's the reason

>>> Because pickle handles unicode objects. >>> Because pickle is written in pure Python, it's easier to debug.

For further reading i suggest the official pickle documentation or if you want to read more tutorials then check out the sqlite tutorial. Now we come to the end of today's post. I hope you liked it. Do follow my blog to get regular updates. If you have any suggestions or comments then post them below.

10 inspirations for your next python project

2013-08-01T23:27:00-04:00

Hi there fellow coders. I know those times when we don't know what to code. I am sure you must have experienced that time as well. Today i am going to share with you 10 python projects that will serve as an inspiration for your next python project. So without wasting a time here's the list.

1. Pillow . Pillow is the "friendly" PIL fork by Alex Clark and Contributors. PIL is the Python Imaging Library by Fredrik Lundh and Contributors. Alex teaches us that if you don't like something then make a fork.

2. Inbox.py. This is the simplest SMTP server you'll ever see. It's asynchronous. One instance should handle over one thousand emails per second. Pretty impressive.

3. PyMinecraft. Have you ever thought of implementing the famous minecraft game into python ? Hawstein has.

4. fuqit. The latest and greatest from Zed Shaw (the writer of learn python the hard way) , fuqit is a web framework that attempts to make you forget everything you have learned from MVC frameworks and focuses on simplicity. Zed Shaw teaches us one thing through this project "Don't panic while doing something against the trend".

5. PyPI Notifier. Watches your requirements.txt files and notifies you via email when a project is updated. Although i have not used it myself but i think this project is great.

6. Vol. A simple project but great inspiration. vol lets you control your mac's audio volume from the commandline. It also has support for profiles.

7. Youtube-dl. A great python project. It allows you to download videos from various websites. It's a great project make sure that you check it out.

8. Jukebox. Ever wanted to listen to music with a larger group of people e.g. in your office? Who decides what to play? Make your music player democratic and give everyone the chance to promote their favourite song. This script will relieve you from numerous fights.

9. Mitmproxy. mitmproxy is a an SSL capable man-in-the-middle-proxy. It allows HTTP(and other) traffic to be inspected and edited on the fly. Don’t be caught unaware by apps like Path any more, now you can know what data you are sending and receiving from remote servers.

10. Thumbor. It is a smart imaging service. It enables on-demand crop, resizing and flipping of images. I haven't checked it out though but it seems promising. Do check it out if you want a server side image cropper.

So finally we come to the end of this post. I hope you found some useful python projects in there to serve as an inspiration. Share your views in the comments below and tell me which project you liked the most or if you want to read further then check out my another blog post about 20 python libraries you can't live without.

Packaging and distributing your python libraries

2013-08-01T14:51:00-04:00

Hi there fellow coders. This post will go over the basics of packaging and deploying your python libraries. So without wasting a minute lets get started.

Folder structure

So first thing first. What should be your folder structure ? At minimum you can have only two files. The first one should be the setup.py file and the second one should be your module file. But today I am not going to talk about the very basics. Today we are going to follow this folder structure :

My_module/
    LICENSE.txt
    README.txt
    setup.py
    My_module/
        __init__.py

Here I have 4 files. The License file contains the license under which you want to distribute your module. So lets move on. What should be our second step ? Obviously it should be to describe our module and write some meta data into the setup.py file.

Describing your module

The setup.py file is the heart of any python module or library. It describes the module and lists some other useful info about the module like it lists any dependencies on which a module depends and it also tells distutils where to find the necessary scripts of this module. So lets describe our module with the help of our setup.py file.

from distutils.core import setup

setup(
    name='My super module',
    version='0.1dev',
    packages=['My_module',],
    license='Creative Commons Attribution-Noncommercial-Share Alike license',
    long_description=open('README.txt').read(),
)

Here we could have rewritten the long_description ourselves but we reused our README file. We wrote 'dev' in the version because we still do not have anything in our module but we are moving toward the 0.1 release. When you have enough code in your module then feel free to drop 'dev' from the version field. Now the next step is to make our first release.

Making your first release

So how do we make our first release ? Just follow me. Your release should have a single archive file. It can be easily made with this command.

$ python setup.py sdist

Just go to the root of your module folder and execute this command. It will create a subdirectory with the name of dist and will package all of your module scripts and other files into a single archive file ready to be uploaded on PyPI (Python Package Index). This archive file will have the default extension of zip on windows and tar.gz on POSIX systems.

Publishing your module

Now that archive file can be uploaded anywhere for distribution but we will focus on PyPI. In order to upload on PyPI you will first have to make an account on http://pypi.python.org/pypi . After that you will have to register your package on PyPI like this:

$ cd path/to/My_module
$ python setup.py register

Just use your existing login. Afte that run the following command to upload it to PyPI.

$ python setup.py sdist upload

This will make the distribution one time more and will upload it to PyPI. Now you should be proud of yourself because you have just contributed to the opensource world. I hope you liked my post. If you want to read about setup.py then go here. For further study i recommend the Hichhikers guide to packaging. I hope you liked today's post. Do share your views in the comments below and stay tuned for the next post.

Connecting to Sqlite databases

2013-08-01T00:48:00-04:00

Hi there fellas. Today i am going to teach you how to use sqlite databases with python. This post will cover the basics of making and using a sqlite database with python using the sqlite3 library. Okay lets get started. Firstly if you are using python 2.5 or greater then you will have sqlite3 installed otherwise you will have to install it.

Creating and connecting to a database

So how do you make a database in python using the sqlite3 library ? It's pretty simple. Just follow the code below and you will be able to make it out on your own.

#!/usr/bin/python

import sqlite3

# If the database is already created then this
# code will connect to it instead of making a new one
conn = sqlite3.connect('test.db')

print "Created database successfully"

So was that difficult ? I hope not. So lets continue. The next thing is to make tables in our database. So how do you go about doing it ? Just follow me.

Making tables in a database

import sqlite3

conn = sqlite3.connect("test.db")

with conn:
    cursor = conn.cursor()

    # create a table
    cursor.execute("""CREATE TABLE books
               (title text, author text)""")

In the above code we made a table with the name of book. It has the following fields : title and author. Both of these fields have the data type of text. First of all we made a database with the name of test.db and after that we made a cursor object which allows us to interface with our database and execute queries. So what now. We have created a database and made a table. Now we have to insert some data in our table. Lets continue.

Inserting data to the database

# insert some data
cursor.execute("INSERT INTO books VALUES ('Pride and Prejudice', 'Jane Austen')")

# save data to database
conn.commit()

# insert multiple records using the more secure "?" method
books = [('Harry Potter', 'J.K Rowling'),
          ('The Lord of the Rings', 'J. R. R. Tolkien'),
          ('The Hobbit','J. R. R. Tolkien')]
cursor.executemany("INSERT INTO books VALUES (?,?)", books)
conn.commit()

So in the above code i showed you two ways to put some data into the database. The first method is to execute a single query and the second method is to execute multiple queries in the same time. In the second method we could have used the string substitution '%s' but it is known to be potentially dangerous and can lead to SQL Injection. So whats left now ? Removing and updating the data ? No problem i will cover that as well. Just examine the code below.

Updating data in the database

import sqlite3

conn = sqlite3.connect("test.db")

with conn:
    cursor = conn.cursor()

    sql = """
        UPDATE books 
        SET author = 'Yasoob' 
        WHERE author = 'J.K Rowling'
    """
    cursor.execute(sql)

In the above code we updated our record by replacing J.K Rowling with Yasoob. Take a look at the below code for deleting the records.

Deleting records from the database

import sqlite3

conn = sqlite3.connect("test.db")

with conn:
    cursor = conn.cursor()

    sql = """
        DELETE FROM books
        WHERE author = 'Yasoob'
    """
    cursor.execute(sql)

In the above code we deleted the record of those books whose writer was 'Yasoob'. Now i am going to show you how to display data from the table. It is easy. Just a few lines of code.

Displaying data from the database

import sqlite3

conn = sqlite3.connect('test.db')
print "Opened database successfully";

cursor = conn.execute("SELECT title, author  from books")
for row in cursor:
   print "Title = ", row[0]
   print "Author = ", row[1], "\n"

print "Operation done successfully";
conn.close()

In the above code we opened our database file and displayed the records on the screen. Do you know that we could have also done:

conn = sqlite3.connect(':memory:')

So what is this code doing ? The :memory: is a special name which creates the database in the ram and lets you execute any query. Lastly I would like to tell you about sqlalchemy which is a database library for python and takes care of a lot of things for you. It does all the escaping for you so you won't have to mess with the annoyances of converting embedded single quotes into something that SQLite will accept. It's a part of my previous blog post as well.

So now goodbye to you all. I hope you liked reading today's post as much as i enjoyed writing it. Stay tuned for my next post.

For further reading i suggest the zetcode database tutorial.

Give me an idea for an Ebook

2013-07-31T20:41:00-04:00

Hi there fellas. Today i am going to ask you for a favour. I am going to write an ebook about python. I want you to give me some ideas about what i should write. This Ebook is going to be short. It will be between 10 to 20 pages because it's my first ebook. Currently i have two topics in mind. The first one is to write an introductory ebook about how to get started with python. This will be for beginners and will include the installation and setting up of python as well as some introductory tutorials. The other topic is to write about how i made the soundcloud searcher and downloader . All the backend programming of soundcloud searcher and downloader is done in python. This ebook will contain all the methods and resources I used to make this soundcloud searcher and downloader as well as how I got it working on google app engine. In my opinion the second topic will be more useful for you guys as most of us are looking for these type of ebooks.

What do you say ? If you want me to write about any other topic then do tell me in the comments below and I will see.

10 python blogs worth following

2013-07-31T14:19:00-04:00

Hi there fellow programmers. I remember those days when i started learning python. I was always eager to find new blogs about python but found just a handful of them. This is one of the reason i started this blog. To spread my knowledge about python to other fellow coders. Today i am going to try to provide you 10 links to different python blogs that are worth following. By that i mean they are regularly updated and have useful info. So here's the list:

1. The first one is Planet python undoubtedly one of the most famous python blogs out there. Go check it out. I hope you will find it helpful. It was last updated today.

2. The second blog is lucumr. The home of the developer of flask (a python micro web framework).

3. The third blog is love-python. I don't know how i stumbled over it but i really like this blog as it has a lot of useful info and code snippets.

4. The fourth blog is ofDoug Hellmann. He is the one behind PYMOTW (Python Module Of The Week). This blog covers a lot of python libraries. If you are searching for some new useful python libraries then go check this out.

5. The next blog is Coder Who Says Py. It's a nice blog. Although it's not regularly updated but still you will find some helpful info in there.

6. The sixth blog is effbot. Full of helpful info, code snippets and everything else you can expect from a python blog. Go check it out before it's too late.

7. The seventh blog is of the writer of "Two Scoops Of Django". It is pydanny. This is mainly about django but still a lot can be learned from this blog.

8. The next blog is of Al Sweigat. He is the writer of numerous python books including his latest book "Hacking Secret Ciphers With Python".

9. The ninth blog is this blog on which you currently are. This is your all in one guide to python.

10. The last blog is one of the most useful blog which got me up to speed with python. It is called Mouse Vs Python.

I hope you liked today's post and i am sure i helped you out of your frustration of finding good python blogs. Do share your views in the comments below and if you think that i have missed a blog then put it in the comments as well.

20 Python libraries you can't live without

2013-07-30T23:44:00-04:00

Hi there fellas. Today i am going to list 20 python libraries which have been a part of my toolbelt and should be a part of yours as well. So here they are:

1. Requests. The most famous http library written by kenneth reitz. It's a must have for every python developer.

2. Scrapy. If you are involved in webscraping then this is a must have library for you. After using this library you won't use any other.

3. wxPython. A gui toolkit for python. I have primarily used it in place of tkinter. You will really love it.

4. Pillow. A friendly fork of PIL (Python Imaging Library). It is more user friendly than PIL and is a must have for anyone who works with images.

5. SQLAlchemy. A database library. Many love it and many hate it. The choice is yours.

6. BeautifulSoup. I know it's slow but this xml and html parsing library is very useful for beginners.

7. Twisted. The most important tool for any network application developer. It has a very beautiful api and is used by a lot of famous python developers.

8. NumPy. How can we leave this very important library ? It provides some advance math functionalities to python.

9. SciPy. When we talk about NumPy then we have to talk about scipy. It is a library of algorithms and mathematical tools for python and has caused many scientists to switch from ruby to python.

10. matplotlib. A numerical plotting library. It is very useful for any data scientist or any data analyzer.

11. Pygame. Which developer does not like to play games and develop them ? This library will help you achieve your goal of 2d game development.

12. Pyglet. A 3d animation and game creation engine. This is the engine in which the famous python port of minecraft was made

13. pyQT. A GUI toolkit for python. It is my second choice after wxpython for developing GUI's for my python scripts.

14. pyGtk. Another python GUI library. It is the same library in which the famous Bittorrent client is created.

15. Scapy. A packet sniffer and analyzer for python made in python.

16. pywin32. A python library which provides some useful methods and classes for interacting with windows.

17. nltk. Natural Language Toolkit – I realize most people won’t be using this one, but it’s generic enough. It is a very useful library if you want to manipulate strings. But it's capacity is beyond that. Do check it out.

18. nose. A testing framework for python. It is used by millions of python developers. It is a must have if you do test driven development.

19. SymPy. SymPy can do algebraic evaluation, differentiation, expansion, complex numbers, etc. It is contained in a pure Python distribution.

20. IPython. I just can’t stress enough how useful this tool is. It is a python prompt on steroids. It has completion, history, shell capabilities, and a lot more. Make sure that you take a look at it.

I hope you liked today's post. Do share your views in the comments below and if you think that i have missed an important library which should be listed up there post it in the comments and i will add it. Stay tuned for the next post.

Targeting python 2 and 3 at the same time.

2013-07-30T19:58:00-04:00

Hi there pythonistas. Today i am not going to write a lengthy tutorial about how to make your programs compatible with python 2 and 3 at the same time, but i am going to share a simple tip with you guys which will help you achieve this target. Just imagine that you have a very popular python module which is use by hundreds of people but not all of them have python 2 or 3. In that case you have two choices. The first one is to distribute 2 modules. One for python 2 and the other for python 3 and the other choice is to modify your current code so that it can be used with both python 2 and 3. I am going to talk about the second choice. First tell me how you import packages in your script ? Most of us do this :

import foo
# or
from foo import bar

Do you know that you can do something like this as well?

import foo as foo

I know it's function is the same as above listed code but it is vital for making your script compatible with python 2 and 3. Now examine the code below :

try:
    import urllib.request as urllib_request #for python 3
except ImportError:
    import urllib2 as urllib_request # for python 2

Did you find something noteworthy in the above code ? So let me explain the above code a little. We are wrapping our importing code in a try except clause. Why are we doing that ? We are doing it because in python2 there is no urllib.request module and will result in an ImportError. The functionality of urllib.request is provided by urllib2 module in python2. So now when python2 try to import urllib.request and get an import error we tell it to import urllib2 instead. The final thing you need to know about is the as keyword. It is mapping the imported module to urllib_request. So that now all of the Classes and methods of urllib2 are available to us by urllib_request.

So how did this help us in targeting python 2 and 3 at the same time ? Okay here is the trick. Once you have got a library that works in python 2 but is not available in python 3 then you have to find out it's alternative in python 3 (most libraries have their alternatives for python 3 but some don't). I know it's trivial and that is why a lot of modules have still not been ported to python 3. After that you have to wrap your module importing code in a try-except clause with a common variable following the as keyword. After that use the imported module. Here is a complete example which should work both in python 2 and 3:

try:
    import urllib.request as compat_urllib_request
except ImportError: # Python 2
    import urllib2 as compat_urllib_request
html = compat_urllib_request.urlopen("http://www.google.com/")
print html.headers['content-type']
'text/html; charset=ISO-8859-1'

Another example which involves the json library:

try:
    import json
except ImportError:
    import simplejson as json

I hope you liked today's post. Stay tuned for the next one. If you have any suggestions then do post them in the comments below.

Exception handling

2013-07-30T02:03:00-04:00

Okay so the chances are you already know about exception handling in python but some new programmers don't. How do you handle exceptions in python ? First let me tell you what exceptions are. Exceptions are when something unexpected happens with your code. Just think that you writing a huge program which browses the internet and accesses web pages, the program works fine on your side and you consider packaging and distributing it. But something unexpected occurs on the pc of the user who downloaded your package from the internet and tried to run it. His internet connection dropped. What will happen ?

>>> import urllib2
>>> urllib2.urlopen('http://www.google.com/').read()

Traceback (most recent call last):
  File "", line 1, in 
    urllib2.urlopen('http://www.google.com/').read()
  File "C:\Python27\lib\urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "C:\Python27\lib\urllib2.py", line 404, in open
    response = self._open(req, data)
  File "C:\Python27\lib\urllib2.py", line 422, in _open
    '_open', req)
  File "C:\Python27\lib\urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "C:\Python27\lib\urllib2.py", line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "C:\Python27\lib\urllib2.py", line 1184, in do_open
    raise URLError(err)

So in order to save yourself from this you have to use Exception Handling. There are three keywords available within python which help you to handle exceptions. They are usually referred to as try-except clause. Just watch the below code.

>>> try:
    urllib2.urlopen('http://www.google.com/').read()
except urllib2.URLError as e:
    print "An error "

An error
>>>

Here we are doing the same thing but using the try except clause. Lets get somewhat deeper into it. We wrap our main code in the try clause. After that we wrap some code in except clause which gets executed if an exception occurs in the code wrapped in try clause. But wait where is the third keyword ? The third keyword is finally. It comes after the except clause and gets executed even if no exception occurs. Let me show you the same example but this time with the finally clause as well.

>>> try:
    urllib2.urlopen('http://www.google.com/').read()
except urllib2.URLError as e:
    print "An error %s " %e
finally:
    print "This is going to be printed even if no exception occurs"

'Google(function(){\nwindow.google={kEI:"sdD2UbmEOMzBtAbyqYCgDQ",ge

------------------------trancuated------------------- 
This is going to be printed even if no exception occurs

So as you can see the finally clause got executed even when there was no exception. Okay now i am going to tell you a little secret which most of us don't know and even i got to know it just few days ago. While using the except clause if we want to handle any exception that we are not aware of then what should we do ? Here in this situation we can put more except clauses for each exception which is likely to occur and in the end we can put another except clause which caches all exceptions which were not likely to occur. Here is an example :

>>> try:
    print ali
except KeyboardInterrupt:
    print "'E's pining!"
except :
    print "This is going to handle every type of exception"
finally:
    print "okay the end"

This is going to handle every type of exception
okay the end

Okay i guess thats it for now. However i just gave you an intro of exception handling in python and if you want to study further then go to this link. I hope you liked today's post. Do share your views about todays post in the comments below.

Source : http://docs.python.org/2/tutorial/errors.html

Python install location and version

2013-07-30T00:17:00-04:00

Have you ever wondered how to check the install path of python and it's version ? If you have then this post is for you. It is very easy to check the version and install location of python on linux as well as on windows. First I will show the method of checking this on linux and then on windows.

1. On linux just type these two commands in the terminal:

# for python install path
root@bt:/$ which python
/usr/bin/python

# for python version;
root@bt:/usr/bin$ python -V
Python 2.7.5

2. On windows type this commands in the command prompt for python version:

python -V

And type these commands in the python shell for python install path

>>> import os
>>> import sys
>>> os.path.dirname(sys.executable)
'C:\\Python27'

This comes in handy when you need to know the version and path of python real quick. I hope you liked today's post. Stay tuned for the next one. Do share your views in the comments below.

Coding styles for python

2013-07-29T20:08:00-04:00

Yet another useful post and this one is about coding styles in python. Everyone has his own coding style in python. Here i am going to post some links that will help you to choose your own coding style in python.

The first link is undoubtedly from the official python documentation as python documentation has everything to get you started. This link will teach you how to use python and how not to use python.
http://docs.python.org/2/howto/doanddont.html
The second link is from the famous PEP-8 which is python's official style guide. It determines the python's official coding standards. You should definitely take a look at them and try to incorporate them in your code.
http://www.python.org/dev/peps/pep-0008/
The third link is from google because as you know google is a huge user of python and most of their infrastructure is based on python as is their online support forum. So it's not surprising that google has it's own style guide for python.
http://google-styleguide.googlecode.com/svn/trunk/pyguide.html
So the last link is from khanacademy. As you know it's the biggest online academy and whole of their online infrastructure is based on python. I really love their python style guide.
https://sites.google.com/a/khanacademy.org/forge/for-developers/styleguide/python

I hope you liked this post. Do share your views in the comments below.

Setting up python on windows

2013-07-29T16:44:00-04:00

Hi there pythonistas. Most of the new python programmers don't know how to correctly install python on windows. Today I am going to share clean and clear instructions on how to install python on a Windows 7 machine. There are only 4 steps involved so without wasting a minute lets get started.

1. Visit the official Python download page and grab the Windows installer. Choose the 32-bit version. (A 64-bit version is available, but there are compatibility issues with some modules you may want to install later.)

Note: Python currently exists in two versions. One is 2.xx version and the other is 3.xx version. In this post i am going to focus only on 2.xx version as it is more widely used right now and some modules don't support 3.xx version as yet.

2. Run the installer and accept all the default settings. Don't change the default directory it creates.

3. Now comes the trickiest part. We have to add python to our environment variable. You do not have to understand what environment variable is. Just follow along with me. We set the system’s PATH variable to include directories that include Python components and packages so that they are available in the command prompt. Without doing this we won't be able to use python from command line. To do this:

Right-click Computer and select Properties.
In the dialog box, select Advanced Â System Settings.
In the next dialog, select Environment Variables.
In the User Variables section, edit the PATH statement to include this:

C:\Python27;C:\Python27\Lib\site-packages\;C:\Python27\Scripts\;

4. Now, you can open a command prompt (Start Menu|Accessories or Start Menu|Run|cmd) and type:

C:\> python

That will load the Python interpreter:

Python 2.7.3  (default, Apr 10 2012, 14:24) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or license for more information.
>>>

Just because of the changes that you did to the path variable you are now able to use python from command prompt. Now press Control-Z to exit the interpreter and get back to a C: prompt.

Till now the the installation of python is complete but you may want to install setuptools which offers the helpful easy_install utility for installing Python packages. Grab the appropriate version for your system and install and that's it for now. Stay tuned for the next post.

Packaging your python scripts.

2013-07-29T14:42:00-04:00

Oh hi there! Welcome to another useful post. This post is going to be about how to package your python scripts and packages for distribution on PyPI or some other place.Here I won't go too deep into explaining everything as most of us just need to know the basics of packaging. However i will provide you with different links for further study. Okay lets talk about

>>> setuptools

first. What is it ? It's a Python module which allows us to easily package our python scripts and modules for distribution. However there are other packaging libraries as well but here i will talk about setuptools only.

So what should be the basic example to show the usage of setuptools ? Here you go. For basic use of setuptools, just import things from setuptools and then look below for the minimal setup script using setuptools.

from setuptools import setup, find_packages
setup(
    name = "HelloWorld",
    version = "0.1",
    packages = find_packages(),
)

As you see we don't have to specify much in order to use setuptools in a project. Just by doing the above, this project will be able to produce eggs, upload to PyPI, and automatically include all packages in the directory where the setup.py lives. But when you are releasing your projects on PyPI then you should add a bit more information about yourself and this package and if your project relies on some external dependencies then list them there as well. Here is another script which can do all that :

from setuptools import setup, find_packages
setup(
    name = "HelloWorld",
    version = "0.1",
    packages = find_packages(),
    scripts = ['say_hello.py'],

    # Project uses reStructuredText, so ensure that the 
    # docutils get installed or upgraded on the target 
    # machine
    install_requires = ['docutils>=0.3'],

    package_data = {
        # If any package contains *.txt or *.rst files,
        # include them:
        '': ['*.txt', '*.rst'],
        # And include any *.msg files found in the 
        # 'hello' package, too:
        'hello': ['*.msg'],
    },

    # metadata for upload to PyPI
    author = "Me",
    author_email = "me@example.com",
    description = "This is an Example Package",
    license = "PSF",
    keywords = "hello world example examples",
    # project home page, if any :
    url = "http://example.com/HelloWorld/",

    # could also include long_description, download_url,
    # classifiers, etc.
)

I hope this is enough for now. However here are some other packaging libraries in case you were wondering :

Distutils is the standard tool used for packaging. It works rather well for simple needs, but is limited and not trivial to extend.
Setuptools is a project born from the desire to fill missing distutils functionality and explore new directions. In some sub-communities, it’s a de facto standard. It uses monkey-patching and magic that is frowned upon by Python core developers.
Distribute is a fork of Setuptools that was started by developers feeling that its development pace was too slow and that it was not possible to evolve it. Its development was considerably slowed when distutils2 was started by the same group.
Distutils2 is a new distutils library, started as a fork of the distutils codebase, with good ideas taken from setup tools (of which some were thoroughly discussed in PEPs), and a basic installer inspired by pip. Distutils2 did not make the Python 3.3 release, and it was put on hold.

>>> source

For further study i recommend: http://pythonhosted.org/distribute/setuptools.html

Dictionaries have a get() method

2013-07-29T00:59:00-04:00

Okay here we are with yet another post. However this post is going to be short as in this post i am going to teach you a small, simple and very useful trick. Have you ever tried to access a value from a dictionary and got an exception that there is no value mapped to that key ? Like this :

>>> dict = {'key1':2,'key2':3}
>>> dict['key1']
2
>>> dict['key3']

Traceback (most recent call last):
  File "", line 1, in 
    dict['key3']
KeyError: 'key3'

Sometimes you do not want anything like this because this error can break your whole program and make it useless until you fix this issue. Is there any way through which we can see if a key is present in a dictionary or not and if it is not present then get a default value in it's place ? the chances are that you don't know of any such method. Okay so here's the trick. Just examine the code below and i will explain it later.

>>> dict = {'key1':2,'key2':3}
>>> dict['key1']
2
>>> a = dict.get('key3', 0)
>>> print a
0

So what is happening above ? We are still accessing the same key but where has the exception gone ? Let me introduce you to get() method which is available with a dictionary. What this method does is that it allows you to specify a default value which is returned in case the key is not present in the dictionary. In our case there is no key3

in our dict but when we are using the get() method we are specifying the default value which should be returned in case if key3 is not present and the default value which i have set is 0. The syntax of the get method is : dict.get('anykey','the default value to return if the key is not present') . However if we do not set a default value then this returns `None` if the key is not present. I hope it's enough for you guys however if you want to further explore this get() method then google is your friend. Do share your views in the comments below.

The use of return and global keywords.

2013-07-28T23:52:00-04:00

Okay so here we have another post. This post is about the return keyword. you might have encountered some functions written in python which have a return keyword in the end of the function. Do you know what it does ? Lets examine this little function:

>>> def add(value1,value2):
...     return value1 + value2

>>> result = add(3,5)
>>> print result
8

The function above takes two values as input and then output their addition. We could have also done:

>>> def add(value1,value2):
...     global result
...     result = value1 + value2

>>> add(3,5)
>>> print result
8

So first lets talk about the first bit of code which involves the return keyword. What that function is doing is that it is assigning the value to the variable which is calling that function which in our case is result.
It's pretty handy in most cases and you won't need to use the global keyword. However lets examine the other bit of code as well which includes the global keyword. So what that function is doing is that it is making a global variable result. What does global mean here ? Global variable means that we can access that variable outside the function as well. Let me demonstrate it with an example :

# first without the global variable
>>> def add(value1,value2):
    result = value1 + value2

>>> add(2,4)
>>> result

# Oh crap we encountered an exception. Why is it so ?
# the python interpreter is telling us that we do not 
# have any variable with the name of result. It is so 
# because the result variable is only accessible inside 
# the function in which it is created if it is not global.
Traceback (most recent call last):
  File "", line 1, in 
    result
NameError: name 'result' is not defined

# Now lets run the same code but after making the result 
# variable global
>>> def add(value1,value2):
    global result
    result = value1 + value2


>>> add(2,4)
>>> result
6

So hopefully there were no errors in the second run as expected. If you want to further explore them you should check out the following links.
1.
2. Stackoverflow

Generating a random string.

2013-07-28T17:40:00-04:00

Okay so most of us do not know how to generate random strings which include letters and digits. This can be really useful for generating a password or something else just use your evil mind. Okay so how do we generate a random string. Have you ever heard of the string module available in python ? The chances are that you have not heard about it. So what does this module provide us ? Okay here you go lets understand this module by example:

# first we have to import random module as this
# provides the backbone for our random string
# generator and then we have to import the string
# module.

>>> import random
>>> import string

# now lets see what this string module provide us.
# I wont be going into depth because the python
# documentation provides ample information.
# so lets generate a random string with 32 characters.

>>> random = ''.join([random.choice(string.ascii_letters + string.digits) for n in xrange(32)])
>>> print random
'9fAgMmc9hl6VXIsFu3ddb5MJ2U86qEad'
>>> print len(random)
32

Another example with a function:

>>> import string
>>> import random
>>> def random_generator(size=6, chars=string.ascii_uppercase + string.digits):
...    return ''.join(random.choice(chars) for x in range(size))
...
>>> random_generator()
'G5G74W'
>>> random_generator(3, "6793YUIO")
'Y3U'

So thats it for now. If you want to further explore the string module then go to the official documentation

What is init.py ?

2013-07-28T16:00:00-04:00

Okay yet another useful post. This post is really important and useful for anyone just starting out with python. So what is the __init__.py file ?

Files name __init__.py are used to mark directories on disk as Python package directories. If you have the files

mydir/spam/__init__.py
midir/spam/module.py

and mydir is on your path, you can import the code in module.py as

import spam.module

from spam import module

If you remove the __init__.py file, Python will no longer look for submodules inside that directory, so attempts to import the module will fail.

The __init__.py file is usually empty, but can be used to export selected portions of the package under more convenient name, hold convenience functions, etc. Given the example above, the contents of the init module can be accessed as

import spam

And finally here is what the official documentation has to say about this file:

The __init__.py files are required to make Python treat the 
directories as containing packages; this is done to prevent 
directories with a common name, such as string, from 
unintentionally hiding valid modules that occur later on the
module search path. In the simplest case,__init__.py can just
be an empty file, but it can also execute initialization code
for the package or set the __all__ variable, described later.

Source : http://effbot.org/pyfaq/what-is-init-py-used-for.htm

The with statement

2013-07-28T15:23:00-04:00

So the chances are that you already know about the with statement but some of us do not know about it. Lets face the reality the with statement saves us a lot of time and reduces our code base. Just imagine you are opening a file width python and then saving something to it. You would normally do :

file = open('file.txt','w')
file.write("freepythontips.wordpress.com")
file.close()

But what if an exception occurs while writing to file ? then the file won't be closed properly. I know how much trouble it can cause. It once happened with me that i was running a web scrapper and it was running for past three hours and then unexpectedly an exception occured and my whole csv file got corrupted and i had to run the scrapper again. So lets come to the point. What method can you adopt to prevent such disaster. Obviously you can not prevent the exception when it is unexpected but you can take some precautions. Firstly you can wrap up your code in a try-except clause or better yet you can use a with statement. Let's first talk about try-except clause. You would normally do something like this :

try:
    file = open('file.txt','w')
    file.write(data)
except:
    # do something in case of an exception
finally:
    # here comes the trick. The finally clause gets 
    # executed even if an exception occured in try 
    # or except clause. now we can safely close this
    # file.
    file.close()

Now lets talk about the with statement. While using a with statement you would normally do:

with open('file.txt','w') as file:
    file.write(data)

Okay so here while using the with statement we do not have to close the file explicitly. It is automatically closed after the data is written to the file.

Do share your views about this post below in the comments.

A simple python server

2013-07-28T02:25:00-04:00

I guess it's not a simple python server. It is really simple and easy to deploy python server. Just imagine you want to share a folder on your pc with someone on your network. What you will do ? Obviously you will start the sharing service but what if you have python installed and want an easier solution ? Here you go. You can just type in :

$ python -m SimpleHTTPServer

This should be typed in the terminal or command prompt if you are a windows guy. Just open command prompt. Navigate to the directory which you want to share and then run this command. Unlike the normal sharing services this server will also show you the IP addresses of the PCs which are accessing that folder. Pretty slick right ?

Do share your views about this post below in the comments.

In-place value swapping

2013-07-28T02:08:00-04:00

Okay so this is kinda new for some of us. If you are a seasoned programmer then this might not be surprising for you but if you are a new programmer then this is something you really need to know. Okay so here we go. In normal situations if you would want to swap the values of two variables then this would be something you would go after :

a = 1
b = 2
c = a
a = b
b = c

But there's a more simple method which will merely take three lines. Here it is :

a = 1
b = 2
a, b = b, a

This is surely a simple way to swap values. Do comment below and tell me your views about this.

The beginning of a new blog.

2013-07-28T01:29:00-04:00

Hi there, Today i am beginning my very own personal blog related to python. Let me first tell you who i am. I am a computer programmer from Pakistan. I have made a lot open source contributions. I have contributed mainly to youtube-dl.It's a python script which allows you to download videos from various websites like youtube, facebook, dailymotion, metacafe, etc . I also run a website with the name of http://www.soundcloud-dl.appspot.com/ It allows you to search and download music from soundcloud.com . I know that sounds cool :D . I have done all of it's backend programming in python. I also have some useful scripts on my github page and if you would like to check them out then you are most welcome. This blog will serve you new python tips every other day. This blog is not for seasoned programmers but those who are starting out with python or want to enhance their knowledge about this language. lastly if you have any comments, suggestions or want my help with something then feel free to email me on yasoob.khld@gmail.com . I guess now it's time to say goodbye.

Bite Sized Python Tips

Python decorators finally demystified

- Everything in python is an object (Functions too!):

- Defining functions within functions:

- Returning functions from within functions:

- Giving a function as an argument to another function:

- Writing your first decorator:

- Decorators Demystified:

Kivy 101: How to Use BoxLayouts

Kivy, Python and BoxLayout

Nesting BoxLayouts

Kv+Python and BoxLayout

Nesting BoxLayouts with Kv

Wrapping Up

Related Readings

Download the Source

Reverse engineering myspace

Tools :

Lets start :

Writing Python script :

Introduction to unittest

A simple script

Our first test

Test Discovery

Furthur

Recent posts by Guido Van Rossum

New Python books for GUI development

* Tkinter GUI Application Development Hotshot :

* Test-Driven Development with Python :

Some Python projects worth looking at

PyDev 3.0 Released

Python 3.4 to be equipped with pip by default

PyCharm 3.0 Community Edition

Python News 13 Oct 2013

All About Decorators in Python

Python's functions are objects

Functions references

Handcrafted decorators

Decorators demystified

Eventually answering the question

Passing arguments to the decorated function

Decorating methods

Passing arguments to the decorator

Let's practice: a decorator to decorate a decorator

Best practices while using decorators

How can the decorators be useful?

The Python yield keyword explained

Iterables

Generators

Yield

Your code explained

Controlling a generator exhaustion

Itertools, your best friend

Understanding the inner mechanisms of iteration

Python Quick Reference v2.7

All about the metaclasses in Python!

Classes as objects

Creating classes dynamically

What are metaclasses (finally)

The __metaclass__ attribute

Custom metaclasses

Why would you use metaclasses classes instead of functions?

Why the hell would you use metaclasses?

The last word

Making a Reddit + twitter Bot

A surprize for you

Making a twitter bot in python

Python Resources

Beginner's Delight

Style Guide and Idioms

Dictionary

Decorators

Generators

Yield

Context Managers

Unicode

Metaclasses

Documentation

Sphinx

Debugging

The `metaclass` attribute

What is init.py ?