GSoC 2014 - Porting to Python3 Round Up

Hello Everyone,

This is my round up post for my GSoC 2014 - Porting to Python3 Project.

I know I am almost a month late but it's better late than never i guess.
Just after the GSoC coding period was over my internet broke down and I was completely without internet for almost two weeks and just after that I got busy with my exams. My exams finished yesterday, so here I am today.

When I started the project I expected to complete it even before the deadline, but the more I got into it I realized how wrong I was. The actual changes I made were trivial Python3 syntax changes but finding where the problem was kind of difficult because of the subtle differences between Python2 and 3 due to which building the modules wouldn't completely stop but gave unexpected results. So most of my time this summer went into researching, debugging and handling multiple modules together.

So, in the beginning of the project i researched the changes that needed to be made from Python2 to Python3 then i started with porting sugar-build to Python3 which took quite some time because it was the first module. All the modules that I ported and the changes I made can be found in my previous posts.

I also researched the changes that needed to be made in telepathy-python which is now deprecated and not compatible with Python3.Basically , we would need to replace telepathy-python with gobject-introspection in sugar in order for everything to work correctly in Python3.

By the end I was able to port and build all the modules of sugar except sugar-datastore to Python3. Now all those modules are compatible with Python2 as well as Python3.

Although this is a big step in the right direction , it will still take along time to completely shift all the sugar modules to Python3.

In the end I just wanna thank the Sugarlabs for making me a part of their community and specially my mentors Walter Bender, Gonzalo Odiard and Sameer Verma for helping me with any problems that I faced.

Although I think that I could have learnt a lot more if I had a more personal and bonding interaction with any of my mentors but I still learnt a lot during this summer and I am grateful for that.
I'll also try to stay in touch with the happenings of Sugarlabs and try to contribute whenever possible.

I guess that's it for today , thanks for reading and goodbye!

Changes in string function in python

error: <class 'AttributeError'>: 'module' object has no attribute 'strip'
Line of Error: _extraArguments = string.split(_extraArguments)

This is one of the errors i got while building sugar.
As it turns out Python3 does not support string functions from the string module anymore and Python2 discourages it too. All the functions are now methods for the string object.
So the correct way to solve the above error would be to use the split method instead of the string.split function.
Correct Code:  _extraArguments = _extraArguments.split()

These string functions are used throughout sugar in various modules so i changed these functions everywhere in most of the sugar modules to their respective methods.

Porting to Python3 Progress

In the last couple of weeks I worked on porting the following sugar modules and submodules to Python3, you can check out  my progress in my own repositories(they are not complete yet , so there might still be some errors):

1) Original repo: https://github.com/sugarlabs/sugar-build
    My repo: https://github.com/curiousguy13/sugar-build
    Changes: Changes in Python3 syntax ,string usage etc in osbuild.py and and different commands in the command folder and called virtualenv with Python3.
2) Original repo: https://github.com/dnarvaez/osbuild
    My repo: https://github.com/curiousguy13/osbuild
    Changes: These are the required changes that I made in the osbuild module:      https://github.com/curiousguy13/osbuild/compare/dnarvaez:master...curiousguy13:master
3) Original repo: https://github.com/dnarvaez/broot
    My repo: https://github.com/curiousguy13/broot
    Changes: Some other basic Python3 syntax changes and some errors that i reported in my common porting errors wiki blog post.
4) Original repo: https://github.com/dnarvaez/plog
    My repo: https://github.com/curiousguy13/plog
    Changes: Just a couple of changes as the module name Queue was changed to queue in python3 and the name basestring has disappeared.
5) Original repo: https://github.com/dnarvaez/sourcestamp
    My repo: https://github.com/curiousguy13/sourcestamp
    Changes: This was a C module written for Python2 but there have been some changes in the C-api for Python3.
The new PEP standard for initialization of extension module: http://legacy.python.org/dev/peps/pep-3121/
Porting Extension modules to Python3: https://docs.python.org/3/howto/cporting.html.
So with the help of  the above links , I did the necessary changes in the sourcestamp module.

6) Original repo: https://github.com/dnarvaez/gwebsockets
    My repo: https://github.com/curiousguy13/gwebsockets
    Changes: Basic python3 changes using and iteration of 2to3 and they seem to be working for now.

In the following modules, I have done all the trivial Python3 syntax changes and currently working on removing the remaining bugs and errors.

7) Original repo: https://github.com/sugarlabs/sugar
    My repo: https://github.com/curiousguy13/sugar

8) Original repo: https://github.com/sugarlabs/sugar-toolkit-gtk3
    My repo: https://github.com/curiousguy13/sugar-toolkit-gtk3

9) Original repo: https://github.com/sugarlabs/sugar-datastore
    My repo: https://github.com/curiousguy13/sugar-datastore

10) Original repo: https://github.com/sugarlabs/sugar-artwork
    My repo: https://github.com/curiousguy13/sugar-artwork

So, currently I am working on building and integrating all the modules successfully together and sorting out the errors and bugs that come in my way.

Common porting errors wiki

I think it could be helpful to someone (at the very least it would be helpful to future me) if i mention here the problems and errors i faced in porting sugar to python3 and their solution according to me.
So here are the errors: 

SyntaxError: invalid syntax
Line of error: except Exception, e:
Explanation: The correct syntax for exception in Python3 use as instead of a comma
Correct code: except Exception as e:

ImportError: No module named 'urllib2'
Line of error: import urllib2
Explanation: In Python3 urllib2 has been merged with urllib , so there is no module named urllib2. You can directly import the required module like urllib.request or urllib.error .
http://legacy.python.org/dev/peps/pep-3108/#urllib-package
Correct Code: import urllib.request, urllib.error, urllib.parse

ImportError: No module named 'StringIO'
Line of error: import StringIO
Explanation : The StringIO and cStringIO modules are gone. Instead, import the io module and use io.StringIO or io.BytesIO for text and data respectively.
Correct code: import io

ValueError: can't have unbuffered text I/O
Line of Error: sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
Explanation: As the error says, we cannot have unbuffered text I/O in Python3. So i have removed this line for now, although there are some other ways to get unbuffered text I/O like using the -u argument.
Correct Code:

/usr/bin/env: python3: No such file or directory
Explanation : On some debugging, i found out that in my case the error was due to the virtualenv . As the virtualenv was running using python2 , it couldn't find python3 inside the virtual environment. So this error may vary from case to case and we would need to find out why the interpreter cannot find the python3 directory
Line of error: subprocess.check_call(["python2.7",
                           os.path.join(source_dir, "virtualenv.py"),
                           "-q", "--system-site-packages",
                            get_virtualenv_dir()])
Correct Code: subprocess.check_call(["python3.3",
                           os.path.join(source_dir, "virtualenv.py"),
                           "-q", "--system-site-packages",
                            get_virtualenv_dir()])

TypeError: Unicode-objects must be encoded before hashing
 Line of Error: path_hash.update(self._config_path)
Explanation: Unlike Python2, in Python3 all the hashlib fucntions take in bytes rather than strings so we need to encode the string before passing.
Correct Code: path_hash.update((self._config_path).encode('utf-8'))

TypeError: expected bytes, bytearray or buffer compatible object
Line of error: base64_hash = base64_hash.replace("+", "0")
Explanation: The .replace method requires a string input , hence we need to decode what we encodeed in the previous error.
Correct Code: base64_hash = base64_hash.decode('utf-8')
                        base64_hash = base64_hash.replace("+", "0")

TypeError: Type str doesn't support the buffer API
Line of Error: for mounted in mount_output.split("\n"):
Explanation: This is also a type error like the previous one which means we are again confusing bytes and strings. On debugging, i found out that mount_output is actually a byte object and thus string method split is not compatible with it, so we first need to decode it.
Correct Code: mount_output = mount_output.decode('utf-8')
                        for mounted in mount_output.split("\n"):

SyntaxError: assignment to keyword

Line of Error: False, True = 0, 1

Explanation: In Python3 True and False are keywords instead of literals , so they cannot be reassigned.

Refer to this article for the history of True/False keywords: http://python-history.blogspot.in/2013/11/story-of-none-true-false.html

Correct code: Just remove this line because True and False are already defined as keywords

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 79: ordinal not in range(128)

Explanation: seems to be another one of those strings vs unicode vs bytes problem.
Correct code : b'' instead of ''

AttributeError: 'module' object has no attribute 'find'
Line Of Error: if string.find(thing, '=') >= 0:
Explanation: The string.find method has been deprecated in python3 , use the .find() method of strings instead
Correct code: if thing.find('=') >= 0:

FileNotFoundError: [Errno 2] No such file or directory: '/home/curiousguy13/py3sugar/sugar-build/build/commands/common.pyc'
Line Of Error: os.unlink(os.path.join(common.commands_dir, "common.pyc"))
Explanation: In python3 the the compiled header files are stored in __pycache__ instead of the same directory. see more here : http://stackoverflow.com/questions/154443/how-to-avoid-pyc-files
So, just changing the directory address to __pycache__ in common.py solved the issue. 

PS: The list is not exhaustive and i will keep on adding new errors and problems that i face with their solutions

WARNING: Use the above post at your own risk. The explanation and the solution of errors are according to my own knowledge of Python and some online research and they might be wrong in which case you can point out the mistake in the comments and i can correct it.

Porting to Python3 Progress

Last week I tried to solve the telepathy issue. The telepathy issue is basically that the telepathy library used in sugar is deprecated and not compatible with Python3 so we had to do something about it.

On researching, I found out that telepathy-glib is the new library which has taken over for telepathy-python and we need to write the telepathy-glib code in gobject convention in order for it to be compatible with introspection and also telepathy-glib is compatible with Python3.

But i had no idea what all these were, so i started reading up on telepathy,telepathy-glib,gobject,introspection,dbus etc. So all this new stuff was a little overwhelming and was taking too much of my time so in friday's sugar meeting, Walter Bender suggested to leave telepathy for now and port other modules first and that is exactly what i am going to do this week and i'll try to port as much code as possible this week and try to increase the test coverage too and after that we'll take care of telepathy.

Also side by side I continued working on porting sugar-build. 

Porting Sugar to Python3 Progress

I started with porting sugar-build to Python3 as it is used to build the rest of the sugar environment.

Until now, I have used 2to3[1] script with all fixers enabled on the sugar-build module which contained an osbuild script, python scripts for various commands and the test. And now I am executing the osbuild script and removing the errors one by one by doing the necessary changed in the code. Right now, it is taking time to deal with the errors which mainly include the proper usage and handling of strings in Python3[2].
It is worth noting though that as I learn how to deal with these errors, the porting further is going to get easier and easier.

I have cloned the sugar-build repo on GitHub and am working on it.
So although, it doesn't work perfectly right now you can follow my progress here:

https://github.com/curiousguy13/sugar-build/tree/py3

[1]: https://docs.python.org/2/library/2to3.html
[2]: http://www.diveintopython3.net/strings.html

Virtual OS trouble

While I was just getting ready to get started with the porting of sugar, the network configuration in my virtual OS (fedora) changed and I could not connect to the internet anymore. I looked for solutions online and even  asked on the vmware irc but still no solution . So, after a couple of days of trying everything from changing network setting in the vmware and fedora to doing a system restore, I still couldn't figure out the problem , so I finally decided to reinstall fedora. Yesterday , I reinstalled fedora and it's updates and also had to reinstall sugar. So, most of the time yesterday went in setting up the development environment again. :(

Back After Exams!

So my exams just finished yesterday (thankfully!). As much as I would have loved to get a head start to the project, I could not do anything for the first 10 days of the coding period .

I can now finally get back onto the project.

I'll try to cover up the lost time and try to finish the project as soon as possible.
I am updating sugar right now and i am gonna start with porting sugar-build first.

Many more posts will be coming from now on.

I'm back!!

Porting To Python3

So, i did some research and found that it would be best to port sugar in the following way:
1) Resolve the dependency issue - mainly telepathy . Telepathy-python has been deprecated , so what would be the best way to deal with this? Would it best to just port it to python3 or something else?

2)Port sugar-build to Python3

3)Port Sugar-shell to Python3

4)Create a new Sugar-toolkit-gtk3-python3 from the old one and also keep the Sugar-toolkit-gtk3 for activities that haven't yet ported to python3 and slowly deprecate it.

How the porting would be done:
1) Choose a single module.
2)Port the tests carefully and if there is not full test coverage , write some additional tests.
3) Update the code carefully to pass the tests.


These links were and will be helpful as we can draw on the experience of organisations who have already ported their codebase to python3.
http://twistedmatrix.com/trac/wiki/Plan/Python3
https://wiki.openstack.org/wiki/Python3#Port_Python_2_code_to_Python_3

PS: This post was discussed in the mailing list and there have been some changes in the strategy which i will mention in one of the upcoming blog posts.

SELECTED FOR GSoC 2014!

It has been a while since the list of accepted students were announced but I was so busy with college work that I forgot to update the blog. So here goes my first blog post ever:
On the day that the result for GSoC was to be announced I was actually at peace with the thought of not getting selected as I had learnt so much in the last 3 months from the open source community. Even if I didn't get selected I would've continued working with many interesting open source organisations (and I still intend to).Even though apparently I was at peace with not getting selected , my hands were shaking with excitement just before the announcement of results.
When on the night of 21st April, i got the e-mail that I had been selected for GSoC 2014 , I felt the euphoria that I hadn't felt in a long time . It felt amazing to be selected for something that 6 months earlier I had only dreamt of .
The project that I am selected for is : Porting Sugar to Python3 under Sugar Labs.
I want to thank Walter Bender and the Sugar Labs community for being there whenever I needed them and for accepting me to be a part of the sugar community.
I am really excited to be a part of the Sugar and look forward to working for it and making it better through GSoC and beyond.