Infinitely nested lists in Python

Readers of this blog know that I sometimes like to write about some strange, unexpected, and unusual things in Python that I stumble across. This post is another one of those.

First, look at this

>>> a = []
>>> a.append(a)
>>> a
[[...]]

What am I doing here? I’m creating a list, a, and I’m adding it to itself. What you end up with is an infinitely nested list. The first interesting thing about this is that Python is smart enough to not explode when printing this list. The following should convince you that a does indeed contain itself.

>>> a[0] is a
True
>>> a[0] == a
True

Now, if you have programmed in C, or a similar language that uses pointers, this should not come as a surprise to you. Lists in Python, like most things, do not actually contain the items inside them. Rather, they contain references (in C terminology, “pointers”) to the items inside them. From this perspective, there is no issue at all with a containing a pointer to itself.

The first thing I wondered when I saw this was just how clever the printer was at noticing that the list was infinitely nested. What if we make the cycle a little more complex?

>>> a = []
>>> b = []
>>> a.append(b)
>>> b.append(a)
>>> a
[[[...]]]
>>> b
[[[...]]]
>>> a[0] is b
True
>>> b[0] is a
True

So it still works. I had thought that maybe repr just catches RuntimeError and falls back to printing ... when the list is nested too deeply, but it turns out that is not true:

>>> a = []
>>> for i in range(10000):
...     a = [a]
... 
>>> a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: maximum recursion depth exceeded while getting the repr of a list

And by the way, in case you were wondering, it is possible to catch a RuntimeError (using the same a as the previous code block)

>>> try:
...     print(a)
... except RuntimeError:
...     print("no way")
... 
no way

(and you also may notice that this is Python 3. Things behave the same way in Python 2)

Back to infinitely nested lists, we saw that printing works, but there are some things that don’t work.

>>> a[0] == b
True
>>> a[0] == a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: maximum recursion depth exceeded in comparison

a[0] is b holds (i.e., they are exactly the same object in memory), so == is able to short-circuit on them. But to test a[0] == a it has to recursively compare the elements of a and a[0]. Since it is infinitely nested, this leads to a recursion error. Now an interesting question: why does this happen? Is it because == on lists uses a depth first search? If it were somehow possible to compare these two objects, would they be equal?

One is reminded of Russel’s paradox, and the reason why in axiomatic set theory, sets are not allowed to contain themselves.

Thinking of this brought me to my final question. Is it possible to make a Python set that contains itself? The answer is obviously no, because set objects can only contain hashable objects, and set is not hashable. But frozenset, set‘s counterpart, is hashable. So can you create a frozenset that contains itself? The same for tuple. The method I used for a above won’t work, because a must be mutable to append it to itself.

isympy -I: A saner interactive environment

As promised, here is another post describing a new feature in the upcoming SymPy 0.7.2.

Automatic Symbol Definition

While not as ground breaking as the feature I described in my last post, this feature is still quite useful. As you may know, SymPy is inherently a Python library, meaning that it lives by the rules of Python. If you want to use any name, whether it be a Symbol or a function (like cos), you need to define it (in the case of Symbols), or import it (in the case of functions that come with SymPy). We provide the script isympy with SymPy to assist with this. This script automatically runs IPython (if it’s installed), imports all names from sympy (from sympy import *), and defines common symbol names (like x, y, and z).

But if you want to use a Symbol that is not one of the ones predefined by isympy, you will get something like

In [1]: r*x
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
 in ()
----> 1 r*x

NameError: name 'r' is not defined

The best solution for this has been either to type var('r'), which will create the Symbol r and inject it into the namespace, or to wrap your text in a string and pass it to sympify(), like sympify("r*x"). Neither of these are very friendly in interactive mode.

In SymPy 0.7.2, isympy has a new command line option, isympy -a, which will enable a mechanism that will automatically define all undefined names as Symbols for you:

In [1]: r*x
Out[1]: r⋅x

There are some caveats to be aware of when using this feature:

• Names must be undefined for isympy -a to work. If you type something like S*x, you’ll get:

  In [3]: S*x
  ---------------------------------------------------------------------------
  TypeError                                 Traceback (most recent call last)
  <ipython-input-3-6656a97ea7b0> in <module>()
  ----> 1 S*x
  
  TypeError: unsupported operand type(s) for *: 'SingletonRegistry' and 'Symbol'
  

  That’s because S is already defined (it’s the SingletonRegistry, and also a shortcut to sympify()). To use a name that’s already defined, either create it manually with var() or delete it using del.

• This only works on the top level namespace. If you define a function with an undefined name, it will not automatically define that symbol when run.
• This works by catching NameError, defining the name, and then re-running the expression. If you have a multiline statement, any lines before the undefined name will be run before the NameError will be caught. This usually won’t happen, but it’s a potential side-effect to be aware of. We plan to rewrite it using either ast or tokenize to avoid this issue.
• Obviously, this is intended for interactive use only. If you copy code and put it in a script, or in some other place where someone might be expected to run it, but not necessarily from isympy -a, you should include symbol definitions.

Automatic int to Integer Conversion

A second thing that is annoying with Python and SymPy is that something like 1/2 will be interpreted completely by Python, without any SymPy. This means that something like 1/2 + x will give either 0 + x or 0.5 + x, depending on whether or not __future__.division has been imported. isympy has always ran from __future__ import division, so that you’ll get the latter, but we usually would prefer to get Rational(1, 2). Previously, the best way to do this was again to either run it through sympify() as a string, or to sympify at least one of the numbers (here the S() shortcut to sympify() is useful, because you can type just S(1)/2).

With SymPy 0.7.2, you can run isympy -i, and it will automatically wrap all integers literals with Integer(). The result is that 1/2 produces Rational(1, 2):

In [1]: 1/2 + x
Out[1]: x + 1/2

Again, there are a couple of caveats:

• If you want to get Python style division, you just need to wrap both arguments in int():

  In [2]: int(1)/int(2)
  Out[2]: 0.5
  

  Of course, if you just want a floating point number, you can just use N() or .evalf()

• This works by parsing the text and wrapping all integer literals with Integer(). This means that if you have a variable set to a Python int, it will still act like a Python int:

  In [6]: a = int(1)
  
  In [7]: b = int(2)
  
  In [8]: a/b
  Out[8]: 0.5
  

  Note that to even do that example, I had to manually make a and b Python ints by wrapping them in int(). If I had just done a = 1, it would have been parsed as a = Integer(1), and I would have gotten a SymPy Integer. But this can be an issue if you use the result of some function that returns an int (again, note that most functions in SymPy that return integers return Integer, not int).

• The same as before: this will only work interactively. If you want to reuse your code outside of isympy -i, you should take care of any int/int by rewriting it as S(int)/int.

Since these are both useful features, we’ve added a way that you can get them both at once: by doing isympy -I (the “I” stands for “Interactive”). If we add similar features in the future, we will also add them to the -I shortcut (for example, we may add an option to allow ^ to automatically be replaced with **).

SymPy Live Sphinx Extension

I didn’t blog about SymPy all summer, so I thought I would write a post about my favorite feature of the upcoming SymPy 0.7.2 release.  In fact, this feature has got me more excited than any other feature from any version of SymPy.  Yeah, it’s that good.

The feature is the SymPy Live Sphinx extension.  To start, if you don’t know about it, check out SymPy Live.  This is a console that runs on the App Engine.  We’ve actually had this for quite some time, but this winter, it got a huge upgrade thanks to the contribution of some GCI students.  Basically, SymPy Live lets you try out SymPy in your browser completely for free, because it runs all the code on the App Engine.  Actually, the console is a full Python console, so you can actually run any valid Python command on it.  This past winter, GCI students upgraded the look of the site, added a mobile version (visit live.sympy.org on your phone), and added other neat features like search history and autocompletion.

Now, Sphinx is the documentation system that we use to generate SymPy’s html documentation. Last year, when I was at the SciPy Conference, Mateusz had an idea at the sprints to create an extension linking SymPy Live and Sphinx, so that the examples in Sphinx could be easily run in SymPy Live.  He didn’t finish the extension, but I’m happy to report that thanks to David Li, who was also one of the aforementioned GCI students, the extension is now complete, and is running live on our development docs.  When SymPy 0.7.2 is released (soon I promise), it will be part of the oficial documentation.

The best way to see how awesome this is is to visit the website and check it out.  You will need a modern browser (the latest version of Firefox, Safari, or Chrome will work, IE might work too).  Go to a page in the development docs with documentation examples, for example, http://docs.sympy.org/dev/tutorial.html#algebra, and click on one of the examples (or click on one of the green “Run code block in SymPy Live” buttons). You should see a console pop up from the bottom-right of the screen, and run your code.  For example:

Example of the SymPy Live Sphinx extension at http://docs.sympy.org/dev/tutorial.html#algebra. Click for larger image.

 

You can access or hide the console at any time by clicking on the green box at the bottom-right of the page.  If you click on “Settings”, you will see that you can change all the same settings as the regular SymPy Live console, such as the printer type, and the keys for execution and autocompletion.  Additionally, there is a new setting, “Evaluation Mode”, which changes how the Sphinx examples are evaluated.  The default is “Evaluate”.  In this mode, if you click on an example, it is executed immediately.  The other option is “Copy”.  In this mode, if you click an example, it is copied to the console, but not executed right away. This way, you can edit the code to try something different.  Remember, this is a full fledged Python console running SymPy, so you can try literally anything

So play with this and let us know what you think.  We would love to hear ways that we can improve the experience even further.  In particular, I think we should think about ways to make the “Copy” mode more user-friendly.  Suggestions welcome!  Also, please report any bugs.

And one word of warning:  even though these are the development docs, SymPy Live is still running SymPy 0.7.1.  So some examples may not work until 0.7.2 is released, at which point we will update SymPy Live.

I believe that this extension represents the future of interactive documentation. I hope you enjoy.

Emacs: 7 months later

In my final post about my switching to Emacs, a commenter, Scott, asked me, “It has been a while since you started using Emacs. I’m just curious. How is your experience so far now that you have more experience and a more complete configuration?” My reply was getting quite long, so I figured it would be best suited as a new post.

The short answer is, mostly the same since I wrote that Vim vs. Emacs (part 3). Once you use something a lot, you notice all kinds of things that could use improvements. Some of them are just minor annoyances. For example, many interactive commands in Emacs (but not all!) require you to type out “yes” instead of just “y” as a confirmation. Others are more serious, like the need for a real replacement of SuperTab from vim.

I actually didn’t have much free time to work on configuring Emacs during the school year, and once the summer started, my computer died, and I’ve been working of an old laptop running Linux until I can get a new one. Fortunately, I had the foresight to put all my Emacs configuration online on GitHub, so it was easy to get my configuration again. I’ve noticed that in Linux, the Alt key (i.e., Meta) is used for other things, so it doesn’t work so well in Emacs (e.g., pressing Alt without any other keys sometimes activates a menu that removes the keyboard focus, and also C-M shortcuts don’t seem to work at all).

I’ve memorized very few keyboard shortcuts, even ones that might be useful to me (e.g., I don’t remember the shortcut to jump to a matching parenthesis). Usually, if I am using some mode or something and I want to know how to do something, I just Google it, and generally find the answer within a few seconds.

There are several major configuration issues that I’ve yet to address, either due to lack of time or because I couldn’t find a suitable solution. A SuperTab replacement is one. This is actually a big one, because scrolling through a file just to see what’s there is getting older and older, as is searching just to jump to a function definition. If anyone knows of a good way to do this, please let me know. I mainly need it for Python files, but having it other modes as well would be nice. Basically, I just want something that shows me all the class and function definitions in the file, in order, that I can easily select one and jump to it.

Related to searching, searching in Emacs sucks. I’m using isearch+, which is an improvement, but it still bugs me that search does not wrap around by default. Also, for some reason, pressing delete doesn’t delete the last character you typed, but the last character that it matched. That may sound minor, but I use it a lot, so it’s really gotten on my nerves.

Regular expression searching in Emacs is useless. I can never get it to work (usually because of differences between () and \(\)). What I really want is an interactive, user friendly, regular expression search/search and replace tool. There’s regexp-builder, but that’s useless because once you build the regular expression, you have to manually copy it and paste it into the real regular expression search function to actually use it. And it doesn’t work with search and replace.

This last semester I had a semester long project in C. For that, flymake-mode was a godsend. It requires a bit of manual configuration (you have to add something to your Makefile, and you have to add some stuff to .emacs as always to enable it by default), but once you do that, it just works. If you don’t know what this is, basically, it highlights the compiler errors in your source in real time, as you type it. So instead of doing something stupid twenty times, and then compiling and finding them all, you do something stupid once, see the error, and don’t do make the mistake any more. It’s also nice to close your editor and know that your code will compile.

The Python mode I am mixed about. On the one hand, it’s really awesome how smart it is about indentation. On the other hand, the syntax highlighting is just shy of what I want (granted, it’s pretty good, but I want better than that). For example, I want to be able to color docstrings, single quoted strings, and double quoted strings differently. It would also be awesome to get some coloring in docstrings itself. I’m thinking markdown mode for any text that’s in a docstring, except for doctests, which are colored in Python mode (or some variant).

Some things I’ve not really cared much about yet because I haven’t used that type of file yet. For example, I’m currently writing this post in Emacs, and just now noticing the deficiencies in html-mode (e.g., I want an easy way to select text and turn it into a link, just like in the WordPress editor).

Finally, I’ve been trying to write my own theme. That process has been slow and slightly painful. Emacs is currently in the process of moving to themes, though, so this is to be expected. When Emacs 24 is actually released I think it will be fair to judge how well this feature works.

That’s my wishlist (or most of it anyway). But there are positive things too. auto-complete-mode, which I mentioned at the top of my previous blog post, is absolutely awesome. I think this extension alone has made me more productive.

Some things I take for granted, like automatic spell checking of strings and comments in Python (not enabled by default, but not hard to configure either). Thanks to someone on an Emacs mailing list, I have the perfect automatic clearing of trailing whitespace, that automatically leaves your whitespace before the cursor in the buffer, but still writes the clear to the file (see my .emacs file from my dotfiles repo linked to above for details).

I’ve been hoping to learn Emacs lisp, so that I could remedy many of these problems on my own, but so far I haven’t really had the free time. Lisp is a very confusing language, so it’s not easy to jump into (compared to the language vim uses, which I found easy enough to hack on without knowing at all).

Ultimately, I’m quite pleased with how user friendly Emacs is, and with how easy it is to find out how to do almost anything I want just by Googling it. Configuration is an uphill battle. Emacs has a ton of great packages, many of which are included, but almost none are enabled by default. Just today I discovered Ido mode, thanks to David Li. I feel that in the long term, as I learn Emacs Lisp, I can make it do whatever I want. It provides a good baseline editing experience, and a good framework for configuring it to do whatever you want, and also enough people use it that 99% of the things you want are already done by somebody.

How to install the development version of IPython Qtconsole and Notebook in Ubuntu

Both the awesome IPython notebook and Qtconsole are in the Ubuntu repositories, so if you just want to use the stable released versions, you can just do

sudo apt-get install ipython-notebook ipython-qtconsole

and be on your way. But the git development version has a lot of cool new features, and you may not want to wait for 0.13 to be released and make its way to the Ubuntu repos. But you may be thinking that to use those you will have to figure out all the dependencies yourself. Actually, it’s pretty easy:

# First install git, if you don't already have it
sudo apt-get install git
# Then, clone the IPython repo, if you haven't already.
git clone git://github.com/ipython/ipython.git
cd ipython
# Now just install IPython with apt, then uninstall it.  The dependencies will remain
sudo apt-get install ipython-notebook ipython-qtconsole
sudo apt-get remove ipython-notebook ipython -qtconsole ipython
# Now install the IPython git version in such a way that will keep up to date when you pull
sudo python setup.py develop

To update, just cd into that ipython directory and type git pull. That’s it. Now type ipython notebook or ipython qtconsole to get the magic.

EDIT: After you do this, apt-get will start bugging you every time that you use it that a bunch of packages are no longer needed. These are the ones that you do need for the qtconsole and the notebook, so you should not autoremove them as it says. Rather, set them as manually installed by copying the list of packages that it tells you about and sudo apt-get installing them.

Vim vs. Emacs (Part 3)

See parts 1 and 2.

Some more comments after using emacs for a while:

I finally found the perfect tab completion solution. It took way too much searching for how awesome it is. It’s called auto-complete-mode. The best way to get an idea of what this is is to watch this screencast. Basically, it shows you a completion list automatically. It uses the TAB key to do completion (to me, this is a no brainer, but for some reason, no other completion extension that I found did this, requiring you to do all kinds of nonsense in your .emacs file). It’s got cool features like simple fuzzy matching and intelligent matching (so the first completion is what you tend to use, instead of just the first one that matches). To quote the author, “a goal of auto-complete-mode is to provide a system that does what users want without any command.” I couldn’t agree with that goal more. If you install it, I recommend adding (define-key ac-mode-map (kbd "M-TAB") 'auto-complete) to your .emacs, so that you can use M-TAB to force the completion menu to come up. This generally happens automatically, but I think this is the only way to get fuzzy matching, for example. Actually, you can also just use (ac-set-trigger-key "TAB"), which intelligently sets TAB to complete or indent, based on which one you more likely want. This seems to work pretty well to me.

Speaking of indenting, emacs has a pretty nice indentation feature for Python. You just press TAB repeatedly, and it cycles through all the syntactically legal indentations. I find this to be more useful than the usual TAB indents behavior of most editors. Note that by default, it won’t automatically indent, even with trivial indentations (i.e., keeping the previous indentation). This is easy to fix, though. Just add (define-key global-map (kbd "RET") 'newline-and-indent) to your .emacs file. This will make RET do the same thing as C-j, i.e., basically the equivalent of RET TAB.

emacs comes with an extension that lets you work with version control systems, called VC. I don’t use it. I don’t like stuff messing with my git stuff behind my back (sounds like a good way to lose data to me), and I’m good enough with git commands straight that I don’t need the help.

But unlike all the other hundreds of emacs features that I don’t use, this one was seriously slowing down my workflow. It adds three or four seconds to the startup time of emacs when loading from within a git repository. So I did some Googling and added this to my .emacs file:

;; Disable all the version control stuff         
;; Makes emacs load much faster inside git repos 

(setq vc-handled-backends nil)

(unrelated: Why doesn’t WordPress support lisp as a language for syntax highlighting?)

This disables the version control stuff, making emacs load fast again (virtually as fast as vim, actually).

Speaking of making emacs go faster, make sure you compile all your extensions into byte code. For whatever reason, emacs doesn’t do this automatically, even though compiled files run much faster, and it doesn’t take very long. The easiest way is to use M-x byte-compile-file from within emacs. Just make sure that if you modify the .el file that you recompile the byte code, or it will continue to use the old version.

I finally figured out how to enable mouse support. For whatever reason, Googling got me nowhere with this, so I ended up asking on the help-gnu-emacs list, which was very helpful. The solution is to put

;; ===== Enable mouse support ====
                                      
(require 'xt-mouse)                   
(xterm-mouse-mode)

in your .emacs file. And then it just works. It needs some tweaking (e.g., it doesn’t play so well with momentum scrolling), but at least it works. I thought I was going to hang myself without mouse support. Because frankly, as good as the movement commands are, moving with the mouse is so much easier sometimes (the same is true for vim too, btw).

I compiled the git version of emacs (it’s not very hard btw). I did this to see if the mouse suport “bug” was fixed there, but I’ve gone ahead and kept using it, as it’s nicer. But I didn’t figure out how to configure it to not load in an X window. So for now, I’ve aliased emacs to emacs -nw. I’m sure I just need to add some flag to configure, but I haven’t gotten around to looking it up yet.

I found out how to allow editing in the Isearch mode (again, thanks to the help-gnu-emacs list). You need to install the isearch+ extension, add the following to your .emacs,

;; ===== isearch+ =====         

(require 'isearch+)

and most importantly, you need to edit the file and uncomment all the commmands you want to allow. If you follow my link above, it goes to my personal dotfiles repo, where I’ve already done that.

On a related note, this is the first of several emacs extensions I’ve installed that I’ve edited the extension file itself for. The rest, I just had to add some code to .emacs. In most cases, there was already a variable or suggested code snippet to add to .emacs to get what I wanted.

On the other hand, with vim, I had to edit virtually every extension I installed to make it do what I want. I’m not sure what this means, though. It could be a statement about one of many things: how the emacs community provides nicer defaults, how the vim language is easier to use, and hence more inviting for me to edit the files, or how I haven’t gotten around to messing with certain things yet.

If you do a lot of work with LaTeX, check out AUCTeX. I haven’t used it enough yet to say much about it, but from what I’ve played around with, it’s pretty awesome. And if you use a windowed version of emacs, it’s got a really awesome preview mode.

If you’re bored, check out the predictive extension. It’s actually not as helpful as you’d think (unlike the very similar auto-complete-mode module mentioned above). But it’s kind of cool to turn on and play around with when you’re typing something. Maybe you’ll learn new words or something.

I could go on and on. I haven’t mentioned the most basic customizations (like how to setup four-space tabs). If you are starting to use emacs, I recommend going through M-x customize, and reading my .emacs file. And my best advice: if you want emacs to do something, first do M-x customize and search for what you want (EDIT: apparently searching customize requires emacs 24, i.e., the development version). If you don’t find what you want there (and you will surprisingly often), search Google. There are so many emacs users, that the chances of someone else wanting what you want are very likely. I’ve found the results from the emacs wiki to be particularly helpful. And one more thing: if you find an extension you like, double check first to see if it’s not already included in emacs. Emacs seems to like including good extensions in future releases, so an older extension has a good chance of already being included.

Some emacs questions:

I tried (define-abbrev global-abbrev-table "Ondrej" "Ondřej"), so that when I type Ondrej it give me Ondřej. But it doesn’t work. Is this a bug or what? If I do (define-abbrev global-abbrev-table "foo" "bar") and type “foo”, it turns into “bar”, but the above leaves Ondrej alone. EDIT: I guess this was an emacs bug. It doesn’t seem to be there any more (perhaps it was fixed with the git version or something).

Is there a way to reload .emacs without closing emacs? I’m doing that a lot these days. EDIT: I found it. Do M-x load-file RET ~/.emacs

Is there a good emacs equivalent of the vim tag list plugin (thanks for commenter Scott for pointing me to that in the first place)? I just want something that lists all the class and function definitions in a Python file in order, so I can easily jump to the one I want, or just get an overview of the file.

This Tuesday will mark the point where I will have spend as long using emacs as I did using vim. But already, I feel more competent with emacs. I won’t repeat what I said in my last post, but I just want to say that the ability to edit and write at the same time makes me way more productive. The fact that it uses keyboard shortcuts that I’m already used to probably helps a lot too. Even so, I’ve not used any kind of cheat sheet for emacs (since I never really found any that were any good), and yet I feel like I’ve memorized more key commands now than I ever did with vim, for which I did use a cheat sheet.

So I really don’t see myself going back to vim at this point.

I’m actually surprised. Virtually everyone I know who uses a command line editor uses vim. It’s definitely the more popular of the two. But having tried both, I can only speculate as to why. Vim has a much higher learning curve than emacs. Everybody grows up learning how to write text in editors like Microsoft Word, TextEdit, Notepad, etc., that all work fundamentally like emacs: if you type text, it enters the text. If you want to do advanced editing with the keyboard, you hold down some meta keys and type chorded keyboard shortcuts. The vim modal editing methodology is so different from this, that it surprises me that so many people go to the trouble of learning it (I mean, to the point that they are more efficient with it). I can see the benefit over GUI editors, which have nothing on either vim or emacs with regards to customization, or just the plain editing power that is really necessary for coding. My guesses why people use vim:

They are shown vim first, so just use it.

They are turned off by the massiveness of emacs (it seems contradictory to me, since the whole point of using a command line editor is to get more power, but I could see it).

They are turned off by emacs lisp.

Some combination of those.

Maybe the vim users out there could comment why they use vim. Am I missing something? Or are your heads just wired differently from mine? And if you use emacs (or anything else), I’d love to hear from you too?

At any rate, I recommend that anyone who wants to give command line editors a chance do what I did: learn both vim and emacs. My blog posts should be enough to give you some good advice. I went cold-turkey, and I recommend that you do too, but only do it if you won’t have any important editing to do for a few weeks, as your editing rate will slow down a lot as you are learning for both editors. And even though I think I am going to stick with emacs, learning vim was still valuable. Unlike emacs, vi is part of the POSIX standard, so it’s included in pretty much every UNIX distribution. I’ll be glad when I find myself on a minimal command line and know how to use a decent text editor. And anyway, you can’t really know which one will be your way until you try them both. I really thought I would end up using vim, as it was so popular among all the people I know who use command line editors. But I guess there is only One True Editor.

EDIT: I found out how to make emacs really fast. The key is to run one process of emacs in daemon mode, and have the rest connect to that. Then you only have to wait for the startup once (per computer session). To do it, just set your EDITOR to 'emacsclient -a "" -nw' (and you might also want to alias emacs to that as well). What this does is connect to the emacs daemon. The -a "" starts one if it isn’t already started (you can also do this yourself with emacs --daemon. If you only want to use the daemon version if you’ve specifically started it, replace "" with emacs. This will connect to the daemon if it’s running, and otherwise just start a new emacs process.

The -nw keeps it from running in window mode. Remove this if you use the GUI version of emacs. This is necessary to make it work correctly with multiple tabs. This is so fast that you should never really even need to use C-z to quickly exit emacs. C-x C-c is just fine, because reopening will be instantaneous. I like this because I was starting to accumulate background emacs processes that I forgot about.

This probably requires a fairly new version of emacs, possibly even the development version.

Vim vs. Emacs (Part 2)

As I noted in part 1, I have decided to switch to a command line text editor. I decided that, to be fair, I would try both vim and emacs. And to force myself to learn them, I decided to use them cold-turkey.

Since I’m going cold-turkey, I am doing this over my break from classes, so that I can weed out any difficulties during a period when I can live with slow text editing if necessary. This is a one month break. I have reached (roughly) the half way point. For the first half, I used nothing but vim to edit text. Now, I will use nothing but emacs.

Now that I’ve stopped using vim (for now anyway), my view of it isn’t much different from what I wrote in the first part. A lot of things there were addressed by commenters (or rather commenter). I still feel that it’s not an a method of text editing that fits my head. My entire life, I’ve used text editors where typing inserts text, and various control characters do things like move around faster.

Enter emacs. It does exactly this. Also a ton more.

I’ve only been using emacs for two days, but here are my impressions so far:

The tutorial is better. When you start emacs, it tells you how to start the tutorial. Just type C-h t (if you don’t already know, in emacs C- means CTRL- and M- means ALT-). Like I said last time, the very first thing you learn is how to scroll by more than one line at a time. That turns out to be a very useful thing to do. Also, the emacs tutorial did a better job of explaining how to use multiple files at once in emacs, which is something that I still don’t really know how to do very well in vim.

I have to give the vim tutorial some credit for one thing, though. It has better interactive examples. For example, in the vim tutorial, you have stuff like

  1. Move the cursor to the second line in the phrase below.
  2. Type  dd  to delete the line.
  3. Now move to the fourth line.
  4. Type   2dd   to delete two lines.

--->  1)  Roses are red,
--->  2)  Mud is fun,
--->  3)  Violets are blue,
--->  4)  I have a car,
--->  5)  Clocks tell time,
--->  6)  Sugar is sweet
--->  7)  And so are you.

whereas in the emacs tutorial, you just have

>> Kill a line, move around, kill another line.
   Then do C-y to get back the second killed line.
   Then do M-y and it will be replaced by the first killed line.
   Do more M-y's and see what you get.  Keep doing them until
   the second kill line comes back, and then a few more.
   If you like, you can try giving M-y positive and negative
   arguments.

which is a little more vague. So I have to give vim credit for that.

Everything’s a buffer. This line from the emacs tutorial really stuck with me: “ANY text you see in an Emacs window is always part of some buffer.” Emacs has really a awesome editing model, even simple things like M-f and M-b to move around words at a time, or M-DEL to delete whole words make things way faster. Vim of course has all of these too, albiet in a different way, but they aren’t everywhere. In emacs, everything is a buffer, which just means that everything supports all the standard emacs commands. So if you type M-x (roughly the equivalent of vim’s :) and start typing a command, you can move around and edit your command with emacs commands. One of the things that bothered me about vim was that when I was typing something with :, I couldn’t use vim’s text moving/modifying commands to manipulate the text. Typing ESC just canceled the command.

Exceptions: There are at least two exceptions I’ve found to this rule. First, if you do a search with C-s or C-r, no control commands work. If you type a search string, and then type M-DEL to try to delete the last word in your search string, you will instead delete the word where the cursor is! The solution I think is to use something like M-x re-builder instead. This was a little slow in my tests.

Second, the emacs manual is presented in the info program, which uses completely different key commands from every other program. This irked me quite a bit, because as soon as I finished the emacs tutorial, it pointed me to the manual, which was in info. Then, the first thing in info is a tutorial on how to use info! I opted to skip this. If I need any information on emacs, I’ll just do a Google search anyway, so I found this to be a waste of time.

It’s a little slower. I do notice a speed difference between emacs and vim. vim is much more lightweight, and it shows. Starting up emacs takes a second or two. Also, since a lot of the features are more interactive, they suffer from a speed delay. It’s not nearly slow enough to be a serious issue, though, and it’s still way faster than the GUI program I was using before (start up time).

The emacs tutorial suggests using C-z whenever you want to only temporarily close emacs. This seems like a good idea, and has worked pretty well for me so far (though I still usually close the whole thing with C-x C-c out of habit).

On a related note, I noticed that doing type-ahead while waiting for emacs to start up didn’t always work, whereas it always worked in vim (I do this, e.g., when waiting for the editor to start up when writing commit messages).

It’s way more user-friendly. Note that this is of course a relative term. I mean more user-friendly than vim, and pretty user-friendly for a command line program. Obviously, the most user-friendly text editors are the GUI ones used by the majority of the population (for that very reason). Actually, both vim and emacs are user-unfriendly in that if you accidentally open them and don’t know what they are or how to use them, you have no idea how to close them. But even less (i.e., man) is technically like this.

I’m not even referring to the different editing “modes” of the two editors, though you could easily argue that emacs style editing is more user-friendly than vim style editing. What I mean here is that emacs interaction is nice. When you type : in vim, start typing a command, and type TAB, it enters the first completion, regardless if it’s unique. Pressing TAB multiple times give the rest. In emacs, if you type M-x and start typing a command and type TAB, it pops up a temporary window with the list of all completions. It even colors the next character, so you can easily see what to type next to get what you want. As soon as you enter the command, the window disappears. (yes, I know about CTRL-D in vim, but to me tab completion should always work like it does in bash: complete characters if and only if they are unique in the list of completions)

By the way, when I said everything’s a buffer, I mean everything. If you want, you can exit the M-x entry (type C-g), type C-x C-b to show the list of buffers, C-x o to switch to it, scroll down to “Completions”, press Enter, and actually get in the completion list, as a buffer (there’s probably a less complicated way to get to it, by the way). You can then do whatever your heart fancies with it (save it to a file, copy it, whatever).

Customization is harder. This was expected, since I already knew that emacs used lisp. vim uses a language that is really easy to understand. I was able to modify all the vim plugins I installed very easily. If you want to change a setting globally in vim, just Google it and add one line to your .vimrc. In emacs, everything is in Emacs Lisp. I suppose prior experience with Lisp would probably help here.

In the vim tutorial, near the end, it told how to create a .vimrc file, and even gave a very useful sample one as a starter. In emacs, it took me a while to figure out how to do the equivalent (it took me a few Google searches just to figure out that the name of the configuration file in emacs is .emacs).

Actually, the emacs equivalent is way better than in vim, but it isn’t really mentioned anywhere. It took me probably a dozen Google searches before I learned about it (granted, I was looking for things in the same way I did for vim, lines to add to .emacs). What you have to do is type M-x configure. This opens what is basically a huge preferences dialog for emacs. You can then go through and set just about every settable emacs setting from there. The interface is very nice, as it’s interactive and tells you all about each setting. And you never have to touch Lisp. I’m still going through it, so I can’t comment more on it yet. But I recommend doing M-x configure as soon as you have finished the tutorial and have gotten used to editing with emacs, as you are invariably going to want to change some things (though I should note that emacs generally has nicer defaults than vim).

Better text editing methodology? Like I’ve already mentioned a bunch of times, the emacs editing model seems to fit my head better than the vim model. In emacs, you type text, and it inserts the text. If you want to do some advanced modification or move around, you type a control sequence. In vim, you type characters, and it does modifications or moves around. If you want to type text, you type i (or one of a few other characters) and type it. Then, if you want to move around or modify the text, you have to press ESC. This so-called “modular editing” doesn’t seem to work for me. For one thing, I like to rapidly switch back and forth between these two “modes” (editing and inserting) when I write things. I type too fast and write something wrong, and have to delete some stuff. The M-DEL emacs command is probably my most used (this also works in Mac OS X text dialogs, so I’m used to it already). In vim, there is CTRL-w and a few others, but if I want to do something more advanced, like rearranging a sentence, then half of my key presses would be ESC or i, i.e., just moving between the modes. In emacs, I can always have my pinky by Control and Alt (especially as soon as I remap CAPS-LOCK to Control).

Also, it really irks me how in vim, if you are at the end of a line and press l (or right-arrow), instead of moving to the beginning of the next line, it beeps! In emacs, if you are at the end of a the line and type C-f, it moves to the beginning of the next line (actually, it technically moves just beyond the line, in case you want to append, which is another annoying thing about vim: you have to use A, not i, to add text to the end of a line).

Well, that’s it for now. I will hold off on the questions until after I go through all the customizations, as it seems that, unlike vim, emacs has many things already built-in (but we already knew that, didn’t we :). So I have just one question for readers: does anyone know of a really good emacs cheatsheet? The one I used for vim was really awesome, but I haven’t found anything equal for emacs. I find myself searching the tutorial whenever I forget something, which is not very efficient, so I would appreciate something better. Otherwise, I’ll just find something decent and print it out, as it would be better than nothing.

And if anyone cares, you can see what I’ve got for my .emacs file so far at https://github.com/asmeurer/dotfiles/blob/master/.emacs.

2011 in review

The WordPress.com stats helper monkeys prepared a 2011 annual report for this blog.

Here’s an excerpt:

The concert hall at the Syndey Opera House holds 2,700 people. This blog was viewed about 11,000 times in 2011. If it were a concert at Sydney Opera House, it would take about 4 sold-out performances for that many people to see it.

Click here to see the complete report.

Vim vs. Emacs (Part 1)

So about a month or so ago, I decided that I needed to start learning a command line text editor. XCode, the editor I had been using for Python files, didn’t work very well with the new version (in particular, the essential plugin that I’d been using to clear trailing whitespace on save doesn’t yet work in XCode 4). I’d been using TextWrangler for other things, and started to switch to it for Python editing too. As far as free GUI text editors on the Mac go, TextWrangler is the best.

But I’d seen some of the nice features that vim has, like automatically keeping all lines under 80 characters, on a friend’s computer, and I decided that I should try it.

Now, I had had a little prior experience with both vim and emacs, but all that I remembered was for vim that i inserts and ZZ quits (for when I accidentally open it) and for emacs, that M-X doctor starts the psychiatrist.

So I’ve decided to try them out, doing it cold turkey. To make sure that I choose the better one, I’ve decided to try both. So, starting about a week ago, I’ve been using nothing but vim for all my text editing. Starting in January, I will try using emacs, and after two weeks, I will see what I like better.

My opinions so far on vim:

The tutorials suck. The best tutorial is vimtutor (type that in the command line), which I think comes with vim. It’s not bad, but it leaves out a few things that I would consider to be essential to a tutorial, for example, how to scroll (answer: use CTRL-D and CTRL-U). I started the emacs tutorial a while back, and while I never finished it, from what I remember, it was much better (and I also remember that the first thing it talked about was how to scroll by more than one line at a time). It also left out the . command, which I think is rather useful. I did print out this cheatsheet and have it sitting next to me on my desk. That has helped a lot. I hope I can find something similar for emacs when I get to it.

vim is too line oriented. vi started out as an extension to ed, the line editor, so this is not surprising. But I still can’t understand why pressing l at the end of a line can’t bring me to the beginning of the next line. Maybe I’m just still doing it wrong (supposedly, you should rarely use h and l over more efficient moving commands).

Somewhat related to the last point, vim really likes to ring the terminal bell a lot. To quote Wikipedia, “vi has two modes – ‘beep repeatedly’ and ‘break everything'”

I managed to customize it to the point of usability (there are still several things I need to go in and figure out how to fix). See https://github.com/asmeurer/dotfiles for my .vimrc and .vim/ files. I found a decent Python syntax file, but it’s actually not that great. I modified it to color single quoted strings different from double quoted strings (a feature I missed from Xcode). I still need to make a better color scheme (preferably the same as Xcode’s midnight), but this is enough work that I’ve put it off.

Pressing ESC all the time is really annoying. Sometimes, I just arrow over, even though I know you’re not “supposed to”, just because my fingers don’t want to reach over and press ESC. I’m also really used to using control sequences to move around while typing, which of course doesn’t work in vim. In fact, so far, I’m suspecting that I’ll like emacs better. But I’ve vowed to give both a fair chance. But so far, my impression is that vim is a great for text editing, but not so hot for text writing (unless you always write text perfectly, so that you never need to leave insert mode until you are done typing). Just the simple act of deleting a mistyped word (yes, word, that happens a lot when you are decently fast touch typist) takes several keystrokes, when it should in my opinion only take one (two if you count the meta-key).

The customizability is really nice. So far, everything that I’ve thought of to change has been changeable. Also, language is easy enough to understand that I was able to modify the Python syntax file without any difficulty.

I like how it syntax highlights virtually everything I throw at it.

If there are any vim experts out there reading this, I have some questions:

Is there an easy way to get a list of and jump to a function/class definition in a Python file? In Xcode and TextWrangler, there was a nice popup at the top of the window that I could access these from. In vim, so far the best I’ve found is searching for it, which isn’t very efficient.

I got TAB to indent 4 spaces in Python, but for some reason, when I create a new line after a :, it puts 8 extra spaces. I wouldn’t be surprised if this is the result of some mismatch/error in my .vimrc or .vim/ files, but I don’t know how to fix it

Any useful tricks to share? Especially for editing Python files.

How long did it take you to become reasonably efficient with vim?

EDIT: I thought of some more questions:

Is there a way to make vim consider camelCase to be word boundaries?

Finally, if anyone else is thinking of starting vim, I have some useful things I’ve already found in my .vimrc. So you might take a look at that, and add the ones that you like to your .vimrc. Finally, if you are on Mac OS X, you should use iTerm2. Actually, you should use this regardless of what text editor you use. It’s a very good Terminal.app replacement that has virtually all the features (with a couple of exceptions) as Terminal.app, and a ton of extra ones. The one I want to mention here is mouse reporting support, so you can use your mouse to do things in vim. This is very useful, as sometimes, e.g., when selecting text, using the mouse is just more efficient. Also, if you get frustrated trying to remember the commands that will move around you faster than h, j, k, and l, you can just click on where you want to go.

:wq

sqrt(x) now prints as "sqrt(x)"

Just a few moments ago, a branch was pushed in that fixed one of my biggest grievances in SymPy, if not the biggest. Previously we had this behavior:

In [1]: sqrt(x)
Out[1]: x**(1/2)

In [2]: solve(x**2 - 2, x)
Out[2]: [-2**(1/2), 2**(1/2)]

Now suppose you took the output of those expressions and pasted them into isympy:

In [3]: x**(1/2)
Out[3]: x**0.5

In [4]: [-2**(1/2), 2**(1/2)]
Out[4]: [-1.41421356237, 1.41421356237]

That’s with __future__.division. Here’s what would happen with old division:

In [2]: x**(1/2)
Out[2]: 1

In [3]: [-2**(1/2), 2**(1/2)]
Out[3]: [-1, 1]

This is because with old division, 1/2 evaluates to 0.

The problem is that Python evaluates 1/2 to 0.5 (or 0) before SymPy has a change to convert it to a Rational. There were several ways that people got around this. If you copy an expression with number division in it and want to paste it into a SymPy session, the easiest way to do this was to pass it as a string to sympify():

In [1]: sympify("x**(1/2)")
Out[1]: x**(1/2)

In [2]: sympify("[-2**(1/2), 2**(1/2)]")
Out[2]: [-2**(1/2), 2**(1/2)]

If that was too much typing for you, you could use the S() shortcut to sympify()

In [3]: S("x**(1/2)")
Out[3]: x**(1/2)

In [4]: S("[-2**(1/2), 2**(1/2)]")
Out[4]: [-2**(1/2), 2**(1/2)]

This solution is fine if you want to paste an expression into a SymPy session, but it’s not a very clean one if you want to paste code into a script. For that, you need to modify the code so that it no longer contains Python int/Python int. The easiest way to do this is to sympify one of the ints. So you would do something like

In [5]: x**(S(1)/2)
Out[5]: x**(1/2)

In [6]: [-2**(S(1)/2), 2**(S(1)/2)]
Out[6]: [-2**(1/2), 2**(1/2)]

This wasn’t terribly readable, though. The best way to fix the problem when you had a power of one half was to use sqrt(), which is a shortcut to Pow(…, Rational(1, 2)).

Well, this last item should make you think. If sqrt(x) is more readable than x**(S(1)/2) or even x**(1/2), why not print it like that in the first place. Well, I thought so, so I changed the string printer, and now this is the way that SymPy works. So 90% of the time, you can just paste the result of str() or print, and it will just work, because there won’t be any **(1/2), which was by far the most common problem of “Python evaluating the expression to something before we can.” In the git master, SymPy now behaves like

In [1]: sqrt(x)
Out[1]: sqrt(x)

In [2]: solve(x**2 - 2, x)
Out[2]: [-sqrt(2), sqrt(2)]

You can obviously just copy and paste these results, and you get the exact same thing back. Not only does this make expressions more copy-and-pastable, but the output is much nicer in terms of readability. Here are some before and afters that come from actual SymPy doctests that I had to change after fixing the printer:

Before:
>>> e = ((2+2*sqrt(2))*x+(2+sqrt(8))*y)/(2+sqrt(2))
>>> radsimp(e)
2**(1/2)*x + 2**(1/2)*y

After:
>>> radsimp(e)
sqrt(2)*x + sqrt(2)*y

Before:
>>> b = besselj(n, z)
>>> b.rewrite(jn)
2**(1/2)*z**(1/2)*jn(n - 1/2, z)/pi**(1/2)

After:
>>> b.rewrite(jn)
sqrt(2)*sqrt(z)*jn(n - 1/2, z)/sqrt(pi)

Before:
>>> x = sympify('-1/(-3/2+(1/2)*sqrt(5))*sqrt(3/2-1/2*sqrt(5))')
>>> x
(3/2 - 5**(1/2)/2)**(-1/2)

After
>>> x
1/sqrt(3/2 - sqrt(5)/2)

And not only is sqrt(x) easier to read than x**(1/2) but it’s fewer characters.

In the course of changing this, I went ahead and did some greps of the repository to get rid of all **(S(1)/2), **Rational(1, 2) and similar throughout the code base (not just in the output of doctests where the change had to be made), replacing them with just sqrt. Big thanks to Chris Smith for helping me catch all instances of this. Now the code should be a little easier to read and maintain.

Future Work

This is a big change, and I believe it will fix the copy-paste problem for 90% of expressions. But it does not solve it completely. It is still possible to get int/int in the string form of an expression. Only powers of 1/2 and -1/2 are converted to sqrt, so any other rational power will still print as a/b, like

In [1]: x**Rational(3, 2)
Out[1]: x**(3/2)

Also, as you may have noticed in the last example above, a rational number that sits by itself will still be printed as int/int, like

In [2]: (1 + x)/2
Out[2]: x/2 + 1/2

Therefore, I’m leaving the issue for this open to discuss potential future fixes to the string printer. One idea is to create a root function that is a shortcut to root(x, a) == x**(1/a). This would work for rational powers where the numerator is 1. For other rational powers, we could then denest these with an integer power. It’s important to do this in the right order, though, as they are not equivalent. You can see that SymPy auto-simplifies it when it is mathematically correct in all cases, and not when it is not:

In [3]: sqrt(x**3)
Out[3]: sqrt(x**3)

In [4]: sqrt(x)**3
Out[4]: x**(3/2)

Thus but (to see this, replace with -1).

So the idea would be to print Pow(expr, Rational(a, b)) as root(expr, b)**a.

The merits of this are debatable, but anyway I think we should have this root() function in any case (see issue 2643).

Another idea, which is probably not a good one, is to always print int/int as S(int)/int. So we would get

>>> Rational(1, 2)
S(1)/2
>>> x**Rational(4, 5)
x**(S(4)/5)

This is probably a bad idea because even though expressions would always be copy-pastable, they would be slightly less readable.

By the way, in case you didn’t catch it, all of these changes only affect the string printer. The pretty printer remained unaffected, and would under any additional changes, as it isn’t copy-pastable anyway, and already does a superb job of printing roots.