How to permanently lose data with git (and then retrieve it again)

So I pushed some changes to github so Ondrej could help me debug the nseries tests, when I noticed that the changes that I pushed had some bad comments. So I decided to rebase. But git rebase -i told me that there was already a rebase in progress. I figured that I must have done it a long time ago and forgot to abort, so I ran git rebase –abort.

DON’T DO THAT.

I noticed my editor was telling me that an open file had changed. Then, I noticed that ALL of my uncommited changes were gone! And, being uncommited changes, git did not have them saved anywhere!

So now I started to panic. I had done a lot of work on dsolve that I hadn’t commited yet. Normally, I have hourly backups run by Time Machine, but I am on vacation and my backup drive is at home. So I started to see if I could retrieve it somewhere. grep quickly told me that it wasn’t in the hidden git directory, but it was still in my .pyc files. But a Google search told me that retrieving from that is not so easy, if not impossible with Python 2.6. So then, I decided to see if there was any lingering stuff in my virtual memory from my editor. So I ran grep on my harddrive and waited.

While I was waiting, though, I noticed when I scrolled up in my command history that my lost changes were in my Terminal. It turns out that I had just run git commit –interactive and had used * on my patches, so it gave me everything!

So I copied my Terminal history and will work on putting everything back tomorrow. It should be easy, assuming that git apply works for the format that git gives in commit –interactive.

So the lessons are: Don’t abort a rebase without commiting. Don’t start a rebase and then leave it there. Look in your Terminal history if you loose stuff. And it might be a good idea to make manual backups if you are away from your backup drive for a while.

This also highlights why it is important to try to recover data immediately after realizing that it is gone. If I had closed my Terminal session or filled it past the maximum number of lines, my data would be gone. Even if it were in my virtual memory, that wouldn’t last forever either.

Refactoring Expand

So I have spent the past week refactoring expand so that you can have more control over what expansion methods you use. With the present method, expand takes in hints which defaults to basic. basic distributed multiplication over addition (), expanded multinomial expressions (), expanded logarithms ( and ), and expanded powers ( and ).

If you wanted to do any of these things, you had to use expand_basic, which did all of them. Also, you had no control on how deep the expansion went. It went all they way down in recursion, so if you only wanted, for example, to distribute multiplication on the top level, it was impossible.

So I decided to start and fix issue 1455. I now have a branch ready in my github account (see here).

With my patch, you can now choose to expand using each of the above individually with the log, mul, multinomial, power_exp, power_base. In addition to this, you also now have complete control of how deep the expression recurses in the expand. Previously, you couldn’t, for example, expand to . (It would also distribute the exponent, then expand to ). Now, you can choose to only distribute multiplication over addition, and to only do it on the top level.

Automatic expansion of exponents
I mentioned above that expand would expand to . Actually, in the current version of SymPy, this would not happen because it automatically combines exponents like to .

I have been working for the past few weeks changing this as per issue 252. I have been mostly successful, except there are two nseries tests that I cannot figure out how to fix. If you think you know why nseries would fail without automatic combining of exponents, please let me know. The expand branch in my GitHub repo also has the exponent patches in it, if you want to see what I mean.

Because some things, like the gruntz algorithm, rely on automatic combining of exponents, so I had to rework powsimp, which combined exponents but also combined bases ( to ) so that it could only combine exponents. That way, I could use it to get the old behavior where I needed it. Use powsimp(expr, deep=True, combine='exp') in my branch to get the old automatic behavior.

So what does this have to do with ODEs?
When I started working on separable equations, I wanted to be able to separate things like into , which is separable in and . So that is why I needed to refactor expand (I don’t, for example, want to change to because the later is not easily recognizable as separate. Doing this or course required that SymPy didn’t automatically put back together into , so I had to fix that issue as well.

I am almost finished implementing separable (there are some match issues that I will blog about later whenever I get them straightened out), and the only thing that is holding all of this back is those nseries tests. If anyone is familiar with how those algorithms work and which parts require automatic combining of exponents, that would be great.

Vacation

I am on vacation now. I am visiting the Grand Canyon, Hoover Dam, Yellowstone, and other places. I will be able to do some coding on the road, but not as much as when I am home. The vacation will last about a week. Hopefully I will be able to get a blog post about what I have been doing in SymPy written up in the next few days.

git stash

I’ve always wondered what the command is that lets you do stuff like change branches and checkout to older states without commiting, since git won’t let you do anything if you haven’t commited.

Well, I found the answer. It’s git stash. I think I will be using this a lot, considering how often my workflow gets interrupted. I wish I knew about it before I started working on this exponentiation mess (more on that later).

It is also nicer to test if my code breaks an old feature to stash instead of testing in sympy 0.6.4 installed on my system.

UPDATE: No wonder I didn’t find this earlier. stash isn’t listed in git --help.