Calibre and the Kindle Fire 2
Quick Summary: All you need for Calibre to detect and connect with the Kindle Fire 2 is an updated libmtp.so that has the correct vendor/product for this eReader. That’s all you need for Linux, but an updated DLL will do the same on Windows, and I presume the same is true for OS X.
Calibre is great. I’ve used it since I got my first Kindle, many years ago, to backup my eReader, to push Gutenberg library files to it, and to download RSS feeds. It’s been working great and is real easy to use (with a few exceptions).
The other day I decided to buy a Kindle Fire at the discounted price of $129. It’s not the Fire HD, just the regular Fire with the crappy (2012 standard) screen. But I had seen it in a store and really liked it, and I thought I could use it as a development platform for my books.
When I opened the box and connected the gizmo to my computer, there was an instant case of the crickets. Linux didn’t recognize the thing at all, and Calibre fared no better. I was using the version that comes with the LTS release, Perfect Penguin (somehow misnamed Precise Pangolin).
I ended up making it work, but it took a day. Since part of the day was spent hearing tons of people complaining about the exact same issue, here is the account of my travails. Again, long story short, you just need a version of the libmtp.so that has the correct Product ID and Vendor ID for the Kindle Fire 2. Long version follows.
Updating Calibre
Kovid Goyal, the author, says on the web site that the first thing to do is download an updated version of the software. This is particularly important for Linux users, since the distribution apparently update this piece of software infrequently. Indeed, the version that comes with Perfect is very old.
I downloaded and installed the software, but it didn’t seem to know there was a Kindle attached. So I looked for some advanced configuration option where I could tell it to please find my new Kindle. Under Preferences → Miscellaneus I finally found what I was looking for:
The option Debug device detection will show you what devices are connected to the USB ports (and helps with finding out what’s wrong). The next option, Get information to setup the user defined device, will ask you to disconnect the device, click on a button, then reconnect the device. But looking at what changed, Calibre will be able to determine which device is your eReader.
Long story short, Calibre tells you that the Kindle Fire 2 has the Vendor ID 0x1949 (Amazon) and the Product ID 0x000a, which is the Kindle Fire 2 and the thing that confuses Calibre, since it doesn’t know anything about that combination.
Connecting to the Kindle
Next in line was finding why Linux didn’t detect the Kindle. I assumed the problem was, as usual, a missing udev entry. udev is the system on Linux that tells the computer what to do when something connects or disconnects. It’s usually reliable, but it doesn’t know what to do with hardware it hasn’t encountered before.
udev is also a terrible mess to configure. Instead of having a rational way of defining what a particular device is, udev uses old-style UNIX configuration files in whom variables are set randomly. It works, but adding a new device is not for the faint of heart. In particular, there seems to be no significant probing of the hardware, so that udev isn’t even going to try to guess what thing connected. If it doesn’t know it, it will ignore it – and you have to spend your time trying to figure out how things work.
It’s Google time. Turns out newer Android tablets are switching away from the “hello! I am a portable hard drive” model (UMS) to the Media Transfer Protocol. Someone said it was a better choice, but I hardly believe that’s the case. I was perfectly happy with my phone or tablet looking like a hard drive, and I have a sneaking suspicion that the switch is due to an attempt to hide files.
Linux has a FUSE file system that deals with mtp. It’s called (what else) mptfs, and it’s a joy to use. After you installed it (sudo apt-get install mtpfs) you simply give it a directory to mount to, and you’re done. I did so, giving it the directory /tmp/kindle, and immediately gained access to the files.
Since originally Calibre complained about the missing file system, I thought that adding one would help. Unfortunately, Calibre liked the Kindle Fire as much after I had mounted it as before. Dead end.
Downloading the Source
As any frustrated Linux user with tech skills, I just went back and downloaded the source code. First, I looked for the string 0x1949 throughout the code, since that’s the vendor Amazon. I found several references. I looked at each of them (mucking for the most part in the Python files, since those are easier to muck around in).
In the end, all my trying in Python didn’t get me anywhere. Fortunately (or not), there was a C file that had an entry for the Kindle Fire HD. Since I already knew that the Kindle Fire HD and the Fire 2 were technically almost identical, I figured that was a good place to start.
The file, calibre/src/calibre/devices/mtp/unix/devices.c contains a definition of a structure called calibre_mtp_device_table. In it, you’ll see an entry for the Kindle Fire HD. All you need to do is copy the line for the Kindle Fire HD one line below, replace the string Fire HD with Fire 2, replace the Product ID of 0x0007 with 0x000a, and recompile. Ta-dah!
Compiling and Installing
Unfortunately for us, Linux has the most amazing package management for binaries, but source code management is a total clusterfuck. It’s absolutely beautiful that you can just type sudo apt-get install calibre, and you don’t have to know at all what calibre needs to run. It’s absolutely stupid that if you try the same thing with the sources, you are sent to the Gates of Hades.
First, you figure out that to build, you need to call python setup.py. That gives you a very long list of options, and you figure out that the one you need for the library to build is, well, build. That’s when the nightmare starts.
You get tons of error messages, Python stack traces, etc. When you go and try to figure out what’s wrong, it’s some dependency that is missing. It’s a real tragedy, because you spend so much time trying to figure out which library is required just from the file of it it’s missing. It’s really, really stupid and needs to change – I have the same problem no matter what I build from sources.
To compound the problem, MTP support is one of the newest features in Calibre, which means the corresponding library is built last. I had to figure out PDF libraries with no relevance to my current quest to get this to work.
Once all was said and done, though, there was a directory calibre/src/calibre/plugins with the precious libmtp.so I so craved. I backed up the old version and replaced it with the new one. (Technically, I symlinked it so that I could continue mucking around without having to copy every time I figured out something new.)
Calibre and Kindle Fire 2
Once I had the new libmtp.so in place, I started the old Calibre and – voila! – it immediately recognized the Kindle Fire 2. It was excited about it, too – as soon as it started, Calibre started copying tons of RSS feeds onto it, and in general managing the device.
I disconnected from Calibre and from the computer and looked at my Books section. Suddenly, there were all the subscriptions I wanted, plus copies of my novels for me to look at in gorgeous color!
Calibre and MTP Support
I guess MTP support is new to Calibre because it’s new to eReaders (and Android devices in general). That explains why such a wonderful piece of software has a component that required downloading hundreds of megs of development libraries just to add a single new device.
Ideally, and presumably in the near future, Calibre will include automatic MTP detection. Once an MTP device is connected, Calibre would figure out if it’s an eReader it knows. At the very least, Calibre will have to be able to integrate new devices from the cloud, without a download of the entire software suite. And absolutely certainly in no way should the addition of a new device require a re-compilation of a C library.
But, as said, Calibre is one of the most wonderful open source products available, and I am sure this minuscule fault will be addressed shortly.
Obtaining the Updated libmtp.so
To save you the compilation hassle, you could try and download the libmtp.so I compiled. It will work only on Linux, only with certain versions and distributions, only with certain (current) versions of Calibre. Can’t tell you which, and be warned that what worked for me might do all sorts of horrible things to your computer. Download at your own risk, install it at your own risk, and don’t try to get me to support you if it doesn’t work. You have the instructions – if it doesn’t work out of the box don’t be lazy and do the footwork!!!
That said, you can find the file libmtp.so in the /misc directory on this server. {jd_file file==2}