Latest Event Updates
This is NOT an indepth look at Heimdall, nor is it a step by step tutorial. It’s an overview to try to give a basic understanding of the process – something I found to be missing elsewhere.
The Android ROM flashing and rooting process is confusing, and I think that’s because most of the people blogging about it have done it so many times they forget how confusing it was at first.
Even websites in this space make the assumption that everyone already knows everything. http://www.clockworkmod.com for example, is website for the popular clockworkmod tool, that when installs allows you to control the recovery process much better than the phone stock process. For some bizarre reason the website doesn’t tell you this!
The good news is, that once you get it, it’s easy. The bad news is, upfront it’s daunting, confusing, and there’s a real lack of “ROM Flashing for Dummies” information out there.
With that in mind, let’s start with Android phone hacking 101.
There are really two ways to get a custom ROM on a phone:
a) Flashing an entire ROM to a phone using something like Heimdall (or Odin on a PC)
b) Getting zip files onto a rooted phone, and installing them using a custom recovery tool like clockworkmod.
Option A is trivially easy after you’ve:
- found the right file to flash
- worked out how to use Heimdall
- done it once
Option B is so utterly trivially easy after you’ve:
- worked out how to root your phone
- worked out what ROM file you want
- worked out what the right “gapp” file is the right one for this ROM (you need to install the bare minimum of Google apps like Google Play, and these are not bundled with ROMS because Google doesn’t like that.)
- you’ve done it once
So – as you can see, the trick is getting it done once!
Which method should you use?
When choosing between A and B, my advice is go with B, but know that if things get messed up, you’re going to have to do A.
I’m not going to go into step B in detail, but what you need to know are:
- how to root your phone (increasingly easy)
- how to install a recovery manager like clockworkmod
- the key combinations to boot your phone in to recovery mode
- what “gapp” is, if you need it for your ROM and where to get the right one
- how to follow instructions for any given ROM which will be all or some of:
- – creating backups
- – clearing different kinds of caches
- – formatting partitions (sometimes)
- – installing zips from SD card
Heimdall for Ubuntu (and OSX)
Heimdall and Heimdall Frontend. What are they?
They are a command line tool, and an optional graphical user interface that are in many ways equivalent to Odin, a very popular PC based ROM flashing tool that I haven’t used.
How do you instal Heimdall on Ubuntu?
You can download the latest DEB files from https://github.com/Benjamin-Dobell/Heimdall/downloads
If you don’t know how to install them once you have them, then flashing ROMs might be a bit beyond you.
Heimdall Gotcha 1: Heimdall Package Files.
Heimdall-frontend is at first confusing because the focus / main screen is for uploading Heimdall Package Files.. and boy did I search the world for them. You probably won’t find for your phone and desired ROM.
Heimdall package files are a great idea by the developer of Heimdall. Instead of having to work out how to flash your phone, all of the ROM source files need for a specific ROM and phone specific partition information are bundled up with a configuration file that makes it “just work”. In one click, Heimdall flashes the phone. I couldn’t find any for the Samsung S3 which is a metiorically popular phone, so based on that I’m thinking they are rare.
That doesn’t mean you can’t use Heimdall – it just means you need to understand the process a little more.
Heimdall Gotcha 2: You should run Heimdall as root!
So to launch, open up a terminal and type <i>sudo heimdall-frontend</i>. Without doing this it’s very possible Heimdall won’t be able to access your phone at the level it needs to.
Heimdall Gotcha 3: Where is all the information about using Heimdall with Ubuntu??
Yes – there isn’t really much out there. Odin for Windows is the much more common tool, but good news, OS X people use Heimdall too – so you might have more luck with more OS X specific Google searches for your phone.
Heimdall Gotcha 4: What ROM files do you need???
This is personal opinion. Option (B) is the way to go, so think of Heimdall as a tool to use when things go horribly wrong.
With that in mind, all you really need is ONE ROM FILE for Heimdall, and it’s the one that will get your phone back to stock. Even finding that can be troublesome, but you are looking for, e.g. “Verizon Samsung S3 Stock ROM download” is the way to go.
You are likely to end up with a file with extension .tar.md5 (which you rename to .tar and extract) or just a .tar, or a zip and it’s going to be pretty big.. a solid Gig or so.
Bottom line you are looking for a big old file, which contain files like (Samsung S3 files here):
These are also pretty big (especially system).
These are images of what specific partitions on your phone would contain when stock.
Your phone has a partition called SYSTEM, and system.img.ext4 contains the data for that partition.
Your phone has a partition called TZ, and tz.mbn contains the data for that partition.
There are two odd-files-out here.
SS_DL.dll. This is I expect something that relates to using Odin (Windows) to flash.. ignore.
NON-HLOS.bin. This is a Samsung S3 ROM file, and not sure if this is a standard, but I’m using it to demonstrate that not all files match up with the partitions they are associated with nicely. This is for the MODEM on a samsung S3.
Flashing a phone using Ubuntu and Heimdall.
A) Extract the PIT file from the phone.
Bottom line, you need to pull information from the phone that relates to the file system – specifically what partitions the disk has.
1) Start Heimdall as root user (e.g. type sudo heimdall-frontend in a terminal window)
2) IGNORE the first page you see! Skip to the Utilities tab.
3) Get your phone in “Download Mode” (you might have to google the key combinations for that)
4) Connect it by USB
5) Click DETECT and hopefully it will find your phone
6) Use the “Download PIT” section to create a PIT file, and DOWNLOAD that to your Ubuntu PC.
If all goes well you now have part of the building blocks you need.
B) Configure which partitions are going to be flashed with which partition files
So at this stage, you have a PIT file containing all the partitions on the phone, and a bunch of .img / .mbn / .img.ext4 / .bin etc. files that are going to be loaded into those partitions.
1) Go to the FLASH tab in Heimdall.
2) Use the Options section to load the PIT file you just created
3) Make sure the “RE-PARTITION” check box is UNCHECKED.
4) You are now going to click the ADD button for every file that you got in that big old ROM file, and use the Partition Details section to map that to the appropriate partition on the phone.
That is the hard bit, and might need some googling. You might have some weird file name that doesn’t seem to map to one of the partitions in the “partition name” drop down.. Google is your friend. Also, not every partition on your phone will have a file associated with it.
When you have a match up for each file you have to one of the available partitions, you are good to go.
5) Click START and hold your breath.
6) If you get an error in the status screen it’s probably got something to do with the status of the connection to the phone. Disconnect and reconnect. Go to the Utilities tab and DETECT the phone again. It took me a few times before START actually did the flash, but once the flash started, it took about 10 minutes, and went smoothly.
After that, the phone was back to it’s stock self (with all that Verizon crapware all over it.)
Hope this helps, and at the very least provides the baseline understanding of the process to make other more in depth blog articles make sense!
Frequently Asked Questions.
Question: Does the phone have to be rooted to use Heimdall?
Answer: No.. but you have to be able to get your phone in to “Download” mode so that Heimdall can talk to it down a USB cable. Google it for your phone.. usually a combination of holding specific keys to start the phone.
Question: Is it better to have a rooted phone to use Heimdall?
Answer: If you are just using Heimdall to fix a trashed phone ROM and restore it to factory, then not so much, but if you are going to get into flashing ROMS then rooting is really the first step.. and the second step is installing something like <i>clockworkmod</i> so you have better control of recovery process. E.g. it lets you create full backup images of your current install before trying something new for easy restore.
Question: After flashing a custom ROM (or even rooting) what’s next?
- a) Install something like ROM Manager and play with performance settings, e.g. Performance: On Demand will change your life.
- b) Install Titanium Backup, and your phone has an SD card slot, regularily use it to back up “User installed apps” to somewhere on your external SD card. That allows you to flash a new ROM, and reinstall all your downloaded apps WITH SETTINGS to the new rom.
- c) Buy some NFC tags and use them places in your life where you can toggle on off things like wifi / bluetooth / cellular data and more, because you are going to need to cut back on battery usages where you can now that your phone is cranking full speed when in use (and they are cool).
The world is falling down with books, methodologies, techniques, tips and tricks on how to be more efficient in life, work, and everything else you can think of. What isn’t often discussed is how the desktops, both real and virtual can be tailored to act as a strong foundation, and take things to the next level.
I’ve been thinking about this ever since I tidied my (physical) desktop a couple of weeks ago. Wow.. what a difference. Not my strength keeping that tidy, but it got me thinking about my virtual desktop too. As a Ubuntu User with multiple virtual desktops available to me, I’ve always had a strong sense of standard placement of specific applications but after some thought, have taken that to the next level.
The Physical Desktop
I love my latpop, and there are times when I get in the habit of sitting in front of the fire for a few days with it on my lap, but nothing beats the productivity benefits of a desk, a second monitor, a real keyboard and mouse, kick ass sound, and somewhere to set your coffee.
I’m not really going to say much more about it than that – the pictures says 1000 words.
The Virtual Desktop(s)
Now things get interesting. If you are a Windows user then unless things have changed since I last braved using one, you are SOL when it comes to virtual desktops. For Mac and Linux uses, multiple virtual desktops are things that we’ve been using (or haven’t bothered to use) for years.
Anyhoo. Ubuntu and Mac users can have as many virtual desktops as they like. Not sure about Mac, but with Ubuntu you can configure how they are arranged, and maybe because of my old Cube days, I like 4 virtual desktops side by side, and configured so you when you get to one end you wrap straight around to the other.
That means I have four full desktops that I can flick back and forward between, and each has 2 monitors.. 8 distinct areas.
So the secret to making this a haven of joy and efficiency is always keeping things in the same place relative to each other. For example, say all you use is a browser and Word Processor all day long as part of your core job, then an email client, and a music player.
You might do something like this:
Virtual Desktop 1
Laptop: Music Player
Virtual Desktop 2
Monitor: Word Processor
Why is that good? If you are working on a document, and you need to send an email you know that the email client is just over there on the Virtual Desktop to the right. Not a great example, but what happens when things get a bit more complex. I’m a rails developer, and at the very least that involves:
- a browser
- a terminal running the rails server (and putting out useful information)
- an editor
- a terminal running a rails console (used constantly while writing code)
- often a MySQL GUI
As if this isn’t enough, in development one is typically working with many open files at the same time, so multiple tabs within the editor. It’s this last part that made me really rethink my old strategy and for the first time move editing and the browser to different virtual desktops.. and try something that has turned out to be incredibly valuable.
I now have 4 different editors open, with the folder tree in each open to a specific folder of a rails project.
- Project Root (for other.. migrations, configs, helpers, css / js)
Each of these editors
- is not full screen
- has a corner visible no matter what (so you can get to it with a click)
- is always in the same on the screen (I go M/V/C/Other clockwise starting top corner).
It’s all kinds of awesome. The result of this separation of browser and editor left some really great gaps for other things. Here’s a rundown of my 4 Virtual Desktops going left to right.
I have to say after running with this for a couple of weeks, I can’t imagine going back. My fingers have absolutely learned things like CTRL-S, CTRL-ALT-LEFT,F5 (Save code, move left one desktop, refresh browser) and having the Rails server beside the browser and the rails server beside the editor makes so much sense.
Give it a go!
Understanding Proxy Servers. What are they for? Why would you use one?
Proxy servers are really used for one of two things:
1) A stepping stone.
Think of a Proxy Server as a step between you and where you want to go. For example, you can configure a proxy server in most browser configurations. The result is something like this:
– you want to view google.com
– your browser sends the request to the proxy server
– the proxy server pulls up google.com
– the proxy server tunnels whatever it gets back to you.
– ANONYMITY. Google.com doesn’t see your address in the request. It just sees the Proxy Server. Of course there is no real anonymity on the web – but we are talking about what someone sys admin at google can tell about the traffic.
– SPOOFING LOCATION. If you are in the US, and there’s a website in the UK that is locked down to only allow people in the UK to access content, then by using a UK Proxy Server, that website thinks you are in the UK and you are good to go.
2) A Gateway
This is frequently more of a corporate thing, where a company comes up with various reasons why they need a Proxy Server, but in general it’s so they can control the crap out of you. If you work for a big company then while on the internal network you might only be able to surf the internet through corporate Proxy Server. There are legit and useful reasons to do things this way, but more often than not, a biggie is that with all traffic tunneling through a single point it’s much easier for the IT dept to block access to certain sites or make sure you aren’t watching donkey porn at work.
For this post, we are more interested in (1), the stepping stone, and what options you have, and that really depends on what you are doing.
TYPE A: FREE (or pay once) PROXY SERVICES
If you are on vacation in Mexico and you want to use http://www.hulu.com, then the best thing to do is go to somewhere like http://hidemyass.com/proxy-list which publishes lists of Open Proxy Servers. The more recent the listing, the more likely it’s going to work.
Results will vary. A proxy might work, or it might not, and it might be horribly slow. For sure, a proxy that works today probably won’t work tomorrow. You also have no idea WHY this proxy is available, and it’s definitely a possibility that it’s sniffing whatever traffic you are putting through it, so not the best time to Instant Message your credit card details to someone.
If these services have a Premium offering, the basic message is the same – you just get the the list delivered to you in a better format.
TYPE B: PAID PROXY SERVICES
There are a bunch out there, and if you are doing some sort of web scraping or web spidering or similar where reliability and performance is important to you, there is no other way to go. PAID proxies typically ensure that they can keep up with needs by charging by throughput . That makes them a BAD idea for your Vacation Property in Mexico for Hulu / Netflix because media bandwidth adds up quickly.
Here are some of the things that differentiate offerings.
1) Revolving IPs
This is a good thing. Basically this means that you always call the same proxy server, but every time that proxy server makes an outbound call it cycles round-robin through a bunch of IP addresses.. so if there are 10 revolving IP addresses and you hit a page 10 times, then in theory each time is with a different IP.
Typically the IP addresses are sequential, so it’s not exactly rocket science for someone on the receiving end to see what is going on, but it’s still a bonus.
2) Multiple Proxy Servers
A decent service will give you credentials for a number of proxy servers, possibly in different countries. That means that you can round-robin call each of them from your application to further tangle the path between you and your final destination (See this post for an example of how you can easily do this with Ruby).
3) Short Term IP use
Not only do decent services round-robin between different IP addresses, they often throw those IP addresses away periodically and start with new ones. The advantage here is that if someone sees incoming traffic and blocks an IP, then that block is only good until the Proxy Service throws that IP away.
A really good PAID proxy service. ProxyMesh.com
It’s been about 6 months since I really dug in, but I really like ProxyMesh.com
– Their prices are reasonable starting at $1 / gig.
– They have highly maintained US and UK proxies with revolving Short Term IPs
– They also manage a list of Open Proxie servers (like the ones listed at hidemyass.com) but THEY manage the list on their end. You just call the same ProxyMesh Proxy, and they farm out the request to any one of hundreds of proxy servers – and that list is very fluid as proxy servers come and go worldwide. These proxy servers are of course much less reliable than their core service, but offer significantly more anonymity.
That’s all folks.
Here’s the problem in a nutshell. People like me who drooled over XBMC for years did so because it did an incredible job of dealing with local media files. We then got really excited because XBMC did an adequate job of dealing with streaming services that started to creep into our lives, like Hulu, Spotify, Amazon Video etc.
That was a while ago – but here we are now, and the world has changed. If you are anything like me, you’ve gone through this evolution:
1) EVERYTHING was local media, acqured by whatever means necessary.
2) Streaming services crept into my life, but still – 90% local, 10% streaming
3) Today.. > 50% streaming, < 50% local.
That’s BAD for XBMC.
It’s not XBMC’s fault.. it’s a great product, created and maintained by a great bunch of open source guys, but the real problem is the plugins. Hulu could give two shizzles about the XBMC market.. likewise for all the rest. They just don’t care.. and that means that the plugin creators have an uphill battle.
It’s one thing to be a part of that battle when the plugins account for 10% of viewing pleasure, but when it’s more than 50% (for me 90!) then it gets really painful. Put that a different way, 3 years ago, XBMC did 90% of what I wanted really well – and I had to fight with plugins for the other 10%. Now It does 10% well – and I have to fight with plugins for the 90%.
MAKES NO SENSE ANYMORE!
For me, the nail in XBMC coffin was a little bundle of joy called Plex. Plex server in it’s current incarnation is all kinds of sexy. It runs on pretty much anything (and for me that’s the only remaining linux Desktop in the house) and merrily does things very very well. In a way it’s XBMC center without the center. It scans specific folders for specific media, works out what it is, adds cover art, and then joy of all joys, makes that content available through a browser.. and.. to Plex Clients.
What Plex clients are there I here you ask? Well that’s where things get really nice..
– Others (including iProducts)
I’m just going to leave the “others” up to you, because as far as I’m concerned that’s already mindblowing.
Let’s switch gears for a sec and talk about Roku. Roku IMHO is the Apple TV for non fanboys. I didn’t realize how astonishingly awesome it was until the Plex server features made stop and say, “Hey.. wait a minute” and take off my XBMC tunnel vision goggles.
Roku Rocks. The Roku 3 is a powerful little beast in the palm of your hand, with a great interface and a nifty little idea – headphone socket on the remote. Love it. But can it really make all of these big PCs go away that I have scattered around the house connected to TVs?
Hell yes.. because of Plex. Thanks to the Plex plugin for Roku, that shrinking yet vital media set that one acquires through various means (because it’s not yet on Hulu, Amazon or Netflix) is still right there in a beautiful package, as is your entire music and photo collection.
Nuts and bolts.. here’s the choice:
– great support for local media
– buggy support for Hulu
– buggy support for Amazon
– ugly support for Spotify
– buggy smartphone remotes
– various fun things
— great people out there working hard to provide plugins in an uphill battle against service providors e.g. PBS
– beautiful support for local media through Plex
– flawless support for Hulu
– flawless support for Amazon
– flawless support for Netflix
– flawless support for Spotify
– excelent Roku smartphone app
– adequate stream from smartphone apps
— service providors falling over themselves trying to work with Roku to add channels e.g. PBS.
I’m sad to say that for me XBMC is dead. I’ve gone from 2 hefty XBMC to two Roku 3s I picked up for less than the price of the PC video cards running XBMC and I couldn’t be happier.
At home I have various devices doing various things.. and it’s important to me that I know they are working. There are many tools out there designed to keep an eye on server health, but they don’t do everything that I want in the way that I want, and I’m a big believer that coding = creating, so there’s nothing wrong with reinventing the wheel just for the sake of it.
For me it started recently when I repurposed an old android phone as an IP webcam pointing out the front window. After that it was a natural progression to install the excelent linux app Motion to pick up movement from that feed, and store images and mpegs to disk – for security. Well no point in storing the images on the nice machine that is likely to be stolen in event of a break in.. so let’s store them on an old EEE-Box that’s running headless in a hidden corner of the basement 😉
Well there you have it. Definite needs for a frequent health check:
– is the camera working?
– is the EEE nfs mount mounted?
I wrote a ruby script and cron to solve that problem, but then it just grew. Now I have the system running on multiple machines:
– checking each other
– making sure all my websites are responding to ping
– validating that certain URLs are giving a 200 response
– making sure that disks aren’t filling up
– checking that certain processes / background applications are running
On top of that:
– my laptop knows if I’m at home or away and tests accordingly
– the system creates Desktop icons for each problem (and removes them when problems resolve)
– it can generate Ubuntu desktop notifications
– it can notify me of issues with a text message
– each machine manages hostname|issue style files on Dropbox so whereever I am my laptop can tell me what’s going on at home.
– it shuts up at night
– for every problem it finds, it checks to see if there are instructions to try to correct it
All this is achieved with:
– 200 lines of core ruby methods that perform all the tests
– 50 lines of code relating to specific issue resolution
– 100 lines of control code – one liners which are basically “If you are this machine, then do this test”
In Part II we’ll take the very basics of that code, and create a simple single ruby script to keep an eye on a machines own health.
Dropbox and Google Drive (with Insync) – let’s just call them “Netdrives” – are great ways to sync files across multiple linux (Ubuntu) machines.
- Utterly reliable sync
- Online login to get your files from anywhere
- Great mobile support
- Can be used headless on linux servers
- Cross platform (Mac, PC, Linux)
When it comes to using them as a repo for scripts though they have inherant issues. They don’t retain file attributes.. so you create a file on one machine, chmod +x and it gets synced to all other machines, but without the +x.
- Ubuntu One.. but every time I try it I walk away thinking it’s not really an option at all. Why can’t Ubuntu get this right!
- NFS mounts. A good option if everythings on the same Lan, but what if it’s not?
- sshfs mounts. I still love sshfs mounts for certain things, but not when I want persistency.
- Github.. a bit clunky.
So, here’s what I do to solve this issue. A BETTER way (and I’ll blog that soon) is to use incrond inotify, but here’s a quick and easy way that works well and is really painless.
1) Put all the files in the same place.
I use /Dropbox/config/bin for Bash files
2) Create two files that do the “work” for you.
You need two files. One of which you actually have to chmod +x yourself at some point, but it’s not a file that will EVER change.
I use /Dropbox/config/bin/worker as a file that has the job of making sure all other files in this tree are executable
chmod -R +x $HOME/Dropbox/config/bin
Note that it launches the file worker2
Here’s MY worker2
It just launches something that I want to run often. The reason for worker2 is that you can modify worker2 from any machine, and worker will make it executable just before executing it.
3) Use Cron to run worker as often as you like.
* * * * * export DISPLAY=:0.0 && export XAUTHORITY=/home/keith/.Xauthority && sudo -u keith /home/keith/Dropbox/config/bin/worker
I run it every minute.. because I want that pulse_check (makes sure all my stuff is healthy) to run every minute anyway. Note the overly complex way that this is called in cron. All that XAUTHORITY stuff on a Ubuntu box means that if notify-send (create desktop notifications) can be used in anything downstream launched by worker, and they show up on the desktop.
That’s it! It’s not rocket science, and in a way it’s a bit ghetto – but it works well.