pofHQ

SSF2T: The quest for the perfect training mode

2014-04-21T23:37:29+00:00

Since I started playing ST a few months ago I wanted to have a training mode on MAME like the one available on the Sega Dreamcast port. I collected all the available training mode cheats but none of them convinced me, so I studied all them, fixed some of the glitches and finally end up taking the best of each one to write my own.

The first usable training mode cheat was made by Pasky, he explained it very well in this comment, the main problem with his cheat is that the stun meter stops working properly after the health bar is recharged and the cheat has to disable stun, so the players never get dizzy. The cheat is interesting because he hooks the game code to make it jump into his own subroutine that refills the health bar for both players. This cheat only recharges the bars after hit damage, but not after throw damage.

After that, there was a new training mode cheat made by d9x/dammit. It is way easier because it only recharges the health bar when the dummies are at a certain state (for example after being hit). We can see the value of this state at memory address FF8451 (or 0x400 more for P2). This allows the stun meters to work properly, so characters can get dizzy normally, however it still has some glitches like sometimes when the health is recharged opponent gets hit or pushed back.

In both Pasky and d9x cheats, the health bar can’t be empty (if this happens the round will end). This is one of the things that bothered me, because you can’t get an idea of how much damage you would do, for example with a 5-hit combo, as the bar is refilled very quickly after every hit or when it reaches a certain value.

Not long ago, jedpossum published a new training mode, which has the particularity that even after the health bar is empty, the game continues. However it has a few new glitches, like the player’s vertical position after a wall throw is sometimes messed up, and there’s the K.O. slowdown present when the health bar reaches zero.

So I took the ability to reach an empty health bar from jedpossum’s cheat, and the memory position to control the player state from d9x cheat and created my own training mode cheat, here is an explanation of the memory values I used:

ff8451: player state, a value of 0 means standing, a value of 2 means crouching, a value of 0xE means after a hit, a value of 0x14 means after a throw, etc..
ff8478: the real health value for PL1, range from 0 to 0x90, set to ffff when the round ends.
ff847a: a copy of the health value for PL1, range from 0 to 0x90, it’s not modified when the round ends.
ff860a: the health bar meter HUD display value, range from 0 to 0x90.
4c94,4a78,4db0,be64e,bebae: when the round ends, the game engine will take the value from here (0xffff) and change the player’s health value at ff8478 with that.
FF82F2: Slowdown value, when set to 0 there’s no slowdown.
FF84AD, FF84AE, FF84AB: stun meter, stun counter, dizzy meter

Basically what my cheat does is recharge health when both players are idle (with a “timer” so we can delay it, useful if we want to look at how much damage our last hit has done). It will also recharge health when the round is about to end, in two different ways:

when the player is hit or thrown, and his energy is less than 0xf
when the round has actually ended, preventing it to really end

If the last hit (or throw) does less damage than 0xf, the energy will be recharged before the round actually really ends. If the last hit (or throw) does more damage than 0xf the round will “end”, but we disable the K.O. slowdown and recharge the health really quick so the round continues and no noticeable slowdown happens.

Here you can see the training mode in action:

Another thing that the Dreamcast training mode has is that you can force the PL2 dummy to do a certain action, for example always crouch or jump. This is useful for practicing some combos and can be done with macros using mame-rr, but Ange Albertini suggested me to do it on a mame cheat, so I thought challenge accepted, and I also wrote a complementary cheat to control the dummy’s repetitive movement… so it’s even better than the dreamcast training mode now :)

This time I’ve decided to share my cheat file on github, instead of publishing all the cheats separately, so grab it and if you try my new training mode let me know what do you think in the comments!

Enjoy! :)

UPDATE: New download bundle with mame-rr, training mode cheat, hitbox viewer, input display and Sako training. All you need to do is put the roms (ssf2.zip ssf2t.zip and ssf2xj.zip in the “roms” folder):

DOWNLOAD

Hacking Super Street Fighter II Turbo (Part 2)

2014-04-08T06:08:34+00:00

In today’s post I will try to illustrate the difference between a RAM cheat and a ROM cheat. RAM cheats usually change the data the game has in RAM, for example the previous post showed how to change the value in a fixed memory address to adjust the game difficulty during gameplay. ROM cheats patch the game’s program code to force the game engine take a different path.

One thing I’ve always wanted to see is the combo messages that appear on the side of the screen when you do a multiple hit combo, but for the combos that the CPU does, which for some reason don’t appear. So, that’s what I’ll show you how to do today: hack the ST rom to see the CPU combo messages, plus some other bonus cheats we’ll discover while getting there :)

First thing we want to do is locate the memory region or address where the game stores who controls a character, the CPU or a human player. From there, we’ll see where in the code this memory region is accessed, and that should lead us to some point where the game engine decides “it’s a human player so I will show the combo message, or it’s a computer so I will not show it”. What we will try to do is patch that part of the code to make the game engine always show it!

We can start working on the “CPU Combo Messages” cheat using the memdump & diff method we used in the previous example, but we’ll use a different method now just for the purpose of illustrating the possibilities of MAME’s built in debugger: the “cheat” commands.

Start the game with the debugger enabled, and start playing with 2 human controlled characters (P1 & P2). When the “Round 1” message disappears press ENTER in the debugger screen to break, and type the command cheatinit. This will start a new cheat search in memory by remembering the state of all memory addresses at that point. Now return into the game, and start playing with one human player (P1) against the CPU (P2). Ideally that match should have the same characters as the one before, with the same colors, etc… to produce the lesser variations possible in the game’s memory. Now when the “Round 1” message disappears, press ENTER again in the debugger screen and type the command cheatnext decrease,1: this will search for all bytes that have decreased by one since we did the cheatinit. Now we can do a cheatlist to see all the possible memory locations that have changed:

If all went well, one of those memory locations listed here should indicate the game whether the P2 is controlled by a computer or by a human. Now we have to try each one to try to find which is the correct one, there are a few possibilities to do this, you can press ALT+M and change them manually in the memory editor window, or you can change them in the debugger using the command line like this:

But the easiest one is to have the debugger generate a cheat file with all the possibilities for us, using the cheatlist ssf2xj.xml command.

If you are playing against the CPU, you’ll see that immediately after typing the command pb@FF8BDC=01 the CPU stops moving, and you can control the character using the P2 controls. So we have now found the exact memory location that tells the game whether we’re playing against the CPU (memory value equals 0) or against a human (memory value equals 1).

BONUS: we can use that information we have just discovered to write a “Controller Mode” cheat, that should allow us to transfer the control of a cpu-controlled character to a human-controlled character and vice versa. The controller mode cheat could initially be something like this:

https://gist.github.com/10100281

Note that in this cheat I’ve put the address also for P1 (0xFF87DC). You could have used the same method to find it, but it is well known that in Super Turbo ROM the offset for the player 2 base address is 0x400 bytes after the player 1, so to find the P1 address I just subtracted 0x400 to the P2 address we already found using the debugger.

Now it’s time to test the cheat. You’ll notice that you can properly transfer a CPU-controlled character to a human-controlled character anywhere, but when you want to do it the other way round (human to CPU) the game crashes and restarts itself if we are in the middle of a round (you can do it only before starting the match, for example in the character selection screen). We’ll see a possible workaround for that later, but now we can continue to pursue our initial goal.

Let’s go back to MAME’s debugger and put a few watchpoints at the memory address for player one and player two, one will tell us when the memory address is read by the CPU, the other will tell us when the value is changed (write). We will also print the program counter (PC) when the read/write operation occurs, and resume the execution:

wpset 0xFF87DC,1,r,1,{printf "P1 Read @ %X=%X with PC=%X", wpaddr, pb@FF87DC, PC; go}
wpset 0xFF87DC,1,w,1,{printf "P1 Write @ %X=%X with PC=%X", wpaddr, pb@FF87DC, PC; go}
wpset 0xFF8BDC,1,r,1,{printf "P2 Read @ %X=%X with PC=%X", wpaddr, pb@FF8BDC, PC; go}
wpset 0xFF8BDC,1,w,1,{printf "P2 Write @ %X=%X with PC=%X", wpaddr, pb@FF8BDC, PC; go}

With these commands we get the following output during gameplay:

As we can see, there’s a pattern here where the memory position indicating if the character is CPU or human controlled is read always when the program counter is always at the same 6 instructions. We can use bpset to set breakpoints or remove the go command at the end of the previous watchpoints to stop execution when the CPU is at these instructions, so that we will be able to inspect the disassembly. We can also use the debugger’s built-in dasm command to write the disassembled code into a file:

dasm 1.asm,0x597a-20,100
dasm 2.asm,0x78034-20,100
dasm 3.asm,0x77E76-20,100
dasm 4.asm,0x68f58-20,100
dasm 5.asm,0xbe54e-20,100
dasm 6.asm,0xbe56e-20,100

Upon inspection of the disassembled code we can see that we mostly have BEQ and BNE instructions. I’ll explain the basic concepts to be able to NOP an instruction or invert a branch condition using very basic motorola 68000 assembly concepts, which is the main CPU that powers the CPS-2 boards:

The instruction NOP (no operation) has the opcode NOP = 0x4e71
The instruction BEQ (Branch if Equal) has the opcode BEQ = 0x67XXYYYYZZZZ where XXYYYYZZZZ indicates how far we will jump forward if the previous comparison instruction (usually a TST) was found to be equal.
The instruction BNE (Branch if Not Equal) has the opcode BNE = 0x66XXYYYYZZZZ where XXYYYYZZZZ indicates how far we will jump forward if the previous comparison instruction (usually a TST) was not equal.

So if we need to invert the logic we can change the BEQ for BNE by swapping a 67 for a 66 on the first byte of the opcode, or if we want to always force a certain code path we can just NOP the branch instruction and it will always go to the next instruction right after the NOP’ed branch.

Now that we have this basic mortola 68k code patching introduction, if we just print the opcodes at the locations we have found using the debugger we will obtain the following results:

As we can see, the first byte of every opcode is always 66 or 67, meaning it’s a BNE or BEQ instruction. So we can either NOP, or invert the condition on every of these locations and see what happens during gameplay to get an idea of what the code is doing there. If we want to inspect all the details of the code it is better to use an external disassembler; we can use the command save to dump the decrypted CPS-2 opcodes into a file, and load this file in a regular disassembler like radare2 or IDA Pro.

As we can see here (it’s easier to see if we also trace it with the debugger when analyzing the code), the BEQ instruction at 0x68f58 will skip the next 6 instructions if the values in the registers indicate that we’re checking the computer and not a human player. If we skip these instructions, the combo message will not appear, so our goal is not to skip them. If we invert the condition (convert the BEQ in a BNE), the message will only show for the computer and not for the human player, and that’s not what we want. If we want the message to show up always regardless if the character is computer or human controlled, we have to NOP the branch instruction, to make sure the instructions after it will always be executed. The branch instruction at 0x68f58 has only 2 bytes, so to NOP it we just need to enter the command ow@68F58=4E71. Now we’re ready to write the mame cheat for that:

https://gist.github.com/10099906

Remember about the “bonus” controller mode cheat we did before?, it was done by just changing a value in memory (RAM cheat), but now that we know every place of the code where this value is read, we are in disposition to do a ROM cheat for that! So with this simple cheat (again, just NOP’ing an instruction), we can transfer a CPU controlled character to a human controlled character:

https://gist.github.com/10100556

Enjoy :)

Hacking Super Street Fighter II Turbo (Part 1)

2014-03-24T23:49:00+00:00

In this post I will show how to debug the Super Street Fighter II Turbo ROM in MAME, to create a simple cheat. This will (hopefully) be the first post of a series that will show more advanced use of the MAME debugger and dig deeper into reverse engineering Super Turbo. First we need to launch MAME using the -debug parameter, this will launch the MAME debugger. You can use the help command in the debugger to see the help.

For this first example we’ll create a cheat to change the game difficulty on the fly. To do this, we’ll press F5 (or type go in the debugger console) and we’ll press F2 (the “test switch” key) to access the game’s Test Menu. In the test menu, under “System Configuration” we can change the settings of the game, one of those settings is the Game Difficulty, ranging from 1/Easieast to 8/Hardest.

Now we need to discover what changes in the game’s memory when we change that setting. To do this we’ll use the debugger’s ‘dump’ command, that dumps program memory as text. We can find the right syntax for the ‘dump’ command using the debugger’s command ‘help memory’. It expects a filename as first argument, the start address and the length of the memory portion we want to dump. So we’ll put the Game Difficulty setting to 1/Easiest and issue the following dump command: dump easiest.txt,0,0xffffff, then we’ll set the Game difficulty setting to 8/Hardest and repeat the command: dump hardest.txt,0,0xffffff. Now we will use the diff utility to see the difference between both files:

$ diff -urN easiest.txt hardest.txt
--- easiest.txt	2014-03-25 00:39:10.038344425 +0100
+++ hardest.txt	2014-03-25 00:39:24.222343789 +0100
@@ -593047,7 +593047,7 @@
 90C960:  0020 0000 0020 0000 0020 0000 0050 0017  . ... ... ...P..
 90C970:  0045 0017 0020 001B 0020 0000 0020 0000  .E... ... ... ..
 90C980:  0020 0000 0020 0000 0020 0000 0043 000D  . ... ... ...C..
-90C990:  0020 0000 0031 0003 0020 0000 0031 001B  . ...1... ...1..
+90C990:  0020 0000 0031 0003 0020 0000 0038 001B  . ...1... ...8..
 90C9A0:  0020 0000 0053 0003 0020 0000 004F 0003  . ...S... ...O..
 90C9B0:  0020 0000 004F 0003 0020 0000 004F 0003  . ...O... ...O..
 90C9C0:  0020 0000 004F 0003 0020 0000 0054 001B  . ...O... ...T..
@@ -1046571,7 +1046571,7 @@
 FF82A0:  0000 0000 0000 0000 0000 0000 0000 0101  ................
 FF82B0:  0000 0000 0000 0101 0000 0000 02C0 0000  ................
 FF82C0:  0000 C000 0000 0000 0000 0000 0000 0000  ................
-FF82D0:  C91C 0000 0000 0000 0000 0100 0000 0000  ................
+FF82D0:  C91C 0007 0000 0000 0000 0100 0000 0000  ................
 FF82E0:  0000 0000 0100 0100 0000 0000 0000 0000  ................
 FF82F0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
 FF8300:  0000 0000 0000 0000 0000 0000 0000 0000  ................

We can see two values have changed, one at position 0x90C99D and the other at position 0xFF82D3. The first one is the probably the text displayed on the screen, while the second one is probably the position in memory where the difficulty value is stored, as we can guess it ranges from 0 (Easiest) to 7 (Hardest). To confirm our guess we’ll launch the Memory View window in MAME’s debugger, to do this just press CTRL+M and the Memory View window will appear. Type in the memory address of our previous finding and then switch the Game Difficulty in the System Configuration Menu to see if the memory value at 0xFF82D3 changes when we change the difficulty.

It appears that our guess was right, and we have found the place in memory where the Game Difficulty setting is stored. Now let’s do the MAME cheat to be able to change this setting on the fly during gameplay, so we can adjust the difficulty while playing against the CPU.

The MAME cheat system XML format is explained in the comments on the src/emu/cheat.c file inside MAME source code. We have to create a cheat file inside MAME’s “cheat” folder named the same as the ROM zip file but with xml extension, so for Super Street Fighter 2X (Japanese version of Super Turbo), the filename will be “ssf2xj.xml”. First we open the mamecheat tag, then we open the cheat tag and add the description for example “Select Difficulty”. Then we will use the parameter tag to let the user chose the difficulty using item values from 0 to 7. Then we’ll use a temporary variable temp0 to store the current value of the Game Difficulty when the cheat starts (script state="on"), we’ll use this variable to restore the value to the user’s setting when the cheat ends (script state="off"). While the cheat is being run (script state="run") we’ll just overwrite the memory value at address 0xFF82D3 with the param value selected in the cheat menu. The final cheat code looks like this:

https://gist.github.com/10100976

So now using our newly created cheat we can adjust the difficulty during gameplay against the CPU by accessing the cheats menu (press TAB on MAME, it must be started with -cheat option):

Enjoy :)

UPDATE: As jedpossum points out, the following memory map of the CPS2 would have helped to decide what needed to be changed instead of guessing (thank you!):

CPS2:
0x000000 - 0x3FFFFF Main Program
0x400000 - 0x40000A Encryption (aka the battery memory on pheonixed roms it's 0xFFFFF0 - 0xFFFFFA)
0x618000 - 0x619FFF Shared ram for the Z80 aka tells what sfx or music to play.
0x660000 - 0x663FFF Network Memory
0x900000 Start of Graphic memory (can change with each game)
	
ST:
0x900000 - 0x903FFF Palette
0x904000 - 0x907FFF 16x16
0x908000 - 0x90BFFF 32x32
0x90C000 - 0x90FFFF 8x8
0x910000 - 0x913FFF 16x16 mainly hud and character names on select screen
	
0xFF0000 - 0xFFFFFF Main Memory

ZTE Open FirefoxOS Phone, root and first impressions

2013-07-05T12:32:23+00:00

ZTE Open is the first non-developer FirefoxOS phone, sold commercially in Spain by Movistar.

It can be rooted using CVE-2012-4220 aka Qualcomm DIAG root discovered by Giantpune. This security advisory was released by Qualcomm on November 15, 2012. The ZTE Open has been launched commercially 7 months later and neither ZTE nor Movistar have bothered to patch this security hole, shame on them for selling vulnerable devices to customers.

The ZTE Open comes with kernel 3.0.8 which is also vulnerable to CVE-2013-2094 (perf_event) exploit.

Root

I took the exploit by Hiroyuki Ikezoe and adapted it to work on the ZTE Open. The source code is available here, and a redy-to-use compiled exploit is here: DOWNLOAD.

These are the details of the original firmware, as hopefully ZTE will patch the security hole and this exploit might not work in future versions:

ro.build.display.id=OPEN_FFOS_V1.0.0B04_TME
ro.build.sw_internal_version=B2G_P752D04V1.0.0B08_TME
ro.build.firmware_revision=V1.01.00.01.019.120
ro.build.date=Fri May 31 23:10:17 CST 2013

To run the exploit connect your phone to your computer using the USB cable, and make sure ‘Remote debugging’ is enabled on your phone in Settings -> Device information -> More Information -> Developer.
You need to have the adb binary in your computer’s path, (if you don’t know what ADB is don’t bother rooting your phone) then execute “run.sh” on Linux or OS X, or “run.bat” on Windows.
If the exploit fails, reboot your ZTE Open and try again (the linux/MAC version will attempt to do that automatically). Once the exploit is successful it will remount the system partition in read/write mode and copy a setuid “su” binary into /system/xbin/su.

Custom ROMs

The bootloader on the ZTE Open does not allow to flash or boot unsigned code through fastboot protocol. The stock recovery image will verify the signature of update packages and not allow you to flash self-signed updates. To overcome that limitation you can flash a custom recovery image that will allow you to backup your current ROM to SD card and flash your customized build of FirefoxOS (or if you want, your own Android port).

You can download ClockWorkMod recovery for ZTE Open here: recovery-clockwork-6.0.3.3-roamer2.img.
To flash it:

# first backup your existing recovery
adb shell dd if=/dev/mtd/mtd0 of=/sdcard/stock-recovery.img bs=4k
adb pull /sdcard/stock-recovery.img
	
# then flash clockworkmod recovery
adb push recovery-clockwork-6.0.3.3-roamer2.img /sdcard/cwm.img
adb shell flash_image recovery /sdcard/cwm.img

To boot into recovery mode, hold both volume ~~down~~ up and the power button while powering on the phone.

Enjoy! :)

SSD alignment on linux with ext4 and LVM

2013-01-12T00:08:36+00:00

First make sure to create partitions aligned to your SSD erase block size (in my case 512k):
sudo fdisk -H32 -S32 /dev/sdb
You can check with fdisk -lu /dev/sdb that the start of each partition is divisible by 512.

Then initialize the desired partition to use with LVM2 using the dataalignment parameter:
pvcreate --dataalignment 512k /dev/sdb1
Make sure your /etc/lvm/lvm.conf contains the following options:

md_chunk_alignment = 1
data_alignment_detection = 1
data_alignment = 0
data_alignment_offset_detection = 1

Now you can use vgcreate to create your volume grup, and then lvcreate to create the logical volumes.

When creating ext4 filesystems (with TRIM support), use the following command:
mkfs.ext4 -O extent -b 4096 -E stride=128,stripe-width=128 /dev/mapper/vg1-test
stride and stripe-width are calculated as sector size / block size = 512k / 4k = 128

When mounting ext4 filesystems, use the ‘discard’ parameter to enable TRIM support:
mount -o discard,noatime,nodiratime /dev/mapper/vg1-test /mnt/

Extra tip : for more speed you can consider turning off journaling (to avoid double-write overhead), at the cost of an easily corruptable filesystem.
Check if journaling is enabled: dumpe2fs /dev/mapper/vg1-test |grep 'Filesystem features'
Disable journaling: tune2fs -O ^has_journal

Fortifying a Galaxy Nexus with stock-ish image and root access

2012-07-30T09:36:43+00:00

In this post I will describe my recipe to have a Samsung Galaxy Nexus (codename “maguro”) using a rooted factory image, capable of getting OTA updates without loosing root access and with a locked bootloader, keeping the user data safe in case it gets lost or stolen, in the sense that the person getting it will not be able to extract personal details from it like Google accounts, settings, downloaded apps and their data, media, etc.

I assume the reader starts with a stock unmodified factory image, and knows how to use fastboot.

Step 1: Getting root access

The first step is getting root access, to accomplish this the easiest way is to temporarily unlock the bootloader (don’t worry, we will re-lock it later).

Open a shell prompt and type fastboot oem unlock, all data on the phone will be lost as after oem unlock the bootloader performs a factory data reset (also called hard-reset or reset to factory default).

Once the bootloader is unlocked, we can flash “unsigned” data through fastboot, this allows us to flash a customized recovery image, which will allow to flash an usable ‘su’ binary with the proper suid permissions and a superuser app into the system.

There are a few custom recovery images out there to choose from, the most popular being ClockworkMod recovery and TeamWin Recovery Project (TWRP). I recommend the later, because it supports decrypting an encrypted data partition on Galaxy Nexus since version 2.2.0.

To successfully root the device, first flash the custom recovery image using fastboot: fastboot flash recovery recovery-maguro.img, then reboot into recovery without booting the system (this is important, as during boot the recovery image checksum is verified and if it doesn’t match the stock recovery the system will overwrite the custom recovery with the stock one).

Once we’re into the custom recovery, we need to flash the ‘su’ binary and the superuser apk, again there are multiple possibilities out there, the most popular being ChainsDD Superuser (flashable zip) and Chainfire SuperSU (flashable zip).

Finally, when the phone is booted again the stock recovery will be automatically flashed and you will have a rooted phone with unlocked bootloader and stock recovery.

Step 2: Relocking the bootloader

Re-locking the bootloader is important, because with an unlocked bootloader any unauthorized user could access your private data by flashing a custom recovery and backing up or mounting your storage and data partitions. This is why the ‘oem unlock’ process wipes your data, and this is why you should keep the bootloader locked.

The Galaxy Nexus bootloader stores the lock status at position 0x000007C (124 decimal) of the param partition of the device’s internal storage. When the byte is set to ‘0’, bootloader is unlocked, when it is set to ‘1’ bootloader is locked. As you can guess now, if you have root access you can manually change the bootloader status from the system, thus it is possible to unlock and relock it without using fastboot, and without wiping your data. You could do this process manually using ‘dd’, but of course the good folks at XDA have created an open source app to automate this process for you. Install BootUnlocker for Galaxy Nexus from the play store, give it root permissions and re-lock your bootloader now. If you want to have a look at the application source code, check the google code project page.

You now have a rooted phone with stock recovery and locked bootloader.

Step 3: Encrypting the phone

Encryption on Android uses the dm-crypt layer in the Linux kernel, to enable encryption go to Settings, Security, Encryption and click on “Encrypt phone”, for the encryption process to start battery should be fully charged and the phone AC adapter must be plugged in. For more details on how encryption works, read Notes on the implementation of encryption in Android.

Once the phone is encrypted, you need to type a numeric PIN or password to decrypt it each time you power it on. The master key to decrypt the filesystem is encrypted with a hash of the user’s lock screen password (that’s why you can only use pin or password in the lockscreen when encryption is enabled).

Currently, there is only one password for both the encryption and lock screen. This is especially bad because you cannot turn off screen lock and therefore have to type it rather frequently, which makes it easier to get a glance on the screen while typing it. Until Google provides a way to use different passwords for encryption and screen lock, you can manually change password for encryption by issuing the following shell command as root:

vdc cryptfs changepw <new_password>

This command changes only encryption password requested at phone boot. The lock screen PIN or password remains unchanged. Please see (and star) Android issue 29468 for Google to implement this in the UI.

Again, there’s an app for that too! check out Cryptfs Password (github source) if you want to change the password without messing with the command line.

Make sure to choose a good password not based on a dictionary word, as the encryption can be cracked using brute force (see Into the Droid – Gaining Access to Android User Data presentation from Thomas Cannon at Defcon 2012, slide 26 and slide 27).

You have now an encrypted rooted phone with stock recovery and locked bootloader.

Step 4: Keeping root access after OTA updates

When Over The Air updates are applied, the function set_perm_recursive is called from recovery, this removes the setuid bits on the ‘su’ binary and disables your root access after the OTA has been applied.

To prevent this form happening, one could change the file attributes for the ‘su’ binary to immutable using chattr +i /system/xbin/su, again there’s an app that will do this for you automatically (and also keep a backup of your su binary, allowing to “temporarily unroot” your phone): OTA RootKeeper, the source code is available on github.

The same functionality from OTA RootKeper has been recently included in recent versions of ChainsDD Superuser app.

You have now an encrypted rooted phone with stock recovery, locked bootloader and capable of keeping root access through OTAs.

Some notes concerning OTA updates and backups

With this setup, you have not modified any system files so your phone should be able to automatically get OTA updates and apply them cleanly. If you are impatient and want to manually flash OTAs as soon as they are available you should keep in mind your setup is a bit different and instructions posted everywere won’t work exactly for your phone.

To manually apply an OTA update, you should flash a custom recovery first, you can do it either from system using ‘dd’ (you are root now), or through fastboot (but remember to oem-unlock your bootloader first using the BootUnlocker app, and re-lock when done).

To flash the custom recovery from system, you can use the following command as root:

$ su
# dd if=/sdcard/twrp-2.2.0-maguro.img of=/dev/block/platform/omap/omap_hsmmc.0/by-name/recovery

You can then reboot into the custom recovery to apply the OTA from sdcard, or to backup your system, remember to use a custom recovery with support for encrypted partitions like TWRP. Again, remember that when you boot your phone again, the custom recovery will be overwritten with the stock recovery on boot.

Update : An avid reader (thanks Marc!) noticed that factory images don’t include the files /system/etc/install-recovery.sh and /system/recovery-from-boot.p which are responsible of re-installing the factory image on boot when the installed recovery checksum doesn’t match the stock recovery, these files are only present if you have updated through an OTA. This means that if you come from a factory image, you’ll need to reflash the stock recovery manually at the end of the process. To do this first extract the stock recovery from a factory image:

$ tar zxvf takju-factory.tgz
$ cd takju-*/
$ unzip image-takju-*.zip
[...]
  inflating: recovery.img

Then reboot into fastboot mode and flash it using fastboot: fastboot flash recovery recovery.img

Ubuntu 12.04 on ONKYO BX407A4

2012-06-18T19:27:44+00:00

Two years ago I wrote about installing Ubuntu 10.04 on ONKYO BX407A4, now I have upgraded it to 12.04 LTS, and here’s a reviewed procedure in case anyone still uses this UMPC.

Basically everything works so well out of the box, except for a few things that need a little bit of tweaking:

Replace /etc/rc.local with the following: rc.local

Replace /etc/default/grub with the following: grub, when done run ‘sudo update-grub’.

Add the file /etc/modprobe.d/blacklist-onkyo.conf: blacklist-onkyo.conf

Add the file /etc/pm/power.d/99_onkyo and make it executable: 99_onkyo

Add the file /etc/pm/sleep.d/74_onkyo and make it executable: 74_onkyo

Add the file /etc/pm/sleep.d/00_onkyo and make it executable: 00_onkyo

Add the file /etc/pm/config.d/onkyo_fix: onkyo_fix

Install vbetool:

sudo apt-get install vbetool

Install the latest version of opengalax Touch Screen Driver:

$ sudo add-apt-repository ppa:poliva/opengalax
$ sudo apt-get update
$ sudo apt-get install opengalax xinput-calibrator

When done, edit the file /etc/opengalax.conf and set psmouse=1, this together with the changes in grub will make the touchscreen and the optical mouse work.

Reboot, and run xinput_calibrator to calibrate the screen, save the calibration data in /etc/X11/xorg.conf.d/99-calibration.conf.

Some other useful hints:

Rember to add the ‘discard’ parameter to /etc/fstab for SSD trim.

For Ubuntu/Unity:

$ sudo add-apt-repository ppa:poliva/pof
$ sudo apt-get update
$ sudo apt-get install indicator-sysbat
$ echo "coretemp" |sudo tee -a /etc/modules

For Lubuntu/LXDE:
Install lxbat instead.

With all these changes everything works as expected, even suspend/resume :)

Why Broadcom 802.11 Linux STA driver sucks, and how to fix it

2012-05-22T22:49:17+00:00

TL;DR - the broadcom sta linux driver always fails in the first scan request after the interface is brought up, this produces a long delay when connecting to a wireless network. There's an open source driver which does not have this problem, but is not good with power management. In this post I describe the steps I took to pinpoint the problem in the proprietary driver and to fix it.

The story begins when I updated Ubuntu from 11.10 to 12.04 on my MacBook Air, everything worked fine after upgrading except one thing that bothered me a lot: when resuming the laptop after suspending it, it took around 30 seconds to connect to my wireless network. It wouldn’t have bothered me if it had been the same in 11.10, but in 11.10 the time to connect was barely 5 or 6 seconds, so having to wait 30 seconds was totally unacceptable.

Initially I thought it was a bug in NetworkManager, and increased the debug level in the config file to finally come out to the conclusion that I was using a different driver in 12.04 than in 11.10.

There are two drivers available for the Broadcom BCM4353 802.11 Wireless Controller:

Broadcom brcmsmac (mac80211-based softmac PCIe): the completely open source drivers, included in the kernel
Broadcom 802.11 Linux STA driver: the broadcom mixed GPL source + a proprietary hybrid binary file agnostic to the specific version of the Linux kernel

Both wl (broadcom proprietary driver) and brcmsmac (the open source driver) were installed in my Ubuntu 11.10 but the open source driver was used by default, and this driver connected to the wifi network in 5 seconds.

In Ubuntu 12.04, the wl proprietary driver provided by the package bcmwl-kernel-source has been updated from version 5.100.82.38+bdcom-0ubuntu4 to version 5.100.82.38+bdcom-0ubuntu6.1 which includes the following fix:

---------------
bcmwl (5.100.82.38+bdcom-0ubuntu6.1) precise-proposed; urgency=low
	
  * debian/bcmwl-kernel-source.postinst:
    - Blacklist brcmfmac, brcmsmac and bcma so that they don't
      conflict with the closed driver (LP: #873117)
 -- Alberto Milone  Mon, 23 Apr 2012 16:11:56 +0200

Which basically blacklists the open source brcmsmac module, forcing the wl proprietary driver to be in use. When the brcmsmac was not blacklisted, even if the wl driver was loaded it failed silently and brcmsmac was used instead.

So, the easy path to solve my problem would have been to blacklist the wl module, and add the brcmsmac to /etc/modules and live happy with my 5 seconds needed to associate when resuming, *BUT* I compared both drivers and the proprietary driver has better signal and way better power management, which makes my battery last longer, so I decided to go the long route. My goal was to achieve the lowest delay possible to connect to a wireless network when coming from a suspend using the proprietary driver.

I went “down” one level and started looking at wpasupplicant, as NetworkManager communicates with it using the DBus control interface (dbus-monitor showed the problem was not in dbus communication) so, increased the debug level in wpa-supplicant by adding ‘-dd’ switch in /usr/share/dbus-1/system-services/fi.w1.wpa_supplicant1.service, and looked through the logs, which quickly revealed the following:

May 20 11:49:46 maco wpa_supplicant[12610]: Scan requested (ret=0) - scan timeout 5 seconds
May 20 11:49:52 maco wpa_supplicant[12610]: Scan timeout - try to get results
May 20 11:49:52 maco wpa_supplicant[12610]: Failed to get scan results
May 20 11:49:52 maco wpa_supplicant[12610]: Failed to get scan results - try scanning again
May 20 11:50:07 maco wpa_supplicant[12610]: Scan requested (ret=0) - scan timeout 5 seconds

The first scan request (SIOCSIWSCAN), after the wireless interface was brought up always failed (?), and wpa_supplicant tried to get the scan results (SIOCGIWSCAN) because some drivers do not deliver SIOCGIWSCAN events to notify when scan is complete, but this failed too, so wpasupplicant requested a second scan after a timeout, which properly delivered the results this time. This first failing scan was adding 21 seconds of delay to the network association process.

I googled the error and found I was not the only soul affected by this problem, Kalle Valo submitted 4 different patches to the hostap mailing list between October 2010 and March 2011, but the patches were never accepted upstream, nor included in the ubuntu package. The wpasupplicant code has changed a bit since Kalle submitted his patches, so I adapted them to the current wpa_supplicant version in Ubuntu. If you are curious, you can dig through ubuntu bug #994739.

In short, the version 4 patch from Kalle basically patches the WEXT driver from wpasupplicant to check the return value when trying to get scan results (SIOCGIWSCAN) from the wl driver, if the number of last error (errno) is EINVAL on the first scan, it requests another scan, so this one will go through (as only the first one fails) and return the scan results next time wpasupplicant tries to get them. This is far from perfect, but it works (and doesn’t seem to break anything), reducing the time needed to associate to the wireless network after the interface has been brought up from 30 seconds to 12 seconds.

But I was still unhappy with this result, so I patched the wpasupplicant code to request a scan right after the driver init function, so this would be the first “failing” scan, and the real scan requested a bit later would return results. This was a very ugly patch, because it made wpasupplicant request a scan in INACTIVE state (when it should be SCANNING), but it worked and reduced the time from 30 seconds to 10 seconds.

So, still unhappy with the results, I decided to go down one more level and have a look at the GPL’d source of the Broadcom’s Linux STA proprietary driver, and BINGO! this is how the wl_iw_set_scan() function ends:

        (void) dev_wlc_ioctl(dev, WLC_SCAN, &ssid, sizeof(ssid));
	
        return 0;

They always return 0, even when the dev_wlc_ioctl() function fails!! and WTH is it casted to void?? It would have been easier to just return the result of this function!. Patching this shows that the first scan after the interface is up fails with errno EBUSY (device or resource busy), so I added a workaround here to make it request the scan to the underlying hardware until it returned something different than EBUSY and could be correctly handled by wpasupplicant, et voilà, time reduced to 10 seconds.

But hey, now that I looked at their source, it turns out that there’s a newer version available in broadcom’s website: 5.100.82.112. This version now supports the new linux cfg80211 wireless configuration API in addition to the older Wireless Extensions (WEXT), you can choose between CFG80211 or WEXT at compile time, the ubuntu package broadcom-sta-dkms in the development release for 12.10 ‘Quantal Quetzal’ has been updated to this version but still uses the old WEXT which is still broken (always returns 0, remember above?). But, guess what they have done it correctly this time in the new CFG80211 code, see the end of the function __wl_cfg80211_scan():

        err = wl_dev_ioctl(ndev, WLC_SCAN, &sr->ssid, sizeof(sr->ssid));
        if (err) {
                if (err == -EBUSY) {
                        WL_INF((\"system busy : scan for \\"%s\\" \"
                                \"canceled\n\", sr->ssid.SSID));
                } else {
                        WL_ERR((\"WLC_SCAN error (%d)\n\", err));
                }
                goto scan_out;
        }
	
        return 0;
	
scan_out:
        clear_bit(WL_STATUS_SCANNING, &wl->status);
        wl->scan_request = NULL;
        return err;

As you can see, they now return EBUSY when the driver cannot perform the scan, and wpa_supplicant can manage this situation correctly, so I quickly backported the broadcom-sta-dkms package from ubuntu 12.10 to ubuntu 12.04 and added a patch to compile it with CFG80211 enabled, and finally I CAN HAS ONLY 8 SECONDS DELAY!!!!1 to associate to the wifi network after my laptop resumes from suspend using the wl driver, and I’m a happy camper! :D

How to have your patch included in Ubuntu

2012-05-21T20:56:05+00:00

Today I submitted a patch to wpa-supplicant package in Ubuntu (LP: #994739) and I learnt the process of pushing your changes to Launchpad and ask for them to be reviewed and merged. Ubuntu has a great documentation on how to fix bugs in this Wiki and packaging guide, just for future reference here are the steps I followed.

If you don’t have them, get the ubuntu-dev-tools, and bazar VCS:

$ sudo apt-get install bzr ubuntu-dev-tools $ bzr whoami "Your Name \<you@example.org\>" $ bzr launchpad-login poliva

Then get the source of the package you want to fix, and apply your patch:

$ bzr branch lp:ubuntu/precise/wpasupplicant $ cd wpasupplicant $ patch -p1 \< /path/to/your.patch

Prepare to build the ubuntu package:

$ dpkg-source --commit $ dpkg-buildpackage -us -uc

In case dpkg-buildpackage complains about no upstream tarball found, you can obtain the .orig.tar.gz file with apt-get source and copy it to the base folder of your work.

$ apt-get source wpasupplicant

Finally, update the debian changelog and commit the change locally:

$ dch -i $ bzr commit

If all looks good, you can push the fix to launchpad and propose merging it into the official package:

bzr push lp:~\<your-launchpad-id\>/ubuntu/\<release\>/\<package\>/\<branchname\> bzr lp-open

The last command will open the Launchpad page of the remote branch in your browser. There click on the “(+) Propose for merging” link, to get the change reviewed by somebody and included in Ubuntu.

Getting started on Android Development from Command Line

2012-01-25T11:01:14+00:00

As a quick reference, here’s a list of useful and most commonly used commands if you want to do Android development from command line (for example, without using eclipse or any other bloated IDE). You must have installed the Android SDK on your system, and make sure the tools and platform-tools folders from the android SDK are available in your PATH environment variable:

ANDROID_SDK="/home/user/android-sdk-linux_x86"
export PATH="${PATH}:${ANDROID_SDK}/tools:${ANDROID_SDK}/platform-tools"

If your SDK installation is not up to date, you can update it using the android update sdk command. To generate a list of system image targets use android list targets, this will produce output similar to this (stripped down for better readability):

id: 1 or "android-3"
     Name: Android 1.5
     API level: 3
id: 3 or "android-4"
     Name: Android 1.6
     API level: 4
id: 5 or "android-7"
     Name: Android 2.1-update1
     API level: 7
id: 7 or "android-8"
     Name: Android 2.2
     API level: 8
id: 10 or "android-9"
     Name: Android 2.3.1
     API level: 9
id: 12 or "android-10"
     Name: Android 2.3.3
     API level: 10
id: 14 or "android-11"
     Name: Android 3.0
     API level: 11

First thing you want to do is create an AVD (Android Virtual Device), this will create an emulator image where you can test your apps without using a real device, the android version used is specified with the -t command switch, and must be one of the targets available in the command we used avobe.

android create avd -n <name> -t <targetID> [-<option> <value>] ...

For example, to create an AVD running android 2.3.3, you would use the command android create avd -n Android233 -t 12. To list all existing AVDs, you can use the command android list avd, this will produce output similar to this:

Available Android Virtual Devices:
    Name: Android233
  Target: Android 2.3.3 (API level 10)
    Skin: WVGA800
  Sdcard: 15M
---------
    Name: Android16
  Target: Android 1.6 (API level 4)
    Skin: WVGA800

To delete an AVD we no longer need, we can use the command android delete avd -n <name>.
To launch the AVD, just use the command emulator -avd <name>:

Now it’s time to create our first project, we will use the command android create project with the following command line parameters:

android create project --target 1 --name MyApp --path ./MyProject --activity MyActivity --package com.example.myapp

Target specifies the minimum android version our project will be able to run, name is just the name of the application, path specifies the path in your hard drive where the project will be stored, activity specifies the name of the first Activity that will be run when we launch the program (an Activity is an application component that provides a screen with which users can interact in order to do something), and finally package specifies the namespace that will be used (it must be a unique package name across all packages installed on the Android system, following the same rules as for packages in Java).

Now it’s time to fire up your favorite text editor (vim FTW!!), and start coding… if you don’t know where to start, a good starting point is reading the Android Application Fundamentals from the Android Developers guide.

If you’re upgrading a project from an older version of the Android SDK or want to create a new project from existing code, we will use the command android update project as folows:

android update project --name MyApp --target 2 --path ./MyProject

The following commands are useful when you have finished coding your application and want to compile, run or debug it:

Building in debug mode:

cd ./MyProject && ant debug

Building in debug mode and install on already running emulator:

cd ./MyProject && ant install

Automatically launch app on the running emulator after building it:

adb shell 'am start -n com.example.myapp/.MyActivity'

And now, all together: build in debug mode, install and run on emulator:

cd ./MyProject && ant install && adb shell 'am start -n com.example.myapp/.MyActivity'

When you are happy with the results, you might want to build your app in release mode, for example to publish it on the android market:

cd ./MyProject && ant release

Note that output needs to be signed and zipaligned before it can be uploaded on the Android Market, you might want to read the Signing Your Applications chapter in the Development guide for instructions on how to generate your private key for signing your applications.

If you want to build in release mode, with the output signed and aligned, you can do it by modifying the ant.properties file in your project as follows:

echo "key.store=path/to/my.keystore" >> ./MyProject/ant.properties
echo "key.alias=mykeystore" >> ./MyProject/ant.properties
cd ./MyProject && ant release

Finally, to wrap up everything we will create a hello world app from command line:

android create project --package com.example.helloandroid --activity HelloAndroid --target 2 --path ./HelloAndroid
	
tee ./HelloAndroid/src/com/example/helloandroid/HelloAndroid.java <<-EOF
package com.example.helloandroid;
	
import android.app.Activity;
import android.os.Bundle;
import android.widget.TextView;
	
public class HelloAndroid extends Activity
{
    /** Called when the activity is first created. */
    @Override
    public void onCreate(Bundle savedInstanceState)
    {
        super.onCreate(savedInstanceState);
        TextView tv = new TextView(this);
        tv.setText("Hello, Android");
        setContentView(tv);
    }
}
EOF
android list avd
emulator -avd XXXXX &
cd ./MyProject
adb devices
ant install
adb shell 'am start -n com.example.helloandroid/.HelloAndroid'

Happy coding! :D