Twig's Tech Tips: June 2012

Python: How to parse XML/RSS feeds with namespaces using lxml.etree

Posted by twig at 12:35 PM Friday, June 29, 2012 0 Comments

Alright, this one had me stumped for a good hour or two.

Those namespaces are a pain, but it's not too bad if you can sort them out before you use them.

# Some basic setup
from urllib2 import urlopen
from lxml import etree
 
# Namespaces copied straight out of the feed source
namespaces = {
  'media': "http://search.yahoo.com/mrss/",
  'dc': "http://purl.org/dc/elements/1.1/",
  'creativeCommons': "http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html",
}
 
# Untested fetching code, just for understanding
file = urlopen(feed_url)
xml = etree.parse(file)
item = xml.get_root().find('channel')[0]

Now here's the basic structure of an RSS feed item.

<item>
  <title>For the lazy</title> 
  <link>http://www.flickr.com/photos/handles/7429952158/<;/link> 
  <description>blah blah blah</description> 
  <pubDate>Sat, 23 Jun 2012 21:59:07 -0700</pubDate> 
  <dc:date.Taken>2012-06-24T14:58:54-08:00</dc:date.Taken> 
  <author flickr:profile="http://www.flickr.com/people/handles/">Handles</author> 
  <guid isPermaLink="false">tag:flickr.com,2004:/photo/7429952158</guid> 
  <media:content url="http://farm6.staticflickr.com/5347/7429952158_962a849b30_b.jpg" type="image/jpeg" height="1024" width="768"/> 
  <media:title>For the lazy</media:title> 
  <media:thumbnail url="http://farm6.staticflickr.com/5347/7429952158_962a849b30_s.jpg" height="75" width="75" /> 
  <media:credit role="photographer">Handles</media:credit> 
</item>

To get information off those elements, you'll need some slightly different syntax.

# Now to fetch the data from the namespaced elements
media_title = item.find("{%s}title" % namespaces['media']).text
 
media_thumbnail = media_title = item.find("{%s}thumbnail" % namespaces['media'])
thumbnail = {
  'url': media_thumbnail.get('url'),
  'width': media_thumbnail.get('width'),
  'height': media_thumbnail.get('height'),
}

Problem solved, like a boss!

Source

Simple XML parsing with Python and LXML

Diablo III: Extracting graphics from .tex files from MPQ and converting them to PNG (or whatever you want)

Posted by twig at 12:04 AM

As always, once I'm bored with a Blizzard game I usually snoop around to see what's in the MPQ files. Diablo 3 is no exception.

Ingredients

MPQ Editor
D3TexConv
Hex Editor XVI32
IrfanView (you'll also need the plugins/addons)

Recipe

OK, I'll be honest. I'm going to assume you have a fair level of computer browsing competency. Some of these tools are still in their infancy so they're not the most user friendly.

If you're gonna need help figuring out how to extract things, navigating folders on your computer or editing batch files, you might want to get someone to help you.

Start up MPQ Editor
Open "Diablo III\Data_D3\PC\MPQs\Texture.mpq"
Browse to the "Textures" folder and extract the files you want to "C:\d3textures\" (or whatever folder you want)
Feel free to close MPQ Editor if you want.
Also extract "xvi32.exe" and "D3Texconv.exe" into that folder.

Now for a little explanation to the stuff you're about to do.

If you go ahead and attempt to convert the .tex files, you're gonna get an error saying "Invalid Texture or Surface (SNO Header Mismatch)". I'm not going to pretend I know what that means.

The whole reason why we have a hex editor is because we can tweak the files to be correct. But of course, we're lazy and we don't want to do it by hand to all 17,000+ TEX files.

Now create "textures.xsc", edit it in notepad and paste the following:

ADR 0
REPLACE EF BE AD DE 2D BY EF BE AD DE 2B

Save and exit.
Create another file called "texfix.bat", open up for editing in notepad. Paste the following (and change your paths if required)

@for /f "tokens=*" %%a in ('dir /b *.tex') do ( "C:\d3textures\xvi32.exe" %%a /S="C:\d3textures\textures.xsc" )

Save and exit.
Create one last file called "convert.bat", edit and paste:

@for /f "tokens=*" %%a in ('dir /b *.tex') do D3TexConv.exe %%a

Save and exit.

Now we're ready to roll!

To patch up all the TEX file headers so D3TexConv can read it, run "texfix.bat".
Once that's done, run "convert.bat" to extract the DDS data from the TEX files.
You'll now get a whole stack of TEX, DDS and TXT files. The only ones you care about are the DDS ones.
Install IrfanView and the plugins pack.
Now you should be able to view the DDS files.
Select the images you want and batch convert them into PNG (or whatever other format you need)

This should take a good while, but you'll be messing with the graphic files in no time!

Browsing the converted PNG files using ACDSee

Sources

Big thanks all round to all the crazy ass reverse engineers who've worked on this. I really don't know who they are, but only through their hard work has this been possible.

I've done my best to source my information so I hope you guys eventually get the credit you deserve!

D3Inferno Textures *.tex
A point in the right direction. This site is very technical
XeNTaX - View topic - Textures of Beta Diablo 3, ".tex" files converter needed :)
Everything I needed, but didn't realise
XeNTaX - View topic - Convert .UI format Diablo III
Found out that UI files were not the way to go, and that TEX files contained what I was after
D3TexConv - Diablo 3 Dev
One of the magic tools that made this all possible
D3Inferno.com - View topic - tex files
Helped solve the SNO header issue

Windows 7: Automatically connect to wireless network

Posted by twig at 9:49 PM Tuesday, June 19, 2012

I was a little perplexed to why the network wasn't reconnecting after a reboot.

I've set it to automatically connect even if SSID isn't even detected.

To fix it, you'll need to have administrator access.

Open up Windows Explorer
Paste in "C:\ProgramData\Microsoft\Wlansvc\"
Right click the folder "Profiles"
Properties
Security
Click "Edit"
Select "Everyone"
Under "Allow", click tick "Full control"
Click OK to save

Most people recommend deleting the "Interfaces" folder under "Profiles". Try my method first, since you won't lose any data.

Secondly if you delete the folder, you'll have to re-enter all the wireless password and settings back in.

Now get back to what you're doing before you lose any more time aging!

Sources

Dell: Replace all driver "download" buttons with direct links (Australian)

Posted by twig at 1:40 PM Sunday, June 17, 2012

I'm not sure if this works on the US site, but this worked a treat for me on the Australian site.

The main issue was I was getting annoyed with the Javascript download question asking me to use the download manager.

I just wanted to queue up files into my own download manager and batch download the files during my off-peak period.

When you're on the driver download page, open up Firebug and paste this into the console.

var rex = /javascript:DownloadFile\('\d+','\d+','.+','(.+)','\w+','HTTP'\);/;
 
var x = $('a[href*="javascript:DownloadFile"]').each(function(index, item) {
  var obj = $(item);
  var href = obj.attr('href');
  var match = rex.exec(href);
 
  if (match != null) {
    var url = unescape(match[1])
    obj.attr('href', url);
  }
});

This will change all the "download file" buttons so they point to the direct file download link.

I'm sure someone can easily convert this into a GreaseMonkey script.

BOXOd

Now, back to the horrors of formatting!

PostgreSQL: ERROR: item pointer (16895,17) already exists

Posted by twig at 10:48 AM Wednesday, June 13, 2012

The numbers in the error are specific to you, but if you see something like this, be sure to check if your indexes are somehow corrupted.

In the console, type "\d tablename" to bring up the details.
Under the table, you'll see a list of indexes.
For each index, type "reindex INDEX indexname"

Try performing your query again.

Do a barrel roll!

Android: Upgrading Samsung i9000 to Gingerbread 2.3.6 (build XWJW6, baseband XWJW1) with root

Posted by twig at 9:53 PM Monday, June 11, 2012

<rant>Alright, I'll be honest. This whole Android flashing business is way more complicated than it should be.

WTF is with all these stupid build "numbers"!? They're not very comprehensible for the average person.</rant>

My reaction when reading a typical Android-hacker post from XDA

Anyways, I've tried to simplify this process down as much as possible.

Preliminaries

First off, some checking. You need to have the 3-button download/recovery mode fix working in case shit hits fan. It did for me and it'll definitely save your ass if it happens to you. If you decide to proceed without fixing this first, you're either very brave or very stupid.

To test it, turn your phone off. Hold Volume Down + Home + Power to turn it on. If you enter download mode, then you're good to go.

Make sure you've backed up what you need!

Installing the firmware

What you'll need is:

This is probably the easiest part of the process.

Extract the firmware
Run Odin (without the phone plugged in yet)
Select the PDA/Phone/CSC files accordingly
Put the phone into download mode
Connect via USB
When Odin's label turns yellow and assigns the phone a COM port, click Start.
Wait a few minutes while it performs the update.
It should restart by itself.

If Odin3 gets stuck on "SetupConnection", just make sure you plug the phone in quicker after starting up Odin. It's a bit fussy.

Rooting your phone

Initially I was going to write about it here, but it was getting too lengthy already and not everybody will need it.

I tried a few different root techniques but not all were stable or easy. I eventually found one that's super easy and stable, so if you're keen on rooting your phone, see this post.

Fix your APN's

Remember, the downside to flashing your phone is that you will lose some settings. This also includes stuff like settings for internet and MMS.

If you're an Aussie, you can get the settings from Ausdroid APN.

Sources

Android: Samsung Galaxy S i9000 3-button download/recovery mode fix

Posted by twig at 9:35 PM

Man, I've been putting this one off for years. Not sure why it only affect some models, but a lot of phones left the factory with a faulty secondary bootloader (SBL).

Because of this, it won't recognise the Volume Down + Home + Power combo.

I had to filter through a LOT of jargon and version hunting for these steps.

Prerequisites

Removed SIM and SD cards.
Phone in USB debugging mode.
Without charging or USB connection, type *#0228# in the phone dialer and ensure that the "Voltage" is at least 3800 (mV).
Phone detected by computer.
Hex codes on your BML1 block (I won't lie, I have no idea what this means. Just check below for instructions)

Just a word of advice, DO THE CHECKS before applying this bootloader. Otherwise you'll brick it, no questions asked.

This is copy pasted from the thread, so please make sure your phone is compatible before fixing the SBL.

HOW-TO check if your mobile is compatible (Windows):
Step 1: Download the XVI32 Hex Editor
Step 2: Dump your BML1 block (rooted phone needed)

Open a new command prompt and type:
adb shell
su (a superuser request will be displayed on the phone screen, accept it).
dd if=/dev/block/bml1 of=/sdcard/bml1.dump
The dump will be copied to the root of the internal SD card.

Step 3: Open the bml1.dump file with XVI32.
Step 4: Search (in ASCII Mode) for OFNI.

If the block reads "!@" just follow the tutorial below starting with Step 1.
If it reads "x0" then STOP as this FiX will BRICK your phone if applied !!

Downloads

I really wanted to use Heimdall for this, but it just wouldn't let me.

Odin v1.7
I already had adb.exe from the Android SDK package. There are plenty of places to get this.
Modified SBL

Procedure

Open up a command prompt and navigate to adb.exe
Type "adb reboot download" to enter download mode
Run Odin
Wait for the "COM###" label to turn yellow
Click on the PDA button and select "P-SBL.tar.md5.tar"
Tick "Phone BootLoader update"
Tick "Autoreboot"
Tick "F.Reset Time"
Ensure the other options are NOT ticked.
Click start whenever you're ready

The whole process will only take a few seconds. You may need to reboot your phone manually.

Turn the phone off and test if download mode works (Volume Down + Home + Power).

Ensure that your phone still works! For me, it got stuck on the loading screen and refused to start up. Take out the battery and try it again.

If it doesn't work, you'll have to reflash your firmware. This doesn't break the bootloader though, so feel free to reflash.

Sources

Android: Gingerbread 2.3.6 root for Samsung Galaxy S i9000

Posted by twig at 9:08 PM

SuperOneClick. That's all there is to it. No jibberish jargon, checking of build versions, phone models, etc. SO EASY!

There's more information about it here on XDA, but this program worked a charm on my phone where the other methods weren't so "clean".

Preparation

Make sure you've got:

Microsoft .NET Framework 2.0+ or Mono v1.2.6+
One of the supported operating systems (Vista+, Ubuntu Hardy+, Debian Lenny+)
Phone on USB Debugging mode
Computer detects your phone (may require USB drivers for your phone to be installed)
Download "Mount /system (rw/ro)" app

Procedure

Download the package, extract. I tried this on v2.3.3.
Run SuperOneClick.
Plug in the phone
Click root
Wait for it to finish
Unplug and reboot the phone manually

It couldn't be simpler!

*edit 25/6/2012* (Added following bits)

Now the last part.

You still won't be able to edit system files because the /system folder is mounted as read only. To change that to read/write, you'll need the "Mount /system (rw/ro)" app.

Start it up, enable write and away you go!

Twig's Tech Tips

Pages

Search Twig's Tips

About Me

Contact

My Pet Projects

Links

Labels

Blog Archive

Python: How to parse XML/RSS feeds with namespaces using lxml.etree

Source

Diablo III: Extracting graphics from .tex files from MPQ and converting them to PNG (or whatever you want)

Ingredients

Recipe

Sources

Windows 7: Automatically connect to wireless network

Sources

Dell: Replace all driver "download" buttons with direct links (Australian)

PostgreSQL: ERROR: item pointer (16895,17) already exists

Android: Upgrading Samsung i9000 to Gingerbread 2.3.6 (build XWJW6, baseband XWJW1) with root

Preliminaries

Installing the firmware

Rooting your phone

Fix your APN's

Sources

Android: Samsung Galaxy S i9000 3-button download/recovery mode fix

Prerequisites

Downloads

Procedure

Sources

Android: Gingerbread 2.3.6 root for Samsung Galaxy S i9000

Preparation

Procedure

Sources