COVER

JUNE 2013  |  ISSUE 230  |  ANDROID

CONTENTS

JUNE 2013  |  ISSUE 230  |  ANDROID

COVER

CONTENTS

LINUX JOURNAL | MASTHEAD

Current_Issue.tar.gz

The Face of a Toaster

LETTERS

UPFRONT

diff -u

They Said It

One Tail Just Isn’t Enough

Non-Linux FOSS: libnotify, OS X Style

Android Poll

GIS with GRASS

EDITORS’ CHOICE

Weechat, Irssi’s Little Brother

COLUMNS

At the Forge

Unicode

Work the Shell

Cribbage: Pairs and Three of a Kinds

Hack and /

Two Pi R 2: Web Servers

The Open-Source Classroom

Prospecting for Ones and Zeros

NEW PRODUCTS

MSC Embedded Inc.’s Qseven Starter Kit

Peter Gasston’s Modern Web (No Starch Press)

ROSA Media Player

Exablox’s OneBlox

xTuple ERP

Erica Sadun and Steve Sande’s Pitch Perfect: The Art of Promoting Your App on the Web (Addison-Wesley)

6WIND’s 6WINDGate

Wind River Linux Carrier-Grade Profile

FEATURES

AIDE—Developing for Android on Android

Compatibility Test Suite—Test Your Android Implementation

Multi-Booting the Nexus 7 Tablet

INDEPTH

Worms and Linux

OPINION

21st-Century DevOps—an End to the 20th-Century Practice of Writing Static Build and Deploy Scripts

EOF

Android’s Limits

LINUX JOURNAL | MASTHEAD

Executive Editor — Jill Franklin, jill@linuxjournal.com

Senior Editor — Doc Searls, doc@linuxjournal.com

Associate Editor — Shawn Powers, shawn@linuxjournal.com

Art Director — Garrick Antikajian, garrick@linuxjournal.com

Products Editor — James Gray, newproducts@linuxjournal.com

Editor Emeritus — Don Marti, dmarti@linuxjournal.com

Technical Editor — Michael Baxter, mab@cruzio.com

Senior Columnist — Reuven Lerner, reuven@lerner.co.il

Security Editor — Mick Bauer, mick@visi.com

Hack Editor — Kyle Rankin, lj@greenfly.net

Virtual Editor — Bill Childers, bill.childers@linuxjournal.com

Contributing Editors

Ibrahim Haddad • Robert Love • Zack Brown • Dave Phillips • Marco Fioretti • Ludovic Marcotte • Paul Barry • Paul McKenney • Dave Taylor • Dirk Elmendorf • Justin Ryan

Publisher — Carlie Fairchild, publisher@linuxjournal.com

Director of Sales — John Grogan, john@linuxjournal.com

Associate Publisher — Mark Irgang, mark@linuxjournal.com

Webmistress — Katherine Druckman, webmistress@linuxjournal.com

Accountant — Candy Beauchamp, acct@linuxjournal.com

Linux Journal is published by, and is a registered trade name of, Belltown Media, Inc.

PO Box 980985, Houston, TX 77098 USA

Editorial Advisory Panel

Brad Abram Baillio • Nick Baronian • Hari Boukis • Steve Case • Kalyana Krishna Chadalavada • Brian Conner • Caleb S. Cullen • Keir Davis • Michael Eager • Nick Faltys • Dennis Franklin Frey • Alicia Gibb • Victor Gregorio • Philip Jacob • Jay Kruizenga • David A. Lane • Steve Marquez • Dave McAllister • Carson McDonald • Craig Oda • Jeffrey D. Parent • Charnell Pugsley • Thomas Quinlan • Mike Roberts • Kristin Shoemaker • Chris D. Stark • Patrick Swartz • James Walker

Advertising

E-MAIL: ads@linuxjournal.com

URL: www.linuxjournal.com/advertising

PHONE: +1 713-344-1956 ext. 2

Subscriptions

E-MAIL: subs@linuxjournal.com

URL: www.linuxjournal.com/subscribe

MAIL: PO Box 980985, Houston, TX 77098 USA

LINUX is a registered trademark of Linus Torvalds.

AT YOUR SERVICE

SUBSCRIPTIONS: Linux Journal is available in a variety of digital formats, including PDF, .epub, .mobi and an on-line digital edition, as well as apps for iOS and Android devices. Renewing your subscription, changing your e-mail address for issue delivery, paying your invoice, viewing your account details or other subscription inquiries can be done instantly on-line: http://www.linuxjournal.com/subs. E-mail us at subs@linuxjournal.com or reach us via postal mail at Linux Journal, PO Box 980985, Houston, TX 77098 USA. Please remember to include your complete name and address when contacting us.

ACCESSING THE DIGITAL ARCHIVE: Your monthly download notifications will have links to the various formats and to the digital archive. To access the digital archive at any time, log in at http://www.linuxjournal.com/digital.

LETTERS TO THE EDITOR: We welcome your letters and encourage you to submit them at http://www.linuxjournal.com/contact or mail them to Linux Journal, PO Box 980985, Houston, TX 77098 USA. Letters may be edited for space and clarity.

WRITING FOR US: We always are looking for contributed articles, tutorials and real-world stories for the magazine. An author’s guide, a list of topics and due dates can be found on-line: http://www.linuxjournal.com/author.

FREE e-NEWSLETTERS: Linux Journal editors publish newsletters on both a weekly and monthly basis. Receive late-breaking news, technical tips and tricks, an inside look at upcoming issues and links to in-depth stories featured on http://www.linuxjournal.com. Subscribe for free today: http://www.linuxjournal.com/enewsletters.

ADVERTISING: Linux Journal is a great resource for readers and advertisers alike. Request a media kit, view our current editorial calendar and advertising due dates, or learn more about other advertising and marketing opportunities by visiting us on-line: http://ww.linuxjournal.com/advertising. Contact us directly for further information: ads@linuxjournal.com or +1 713-344-1956 ext. 2.

SPONSORS OF THIS ISSUE

1&1  http://www.1and1.com

Cloud World Forum  http://www.cloudwf.com/

Drupalize.me  http://drupalize.me

Emac, Inc.  http://www.emacinc.com

EmperorLinux  http://www.emperorlinux.com

iXsystems, Inc.  http://www.ixsystems.com

KingStar USA  http://www.kingstarusa.com

Manage Engine  http://www.manageengine.com

New Relic  http://www.newrelic.com

O’Reilly OSCON  http://www.oscon.com/oscon2013

OVH  http://www.ovh.com

Sharepoint  http://www.sptechcon.com

Silicon Mechanics  http://www.siliconmechanics.com

USENIX  https://www.usenix.org/conference/fcw13

LINUX JOURNAL (ISSN 1075-3583) is published monthly by Belltown Media, Inc., 2121 Sage Road, Ste. 310, Houston, TX 77056 USA. Subscription rate is $29.50/year. Subscriptions start with the next issue.

CURRENT_ISSUE.TAR.GZ

SHAWN POWERS

The Face of a Toaster

Science-fiction geeks have very specific definitions for their (okay, “our”) jargon. A cyborg is a hybrid mechanical and organic creature. An AI (artificial intelligence) is a purely electronic device or program capable of thought. And an android is a purely mechanical and electronic device with a human appearance. In Star Trek terms, the Borg are cyborgs, and Data is an android. Then Google came and messed everything up. “Android” is now synonymous with the operating system on phones and tablets everywhere. Unless it’s an obscure social commentary on square shapes and human obesity, I think it’s safe to assume Google just went with a cool-sounding, futuristic name. This month, we focus on Google’s Android, the Linux-based operating system that’s taking over the planet.

Reuven M. Lerner starts out the issue with the universal translator of the programming world—namely, Unicode. Although it makes life easier all across the planet, working with Unicode can be challenging. Reuven addresses some of those challenges, and shows how to work through them. Dave Taylor helps us work through some things too as he continues his series on the Cribbage script we’ve been working on. Whether you want to learn scripting, or better understand Cribbage, it’s an awesome series.

Kyle Rankin brings us another Raspberry Pi article this month, this time doing fault tolerance with redundant Web servers. Kyle’s articles always make me hungry for knowledge, but the past couple months, his titles have made me hungry in general—Raspberry Pi, indeed. I, on the other hand, talk about cryptocurrency this month. It’s been a while since I’ve mentioned Bitcoins or Litecoins, and with the recent bubble, perhaps it’s time to dust off those miners and start generating cash. It’s not a surefire way to get rich, but you might be able to earn enough income to buy some Raspberry Pi devices and keep up with Kyle!

Generally, developers develop computer programs on computers. They also develop mobile applications on computers. Joey Bernard shows us AIDE this month, which is a nifty way to develop Android apps while using the Android OS! If you’ve ever wanted to do some coding while on the road, but you’re stuck with only your tablet, you’ll want to check out this article. Nitish Tiwari delves into development this month as well, but he discusses testing side of things. If you want to port your application to new hardware, go directly to his article and start reading.

Bill Childers and I are kindred spirits when it comes to mobile technology. I think his pile of goodies is a little taller than mine, but thankfully, he shares his experiences with us all. A couple months back, I wrote about my Nexus 7 tablet. Bill goes in a slightly different direction with his Nexus 7, and he explains how to boot in to completely different ROMs! Whether you want to try CyanogenMod or Ubuntu on your tablet, MultiROM is your tool, and Bill describes how to use it.

At times, we can get arrogant about security as Linux users. Himanshu Arora looks deeper into viruses and worms. Are we safer by design? Are we setting ourselves up for disaster? Himanshu’s article is a must-read for anyone interested in the debate. And finally, Tracy Ragan’s opinion piece on DevOps is one that hits home to an old-timer like me. The scale of data centers is such that the traditional static scripts and hacks are being phased out by newer, truly automated systems.

If you think Android is a little green robot that lives in your phone, or if you think Google has blasphemed all of science fiction, this Android issue of Linux Journal should be enjoyable. I can’t bring myself to make a cheesy comment like, “set phasers to awesome”, so I’ll just hope you enjoy this issue. We think it’s awesome, phasers or no.■

Shawn Powers is the Associate Editor for Linux Journal. He’s also the Gadget Guy for LinuxJournal.com, and he has an interesting collection of vintage Garfield coffee mugs. Don’t let his silly hairdo fool you, he’s a pretty ordinary guy and can be reached via e-mail at shawn@linuxjournal.com. Or, swing by the #linuxjournal IRC channel on Freenode.net.

LETTERS

Linux and Windows File Servers

I’ve been looking through the April 2013 issue of LJ’s Letters section with particular interest in the uptake of Linux in schools, and I’d agree with most of what Lucian Macu and you have to say.

It’s not just schools though where the adoption of Linux is perceived to be difficult. In my previous job with the Birmingham City Council, I was one of a few users who ran Linux on the desktop. I was on the UNIX team, so it seemed the most natural thing to do. However, initially the back-end home drives and so on were on Novell, and then they moved sadly to Windows. We ran Lotus Notes for e-mail, and I can say the native client for v8 worked very well with Fedora (can’t remember the release).

In my current job with IGT, until recently, I have been forced to use the truly dreadful Windows 7, although recently due to Windows 7 crashing and not starting up yet again (four times in two and a half years), I’m back with a Linux desktop in a totally Windows environment.

To try to streamline living in this environment, I started writing some login scripts that would cope with Novell (ncpfs), Windows (Samba CIFS and so on) and Linux (sshfs) back ends and create a mapped drive similar to a Windows desktop. It wouldn’t be too hard to cope with a Mac desktop to these back ends too. Some embryonic code exists to detect OS X, but as I don’t have a Mac currently, this is on hold.

If there is a mix of desktop systems, it is desirable to have a degree of uniformity to them. Personally, I find the Windows desktop clunky to use, but if I were a user familiar with how Windows desktops work, the idea of mapped drives rather than mounted filesystems grafted on at a mountpoint would be easier to adjust to.

Also, if I am a Linux laptop user, I would not want a remote home directory mounting over my /home/user, and if there are shared team filesystems, there still needs to be somewhere to put them, so on balance, I think the idea of using a drive letter isn’t too bad for a desktop system. Of course, the drive letter is really just a mountpoint, but let’s not tell the users that!

So far, development has halted a bit for various time-related reasons, but the work as it exists is freely available on my Web site: http://www.rainsbrook.co.uk/wiki/doku.php?id=rb:linuxloginscripts.

One main block to progress has been the move by Linux from SMB to CIFS. I used to be able to use smbmount, but CIFS doesn’t want to allow a user to run this without anything in fstab.

One of the golden rules I have tried to set is that all this drive mapping and setting mountpoints should be able to be run as the local user. I’d really like to avoid any root level work. Ultimately, if the login script lives in the user’s local home directory, the cfg part could come from the netlogon shared directory for Windows or an HTTP link. If the cfg file is changed by an administrator, there should be no requirement for the system admin to modify anything locally.

For many years now, I have been a passionate Linux user. I am constantly frustrated by the lack of progress on adoption in education and business. It’s more than capable for use in either environment with some small developments.

I’d hoped that with the Linux adoption by Novell, we might have seen more business/user developments in distros rather than just technical and architectural ones, as it is, Linux is more than capable, but it needs some bits bolted on to help it play well in an alien environment.

I’m not really sure where to go with this. I’d hoped that someone in the local LUGs would have picked up on this and offered to join in, but not much interest has been shown. I think I need to try to open this up and get more ideas in to solve some of the blocks that exist and maybe to clean up the code and so on. So if you know anyone, please point them to the Web page!

—Andrew Stringer

Well, it’s posted as a Letter to the Editor, so all our readers will see your link. Perhaps it will go somewhere.

I will note that I was very excited when Novell turned to Linux as well. Unfortunately, like you, I haven’t seen anything really come from it. I deal with Windows a lot in my day job, so in that regard, I feel your pain too. I try to look at the heterogeneous computing environment as a learning opportunity, but it’s still frustrating at times. I’m not able to use Linux as my main computer during the workday, but thankfully, I can have a virtual machine running full screen. Thanks again for your letter—Shawn Powers

Space Program on Twitter

The best thing I’ve done to re-ignite my enthusiasm for the space program is to join Twitter, so I could follow @Cmdr_Hadfield. He’s the Canadian commander of the ISS for the next two weeks still, and he tweets down some amazing photos of places on Earth and the aurora and such. He’s a big hero up here in Canada, of course.

Also, I hope you get your ISP problems sorted out soon, as I neglected to download the April PDF and epub files; I don’t remember reading it either. I must have been out to lunch on April 1st.

—Neal Murphy

I love Commander Hadfield! I think his foray into social media has done more to excite the younger generation than years of marketing ever did. Oh, and my ISP problems? Yeah, it seems to be getting bett[CONNECTION LOST].—Shawn Powers

The Open-Source Classroom Column

I’ve been a subscriber to Linux Journal for the past 5+ years, and I thoroughly enjoy working in Linux (Ubuntu/Debian). I believe that not enough is done to expose the children of South Africa to Linux from an early age. They are mostly educated on Windows/Office-based systems.

For a while now, I have been seriously considering starting a physical training center to teach them about Linux, open-source software, Raspberry Pi, Arduino and so on, but the details haven’t been finalized yet.

The reason for this e-mail is to find out if there will be any issues if I call the center “The Open-Source Classroom”. I know there probably are a million other choices, but the name is so appropriate for what I have in mind.

—Roelof Oelofsen

I responded right away when I got this, but I’ll reiterate for everyone—yes, by all means, use the name! We chose the title for my column because it sounded cool, but it’s not something unique or specific enough to the magazine to worry about it. Thank you for asking, but even more, thank you for your passion in regards to educating kids!—Shawn Powers

Linux Poetry

I write original pieces that I e-mail to friends and colleagues every Friday. This one, from this week, seemed like something you might like to “print” in Linux Journal. (I am a longtime subscriber and still have issue 1 wrapped in plastic!)

My title is “Linux Operations, Relations And X”

to understand my position, go read the Specs.

They call me the LORAX, I speak about trees.

Rooted, Directed, Acyclic and Bs.

I speak about trees, for the trees have no tongues,

And I’m asking you, sir, at the top of my lungs -

I love backyard flowers, and gardens and seeds,

but Version Control Systems, use the trees in my needs.

It lets people find the files that go missing,

it lets me bring back bad mods, that others were dissing.

There’s CVS, RCS, Subversion and others,

with features and options and ideology that smothers.

I search for myself, I search for my team,

for without simple GUIs,

They’ll simply run out of steam.

I’m looking at Bazaar and I’m looking at Git,

I’m looking, and looking and looking more, a bit.

They call me the LORAX, I speak about trees.

I care about branches, and trunks and tags for these.

It must be easy and it must be simple,

it must make people smile and show me their dimple.

I already got Git, Git I already gat,

don’t correct my conjugation, I don’t wear that hat.

Bazaar they say is simpler,

they say that’s what to get,

But I rely on my gut, my gut says get Git.

Git is bazaar, and Bazaar, I can get,

but for the non-data-arborist, both are a threat.

Git is from Linus, so it must be well done,

Linus did Linux, then Git just for fun.

I can insist, and I will, on my best rule of thumb

smart things from smart people really make me feel dumb.

When I smell Git’s docs, I feel like a sneeze,

How will I ever find the forest, while looking at trees?

Still, they call me the LORAX, I speak about trees.

Rooted, Directed, Acyclic and Bs.

—Dennis Director

Poetry for geeks is cool,

And Dr. Suess rocks,

So truly everyone wins!

—Shawn Powers

Evernote

Just a follow-on from the Everpad article [see Shawn Powers’ Everpad article in the April 2013 Upfront section]—I find that the Windows .exe version works just fine for me in Linux using Wine (1.5). But just as good, if not better, is the Evernote extension for Chromium (or Chrome) as well as the Evernote Web Clipper add-on for Firefox.

—Bill

I find myself using the Evernote Web app in Chromium more often than not. I’ll admit, I haven’t tried the Wine solution yet; I’m not sure why it never occurred to me! I still wish they provided a native Linux client, but at least the Web application is robust. Thanks for the tips!—Shawn Powers

Photo of the Month

When cleaning out a closet with old papers, I found the attached. How many memories this card has brought me. I think it would be a good idea to republish this card and send it to those who renew their subscriptions.

—Manuel Soriano-Vic Pérez

WRITE LJ A LETTER We love hearing from our readers. Please send us your comments and feedback via http://www.linuxjournal.com/contact.

PHOTO OF THE MONTH

Remember, send your Linux-related photos to ljeditor@linuxjournal.com!

UPFRONT

NEWS + FUN

diff -u

What’s New in Kernel Development

Recently, Rob Landley tried to remove a Perl dependency that has cropped up in the Linux build system...and tried...and tried. But, no one appeared to want his patches. Even Andrew Morton argued against removing the Perl dependency, in spite of the fact that Rob’s patches replaced the code with smaller, simpler shell scripts.

It’s unclear what the reluctance stems from—my understanding always has been that any chance to remove a dependency from the Linux build system was a chance worth taking. But apparently, the top kernel developers see a value in this particular dependency.

The e-mail thread was a bit of an eye-opener for me personally, because I’d started to consider Perl to be a legacy language, with no forward progress on Perl 6, and the language itself essentially stalled.

Microsoft’s “secure boot mode” seems to be pitting kernel developer against kernel developer successfully. David Howells recently posted some patches to allow the kernel to load Microsoft-signed cryptographic keys dynamically, and there was some support for the patch. But, Linus Torvalds wouldn’t allow it, and there was some support for his rejection of it.

Linus’ approach seems to be that he’ll support anything that can provide genuine security to the user. And if some new technology is not designed to provide that security, he’ll support it to the extent that it can be torqued into a position that provides genuine security. But, he won’t support technologies that claim to add security when all they really do is take control of the system away from the user.

Evidently he feels that, as designed, “secure boot mode” does more to take control away from the user, than it actually does to secure the system. So he’s rejected David’s patches, as well as apparently any notion of catering to Microsoft as the sole key-signing authority for “secure boot mode” security keys.

According to Thomas Gleixner and others, the Linux kernel’s hotplug code had been developing races, undocumented behaviors and other problems. He posted a patch to rip the guts out of the beast and replace it with something that, to the user, would appear as a simple state machine.

Linus Torvalds liked the idea, but he wanted to be sure that the guts really were fully ripped out and replaced. He didn’t just want to hide the problems. But as Rusty Russell pointed out, Thomas’ patch represented only the first step, in which the horror was hidden from the user. The second step would involve cleaning up the internals themselves.

Rob Landley recently chafed at the increased security on the kernel.org server. Ever since the 2011 security breach, the kernel.org admins have implemented tighter access controls that limit the ways in which developers can contribute their work. Rob in particular missed the ability to rsync his code updates to a kernel.org account. As a result, he hadn’t been able to update the 00-INDEX files in the kernel source tree. Paul Gortmaker also had been updating those files, but he posted his patches to the public mailing list instead. That seems to be the standard approach these days, along with creating public git repositories from which Linus and others may pull.—ZACK BROWN

They Said It

Don’t go around saying the world owes you a living. The world owes you nothing. It was here first.

—Mark Twain

In a networked world, trust is the most important currency.

—Eric Schmidt

When you relinquish the desire to control your future, you can have more happiness.

—Nicole Kidman

Not a shred of evidence exists in favor of the idea that life is serious.

—Brendan Gill

Genius begins great works; labor alone finishes them.

—Joseph Joubert

One Tail Just Isn’t Enough

Figure 1. Two files at once is just the beginning; multitail is awesome (screenshot from http://www.vanheusden.com).

Although it’s difficult for me to look at this piece’s title and not think of mutant felines, it doesn’t make the statement any less true. If you’ve ever used the tail command on log files, you’ll instantly appreciate multitail. My friend (and LJ reader) Nick Danger introduced me to multitail, and I can’t believe how useful it is. multitail will “tail” multiple files, split the screen to display them, notify of log file changes and so on. One of my favorite features is rather than show 100 lines of repeated log, it shows the line only once, and it says, “line repeats 100 times”—simple, but awesome.

multitail has more features than I could list on this page, but chances are if you’ve ever wished you could do something with log files, multitail does it. Check it out at http://www.vanheusden.com/multitail.—SHAWN POWERS

Non-Linux FOSS: libnotify, OS X Style

Figure 1. A quick command-line-ninja move creates a pop-up.

One of the things I dislike about using Irssi in a terminal window on OS X is that I often miss the screen flash when someone mentions my name in IRC. With some fancy SSH tunneling (maybe more on that some other issue) and a really cool pop-up notification tool, if someone mentions my name, I can’t miss it.

terminal-notifier is a command-line tool for creating OS X-native user notifications. It doesn’t rewrite the concept of pop-ups; instead, it gives us nerds a way to add pop-ups to scripts (Figure 1). Because it uses the native notification system, it’s easy to modify what sort of pop-up appears. I prefer the kind that doesn’t go away until dismissed, but you can change that in the notification settings in OS X’s preferences.

If you like pop-up notifications like libnotify, but find yourself on a Macintosh machine more often than not, terminal-notifier might be as useful for you as it is for me at my day job. Plus, now you know that if you mention my name in IRC during the workday, you’ll make a window pop up on my screen! Get it at https://github.com/alloy/terminal-notifier.—SHAWN POWERS

Android Poll

We recently asked our readers about their Android usage, and as predicted, most of our readers own an Android phone, tablet or other Android device. Also not surprising is that our readers are mostly up to date, with the majority of users running the Jelly Bean release. 86% are loyal to Android and have not jumped ship in favor of another platform. E-books also are popular with our readers, but E Ink vs. a backlit color display is a toss up. Read on to see the full results, and as always, thanks for participating!

Do you own an Android smartphone?

■ Yes: 80%

■ No: 20%

Do you own an Android tablet?

■ Yes: 56%

■ No: 44%

Do you own a non-smartphone/tablet Android device?

■ Yes: 56%

■ No: 44%

Which version of Android is running on the majority of your devices?

■ Cupcake (1.5): <1%

■ Donut (1.6): <1%

■ Eclair (2.0/2.1): 1%

■ Froyo (2.2): 5%

■ Gingerbread (2.3): 15%

■ Honeycomb (3.0/3.1/3.2): 2%

■ Ice Cream Sandwich (4.0): 22%

■ Jelly Bean (4.1/4.2): 54%

Did you switch from an iPhone to an Android phone?

■ Yes: 10%

■ No: 78%

■ NA: 12%

Have you switched away from Android to another platform (iOS and so on)?

■ Yes: 6%

■ No: 86%

■ NA: 8%

Have you already or are you planning to buy an Ouya?

■ Yes: 11%

■ No: 89%

Which is worse for developers?

■ iOS’s walled garden: 68%

■ Android’s version fragmentation: 32%

Do you read e-books?

■ Yes: 79%

■ No: 21%

If you read e-books, which do you prefer?

■ An LCD screen: 49%

■ E Ink: 50%

Does DRM limit the amount of digital media you purchase?

■ Yes: 73%

■ No: 27%

What do you do most on your Android device?

■ Consume media (audio/video): 34%

■ Read/create e-mail: 27%

■ Social networking (Twitter, Facebook and so on): 16%

■ Play games: 7%

■ Talk on the phone: 15%

■ Video conference: 1%

Does your device’s battery last all day?

■ Always: 28%

■ Never: 12%

■ Usually: 43%

■ If I don’t use it a lot during the day: 17%

Have you rooted your Android device(s)?

■ Yes: 47%

■ No: 53%

—KATHERINE DRUCKMAN

GIS with GRASS

The major player in the Windows world for GIS programs is the suite of ESRI products. In Linux, we have the package named GRASS (http://grass.osgeo.org). GRASS originally was developed by the US Army Construction Engineering Research Laboratories, starting in 1982. It is used by many large groups, including NASA, NOAA and the National Park Service. In September 2006, management of GRASS was taken over by the GRASS Project Steering Committee, and it now is an official project of the Open Source Geospatial Foundation.

Figure 1. The Main Web Site for the GRASS Project

GRASS may be too much if you just want to do basic GIS tasks. In that case, you may be better served by a program like QGIS. But, if you need to do some serious GIS analysis, GRASS definitely is worth the learning curve. Most distributions should have a set of packages to simplify installation. If you do want the latest source, or need a version of GRASS for Windows or Mac OS X, you always can go to the main Web site.

Figure 2. The download page provides binaries for Linux, Windows and Mac OS X.

When you first start up GRASS, it asks you to set a data directory. The suggestion is ~/grass-dir. Once you select this data directory, you need to set some project information. You can click on the Location Wizard to help set the location information for your project.

Figure 3. When you initially start GRASS, you need to select a location and a mapset.

Once you set the name and data directory, you need to select the method for creating a new location. Just to get started, you simply can accept the defaults. To learn how to work with GRASS, you will want to have some data to play with. Sample datasets are available to download from the main GRASS site (http://grass.osgeo.org/download/sample-data). Choose one or more of them, download the files and uncompress them into the data directory you set above. These sample datasets then will show up in the “Welcome to GRASS” window when you first start up GRASS. At this point, select one of these datasets. You also need to select a mapset, most usually PERMANENT.

Figure 4. You can use several sample data sets while you are learning to use GRASS.

Once GRASS starts up, two windows appear. The first window is a map display, where all of the layers you select will be rendered. The second window is where you select the map layers that you want to apply to the map display window.

Figure 5. On start up, you will have a map display window and a layers window.

To create your first map, click on the “Add raster map layer” button (the one with a checkerboard and a plus sign), or you can press Ctrl-Shift-R. This will pop up a dialog window where you can select which layer you want to add from the mapset you loaded on startup. In this example, I have loaded the PERMANENT mapset from the spearfish location, and I set the 10m elevation as the first layer of my map.

Figure 6. Here you select which layers to load.

Figure 7. Here you can see the elevation of the land in the spearfish location.

One of the first things you will want to do is to change the colors used within the map. To do this, right-click on the layer in question, and select “Set color table” from the drop-down menu. You then can change the color table that GRASS will select from in order to render the layer on your map. To change your layer to grayscale, select “Type of color table:” and select “grey”.

Figure 8. Changing the color map is relatively simple.

When you click on the run button, you are switched to the “Command Output” tab where the results from this command are displayed. If you want to see an idea of the spread of the possible values, you can get a histogram by right-clicking the layer and selecting Histogram. If you need more exact numbers, you actually can calculate univariate statistics on the data in the layer. This is done by right-clicking on the layer and selecting “Univariate raster statistics”.

Figure 9. You can see the results of running the color change immediately.

Figure 10. You can get an idea of the spread of values in a raster map by generating a histogram.

Figure 11. More detailed analysis is available by looking at univariate statistics.

Adding a second layer allows you to start building up the information being displayed on your map. You need to be careful of which order the layers are in the layer list. They are rendered from the bottom up. This means that layers further up the stack may obscure what is being displayed lower down. You may need to change the opacity of the upper layers to allow information from the lower layers to show through. Right-click on the layer in question and select “Change opacity level”. You then can set it to an appropriate level so everything you want to see actually is rendered.

Figure 12. You may need to change the opacity of layers once you start stacking them.

The other type of layer that you can add to your map is a vector layer. In this case, the data is stored as a set of geometrical objects, where each object has some attribute data assigned. With vector layers, the only portions that are rendered are the actual objects. For example, if you add a road layer, you don’t need to worry about opacity, because the roads are small enough not to obstruct anything on the layers below. You can right-click on that layer and edit the attribute data. You then can select which values for each attribute to display. This can be a more complex selection—for example, selecting those values between an upper and lower bound or selecting only those values that match some other criterion.

Figure 13. You also can load vector layers.

You can change display properties for the objects by right-clicking and selecting Properties. For the road layer, you can set properties like line width, line color and what symbols to use for point elements.

Figure 14. You can highlight elements in a vector layer.

You can add extra elements that you normally see on maps by selecting the “Add map elements” button on the main map display. This opens a drop-down box where you can select extra elements to add. These include scalebars, North arrows, legends and text areas. You can click and drag these elements and place them where they need to be on your map.

Figure 15. You can add extra elements to your map, like a scale bar.

Once you have the layout the way you want, you need to save a final copy so you don’t lose all of your work. To do so, click the “Save display to graphic file” button on the main map display. The first step is to choose the output size for the map. Then you can select the filename and the file format.

Figure 16. When saving a map, you need to select the output size.

Figure 17. You can select the filename and file format when you save your map.

Figure 18. Don’t forget to quit when you’re done.

Hopefully, this article introduces you to enough of GRASS to induce you to try it out. If it’s good enough for the US Army, it’s good enough for me. It should be powerful enough to handle any GIS task you have.—JOEY BERNARD

EDITORS’ CHOICE

Weechat, Irssi’s Little Brother

Figure 1. The Linux Journal IRC bot works in Weechat, so I was happy.

It may not be fair to call Weechat the little brother of Irssi, but in my short introduction to it, that’s what it felt like. If Weechat didn’t seem quite as powerful as Irssi to me, I definitely can say that it is better-looking out of the box. So, little brother has one thing going for him!

The other day, I was tweeting with Janne Jokitalo about the sorts of things two nerds tweet about: command-line editors and command-line chat clients. I mentioned Irssi in a screen, and he mentioned Weechat. I’m glad he did! Right out of the box, Weechat does some things I’ve never been able to get Irssi to do well. First off, it has a list of users docked to the right side of the terminal (Figure 1). I always liked that feature in the GUI client X-Chat, but I couldn’t get it to work well in Irssi. I also think the look and feel is far more friendly than that of Irssi. Yes, with the help of Kyle Rankin, I’ve been able to tweak Irssi into the perfect chatting machine, but Weechat seems to have a more gentle learning curve.

It supports IRC and Jabber right now, but the Web site claims more protocols are coming. Weechat is probably already in your distro’s repository, so install it, and give it a whirl. You’ll get all the geek creed of Irssi with some fancy interface additions! Due to its focus on usability and its roots in hard-core nerd-dom on the command line, Weechat is this month’s Editors’ Choice selection. Check it out at http://www.weechat.org.—SHAWN POWERS

COLUMNS

COLUMNS

At the Forge

REUVEN M. LERNER

Unicode

Support legacy data and users better by understanding Unicode.

Let’s give credit where credit’s due: Unicode is a brilliant invention that makes life easier for millions—even billions—of people on our planet. At the same time, dealing with Unicode, as well as the various encoding systems that preceded it, can be an incredibly painful and frustrating experience. I’ve been dealing with some Unicode-related frustrations of my own in recent days, so I thought this might be a good time to revisit a topic that every modern software developer, and especially every Web developer, should understand.

In case you don’t know what Unicode is, or how it affects you, consider this: in C and in older versions of languages like Python and Ruby, a string is nothing more than a bunch of bytes. There’s no rhyme or reason to it; you can read whatever data you want into a string, and the language will be fine with it. For example, if I fire up iPython (which uses Python 2.7), I can read a JPEG image into a string:

s = open(‘Downloads/test.jpg’).read()

Most of the time, you use strings not to hold JPEG images, but rather to hold text. If your text is all in English, you’re in luck, because all the characters used by the English language are defined in ASCII, a standard that defines 128 different characters, each with a unique number. Thus, character 65 is uppercase A, and the space character is number 32. ASCII is great, and it works just fine—until you want to start using languages other than English.

The problem is most languages require characters that are not used in English, and that aren’t defined in ASCII. This means if you want to write words in French, let alone in Arabic or Chinese, you won’t have a way to represent characters using ASCII.

A solution for alphabetic languages was a set of ISO standards (ISO 8859-*), which took advantage of the fact that ASCII uses only 7 bits, but that data is transmitted with 8 bits. If you can take advantage of all 8 bits, you double the number of available characters, from 128 to 256. This is more than enough for languages with a defined alphabet. Thus, Western European languages were defined in ISO-8859-1, Hebrew in ISO-8859-8 and so forth. Moreover, these ISO standards were meant to make it possible to mix the “foreign” language with English. Thus, you could have a document with English and French or English and Arabic. ASCII characters retained their original values, and the non-ASCII characters were defined in the upper 128.

But, what happens when you want to have a document that contains English, Arabic and French? In the ISO-8859 family of standards, there wasn’t any way to accomplish this. The same number that was used to describe an accented character in French also would be used to describe a character in Arabic. The program displaying the text in question was responsible for deciding which language and, thus, which characters, would be displayed. A document written in Russian (ISO 8859-5) but displayed by a program expecting Hebrew (ISO 8859-8) would show Hebrew characters, or rather, gibberish.

Things are even worse if you’re working with non-alphabetic languages, such as Chinese. Even if you would like to use the upper-128 characters to write Chinese, you would be forced to choose from a tiny percentage of the characters that are necessary to use the language. Clearly, something else would be necessary, and indeed, the Chinese (as well as Japanese) invented their own systems for storing text on computers, which were completely incompatible with ASCII.

Unicode was designed to solve all of these problems. Simply put, it gives every individual human-designed character its own unique number, or “code point”. Doing this removes the ambiguity associated with displaying text. So long as a program supports Unicode, it doesn’t need to know the language family that’s being used. English, French, Arabic and Russian all can coexist on the same page, without any interference between the characters. Moreover, Unicode supports a very large number of code points, allowing Chinese and Japanese characters to coexist with alphabetic characters.

Encodings

So far, so good. But, switching over to this new system raised two questions. First, how do you take these individual code points, uniquely identifying just about every character humans have created, and translate them into bytes? Second, what happens to existing documents, which weren’t written in Unicode?

On the one hand, the answers to those questions are relatively straightforward. On the other hand, the answers lead to much of the frustration associated with using Unicode—not because Unicode itself is bad or difficult, but because the mix of different, existing encodings with a Unicode-based system can be frustrating.

The first question, how do you encode the various Unicode characters using bytes, has multiple answers. If you’re using a Unicode-aware language, you no longer can think of characters as being equivalent to bytes. Rather, one character might be a single byte, but it also might be multiple bytes. In the UCS-32 encoding scheme, for example, each Unicode character uses 4 bytes. This provides enough space for all of the defined Unicode characters, which is a good thing, but it also breaks backward compatibility with ASCII documents and quadruples the size of anything written using ASCII or any of the ISO-8859 series.

For these reasons, the de facto standard in the Unicode world is UTF-8, a variable-length encoding scheme invented by famed programmers Rob Pike and Ken Thompson. The basic idea is that all defined ASCII characters, from 0–127, remain as they were. If the high (8th) bit is set, that indicates the character consumes an additional byte (that is, two bytes for the character). In a similar way, high bits are used on succeeding bytes to indicate that the character’s description has not ended. In this way, UTF-8 characters can consume as little as one byte (for ASCII characters) or as many as 6 bytes for truly unusual characters. Languages like Chinese and Japanese will require 4 bytes per character.

UTF-8 provides the best of all possible worlds—ASCII documents remain as they were, alphabetic languages don’t use too many more bytes than necessary, you resolve ambiguity with Unicode, and you can represent all Unicode characters. But, it does introduce a new problem: strings can now be invalid! If you were to use the fixed-width UCS-32 system, just about every byte would point to a valid character. But in UTF-8, it’s possible to have a sequence of bytes that’s invalid according to this encoding scheme.

To return to my example from earlier in this article, let’s say I execute the following code in Python 3, rather than Python 2.7:

s = open(‘Downloads/test.jpg’).read()

Now, in Python 2.7, strings are just collections of bytes. If I want to use Unicode, I need to use a “Unicode string”, a special version of the str type in which characters are all in Unicode (and stored in UTF-8). In Python 3, the default string encoding is UTF-8, which means that executing the above code actually will result in an exception:

UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xff in

position 0: invalid start byte

In other words, Python was expecting to get input in UTF-8, but noticed the byte 0xFF at the start of the file, which is illegal. What you need to do is tell Python that you want to read the file in binary format, by opening it in “read binary” mode:

s = open(‘/Users/reuven/Downloads/test.jpg’, mode=’rb’).read()

Now, given that you’ve read the file in binary mode, you’re treating it as bytes, rather than a string. And sure enough, if you ask Python what type of data was returned:

>>> type(s)

<class ‘bytes’>

In other words, Python won’t create an illegal string. So instead of doing so, read() returns a bytestring, which is roughly the same as the Python 2.x string.

That covers files that were written in Unicode. But what about files written in another encoding scheme, such as ISO-8859-5? In such a case, you need to pass another parameter to “open”, indicating the encoding you should use.

Ruby has undergone a similar change in the past few years. Ruby 1.8 saw strings as collections of bytes, but it really didn’t think or care much about Unicode and other encodings. Ruby 1.9 (as well as 2.0) made a shift in a similar direction to Python, such that every string has an encoding associated with it. Unlike Python, you can read binary data into a Ruby 2.0 string, and the language will be fine with that:

s = File.read(‘Downloads/test.jpg’)

If you ask Ruby what sort of object was returned, it’ll tell you that it was a string:

>> s.class #=> String

>> s.encoding #=> #<Encoding:UTF-8>

>> s.valid_encoding? #=> false

But, you then can set the encoding to something else:

>> s.force_encoding(Encoding.find(‘ASCII-8BIT’))

>> s.encoding #=> #<Encoding:ASCII-8BIT>

>> s.valid_encoding? #=> true

Web Development and Unicode

All of this is well and good, but how does it affect Web developers? Again, none of this would be a problem if you magically could flick a switch and have all documents and computers switch to using UTF-8. But, that’s far from the case. Not only are there many documents out there that were written in non-UTF-8 formats, but also there are many computers whose encoding is still not UTF-8.

This means if you have an HTML form and you accept input from users’ browsers, you likely will get input from users’ browsers in whatever encoding system their computers are using. True, most modern computers and browsers use UTF-8, but you would be amazed by how many old systems exist. You should experiment with your Web application, ensuring that even when someone sends you data in a non-Unicode system, you still can handle it (or gracefully deal with the failure).

Another issue I recently encountered myself wasn’t directly from user input, but rather files that users were uploading. My Web application worked in UTF-8, and everything seemed to be humming along—until it wasn’t. The problem was that part of the application involved people uploading text files. I would read the contents of the file into a string and then store that string in a database. Unfortunately, the application would raise an exception, because the text files—coming from people around the world, in different languages and using many different encodings—often were incompatible with UTF-8. One solution would have been to try to identify the encoding of the uploaded file. In my particular case, I was able to catch the exception and report it to the user, indicating that only files in UTF-8 were acceptable. Whether such an error message will suffice for your application depends on what you’re doing.

And yes, that leads me to my next point, namely databases. All of the major relational and NoSQL databases with which I work support UTF-8 as a default. PostgreSQL, for example, gives each database an encoding, indicating the encoding that will be used in text columns. The good news is that this ensures that all text stored in the database will be valid UTF-8, or whatever other encoding you use. The bad news (to some degree) is that if you want to store both binary and textual data in the same column, you’ll have to find another solution. Binary data, such as the contents of a JPEG file, cannot be stored in a text column, because it’s not legal UTF-8. Instead, you’ll need to store such information in a binary BYTEA column, which accepts any sequence of bytes and doesn’t attempt to ensure its validity. Fortunately, the drivers with which I work understand the difference between TEXT and BYTEA columns and return results using appropriate data types.

Realize that there is a difference, however, between encoding and collation. Encoding refers to the way UTF-8 (or any other character set) is translated into a series of bytes. Collation refers to how the text is sorted and, thus, is language-dependent. Consider that sorting a list of 100 words will have different results in English, Spanish and French, and you’ll understand that your application’s needs (and users) will determine, to a large degree, which collation, if any, you choose to use.

Conclusion

Just about ten years ago, I worked on a multilingual site that required Unicode, and my decision to use it caused a great deal of friction with others working on the project, because they didn’t have editors that supported UTF-8.

Things are quite different today. Just about every piece of Web-related software supports Unicode, from the operating system and language to the database and browser. However, the numerous non-Unicode computers, programs and files out there require that you keep them in mind and are able to work with them. Moreover, working with binary files and data means that you need to get out of the mindset that “everything can be a string”, because modern strings are picky about the data they will let you store.

Understanding Unicode is essential to knowing how modern Web applications work. Once you’ve made sure your application is using the right methods and checking the data in the right places, it’ll work just fine with users from around the world.■

Web developer, trainer and consultant Reuven M. Lerner is finishing his PhD in Learning Sciences at Northwestern University. He lives in Modi’in, Israel, with his wife and three children. You can read more about him at http://lerner.co.il, or contact him at reuven@lerner.co.il.

Send comments or feedback via http://www.linuxjournal.com/contact or to ljeditor@linuxjournal.com.

Resources

Character sets in general, and Unicode in particular, can take a long time to understand. One of the best introductions to the subject is in the first chapter of the O’Reilly book Java Internationalization, published in 2001 and written by Andy Deitsch and David Czarnecki. The book begins by describing many different writing systems, only afterward going into detail about what this means for Unicode.

For more information about Unicode support in Python, take a look at the “HOWTO” document for Python 2.7.4 at http://docs.python.org/2/howto/unicode or for Python 3.x at http://docs.python.org/3/howto/unicode. Unicode support in strings is one of the major changes in Python 3, so be sure to read about the version you’re using.

For information about Unicode support in Ruby 1.9.x (which is virtually identical to Ruby 2.0), I recommend the “Ruby 1.9 Walkthrough”, a long (but excellent!) screencast by Peter Cooper. He spends a lot of time demonstrating the differences between Ruby 1.8 and 1.9, with a great deal of detail about encoding and strings. More information is at https://cooperpress.com/19walkthrough.

The GNU recode program, which allows you to move documents among character sets and encodings, is at http://directory.fsf.org/wiki/Recode. Recode is an essential part of my toolkit when I work on Unicode-related sites.

COLUMNS

Work the Shell

DAVE TAYLOR

Cribbage: Pairs and Three of a Kinds

Debugging his last article’s script and calculating straights in a Cribbage hand keep Dave busy coding this month, with punctuation graffiti included!

The Cribbage game programming continues with further expansion of the subhand evaluation code. You’ll recall that in a two-player game of Cribbage, you’re dealt six cards but have to put two into the “crib”, a third hand that alternates between players.

The challenge is this: which four cards of the six leave you with the most points possible?

There’s a secondary consideration, because you also want to avoid putting points in the crib when it’s not yours, if you can help it, but for now, I’m going to stick with the six-choose-four challenge.

And a challenge it is, because cards are worth points based on whether they have the same rank (for example, 9S and 9C = 2 points for a pair); whether they add up to 15, with all face cards = 10 (for example, 7S and 8C = 15 = 2 points); whether all four cards have the same suit (for example, 3D, 7D, 9D, QD = 4 points); and whether three or four of the cards are in sequential rank order (for example, 9D, 10C and JS = 3 points), even if they aren’t the same suit.

I wrapped up my last article with code that could figure out the six-choose-four combinations (it’s a straight combinatorics problem—I knew that stuff I learned in college eventually would come in handy), then evaluate each four-card set for possible fifteens and pairs. With some debugging code added, the current output looks like this:

As you can see, the third subhand is worth more than the first two. In fact, 2C + 4S + 9C and 6S + 9C are both fifteens, so it’s worth four points. Not too bad.

Further down in the debugging output, subhands start to appear with the pair of eights:

So at this point the code recognizes pairs, but the point accumulator isn’t actually scoring them. That’s not good.

Let’s start by fixing that. The scoring code is getting pretty long, so I’ll just share the two-card code, which is a bit simpler too:

Here’s the line that fixed the scoring problem for pairs:

It’s easy shell math, and something I hope you’re using with some frequency. In fact, for subshells and for math equations that alternatively could be handled by are good.

That single line fixes the problem, as demonstrated in the very first test run:

How did I get six points? 3H + 3D is a pair (2 points), then 3H + 4C + 8H = 15 (2 points) and 3D + 4C + 8H = 15 (2 points). That’s a pretty decent little four-card Cribbage hand, actually.

What about when there are three cards that are the same? It turns out that Cribbage has a very logical scoring system, and three of a kind are scored as 3 * 2-card pairs, which makes sense. Here’s an example to illustrate:

So 4C + 4D, 4C + 4H and 4D + 4H are the three pair and are worth six points. This subhand is really superb, however, because there also are a number of card combinations that add up to fifteen, totaling 12 points. Very good for four cards!

The piece that’s missing with the scoring is straights. This is going to get a bit complicated, so stick with me.

Calculating Straight Runs

There’s already code in place that generates all three-card combinations that catches when three cards sum up to fifteen points, so that’s easily tapped within a “for” loop to extract the three-card index values:

That’s going to be set to “0 1 2”, “0 1 3” and so on.

The card’s normalized rank (for example, J=10, Q=10) is set in the point calculation function as the local array , and the original rank (J=11, Q=12 and so on) is in . These originally were and , but I renamed them to make their purpose a bit clearer in the script.

With “combo” set to the card indices, the full rank of a specific card in the four-card subhand can be referenced like this:

As Douglas Adams would say, don’t panic. Let’s unwrap it instead.

The reference is a string slice and extracts a one-character-long substring starting at index 0. The second value in the combo array is :2:1 and the third is :4:1. That’s used directly, so it’s akin to .

Put the three together and output the three ranks:

Testing the values is easy because the hand’s already sorted by lowest-to-highest rank. There’s more notational complexity because I’m going to use the mathematical shortcut again, but here’s the conditional test to see if the three-card subset is in sequential rank order:

I warned you, it was notationally complex and once the mathematics are added, the for algebraic equals and for the logical “AND” between two statements, well, it’s pretty thick with punctuation, to say the least.

The result (a subset to show it’s working):

This calculation is correct, that both cards 1,2,3 and cards 1,2,4 are runs, so it’s worth twice 3-points. But, there’s another bug looming: the situation where all four cards are a four-card run. That’s worth four points, not six. But we’ll have to figure out that bug fix next month—I’ve already gone way long on this column.■

Dave Taylor has been hacking shell scripts for more than 30 years. Really. He’s the author of the popular Wicked Cool Shell Scripts and can be found on Twitter as @DaveTaylor and more generally at http://www.DaveTaylorOnline.com.

Send comments or feedback via http://www.linuxjournal.com/contact or to ljeditor@linuxjournal.com.

COLUMNS

Hack and /

KYLE RANKIN

Two Pi R 2: Web Servers

Who knew that Raspberry Pis would make such a great redundant Web cluster?

In my last column, I talked about how even though an individual Raspberry Pi is not that redundant, two Pis are. I described how to set up two Raspberry Pis as a fault-tolerant file server using the GlusterFS clustered filesystem. Well, now that we have redundant, fault-tolerant storage shared across two Raspberry Pis, we can use that as a foundation to build other fault-tolerant services. In this article, I describe how to set up a simple Web server cluster on top of the Raspberry Pi foundation we already have.

Just in case you didn’t catch the first column, I’ll go over the setup from last month. I have two Raspberry Pis: Pi1 and Pi2. Pi1 has an IP address of 192.168.0.121, and Pi2 has 192.168.0.122. I’ve set them up as a GlusterFS cluster, and they are sharing a volume named gv0 between them. I also mounted this shared volume on both machines at /mnt/gluster1, so they each could access the shared storage at the same time. Finally, I performed some failure testing. I mounted this shared storage on a third machine and launched a simple script that wrote the date to a file on the shared storage. Then, I experimented with taking down each Raspberry Pi individually to confirm the storage stayed up.

Now that I have the storage up and tested, I’d like to set up these Raspberry Pis as a fault-tolerant Web cluster. Granted, Raspberry Pis don’t have a speedy processor or a lot of RAM, but they still have more than enough resources to act as a Web server for static files. Although the example I’m going to give is very simplistic, that’s intentional—the idea is that once you have validated that a simple static site can be hosted on redundant Raspberry Pis, you can expand that with some more sophisticated content yourself.

Install Nginx

Although I like Apache just fine, for a limited-resource Web server serving static files, something like nginx has the right blend of features, speed and low resource consumption that make it ideal for this site. Nginx is available in the default Raspbian package repository, so I log in to the first Raspberry Pi in the cluster and run:

$ sudo apt-get update

$ sudo apt-get install nginx

Once nginx installed, I created a new basic nginx configuration at /mnt/gluster1/cluster that contains the following config:

server {

root /mnt/gluster1/www;

index index.html index.htm;

server_name twopir twopir.example.com;

location / {

try_files $uri $uri/ /index.html;

}

}

Note: I decided to name the service twopir, but you would change this to whatever hostname you want to use for the site. Also notice that I set the document root to /mnt/gluster1/www. This way, I can put all of my static files onto shared storage so they are available from either host.

Now that I have an nginx config, I need to move the default nginx config out of the way and set up this config to be the default. Under Debian, nginx organizes its files a lot like Apache with sites-available and sites-enabled directories. Virtual host configs are stored in sites-available, and sites-enabled contains symlinks to those configs that you want to enable. Here are the steps I performed on the first Raspberry Pi:

$ cd /etc/nginx/sites-available

$ sudo ln -s /mnt/gluster1/cluster .

$ cd /etc/nginx/sites-enabled

$ sudo rm default

$ sudo ln -s /etc/nginx/sites-available/cluster .

Now I have a configuration in place but no document root to serve. The next step is to create a /mnt/gluster1/www directory and copy over the default nginx index.html file to it. Of course, you probably would want to create your own custom index.html file here instead, but copying a file is a good start:

$ sudo mkdir /mnt/gluster1/www

$ cp /usr/share/nginx/www/index.html /mnt/gluster1/www

With the document root in place, I can restart the nginx service:

$ sudo /etc/init.d/nginx restart

Now I can go to my DNS server and make sure I have an A record for twopir that points to my first Raspberry Pi at 192.168.0.121. In your case, of course, you would update your DNS server with your hostname and IP. Now I would open up http://twopir/ in a browser and confirm that I see the default nginx page. If I look at the /var/log/nginx/access.log file, I should see evidence that I hit the page.

Once I’ve validated that the Web server works on the first Raspberry Pi, it’s time to duplicate some of the work on the second Raspberry Pi. Because I’m storing configurations on the shared GlusterFS storage, really all I need to do is install nginx, create the proper symlinks to enable my custom nginx config and restart nginx:

$ sudo apt-get update

$ sudo apt-get install nginx

$ cd /etc/nginx/sites-available

$ sudo ln -s /mnt/gluster1/cluster .

$ cd /etc/nginx/sites-enabled

$ sudo rm default

$ sudo ln -s /etc/nginx/sites-available/cluster .

$ sudo /etc/init.d/nginx restart

Two DNS A Records

So, now I have two Web hosts that can host the same content, but the next step in this process is an important part of what makes this setup redundant. Although you definitely could set up a service like heartbeat with some sort of floating IP address that changed from one Raspberry Pi to the next depending on what was up, an even better approach is to use two DNS A records for the same hostname that point to each of the Raspberry Pi IPs. Some people refer to this as DNS load balancing, because by default, DNS lookups for a hostname that has multiple A records will return the results in random order each time you make the request:

$ dig twopir.example.com A +short

192.168.0.121

192.168.0.122

$ dig twopir.example.com A +short

192.168.0.122

192.168.0.121

Because the results are returned in random order, clients should get sent evenly between the different hosts, and in effect, multiple A records do result in a form of load balancing. What interests me about a host having multiple A records though isn’t as much the load balancing as how a Web browser handles failure. When a browser gets two A records for a Web host, and the first host is unavailable, the browser almost immediately will fail over to the next A record in the list. This failover is fast enough that in many cases it’s imperceptible to the user and definitely is much faster than the kind of failover you might see in a traditional heartbeat cluster.

So, go to the same DNS server you used to add the first A record and add a second record that references the same hostname but a different IP address—the IP address of the second host in the cluster. Once you save your changes, perform a dig query like I performed above and you should get two IP addresses back.

Once you have two A records set up, the cluster is basically ready for use and is fault-tolerant. Open two terminals and log in to each Raspberry Pi, and run tail -f /var/log/nginx/access.log so you can watch the Web server access then load your page in a Web browser. You should see activity on the access logs on one of the servers but not the other. Now refresh a few times, and you’ll notice that your browser should be sticking to a single Web server. After you feel satisfied that your requests are going to that server successfully, reboot it while refreshing the Web page multiple times. If you see a blip at all, it should be a short one, because the moment the Web server drops, you should be redirected to the second Raspberry Pi and be able to see the same index page. You also should see activity in the access logs. Once the first Raspberry Pi comes back from the reboot, you probably will not even be able to notice from the perspective of the Web browser.

Experiment with rebooting one Raspberry Pi at a time, and you should see that as long as you have one server available, the site stays up. Although this is a simplistic example, all you have to do now is copy over any other static Web content you want to serve into /mnt/gluster1/www, and enjoy your new low-cost fault-tolerant Web cluster.■

Kyle Rankin is a Sr. Systems Administrator in the San Francisco Bay Area and the author of a number of books, including The Official Ubuntu Server Book, Knoppix Hacks and Ubuntu Hacks. He is currently the president of the North Bay Linux Users’ Group.

Send comments or feedback via http://www.linuxjournal.com/contact or to ljeditor@linuxjournal.com.

COLUMNS

The Open-Source Classroom

SHAWN POWERS

Prospecting for Ones and Zeros

Create money out of thin air with cryptocurrency.

At the current market rates (April 24, 2013), a user with a single AMD Radeon 7950HD video card can make $6 a day mining cryptocurrency. Granted, it’ll cost around $1 in electricity, but that’s still a $5-per-day profit. Of course, tomorrow the market could tank, and it’ll be worthless, but at this precise moment, it’s profitable. I’ve written about cryptocurrency before in Linux Journal, but as you can imagine, I’ve gotten lots of questions about mining since the price skyrocketed. In this article, I want to talk specifically about mining. If you’re unsure about cryptocurrency in general, look back at my past article: http://www.linuxjournal.com/content/cryptocurrency-your-total-cost-01001010010. If you just want to get down and nerdy, keep reading.

CPUs or GPUs?

Cryptocurrencies come in several varieties, but the most popular, and most valuable, is the Bitcoin. Since its early days, it’s been silly to mine Bitcoins with a CPU. The raw horsepower of a GPU blows the doors off any CPU, regardless of the CPU’s speed. Because of that wasted opportunity, Litecoin was developed specifically to make mining difficult for GPUs.

For several months, it proved to be a great way to take advantage of idle CPUs. It also opened up mining to the regular desktop user, because a standard workstation CPU would be able to contribute and create coinage. Brilliant programmers being what they are, however, GPUs now are able to mine Litecoins more efficiently than CPUs. It’s still profitable to mine Litecoins with high-end CPUs, but the profits are measured in pennies instead of dollars.

Although Bitcoins still are the most popular cryptocurrency, an interesting mining fact is that it’s more profitable to mine Litecoins and then trade them for Bitcoins. Perhaps that’s my own little trade secret, but it’s held true for several months, and it looks to continue. Once you figure out the hash rate you can achieve with Bitcoins versus your Litecoin hash rate, it’s as simple as plugging the numbers in to the calculator at http://allchains.info/calc to see which is more profitable.

Where to Begin?

If you want to get into mining, it certainly sounds like a good idea to start with CPU mining on your existing desktop computer. Although that’s definitely possible, my fear is that it will cause more frustration than fun. The profits, if any, will be in the pennies. If the market fluctuates (which it always does), you could go from making three cents a day to losing a nickel a day overnight. Plus, your system will be running at 100% CPU usage, making the rest of your computing frustrating. Still, if you just want to dabble, it’s a way to start. If you want to mine with your desktop machine, you’ll need to use a mining pool; otherwise, it will take years before you see any profit.

To start with serious mining, it means purchasing video cards. For Bitcoins, there also are custom FPGA and ASIC boards that can do mining, but most people still use GPUs, so I focus on those here. Also, in this article, I focus more on Litecoins because they’re slightly more profitable to mine, but also because the mining rigs are harder to configure. If you can mine Litecoins, Bitcoins are a breeze, so let’s focus on Litecoins.

Currently, the best price/hashrate/electrical-usage video card is the AMD 7950HD. For about $300, the 7950 produces around 600kH/s on the Litecoin network. There is a great wiki that compares the various GPU chipsets and their wattage versus hashrate. You owe it to yourself to study the wiki to see what will work best for you: https://github.com/litecoin-project/litecoin/wiki/Mining-hardware-comparison.

(There is a similar chart for Bitcoin mining. It’s worth reading that in case the Litecoin market crashes and burns, and being able to switch to mining Bitcoins efficiently is a bonus: https://en.bitcoin.it/wiki/Mining_hardware_comparison.)

In addition to the hash rate and electrical usage, the allchains.info site mentioned previously is a great tool for figuring out how much money a particular set of GPUs will create. Keep in mind that the market is highly volatile, but it’s a nice way to see a snapshot daily rate.

And of course, to use a GPU, you need a motherboard that will hold it. Thankfully, cryptocurrencies don’t require anything more than PCI-E 1x for bandwidth, so as long as you can get the cards into the slots (or use extenders), you’re set. Add a cheap CPU, a gig or two of RAM and a power supply big enough to support your rig, and you’re golden! I know, it’s a lot to think about and a lot to consider, but if this sort of planning is fun, mining cryptocurrency is perfect for you!

My Head Hurts

Did the last section dash your hopes of ever becoming a miner? Too complicated? Too many variables? I understand. Here’s a sample mining rig, part by part:

Motherboard: ASRock 970 EXTREME4—this motherboard is about $99 and has the convenient spacing that allows (three) GPUs to be mounted at the same time (Figure 1). They’ll be crammed together, but I’ll talk about cooling a little later.

Figure 1. The PCI-E slots are far enough apart that all three are usable.

RAM: Crucial 2GB Kit (2x1GB) 240 pin DDR3 1333—for $30 or so, this is more than enough RAM to mine with. When I tried to find a 1GB kit, it cost more, so I got this kit because it was the cheapest.

CPU: AMD Sempron 145 Sargas 2.8GHz—this single-core CPU is less than $40, fits in the AM3+ slot on the motherboard listed above and is horrible for mining anything. You could buy an expensive CPU and get some mining hashes out of it, but the time it would take to recoup the expense is crazy. I recommend slow and cheap.

Hard Drive: anything—if you’re considering building a mining rig, chances are you have a pile of old hard drives sitting in a closet or on a shelf. Just use one of those. Otherwise, any old SATA hard drive will work.

Power Supply: 850Watt 80 PLUS certified or better—don’t go less than 850 watts, even if you don’t start with three full cards. Give yourself room to grow. Also, the more efficient power supply you buy, the more profitable your rig will be in the long run. The price you pay will vary anywhere from $90 to $175 or so. Make sure you have enough PCI-E power connectors for the cards you’re going to be installing. (Note: you can get adapters to convert SATA power connectors to PCI-E connectors, but I wouldn’t recommend using too many of those in a single rig.)

GPUs: AMD Radeon 7950 HD—these seem to be available from a fairly wide variety of on-line sources. The prices range from $299 to around $329, and each card will do around 600kH/s on the Litecoin network. Other cards work, some are faster, and if you can get a deal on something else, just compare on the charts mentioned previously. If you just want a parts list, however, go with the 7950. It’s a great card.

Case: don’t use a case. GPUs produce a ton of heat while they’re mining, so it’s best to have them out in the open air. Plus, you can get a $10 box fan and push more cooling air over an exposed rig than any number of case fans could do.

The Build

If you’ve researched for days and ordered what you found to be the absolute best possible combination of hardware, or if you just used my list, the build is similar. When not using a case, find a fairly secure location where you can set up computer equipment and not worry about anyone bumping it. (I have my rigs set up in the basement; see Figure 2.)

Figure 2. It’s not pretty, but it does the job!

I should add a disclaimer that you should wear a static wrist strap, make sure everything is grounded, and securely mount the motherboards and power supplies onto whatever surface you’re using. Those are all very important things. As you can see in my photo, however, my motherboards are sitting on empty boxes, and I turn my rigs on by shorting the power connector on the motherboard with a screwdriver. My setup is a bit redneck, and I can’t recommend that you do the same, but I won’t judge you if you do.

These rigs will be headless, as you can tell by the photo, but for the operating system install, you’ll need a monitor. I recommend installing Xubuntu. I won’t go through the details on how to install Xubuntu, but you can use a USB stick to install it or a temporary CD drive—whatever floats your boat. Be sure to have the system log in automatically. This is a requirement for later on when you won’t have a monitor connected! Once Xubuntu is installed, you need to go to http://amd.com and get the latest Catalyst driver for Linux. (Make sure to get the version that matches your installed system in regards to 32- or 64-bit.) Please note that although there are proprietary drivers in the Xubuntu distro, those drivers do not have the things required for mining. You’ll need to install the driver from amd.com in order to mine properly.

After you run the installer, you’ll be forced to reboot. Once rebooted, you need to make sure the video card(s) are all initialized. This is a squirrelly procedure, which doesn’t work consistently. Simply open a terminal and type:

sudo aticonfig --adapter=all --initial

The squirrelly part is that sometimes it doesn’t detect all the cards. Look at the output, and make sure it listed the number of cards you have installed. If not, run the same command over, and it likely will detect them. (I know, it’s weird.) Once that’s done, you’ll need to reboot again. At that point, configuring the GPU drivers should be done.

The other tools you’ll need installed (these from the product repositories) are screen and openssh-server. Those come into play when you run the miners in headless mode. Make sure you have a static IP address set on your mining rig, then reboot it one final time, disconnecting the keyboard, mouse and monitor. If everything goes right, you should have a headless miner set to log in automatically, with an SSH server running so you can connect remotely. Leave the dark basement and go back to your comfy workstation. You should be able to do the rest from there.

A Bit about Pools

Before you actually set up the mining programs, it’s important to decide where you’re going to mine. Back in the day before cryptocurrencies were popular, a user would install the litecoin client and mine locally. The problem with that method now is that it would take weeks or months to find a “block” mining on your own. Unless you have many thousands of dollars worth of mining rigs, you’ll want to mine at a pool. Basically, users at a pool will combine their mining horsepower and then split the coins they mine proportionally to the hashing power they contribute. This means instead of mining for months and possibly finding a block of 50 coins, the pool miners get a regular payout daily based on their contribution to the network. With my nine GPUs mining 24/7, I can earn only ten Litecoins a day. If I didn’t mine with a pool, it would become frustrating quickly.

Several mining pools are available for Litecoins and Bitcoins. Regardless of what pool you choose, I recommend withdrawing your earnings on a regular basis, because pool owners are anonymous, and I’m not terribly trusting. (Perhaps Linux Journal should start a pool—hmmm....) Here’s a partial list of pools: https://github.com/litecoin-project/litecoin/wiki/Comparison-of-mining-pools.

One Miner to Rule Them All

I know I’m focusing on GPU mining here, but since I’m talking about miner programs, it seems a good time at least to mention how to CPU mine. If you’re mining Litecoins with a CPU, you want to use pooler’s miner (https://github.com/pooler/cpuminer). For mining with a GPU, whether you want to mine Bitcoins or Litecoins, cgminer is the tool of choice (https://github.com/ckolivas/cgminer). I use cgminer here and configure it to mine Litecoins. Be sure to get the latest binary, because Litecoin support is a fairly recent addition to cgminer.

From the comfort of your workstation, ssh in to your miner. Assuming everything is working correctly, you should be able to log right in. Now that you’re logged in, type screen to get a screen session going, so once you start mining, you can disconnect and it will keep mining. Because mining requires OpenCL, it means that you have to be using the X Window System display. Because you’re SSH’d in, that would be a problem, except that Linux is so very flexible. In the screen session, type:

export DISPLAY=:0

That should give you the ability to use OpenCL. While you’re at it, export a couple other variables that fix a few oddities with how GPU memory is handled:

export GPU_USE_SYNC_OBJECTS=1

export GPU_MAX_ALLOC_PERCENT=100

There should be nothing returned from the system; you’re just setting some environment variables so that cgminer works properly. If you haven’t downloaded and extracted the cgminer program, do so now. Then, cd to that directory:

cd cgminer_folder_name

Now, see if cgminer sees all your GPUs:

./cgminer -n

You should see something like this:

[2013-04-24] CL Platform 0 vendor: Advanced Micro Devices, Inc.

[2013-04-24] CL Platform 0 name: AMD Accelerated Parallel Processing

[2013-04-24] CL Platform 0 version: OpenCL 1.2 AMD-APP (1084.4)

[2013-04-24] Platform 0 devices: 3

[2013-04-24]  0       Cypress

[2013-04-24]  1       Cypress

[2013-04-24]  2       Cypress

[2013-04-24]  3 GPU devices max detected

Assuming all your cards are showing, you’re ready to mine! If not, you might have to fiddle with the aticonfig tool again. Remember, it’s a bit squirrelly. Because you’re inside a screen session, you can start mining and then disconnect the screen session without stopping cgminer. For a quick start, type something like this (all on one line):

./cgminer --scrypt -o stratum+tcp://your.pool:3333

↳-u pool_user -p pool_pass -I 13

You’ll need to fill in your information for pool IP, port, user and password. If everything works correctly, after a few moments, you should see cgminer doing its thing with all your GPUs. (Figure 3 shows one of my rigs.)

Figure 3. These are actually three 5850 cards, not 7950s

Step 73—Profit?

To be honest, all this work got you only to the beginning of configuring your mining rigs. If there’s enough interest, perhaps I’ll do another column on tweaking and overclocking for maximum hash rates and maximum profit. In the meantime, reading forums like https://bitcointalk.org will yield some valuable information. (It also will yield trolls, scammers and so on—the forum is like the dirty underbelly of the Bitcoin mining community.) The next steps I recommend are:

■ Read the cgminer options and configuration options in the README. It’s an amazing piece of software.

■ Tweak your memory speeds, clock speeds, thread concurrency and intensities to get better hash rates.

■ Set up scripts to start your miners, and create config files for cgminer instead of putting everything on a really long command-line argument. Use your script to export those environment variables so you don’t forget to do so.

■ Install the litecoin (or bitcoin) client on a computer, so you can withdraw your coins to your own local wallet.

■ Check out the Resources section for this article. Lots of information is available; it’s a data-geek’s dream.

If you’re a cryptocurrency miner or are about to become one, please send photos of your mining rigs to ljeditor@linuxjournal.com. You can tell by my setup that we don’t care about professionalism, it’s just really cool to see other people’s setups. If you’d like to hear more about Bitcoins or Litecoins, let us know that too!■

Shawn Powers is the Associate Editor for Linux Journal. He’s also the Gadget Guy for LinuxJournal.com, and he has an interesting collection of vintage Garfield coffee mugs. Don’t let his silly hairdo fool you, he’s a pretty ordinary guy and can be reached via e-mail at shawn@linuxjournal.com. Or, swing by the #linuxjournal IRC channel on Freenode.net.

Send comments or feedback via http://www.linuxjournal.com/contact or to ljeditor@linuxjournal.com.

Resources

Bitcoin Forum: https://bitcointalk.org

Chain Data: http://allchains.info

Litecoin Project on GitHub: https://github.com/litecoin-project

Litecoin: http://litecoin.org

Bitcoin: http://bitcoin.org

NEW PRODUCTS

MSC Embedded Inc.’s Qseven Starter Kit

The new Qseven Starter Kit with AMD embedded G-Series APU, says its producer MSC Embedded Inc., frequently will be used to drive an LCD output. The MSC Q7-SK-A50M-EP4 Starter Kit consists of MSC’s 3.5” Qseven baseboard Q7-MB-EP4 with heatspreader and heatsink and an integrated power supply with cable kit. The kit comes with a ready-to-run Linux installation in Flash Disk to enable an out-of-the-box functional experience. An optional TFT kit is available, which provides for a 12.1” LCD panel with XGA resolution (1024 x 768), the appropriate cable kit for operation off the Qseven baseboard and full implementation in the Qseven module’s Graphics BIOS. Users can select the Qseven module with the most suitable processor and clock speed from the MSC range of Q7-A50M modules: AMD G-T40E Dual-Core APU or AMD G-T40R Single-Core APU with optional 7.2GB Flash Disk or AMD G-T16R Single-Core APU.

http://www.mscembedded.com

Peter Gasston’s Modern Web (No Starch Press)

Today’s Web technologies are evolving at near-lightspeed. When users can browse the Web on a 3” phone screen as easily as on a 50” HDTV, what’s a developer to do? Well, flummoxed friends, there is a new book designed to answer just that question, namely Peter Gasston’s Modern Web: Multi-Device Web Development with HTML5, CSS3, and JavaScript. Gasston’s book will guide readers through the latest and most important tools for device-agnostic Web development, including HTML5, CSS3 and JavaScript. His plain-English explanations and practical examples emphasize the techniques, principles and practices one needs to transcend individual browser quirks easily and stay relevant as these technologies are updated. After reading the book, claims publisher No Starch Press, one will be equipped to design content that displays fluidly across multiple devices as well as turn outdated Web sites into flexible, user-friendly ones that take full advantage of the unique capabilities of any device or browser.

http://www.nostarch.com

ROSA Media Player

Handiwork of the Russian firm ROSA, the new ROSA Media Player (ROMP) 1.6 is reportedly the first free media solution with full YouTube support. This support enables users to search, view and save YouTube video clips to their computers. Other advances in the 1.6 release include desktop video and audio capture, DVD menu support and improved IPTV support, as well as the ability to load a list of channels from a remote server. ROMP, available in Linux and Windows versions, is distributed under the GPL 3+ license and available for free download and via various distribution repositories.

http://www.rosalab.ru

Exablox’s OneBlox

After two years in stealth development mode, Exablox has burst out of the lab to debut its OneBlox flagship solution, an affordable storage appliance that combines a paradigm-shifting architecture design with integrated, enterprise-grade software features. The two-year development phase, complete with extensive customer and partner research, chipped away at businesses’ most common storage pain points, including runaway costs, complicated installation, cumbersome data management and a lack of data security. The 100% cloud-based OneBlox’s key product highlights include initial installs in less than five minutes and subsequent configurations in less than three, HTML5 storage management and control that graphically depict storage use and performance, automatically enabled deduplication and compression, dynamic and automatic capacity and performance expansion with new drives or nodes, instant data recovery from any desktop and an encrypted local copy of all data.

http://www.exablox.com

xTuple ERP

Distribution companies have three problems: items, items and items. Managing all the inventory in distribution-heavy industries, such as electrical, HVAC/R, PVF, office products and automotive, can be a management nightmare. These sectors are potential target customers for the updated xTuple ERP, which includes the new xWD extension for sophisticated, enterprise-class inventory control. One feature of xTuple’s xWD is the external catalog that allows distribution companies to merchandise non-stock products readily for immediate sales fulfillment and generation of a corresponding purchase order. The xWD module complements the open-source xTuple’s existing functionality in sales, accounting and operations—including customer and supplier management, inventory control, manufacturing and distribution. xTuple gives customers the ability to tailor solutions with multiplatform support for Linux, Windows, Mac and mobile.

http://xTuple.org

Erica Sadun and Steve Sande’s Pitch Perfect: The Art of Promoting Your App on the Web (Addison-Wesley)

You’re about to pour your heart, soul, skill...and savings, into creating the next big app. How will anyone know how awesome it is? Authors Erica Sadun and Steve Sande are veteran tech bloggers who have seen countless marketing successes and failures in the app world firsthand. Their new Addison-Wesley book Pitch Perfect: The Art of Promoting Your App on the Web is a guide to success with your new app. The book provides an “in” with popular blogs to get those make-or-break product reviews and features extensive tutorials on how to develop short-and-sweet pitches that capture attention. It also offers tips on how to avoid Inbox deletion, how to keep tech bloggers in the loop during the development process and how to maintain good relations with the public and the press.

http://www.informit.com

6WIND’s 6WINDGate

Readers interested in virtualization and enterprise networking solutions will want to read on about improvements to the 6WINDGate networking software from 6WIND. 6WINDGate solves critical performance and scalability challenges for virtual switches while retaining full compatibility with standard virtual switch software, such as Open vSwitch. The company claims that 6WINDGate provides an optimized data plane solution that delivers more than a 10x acceleration for the baseline switching functions and delivers high performance for secure tunneling protocols, such as IPsec, GRE, NVGRE, VLAN and VxLAN. This enables service providers, such as cloud and telecom data centers, to achieve significant CAPEX and OPEX 1 improvements that are not possible without data plane acceleration. The virtual switch acceleration delivered by 6WINDGate is transparent to the applications running on the platform, which do not need to be recompiled or re-verified in order to work with this high-performance solution.

http://www.6wind.com

Wind River Linux Carrier-Grade Profile

Although carrier-grade functionality isn’t new for Wind River, the new Wind River Linux Carrier-Grade Profile for the latest version of Wind River Linux delivers these capabilities on top of a Yocto Project-compatible product. Formally registered for the CGL 5.0 specification with the Linux Foundation, the profile is the first delivery of carrier-grade Linux functionalities on top of a Yocto compatibility, says the company. The Yocto Project is a multivendor, open-source project that provides templates, tools and methods for creating custom Linux-based systems for embedded products, regardless of the hardware architecture. The net result of this combination of features, says Wind River, is a decrease in complexity and costs, an increase in the portability of Linux implementations and improved cross-platform compatibility and component interoperability. Carrier-grade products typically require up to 5 nines or 6 nines (99.999% to 99.9999%) availability, translating to downtime as low as 30 seconds a year.

http://www.windriver.com

Please send information about releases of Linux-related products to

newproducts@linuxjournal.com

or New Products c/o Linux Journal, PO Box 980985, Houston, TX 77098.

Submissions are edited for length and content.

FEATURES

AIDE—Developing for Android on Android

No matter where you find yourself, there you are. And with AIDE, now you can develop code there too.

JOEY BERNARD

Android, as a platform, is one of the fastest growing on the planet. It is available on smartphones and a series of different tablet sizes. Most devices also include a full spectrum of sensors that are available to programs you install, so it’s a very inviting platform for development. The usual workflow involves installing a development environment on some other machine, either a Windows or Linux desktop or laptop. You then do all of your code writing, compiling and debugging there before you actually copy it and install it onto your Android device.

But, there may be times when you want to develop on the road or shorten the cycle by developing on your Android device itself. One of the better options for this is AIDE, the Android Java IDE. AIDE is distributed under a freemium model. The free version allows you to develop, compile and run your code. It also allows you to install to the device on which AIDE is running. However, if you want to generate APK files that can be used to install onto other devices, you need to purchase the full version.

In this article, I start by explaining how to install AIDE and create a new program, and then I cover what is involved in coding, debugging and running your new program. For more information, see the Google+ page (https://plus.google.com/101304250883271700981/about).

Figure 1. AIDE is available as a free download from the Google Play store.

The first step is to install AIDE on your Android device. Open up the Play Store and do a search for “AIDE”. The appropriate package should show up at the top of the list. If you are in doubt, verify that the developer is “appfour GmbH”. AIDE takes up more than 12MB, so if you are running short on space, you can transfer the majority of it to an SD card, leaving 4.45MB in your device’s main storage.

The very first time you start AIDE, it will pop up a dialog box where you can enter the details for beginning a project. In this dialog, you can enter an App Name and a Package Name. You also can select an app template from some built-in ones, such as “Hello World”, “Tetris” and “Analog Clock Widget”. These templates will set up the folders and files for your new project.

Figure 2. When AIDE starts up the first time, you are shown a dialog for your first project.

Once you click create, the files will be created in the folder /mnt/sdcard/appname (where appname is the name you gave your project), and this new project will be opened up in the IDE. The main part of the IDE consists of two panes. Their alignment depends on the size of the device on which you’re running it. On my phone, the panes are one above the other, and on my tablet, the panes are side by side. The first pane is a file listing for your project, containing all the properties, resources and source files needed for an Android project. The second pane is the main editor, where the central file (MainActivity.java) gets loaded on project creation.

Figure 3. When you open a project, it gets pulled up into the IDE.

Figure 4. The tablet interface opens with panes side by side.

If you start with one of the templates, you can compile it and run it right away to see how the process works. Click on the menu button, and select the Run option. This will pop up a dialog, informing you of each step being done. It will compile your code, link it and create an APK file. In order to run it, this APK file needs to be installed. So an installation dialog will appear asking if you want to install it. Once installed, it then will start up, and you will have your very first Android application, developed completely on Android.

Figure 5. Security requires that you approve any app being installed.

Figure 6. After compiling and installing, AIDE will run your new program.

All of the apps that you develop in AIDE are structured as projects. Anyone used to using IDEs, like Eclipse, should be familiar with that. This means if you want to work on a different app, you need to close the current project and open a new one. To close a project, click the menu button, select More and then Close Project. This still will leave you in the main directory of your current project.

To see your other projects, click on the “..” entry in the file pane to move up one directory. Here, you should see three standard entries, then a folder for each of your projects. The first entry is a tool to let you clone a Git repository to your Android device. The dialog that pops up allows you to enter a repository URL and a directory name. If you already have done some development work and have the code on Dropbox, you can download the relevant folder directly within AIDE.

Figure 7. You can create new projects several different ways.

Figure 8. Cloning a Git repository is a fairly easy task.

The third option is to create a new project. Selecting this option brings up the dialog you saw when you started AIDE the very first time. To load a project into the IDE, it isn’t enough simply to select the project folder. Within the folder is an option to “Open this App Project”. This loads all the meta information about your project, like its properties and resource file locations.

Now that you know a little about how projects are handled, let’s start creating your very first app. If you have a project open right now, go ahead and close it. Change directories in the file pane until you are in the main directory for all of your projects. From here, you can select “Create new App Project here”, and name your project “MyFirstApp”. You can set the package name to whatever you like, as long as it follows the usual format. Also, be sure that you have selected the “Hello World” app template.

Once everything is filled in, go ahead and click the create button. This will create all of the required files and open the new project in the IDE. The file MainActivity.java extends the class “Activity”. This is the main class you will be dealing with when developing apps. Your app builds on this class and overrides the methods within it to make your app do its own thing.

The first method to override is “onCreate”. This method is called when your activity first is created. The usual tasks involve creating the graphical elements of your program and getting them initially rendered and displayed. The first step (super.onCreate) runs the code in the main class “Activity” to handle all of the OS-related tasks in creating a new activity. You then add your own code, usually GUI creation.

In the Hello World template, it includes a call to the “setContentView” method, which assigns a particular layout as the content view. The beginning value is the layout “R.layout.main”. Layouts are XML files, containing all the elements that make up your graphical interface. To get to this file, you can change to the directory res, then layout. Within this subdirectory, you will find XML files for each layout that is defined.

Figure 9. Layout files are stored in the subdirectory res/layout.

In this simple program, you just have a single file called main.xml. Selecting it in the file pane loads it into the editor pane. The outermost container in the XML file is the type of layout being defined. A number of options are available. In the Hello World template, the layout type being used is a LinearLayout. You can set a width and height for the entire layout with the properties android:layout_width and android:layout_height. In this example, let’s leave the defaults of “fill_parent” for both. You also can set the display’s orientation; in this case, it is vertical.

The individual elements for the graphical display are defined as internal XML containers within the outermost Layout container. In this example, there is a “TextView” object, where you can set various properties like the width, height or the text to be displayed. The proper way to include things like text strings, or anything else that may be reused, is to store them once as a resource and to refer to this string with a resource identifier. In this example, the identifier is “@string/hello”. The actual text string is stored in the file strings.xml, which is located in the directory res/values. The outermost container of this XML file is the tag “resources”. Within this, is the list of all of the resources available to your program. In this case, there are only two strings: one for the app name and one for the text content of the main window.

Figure 10. Resources for your program are stored in a series of XML files.

To this example, let’s add a button to change the text being displayed. If you need to do any amount of typing, you will want to download and install a more complete soft keyboard or use an external Bluetooth keyboard. A good soft keyboard is the hacker’s keyboard. It includes a lot of extra keys that are useful in code editing, including arrow keys to aid navigation and real control, Tab and Escape keys. Most soft keyboards included on Android devices lack these.

Open the main layout file, and below the TextView, add a new tag for a Button entry with the following code:

<Button

android:layout_width=”wrap_content”

android:layout_height=”wrap_content”

android:text=”@string/button_title” />

AIDE actually does a continuous code check to make sure there aren’t any errors in the code. This means that while you are typing the above additions, you likely will see errors until you finally finish.

The editor includes tab completion, so you can start typing “<Butt” and then press Tab, and the editor will fill out the rest of the word for you. This is because “Button” actually is a special word in Android development.

Once you add the above, you can click the menu button, select More, and then select Refresh Build. This will try to do a full rebuild of your app, and you will see an error about the missing string resource. If you have multiple errors, clicking on the error in the list will bring you to the location where the error appears. To fix this particular error, you need to add the following text to the file res/values/strings.xml:

<string name=”button_title”>My Button</string>

Figure 11. Any errors that crop up during a rebuild are listed here.

When you get to the end of this line and start typing “</”, the editor automatically will fill in the rest of the line for you. When you refresh the build, the error should go away, assuming that you haven’t introduced any typos.

Figure 12. You can add new string resources to the file strings.xml.

Buttons are expected to trigger some reaction, however. This means you likely will want to add some kind of callback to a function in your button. This is handled within the layout file where the button is defined. You can add an extra property, “onClick”, which gives a method name to be called when the button is clicked. For example, you might have the following in the button definition:

android:onClick=”my_method”

You then can add the function “my_method” to the file MainActivity.java. This new method needs to be public and return void. Also, the only input parameter is a View object. Because you want to change the text in the TextView object, you’ll need to add an ID so that you can refer to it. In the main.xml file, add the following property to the TextView entry:

android:id=”@+id/view_text”

Figure 13. Add callbacks for buttons in the main.xml file.

You then can use “view_text” to access the text display. In the method “my_method”, you can get a reference to this text field with the line:

TextView tview = (TextView)findViewById(R.id.view_text);

Figure 14. You need to add ID labels to interact with items in your program.

You then can reset the text to be “The button was clicked” with:

tview.setText(“The button was clicked”);

When you click Run, your program will be recompiled, re-installed and opened. Now you can see what happens when you click on the added button.

Figure 15. The actual callback code goes into MainActivity.java.

Figure 16. Before pressing the button.

Figure 17. After pressing the button.

Figure 18. There are lots of functions in the menu not covered here.

Figure 19. The premium version adds even more capabilities.

I’ve provided only a short introduction to AIDE and all of its super powers here. This article hasn’t looked at more than the most basic features of an Android application. Lots of tutorials exist to get you up to speed in Android development, and now you can run through those tutorials on the go.■

Joey Bernard has a background in both physics and computer science. This serves him well in his day job as a computational research consultant at the University of New Brunswick. He also teaches computational physics and parallel programming. He has been using Linux since the mid-1990s and believes it is the future.

Send comments or feedback via http://www.linuxjournal.com/contact or to ljeditor@linuxjournal.com.

FEATURES

Compatibility Test Suite—Test Your Android Implementation

Porting Android to new hardware? You must read this!

NITISH TIWARI

Imagine having a refrigerator in your kitchen that could give you the nutritional contents and shelf life of the food you keep inside, or an automatic room cleaner that could take your commands via a chat app or SMS and clean your house in your absence. A few years ago, such devices were only dreamed of, but with rapid advancement in mobile operating systems and the ecosystem around them, such devices will be a reality soon.

Android specifically has an edge here. Because it is an open-source OS, developers all over the world are free to contribute, improve or even adapt it specifically to their needs (read: hardware requirements), and with the community growing larger by the day, much innovation is happening. But, this freedom to change the OS and adapt it to different hardware has to be managed somehow. There must be a mechanism to ensure that all the devices running Android provide the same experience to end users. Although developers get a chance to port Android to their hardware, users should not need to know about the underlying changes.

As per Google, Android compatibility consists of three key components:

1. Android source code: the source code itself serves as a specification for the developers aspiring to port it to new hardware, and anyone who understands the code, will learn about the platform, APIs and their behavior.

2. CDD (compatibility definition document): CDD serves as the policy for Android compatibility. It is a document that aims mainly to clarify and remove ambiguity in the implementation. So if you want to port Android to new hardware, you should read the CDD and follow all the guidelines mentioned there.

3. CTS (compatibility test suite): this is the mechanism for Android portability. Because it is difficult—rather, impossible—to test hardware behavior via software, CTS helps with testing the APIs and the platform functionality. So, where CDD tells you what to do, CTS helps you check whether you have done it properly.

In this article, I discuss the CTS (Compatibility Test Suite), an application that allows you to validate your Android porting. CTS helps you check that the app developers get the same set of APIs (and that those APIs will behave properly) and that users get the same overall Android experience in you new device. The testing is as easy as running test cases and generating a report and logs. Looking at the report will tell you whether everything is okay with your implementation. If that’s not enough, you can send the report to Google to claim entry to Google play.

Overview

CTS, like Android, is an open-source tool, and the code for CTS is available with the Android source code. CTS also is available separately as an installable file on the Android developer Web site. So if you just want to run the tool without modifications, you can download it from the Web site. But, if you want to add or delete some test cases, or change a test case, you need to use the source code available with the Android source code, change it and then run the test cases. It is worth mentioning here that for every version of Android released, a corresponding CTS version is available. It’s best to use the same versions of Android and CTS for testing. I use the latest CTS version 4.2_r1 for this article.

CTS is composed of two major components: 1) the application that runs on your desktop and manages the test execution, and 2) the test cases that actually are executed on the mobile device or emulator. The test cases are written in Java as JUnit tests and packaged as .apk files, so that they can be run on the target Android device. Once the testing starts, the test application loads each of the .apk files (in the test scenario that is run) to the target, executes it, records the test result and, finally, removes it from the target device.

In this article, I provide a step-by-step guide to configure, run and generate the test reports. I also show how to add or remove test cases from the test harness. As a open-source enthusiast, I have an inclination toward the Linux OS, so I explain the CTS configuration on a Linux system first. But, Windows users need not worry, because I explain the configuration of Android on Windows also. Configuring it on a Mac machine shouldn’t be much different, and once you configure it, running CTS is the same for all three platforms.

Prerequisites

To run CTS, you must have the CTS build downloaded from the Android developer Web site. If you are planning to add or remove test cases, you need the CTS source code also, which is available with the Android source code. When you change the CTS source code, it has to be built again, and you will need a compiler for that, depending on your personal preference.

Along with the CTS, you also need to download the Android SDK available on the Android Web site. The SDK has the adb (android debug bridge) utility, which is required for CTS to run. Also, using the SDK lets you create virtual devices on which to run your CTS test cases if you don’t want to use a real Android device.

Initial CTS Configuration

Here are the steps for configuring your Linux system to run CTS.

Ensure that adb is in the system path. This can be done with the command:

export PATH=$PATH:/path/to/android-sdk-linux_x86/platform-tools

Once the path is set, you can launch the CTS console by running the cts-tradefed script. To run the script, browse to the folder containing the android-cts folder and run the command:

./android-cts/tools/cts-tradefed

This takes you to the cts command prompt. Now you can run the test cases for your device using the cts commands. The commands are discussed below, but more details on configuring and running CTS are available in the official CTS documentation at http://source.android.com/compatibility/downloads.html.

If you use Windows, here are the steps to configure CTS on your computer.

Put the CTS prebuilt package downloaded from the Android developer site in a suitable location on your PC. Next, create a .bat file with the following contents and put it in the tools folder inside the CTS folder:

SET SDK_ROOT=<Path to the Android developer tools folder>

SET CTS_ROOT=<Path to the CTS folder>

if not exist %SDK_ROOT% GOTO showError

if not exist %CTS_ROOT% GOTO showError

SET PATH=%PATH%%SDK_ROOT%\platform-tools;

if not exist %SDK_ROOT% GOTO showError

java -cp

ddmlib-prebuilt.jar;tradefed-prebuilt.jar;

↳hosttestlib.jar;cts-tradefed.jar

-DCTS_ROOT=<Path to the CTS folder>

com.android.cts.tradefed.command.CtsConsole

:showError

Echo.

Echo “SDK_ROOT/CTS_ROOT not configured properly;

Remember to change the values of SDK_ROOT, CTS_ROOT and the -DCTS root. This batch file sets the required environment variables and prepares your system to run the CTS.

Next, run the .bat file you created in the above step. This should create a cts-tf command prompt. Start your Android virtual device (not required for real devices), and run the command list devices at the cts-tf command prompt. This will list the Android devices that are available currently on your computer. If it doesn’t, you may want check whether the previous steps were followed properly.

That’s it for the CTS configuration, but if you are planning to use the virtual Android device, you need to configure it. To do that, just run the SDK manager.exe in the adt folder. This tool helps you manage the Android packages and the Android virtual devices. You can install your (modified) Android package and then create a virtual device with a target as your Android package. So, you can have a virtual device with your Android code loaded. (Skip this step if you plan to use an actual device, and just plug in your Android device to your computer via USB cable. CTS should be able to identify the device.)

Figure 1. Android Virtual Device Window

Running CTS

The CTS testing mechanism is arranged as test cases, test packages and test plans. As the names suggest, test cases test a particular class, while the test packages are used to test functionality, which can be a combination of several APIs. Test plans are comprehensive testing schemes where features of the OS, such as signature and appSecurity, are tested. Now as you know, CTS is a combination of all these test cases and the application that manages these tests. So, to run a CTS test case, you need to communicate with the application, which internally will start running the tests.

The CTS application can be run via the cts-fd command prompt (which you get after running the .bat file mentioned previously). It has its own set of commands, and you need to use these commands to interact with it. The following are a few of the most important CTS commands:

■ help: lists all the available commands and their options—type this command if you are stuck somewhere.

■ list: the list command can be used with different options to list the test packages, test plans, test results, devices and so forth on the command prompt. For example, if you want to list all the test plans, type list plans.

Figure 2. Device Listing on the CTS Prompt

Figure 3. Plan Listing on the CTS Prompt

■ run cts: this command runs the test cases in CTS. You can use it to run test packages, test plans and so on. For example, if you want to run the android.view package, type run cts -p android.view.

Now that you know the important CTS commands, you can start running the CTS test cases to verify your android implementation.

Report Generation

Once you run the CTS test case/package/plan, you need to wait for the test to finish running. Once the test is finished, CTS automatically generates the reports and logs. The reports generated by CTS are quite comprehensive with clear information about the passed/failed cases. If there are any failures in the test, the report shows it on the top along with the failure details to help you analyze the failed test cases. You can access the report in the folder android-cts\repository\results\<Time-Stamp>\testResult.xml. CTS also generates detailed logs of the test case execution, which can help you if you need to troubleshoot issues. The log is available for both the device and the host system. The logs are available in the folder android-cts\repository\logs\<Time-Stamp>.

Figure 4. Test Failure Summary in Report

Figure 5. Test Report Summary

Adding Tests to CTS

Thus far, I’ve been discussing how to run the default test cases available with CTS, but CTS is open source, so where is the fun if we don’t play around with the code and add something extra? Let’s look at how to write test cases for CTS and then integrate them with the framework.

Getting Started

Probably the best way to get started with adding a test case to the CTS is to look at the existing test cases present in the framework source code. (The CTS source code is available with the Android source code, so you need to download the Android source code.) As you go through the folder structure and the test cases, you will get an idea of how the test cases are written and organized in the framework.

You can see that the default test cases included with CTS in the cts/tests/ folder are segregated according to the different features they are meant to test. There are folders for processtest, signaturetest, accessibility and so on, which correspond to the various features. The folder cts/tests/tests has the general test cases, and most of the time, this is where you can add your own test case. If you go to a folder inside the cts/tests/tests folder, say the bluetooth folder, you can see it has an AndroidManifest file along with a make file. So, these tests essentially are Android apps written based on JUnit.

JUnit Introduction

JUnit is an open source, unit-testing framework for the Java language, which aims at testing the code and then its implementation. So, with JUnit, developers can add the test cases in their code and get feedback while developing the code. The Android CTS uses features like assertion and annotations in the test cases, so I primarily focus on these features here.

■ Assertion: using assertion, you can test whether a given condition is true or false. JUnit provides an assertion method for all the primitive types and objects. So, you can use assertion in your test case to check whether things are as they should be. There also is an optional field for a string message that is output on failure. Assertion methods available in JUnit are assertTrue, assertFalse, assertEquals, assertNull, assertNotNull, assertSame, assertNotSame and so on.

■ Annotations: annotations are used before a method definition to identify whether it is a test method. They also are used to define the execution order of the test case or even ignore a test case. The annotations available in JUnit are @Test, @Before, @After, @Ignore, @BeforeClass, @AfterClass and so on.

Enough theoretical knowledge, let’s look at a working sample! Let’s take a working test case based on JUnit and see how it works. Below is a small code snippet from the CTS source to show how assertions are used in the test cases (you can see the full code here: cts/tests/tests/bluetooth/src/BasicAdapterTest.java):

public class BasicAdapterTest extends AndroidTestCase {

private static final int DISABLE_TIMEOUT = 8000;  // ms timeout for BT

disable private static final int ENABLE_TIMEOUT = 10000;  // ms timeout

↳for BT enable

private static final int POLL_TIME = 400;         // ms

↳to poll BT state

private static final int CHECK_WAIT_TIME = 1000;  // ms

↳to wait before enable/disable

private boolean mHasBluetooth;

public void setUp() throws Exception {

super.setUp();

mHasBluetooth =

↳getContext().getPackageManager().hasSystemFeature(

PackageManager.FEATURE_BLUETOOTH);

}

public void test_getDefaultAdapter() {

/*

* Note: if the target doesn’t support Bluetooth at all,

* then this method should return null.

*/

if (mHasBluetooth) {

assertNotNull(BluetoothAdapter.getDefaultAdapter());

} else {

assertNull(BluetoothAdapter.getDefaultAdapter());

}

}

}

As you can see, the BasicAdapterTest extends the AndroidTestCase. The first method in this class, the setUp method, sets a variable as true or false based on whether the device has a Bluetooth feature available. The second method, test_getDefaultAdapter, tests whether an application can access the default Bluetooth adapter using the assert function available in JUnit. Similar methods to test different features are written. Additionally, you can use Eclipse to write new JUnit test cases—just select JUnit test case in the new Eclipse project window.

Integrate the New Test Case with CTS

Now that you know how to write a test case and where the files are stored, let’s look at how to integrate these newly written tests to the CTS framework so they can be executed as the default tests. Following are the steps to add a test case to the CTS:

1. Develop the test case as a JUnit Android application. Add the folder to the cts/tests/tests folder.

2. Add the line LOCAL_PACKAGE_NAME: = <TestPackageName> to the Android.mk file present in the application folder. Here, TestPackageName is the name of test package you just created.

3. Add the name of the test package to the CTS_CASE_LIST variable in the android/build/core/tasks/cts.mk file.

4. Open the command prompt and change the current directory to the Android source folder. Now you can build the CTS by running the command make cts at the command prompt. (To run the make command on Windows, you need a compiler toolchain installed.)

Conclusion

That’s pretty much everything about the Android CTS tool. Use it whenever you are making changes to the Android source code, so that you can be sure of being inline with the original Android code. Happy coding!■

Nitish Tiwari lives in Bangalore, India, and he currently works as a developer for a FOSS-based startup. He also helps enterprises implement open-source tools based on their needs. In his free time, he likes to try out and test open-source tools. You can reach him at nitish.tiwari@technocube.in.

Send comments or feedback via http://www.linuxjournal.com/contact or to ljeditor@linuxjournal.com.

FEATURES

Multi-Booting the Nexus 7 Tablet

Tired of Jelly Bean on your Nexus 7?

Try something else with MultiROM!

BILL CHILDERS

Anyone who knows me well enough knows I love mobile devices. Phones, tablets and other shiny glowing gadgets are almost an addiction for me. I’ve talked about my addiction in other articles and columns, and Kyle Rankin even made fun of me once in a Point/Counterpoint column because my household has a bunch of iOS devices in it. Well, I was fortunate enough to add an Android device to the mix recently—a Nexus 7 tablet. I actually won this device at the Southern California Linux Expo as part of the Rackspace Break/Fix Contest, but that’s a different story.

If you’ve not seen a Nexus 7, it’s a nice little device. Like all “Nexus”-branded Android devices, it’s a “reference” device for Google’s base Android implementation, so it’s got a well-supported set of hardware. I’m not trying to make this article sound like a full-fledged review of the device, but here’s a few tech specs in case you’re not familiar with it:

■ 7” screen with 1280x800 resolution.

■ 7.81” x 4.72” x 0.41” (198.5mm x 120mm x 10.45mm).

■ 16 or 32GB of Flash storage (mine is the 16GB model).

■ 1GB of RAM.

■ NVIDIA Tegra 3 Quad-Core Processor.

■ Wi-Fi, Bluetooth and optional 3G radios.

■ Android 4.2 Jelly Bean.

The Nexus line of Android devices makes up the reference implementation for Android, so that tends to be the series of device that sees the fastest movement in terms of new builds of the OS, and in unique OS derivatives like CyanogenMod. Right about the time I received the Nexus 7, Canonical released the developer beta of Ubuntu Touch, which targeted the Nexus 7 as its deployment platform.

Because I can’t leave nice things well enough alone, I decided to start trying alternate OS ROMs on my shiny new Nexus 7. Ordinarily, each new OS would require you to reflash the device, losing all your configuration, apps and saved data. However, I found a neat hack called MultiROM that lets you sideload multiple ROMs on your device. How does it work? Well, let’s walk through the installation.

Prep for MultiROM Installation

First, and I can’t stress this enough, back up your device. I really, really mean it. Back up your device. You’re messing around with lots of low-level stuff when you’re installing MultiROM, so you’ll want to have copies of your data. Also, one of the first steps is to wipe the device and return it to an “out-of-the-box” configuration, so you’ll want your stuff safe.

Second, grab copies of the “stock” Nexus 7 ROMs as they shipped from the factory. You will want these in the event something goes wrong, or if you decide you don’t like this MultiROM hackery and want to roll your device back to a stock configuration.

Third, check the links in the Resources section of this article for up-to-date documentation on MultiROM. It’s possible for things to change between this writing and press time, so follow any instructions you see there. Those instructions will supersede anything I type here, as this kind of hack can be a rapidly moving target. Also, do your own homework—lots of great YouTube videos describe this process, and a video sometimes can be worth several thousand words.

Notice: please make sure you follow these three steps, then follow the MultiROM documentation exactly. I’m not responsible if your tablet gets bricked or turns itself into SkyNet and goes on a rampage against humanity. Though I have to say, if that happened, it’d be kind of neat, in a geeky sort of way.

Unlocking Your Bootloader

Your device should be on the latest available factory ROM supported by MultiROM before you begin the installation. At the time of this writing, on my Nexus 7 (Wi-Fi-only) model, that was 4.2.2. The Nexus 7 comes from the factory with a “locked” bootloader. The first thing you’ve got to do is unlock the bootloader before you can proceed.

To unlock the bootloader, you need the Android SDK tools installed on your computer (see the Resources section for a download link). Specifically, you’ll need the fastboot and adb tools for this, so make sure they’re on your system and in your shell’s PATH environment variable.

Next, hook up your tablet to your computer via the USB-to-MicroUSB cable, and then run:

adb reboot bootloader

Your tablet then will reboot, and you’ll be in the Android bootloader. Once you’re in the bootloader, run the following command:

sudo fastboot oem unlock

Next, you’ll be prompted to confirm the command and accept that all data on your device will be erased. The tablet then will reboot, winding up in the setup wizard where you’ll be prompted for all your setup information as if it were fresh out of the box once more.

Installing MultiROM

Now that your bootloader is unlocked, you can proceed to the trickiest part of this process—installing MultiROM. Grab a copy of it from the XDA-Developers MultiROM thread (the link is in the Resources section of this article; currently the filename is multirom_v10_n7-signed.zip). You’ll also need to get the modified TWRP install file (TWRP_multirom_n7_20130404.img) and a patched kernel (kernel_kexec_422.zip). Rename the TWRP install file to recovery.img, then hook your tablet back up to your computer, and place these files in the root of its filesystem (keep the .zip files zipped—don’t unzip them).

Next, from your computer’s command line, you’ll need to run the adb utility from the Android SDK again, but this time, with the proper argument to get the system to boot to “recovery” mode:

adb reboot recovery

This will bring the device to “Clockwork Recovery” mode. From the Recovery menu on the device, choose “Install zip from sdcard”, followed by “choose zip from sdcard”, then specify the MultiROM zip file you moved to the root of your tablet’s filesystem earlier. When it’s done flashing, select “reboot system now”, and your Nexus 7 will reboot.

Once the device boots normally, issue the following command from your computer to get the system back in the bootloader:

adb reboot bootloader

The device will reboot in bootloader mode. Select the fastboot option on the screen, then type the following on your computer:

sudo fastboot flash recovery recovery.img

That’ll flash the modified recovery image that MultiROM requires to your tablet. Next, just tell the tablet to reboot by issuing the following command to it:

sudo fastboot reboot

Your Nexus 7 now is ready to install alternate ROMs.

Adding ROMs to MultiROM

Adding ROMs to MultiROM is fairly straightforward from here. Just hook your tablet up to your computer, drop the .zip file for the ROM you want to install onto the root of the filesystem, and then shut down the tablet. Restart your Nexus 7 in MultiROM by holding the “Volume Down” button while pushing the power switch. You’ll see a screen with what appears to be the Android logo lying on its back (Figure 1). This is the bootloader. Push the “Volume Down” button until the red arrow at the top of the screen indicates “Recovery Mode”, then push the Power button. This will boot the Nexus 7 into MultiROM.

Figure 1. Android Bootloader Screen

Now that your Nexus 7 is actually in MultiROM, select the “Advanced” button in the lower-left corner, then select “MultiROM” in the lower-right corner. Now, to install a ROM, touch “Add ROM” in the upper-left corner (Figure 2).

Figure 2. MultiROM “Add ROM” Screen

Accept the defaults (unless you’re trying the Ubuntu Touch developer release), and just press Next. The next screen will ask you to select a ROM source. Touch the Zip file button, then pick the .zip file of whatever ROM you want to install. The system will go ahead and install it, and it’ll let you know when it’s complete. Push the Reboot button when the install is complete, and your tablet will reboot into the MultiROM selection screen (Figure 3).

Figure 3. MultiROM Boot Menu

Looking at my boot menu, you’ll see I’ve got cm-10.0.0-grouper installed, otherwise known as CyanogenMod. To boot that, I simply touch it, then press the large blue Boot button at the bottom of the screen. It’s as simple as that—the Nexus 7 will just start booting CyanogenMod. At one point, I had the stock ROM, CyanogenMod, AKOP and Ubuntu Touch on my Nexus 7, all coexisting nicely (but they took too much of my limited 16GB storage space, so I pruned back some).

If you decide a particular ROM isn’t for you, you can get rid of it quite easily. Just go back to the MultiROM install by booting with the Power and Volume Down buttons depressed, then select Recovery, and press the Power button again. Dive back into the MultiROM menus, just like you’re installing a ROM, but instead of pressing Add ROM, press List ROMs. Touch the ROM you want to delete, and then just select Delete from the buttons that pop up. This will let you keep your MultiROM install clean, with only the ROMs you want to test active at any given time.

Getting Ubuntu Touch Running

Ubuntu Touch is something I’ve been watching closely, particularly because I spent a little time with an Ubuntu Touch-equipped Nexus 7 at the Southern California Linux Expo. The Ubuntu Touch developer builds can be a little finicky, although they’ve stabilized in recent weeks. The key to getting them going in MultiROM is to select the “Don’t Share” radio button when adding the ROM (Figure 2). The Ubuntu Touch builds come in two parts. Add the smaller hardware-specific zip file first (on my Wi-Fi Nexus 7, it’s quantal-preinstalled-armel+grouper.zip on my Wi-Fi Nexus 7), but do not reboot—go back, list the ROM again, then push Flash Zip, and select the larger ROM file (quantal-preinstalled-phablet-armhf.zip). After that completes, you can reboot your tablet into Ubuntu Touch. Be advised, though, that Ubuntu Touch is under very heavy development, and sometimes the daily builds exhibit issues—and may not work at all. Your mileage may vary. If you do get Ubuntu Touch going, but it seems unresponsive to touch, try sliding from the left bezel toward the center. That’ll bring up a Unity-style launcher, and things should work from there. It took me a few tries to figure this out. I thought my Ubuntu Touch installation was broken or that I had a bad build. It turns out, it’s just a different operating paradigm.

Figure 4. Ubuntu Touch on the Nexus 7!

Conclusion

The Nexus 7 by itself is a great, low-cost, high-power tablet. However, thanks to its status as a reference device, there’s a lot of alternate OSes out there for it. MultiROM lets you try them all without requiring you to wipe your device each time you want to try a new OS or ROM build. Check it out, but back up your data, and read the documentation thoroughly.■

PONG

The programmer who wrote the MultiROM program has a great sense of humor, and he left a “Pong” easter egg in the software. From the main MultiROM boot screen, just touch the MultiROM logo, and you’ll get a proper portrait-orientation port of Pong (say that three times fast!)

Pong!

Bill Childers is an IT Manager in Silicon Valley, where he lives with his wife and two children. He enjoys Linux far too much and probably should get more sun from time to time.

Send comments or feedback via http://www.linuxjournal.com/contact or to ljeditor@linuxjournal.com.

Resources

XDA-Developers MultiROM Install Thread: http://forum.xda-developers.com/showthread.php?t=2011403

Nexus 7 Factory ROM Images: https://developers.google.com/android/nexus/images

Android SDK Tools Download Page: http://developer.android.com/sdk/index.html

CyanogenMod Home Page: http://www.cyanogenmod.org

AKOP Home Page: http://aokp.co

Ubuntu Touch Installation: https://wiki.ubuntu.com/Touch/Install

Ubuntu Touch Download Page: http://cdimage.ubuntu.com/ubuntu-touch-preview/daily-preinstalled/current

INDEPTH

Worms and Linux

Some people believe that Linux as an operating system inherently is not vulnerable to viruses and worms; others believe that Linux is not vulnerable simply because of its limited use as a desktop. The debate remains open.

HIMANSHU ARORA

Computer worms are considered to be one of the best weapons in a computer attacker’s arsenal. Through these computer worms, evil-doers intrude into computer systems to destroy files, attack other computer systems, steal data and so on. The concept of a computer worm is not new; attackers have been using worms to attack computer systems for decades now. If you look back at the history of computer worms, you’ll see that the computer worms that caused the most damage were directed toward the Microsoft Windows OS. Is this because of the number of Windows vulnerabilities, or is it merely due to the number of Windows users? The question remains unanswered. Meanwhile, apart from the Morris worm, very few worms have been directed toward Linux.

An Introduction to Computer Worms

A computer worm is a program that has malicious intent (that is, it can cause damage to a computer system). It can replicate itself and propagate over the network to infect other vulnerable computer systems. You also can think of a computer worm as a computer virus that requires no human action to start and that has the capabilities to self-replicate and traverse over the network.

A worm can arrive on a system through various means, such as an e-mail attachment, a downloaded file from the Internet, a file over a TCP/UDP connection from a trusted neighbor host or removable media like CDs, pendrives and so on. Once it has arrived on a system, it has three tasks to do:

■ Hide from any system-monitoring software, like antivirus software.

■ Start the actual damage for which is it is designed.

■ Send a copy of itself to other hosts on the network to infect them too.

A worm that is able to do all these tasks effectively can be a real threat to a host or a network. Developing an effective worm requires a lot of system-/network-related knowledge, and this is the reason not every computer worm is successful.

Design a Computer Worm—Wearing an Attacker’s Hat

From an attacker’s prospective, three things need to be considered when designing a computer worm.

1) Exploiting system vulnerabilities: The whole idea behind developing a worm is to cause maximum damage, and this can be possible only if the worm exploits one or more system vulnerabilities. Exploits are nothing but loopholes in the standard utilities or in the OS kernel that worms can use to get the required power or privileges to compromise a computer system.

If a worm does not exploit a system vulnerability, the only way for a worm to compromise a system is to wait for the user to make some mistake. But, this requires writing a lot of code; thus, it is not considered a good design for developing a worm. Also, this design strategy does not ensure that a worm always will be successful in compromising and damaging the system.

On the other hand, if a worm exploits some vulnerability, such as a “zero-day” vulnerability (http://en.wikipedia.org/wiki/Zero-day_attack), it becomes very easy for a worm to compromise a system and cause maximum damage.

Note: some worms do not intend to cause any damage to the infected host (see my next point for details).

2) Stealth—the ability to remain undetected: This also is one of the major aspects of worm design. Even a worm with the deadliest intentions can’t do much damage if a system administrator or the monitoring software that runs on a host can detect it easily. So, the ability to remain undetected on a system is yet another important factor that worm authors consider. Countless worms are developed each month, but only very few become potential threats because of most worms’ inferior stealth capability.

The best way for a worm to stay undetected is to piggyback itself onto a trusted process by compromising it. A trusted process is a process running on a computer system that is deemed trustworthy by the system itself, and hence, it faces minimal interference from firewalls and antivirus software running on the system. A process can become trusted if it follows the relevant standards that are set by the US Department of Defense.

Worms have other ways of hiding on a system. For example, my article in the January 2012 issue of LJ discusses a small and not-so-destructive example of a piece of code that can exploit ELF executables on a Linux system: http://www.linuxjournaldigital.com/linuxjournal/201201?pg=92#pg92. The design of this malicious piece of code enables it to compromise the executables on a Linux system, and each time a compromised executable is run, many other executables are compromised. Here, the malicious code remains on the system until all the affected executables are cleaned.

Some worms are designed to hide for a very long period of time on an infected host. Such worms do not intend to cause any damage to the host on which they reside. Rather, the sole purpose of these worms is to provide a back door (to each host on which they reside) to the worm author, who then controls the host remotely and activates the core of the worm at any given point of time in the future to launch DDoS attacks to a particular server. These types of attacks make it practically impossible to catch the actual attackers (worm authors) and generally are used to target government organizations and big corporations.

3) Speed and method of propagation: Every worm has a lifetime of its own before it is nailed down. That’s the reason propagation speed is one of the top priorities for worm authors. Propagation speed directly depends on the method of propagation.

One propagation method worms use is via e-mail, where worms are transmitted in the form of e-mail attachments. If you look at the intricacies of this method, you’ll find that propagation speed is not very high, because all the messages have to go through e-mail servers. This adds an extra delay for a worm to reach its destination.

Another worm propagation method is to establish TCP connections directly. A TCP connection could be between an infected and non-infected host, or it could be between a non-infected host and a server where a fresh copy of a worm can be downloaded. Because TCP is a connection-oriented protocol, the three-way-handshake (http://en.wikipedia.org/wiki/Three_way_handshake#Connection_establishment), or the concept of acknowledgement, can cause unnecessary delays and hamper the worm’s propagation speed. But, this propagation method is still faster than propagation via e-mail.

The fastest of all the propagation methods is over UDP connections. Because UDP is not connection-oriented and is not affected by network congestion, it offers speedier propagation for worms. The only drawback of using UDP is that delivery to the destination is not reliable, but that’s not a real issue in the case of worms where more than one copy of a worm is sent to a host.

You may wonder why e-mail still is used as a propagation method if it has the slowest speed. E-mail provides a way for worms to reach a lot of systems through contacts, while UDP makes sure that propagation is quick. So, there is no best way of impact using either e-mail or UDP. But, if you consider a method where the worm is sent through e-mail over UDP, the worm can propagate to a large audience in no time.

Detect and Counter a Computer Worm—Wearing an Administrator’s Hat

Here are some methods for detecting a worm:

■ Monitor trusted processes: you all know that non-trusted processes are monitored actively by programs like firewalls and antivirus software, and any deviation from the allowed behavior causes those processes to be quarantined and even interrupted. But, what about trusted processes? Although minimal, there are chances that a worm may attach itself to a trusted process. So, what happens in that case? Well, trusted processes also can be monitored to a certain extent. For example, each trusted process has specific work to do. If it can be monitored for any deviation from its work, worms attached to trusted processes also can be detected.

■ Monitor IP address scanning: each process that tries to do some activity over the network tries to connect to an IP address. Although legitimate processes know the IP addresses (on which the connection needs to be established), worms try to scan all the IP addresses that fall in the host’s IP address space. This means that worms would try to connect to all the IP addresses in the IP address space of the host, and any good monitoring software will detect these worms.

■ Install honey pots on the network: a honey pot is a computer node on a network that is set up explicitly as a trap for intruders and malicious programs like worms. Honey pots let the intruder (or malicious program) do all the evil stuff while recording all its activities. This way, an alarm can be raised to all the nodes on a computer network, and the network administrator easily can block such unwanted activity on other nodes.

After detecting a worm, the first step should be to quarantine the process and strip off its permissions to execute code on areas like heap, stack and so on. If it continues to show the same suspicious behavior, affected processes also can be killed. Other than this, by keeping your system updated (with all the latest patches to kernel and system utilities), you can minimize the threat.

Linux Worms—Why Are There Only a Few Successful Ones?

Worms on Linux have had a limited success rate. As I mentioned at the beginning of this article, some people believe that Linux as an operating system is inherently not vulnerable to worms, and others believe that Linux hasn’t been very vulnerable to worms simply because of its limited use as a desktop. The following are some points I’d like to add to this debate:

1. Unlike Windows, most Linux users are not just end users; they know what they are doing on their Linux boxes. Linux users do not easily fall for traps like “Hey, your system has lots of viruses and trojans. Download AntiVirusFake.exe and get rid of this malware.” Also, most Linux users keep their OS updated with the latest patches, which automatically keeps viruses, worms and so forth at bay.

2. The wall of permissions is the other factor that makes Linux a difficult place for viruses and worms. A normal Linux user account does not have much privileges at its disposal, so a worm or virus cannot do much except affect the files and executables that are limited to a single account. Also, unlike Windows, you never are encouraged to log in as administrator or root and play around all day clicking on files and executables with root privileges. The Linux philosophy is that if you want to do something that requires root privileges, acquire root privileges for some amount of time, do the required work and then return to normal user mode.

3. Linux and even most of the software that is run on it is open source. A large number of people constantly are working to improve and polish Linux and other open-source software. This means that even if somebody is able to develop a virus or worm by exploiting a vulnerability in the Linux kernel or some popular piece of open-source software, it will take very little time for the community to fix and patch it. Bugs in the Linux kernel are fixed in no time.

4. Unlike Windows, Linux does not follow the “brain-dead” file extension system. If a.out is an executable file on Linux, and if you try to run it, it will not run just because it has a .OUT extension. The executable bit has to be set on every executable in order to execute it. If you compare a scenario where a Windows user receives an e-mail message with an attachment named “HotBritneyWallpaper.exe”, just by clicking on the attachment, the virus/worm is installed in the system. On the other hand, in Linux, if an e-mail with a similar attachment is delivered, every user first has to save the attachment, then give execution permissions to the file in order for the file to execute. But, if the attachment boasts of just being wallpaper, why would even a normal Linux user provide the file with execution permissions? There you go—problem averted. Linux provides another layer of protection that prevents accidental execution of binary files.

5. Linux comes packaged in various distributions like Ubuntu, Red Hat, Cent OS and so on, and each distribution has its own set of software utilities. For example, several e-mail clients are used across various Linux distributions, such as KMail, mutt, emacs, Mozilla Mail, Evolution, pine and so on. Now if an attacker writes an exploit for one of those, it’s not going to impact all Linux users. So, unlike Microsoft, where an exploit for Outlook rings alarm bells all over the world, because Linux has more variety, this adds another layer of protection.

A Case Study of Some Linux Worms

In this section, I focus on two popular Linux-specific worms. Note: the Morris worm has been the most popular UNIX-based worm to date, but I don’t discuss it here, because so much has been said and written about it already. Read http://en.wikipedia.org/wiki/Morris_worm for more information on the Morris worm.

The Linux Slapper Worm: The Linux Slapper is a worm that targeted Linux machines running an Apache server. This worm was detected in October 2002. The Slapper worm exploits the buffer overflow vulnerability of the OpenSSL software installed on the server.

The worm starts by scanning the Internet randomly for hosts, and when it gets one, it checks whether the host can be attacked. This is done by sending an incorrect HTTP GET request to the target. If an Apache Web server is running on the target, the reply from the server would provide enough information for the worm to decide on the next step. After confirming that an Apache server is running on the host, the worm connects to port 443 (the SSL port) and tries to launch a shell (/bin/ssh) by exploiting the buffer overflow vulnerability of OpenSSL. This exploit specifically targeted Intel x86 platforms. Once the shell is launched, the worm sends a shell script containing an uuencoded copy of its own source code. Using the shell script, the source code is written to a file named .bugtraq.c in the /tmp/ folder.

Once the /tmp/.bugtraq.c file is saved, the worm depends on the availability of GCC on the host and Apache’s permissions to execute the GCC compiler. If these requirements are in place, it compiles the .bugtraq.c file into an executable. This executable then is run with the IP address of the attacking computer as a command-line argument. This way, the worm propagates further by forming a peer-to-peer network. It also opens up a specific UDP port to listen and accept instructions to be carried out on/from the infected system.

The Linux Slapper worm had many variants. The A variant listens on UDP port 2002, while variants B and C listen on 1978 and 4156, respectively. The B variant also acts like a virus and corrupts some other files by overwriting them with itself. It also sends data to a yahoo.com address. The C variant also listens to port 1052 periodically and provides a shell to the attacker who is connecting to this port. Besides these differences, all three variants used different names for the files saved in the /tmp/ directory.

Detecting and removing this worm is not very difficult. All you need to do is:

■ Log in as root.

■ Stop the Apache server.

■ Go to the /tmp/ directory.

■ Stop any process that is running with the same name as the worm executable file present in the /tmp/ directory.

■ Remove the worm files (use ls -la to list the files).

Note: there is another Linux worm called DevNull that worked pretty much the same as the Slapper worm. DevNull also exploited the buffer overflow vulnerability in OpenSSL. It is sometimes referred to as Slapper variant D, but it’s actually a different worm.

The Linux Lion (L10n) Worm: The Linux L10n worm, also known as the Lion worm, was written by a Chinese hacker named Lion. He created this worm to warn the Japanese education department of controversial books being used in Japanese schools. According to the worm author, these books proclaimed the war crimes done by the Japanese against China and Korea as “legitimate”.

The worm exploited a TSIG buffer overflow vulnerability in the BIND (Berkeley Internet Name Domain) server software. BIND server software is used to provide instructions to domain name servers (DNS) to convert Web addresses into IP addresses. The Lion worm scans port 53 of the class B network IP addresses, and if it finds a name server running on the target, it launches its BIND exploit on the target. If the exploit is successful, a copy of the worm is downloaded in form of a package. This package then is extracted and the startup scripts are executed.

The infected machines send the root passwords to a Web site hosted in China where the attackers tried to decrypt the password and gain administrative-level access to the infected systems. Reportedly, the worm also created “back doors” for attackers to gain complete access to the infected system.

The Lion worm installed a rootkit in the infected systems in order to hide itself from being detected. This made system administrators’ jobs really difficult. The SANS (System Administration, Networking and Security) institute created a program known as “Lionfind” for sysadmins to detect whether the Lion worm is present on a suspected machine.

This worm installed the following files on a Linux system:

■ /bin/in.telnetd

■ /bin/mjy

■ /bin/ps

■ /bin/netstat

■ /bin/ls

■ /etc/inetd.conf

■ /sbin/ifconfig

■ /usr/bin/find

■ /usr/sbin/nscd

■ /usr/sbin/in.fingerd

■ /usr/bin/top

■ /usr/bin/du

And, to hide its footprints, the worm deleted these files:

■ /.bash_history

■ /etc/hosts.deny

■ /root/.bash_history

■ /var/log/messages

■ /var/log/maillog

Experts believe that this worm cannot be removed, and infected systems have to be re-formatted in order to clean out the worm.

Antivirus, IDS and IPS Products for Linux

Linux is now the “most-used OS” on servers, and its popularity on desktops is growing day by day. With this increasing acceptance, Linux is gaining more attention from attackers and intruders. More and more viruses, worms, trojans and so on are being made for Linux these days. Although the success rate of Linux malware is low at the moment, this is no excuse for not protecting Linux-based systems, especially if they are being used in sensitive and critical areas.

Many companies actively are working to provide quality antivirus programs for Linux-based systems, such as Computer Associates, Kaspersky, Trend Micro, Sophos, F-Secure and Symantec.

Apart from antivirus software, some products are available that are based on advanced technologies like intrusion detection systems (IDSes) or intrusion prevention systems (IPSes).

An intrusion prevention system uses an intuitive approach to detect and prevent system intrusion attempts directed at a host. An IPS system scans the data flowing to and from a host, and based on its detection system, it detects any suspicious activity taking place on a host or on a network. Unlike firewalls, IPS systems also check the outbound packets from the host.

The detection methods used are:

■ Signature method.

■ Profile method.

■ Stateful protocol method.

Some free and open-source IPS products are Snort, OSSEC and Suricata.

Read http://www.ibm.com/developerworks/library/se-intrusion to learn more about this technology, how it works, what detection techniques it uses and some free and open-source IPS products.

Conclusion

Computer security, as you all know, is a process. It is not a product you can purchase and then forget about. Still, you can make good choices and follow best practices so that your systems suffer the least. Today, Linux is the best choice if you want to build a stable and secure host or network environment. But, with the growing popularity of the Linux OS, attacks and intrusions will increase, so be safe!■

Himanshu Arora is a senior software engineer at STMicroelectronics, India. Most of his experience is on Linux system programming and network protocols, but his goal is to study Linux in its entirety. In his free time, Himanshu writes articles on topics ranging from Linux administration to Linux security. His articles have been featured on IBM DeveloperWorks and in Smashing Magazine and Linux Journal.

Send comments or feedback via http://www.linuxjournal.com/contact or to ljeditor@linuxjournal.com.

OPINION

21st-Century DevOps—an End to the 20th-Century Practice of Writing Static Build and Deploy Scripts

Automating, standardizing and simplifying DevOps requires a model-driven process unchained from one-off back-end scripts.

TRACY RAGAN

Having served as a software developer since leg warmers and shoulder pads were in style, I’ve seen the distributed platform, from UNIX to Windows struggle, with the process of moving software changes from development to production over and over again. News from IBM and CA technologies about acquiring “Release Automation” solutions for the distributed platform has occurred in the recent past and has become in fashion once again. It seems our industry either enjoys the feeling of déjà vu or simply chooses to forget what it already has done to address the problem of DevOps.

DevOps, at its core, is a process that simplifies the hand-off of source code between development and production, allowing test and production release teams to build and deploy binaries as required for the correct technology stack along the way. And, we have tools released in the not so distant past that have tried to solve this problem, but never quite met the challenge. Tivoli, CA Unicenter and a host of other solutions from competing companies always have had the ability to perform software releases. But there is always something missing, and most of today’s “in fashion” solutions do little different from their predecessors.

So what is missing? Is no one else noticing the 800-pound gorilla sitting in the room? The solutions that are paraded around as the latest-and-greatest method of solving the DevOps problem simply do not disrupt the status quo enough to get to the core of the DevOps issue. They serve some ability to centralize logs and to manage server environments and configurations, which is certainly helpful. But they ignore the back-end one-off build and deploy scripts that contain the logic that actually does the work.

Yes, I understand. Ant and Maven are so cool that you may be compelled to tinker with them. But in the corporate enterprise, the time for tinkering is over, and the time has come for a 21st-century solution that is dynamic and model-driven for both build and deploy. Central to most DevOps tools is the claim of a “virtualized” process. However, these “virtualization” solutions choose to ignore the 20th-century static scripts that serve as the foundation of their solutions. A 21st-century “virtualized” solution cannot be built upon brittle, one-off scripts for either the build or the deploy. What is required is for developers to accept the paradigm shift that is needed for 21st-century DevOps, a model-driven, scriptless solution. And yes, change can be scary.

History can be our best teacher, and in the DevOps space, this is particularly true. Looking at how the UNIX administrators and mainframe administrators of the 1980s and 1990s addressed the problem can provide insight into what works and what does not work. On the mainframe, private compile and ship JCL was thrown out when “processors” were introduced. Processors are the way that the mainframe creates dynamic build and deploy “scripts”, based on a model-driven framework. Everyone repeats and reuses the logic for compiling/binding/linking source code and shipping load objects. On the UNIX side, a central administrator, often using ClearCase, managed a central build and deploy script for the different levels and versions of the application moving across the life cycle. Even though they did not succeed in completely eliminating build and deploy scripts, they minimized the number of them down to a manageable level, often one build and deploy script for each environment—smart!

For reasons that are far beyond my understanding, it has been my perception that individuals who sell themselves as visionaries in this particular space avoid the discussion of scripts. Scripts are a huge bottleneck and a hidden cost in the activities of developing and releasing software. There is some hope, however. We may be seeing some movement away from scripting for releases, but few visionaries understand the build and tend to leave it out of the conversation.

So, what is in a build and why does it matter? Actually a substantial amount of information that is critical in the process of managing the DevOps effort is managed in the build. Get the build right, and a substantial amount of time and money is saved across the life cycle. Builds most commonly are managed by static build scripts that are somewhat unintelligent. You pass it a set of commands, and it executes them starting at the top and ending at the bottom. What build scripts cannot do is everything that is needed for managing your DevOps process through release.

A software build needs the ability to be flexible and transparent in what and how it is building the software—for example, incremental builds, dependency management, compile/link/archive options (debug vs. no debug), transitive dependency and the use of third-party libraries that make up the release target technology stack. Scripting languages lack the ability to manage these moving parts dynamically. As a result, you get redundancy—copied scripts for different needs and environments, and you often hear “it worked on my machine.” Scripts also cannot produce reports that provide the insight that allows validation of the binaries before a release. Scripts are black boxes that produce black-box binaries. At best, there is a guessing game as to what the script did, such as what libraries and options did it use to create the deployable objects. And, if you do not know what your build script did, you cannot guarantee consistent deployments, regardless of the level of virtualization you have achieved. A bad build absolutely will result in a bad deploy, no matter how much money you spend on your release automation solution. The two are simply different sides of the same coin.

Deploy scripts are equally as critical. In a deploy script, the steps for performing the release should be super-standard, with no wobbles between one Java application using a Websphere server and another. Scripts are wobbly. It is their nature to be wobbly, as there are so many different ways to write a script. For any DevOps solution that claims automation and virtualization as the core features, one-off scripts delivered by the customer’s development teams should not be required to drive the automation tools. When you are spending top dollars on a new DevOps solution, your developers should not need to deliver the foundation of that new solution. If you purchase a solution that requires build or deploy scripting, Ant, Maven, Make, Python or the next groovy-new-scripting language, you can be guaranteed to be in the market once again in the not-too-distant future, as the problems you are trying to solve today still will be with you in those black-box, one-off build and deploy scripts. The only solution is for your team to embrace the paradigm shift and move to a model-driven process, from build through deploy, for achieving DevOps.

Paradigm shifts often are difficult when there is a deep-rooted culture to overcome. I suspect this is the reason that one-off build and deploy scripts still are used by the distributed platform, even though the mainframe and UNIX administrators eliminated or minimized them close to 30 years ago. I recently purchased a Tesla Model S, a 100%-electric automobile. It is an amazing automobile. Very little can go wrong; two small electric motors push the back wheels—no transmission, no grease, no parts, no hassle. I was showing the car to a neighbor recently, and he argued “I would never buy an electric car. I would miss the sound of the engine racing, and I would not be able to work on it.” His statement reminded me of a developer who once told me he liked his build and release scripts and enjoyed “tweaking” them. He explained it gave him a sense of accomplishment when they worked well, and that he considered himself a craftsman of his trade. I’m not sure what his director would have thought of that statement considering the time and money he spent on tweaking and managing those scripts.

Like the electric car, the time has come for a better way to do DevOps. A model-driven process will allow us to stop spending time and money tinkering with a DevOps engine that is based on hard-coded scripts and instead move to a process that does not require 20th-century techniques to solve the very real 21st-century DevOps challenge.■

Tracy Ragan has had extensive experience in the development and implementation of business applications. She began her consulting career in 1989 consulting to Fortune 500 organizations in the areas of testing, configuration management and build management. During her consulting experiences, Tracy recognized the lack of build management procedures for the distributed platform that had long been considered standard on the mainframe. In the four years leading to the creation of OpenMake Software, she worked with development teams in implementing a team-centric standardized build and deploy process. She served on the Eclipse Foundation Board of Directors as an Add-in Provider Representative for five years. She received her BS in Business Administration from California Polytechnic University and is a first degree black belt in Shotokan Karate.

Send comments or feedback via http://www.linuxjournal.com/contact or to ljeditor@linuxjournal.com.

EOF

DOC SEARLS

Android’s Limits

Android is a lot more free than iOS, but there are limits. We need to break through those.

At its birth, Android was the horizontal and open solution to the problem of Apple’s vertical and closed silo. On Android, hardware makers and software writers could build devices and apps, free to operate outside the walls of any vendor’s closed garden.

This was fine, as long as we ignored the closed and vertical natures of three controlling forces in Android’s market space: 1) mobile-phone companies; 2) Google’s main business, which is advertising; and 3) every e-commerce vendor, each operating its own silo. So let’s visit those, in order.

Mobile-Phone Companies

Before Android standardized a single popular platform for smart mobile devices, the phones we got were co-silo’d by partnerships between phone makers and phone companies. Back in the early 2000s, I sat in a meeting where a parade of software developers presented ground-breaking ideas to Nokia, which had invested in a number of those same developers. At the end, one of the Nokia people explained that these were all interesting ideas, but that the company already had worked out plans for features to be rolled out during the next several years on the company’s phones, in partnership with the carriers. This explained why, for example, my Nokia E62 lacked the Wi-Fi capability of the otherwise identical Nokia E61 (http://en.wikipedia.org/wiki/Nokia_E61). The E62 was built for carriers in the US, while the E61 was built for various European carriers. If AT&T didn’t want Wi-Fi on the E62, it wouldn’t be there.

We can thank Apple for driving a much harder and better bargain with AT&T than any other mobile-phone maker ever did. And we can thank Google for making sure that the smart mobile device market had a white-box operating system, so its hardware base could be as broad and generative as possible.

But those moves did not change the nature of phone companies, or our dependency on them. Back when JP Rangaswami (http://confusedofcalcutta.com) was Chief Scientist at BT (and I worked for him there as a consultant), he observed that the core competence of phone companies is billing, not telephony. This is the biggest reason (among many others) why no phone company ever would have welcomed, much less created, the Internet. After all, there is nothing in the Net’s protocols that welcomes, much less creates, billing opportunities. But today most of the data paths between mobile devices and the rest of the Internet go through phone company connections, and those connections are built for billing. This is done through “plans” for data usage. In some cases (for example, Sprint in the US), there are plans that allow unlimited usage. But in most cases, customers take a plan that obligates them to pay for some number of MB or GB per month, with additional charges for going over “caps”. While phone companies need to make money by charging for services, it’s an open question whether charging for data usage is the best, or the only, billing choice. It’s just one that comes easily to a business that’s already built to charge for minutes and text messages (the latter of which cost nothing on the open Internet).

Mobile phone systems are also walled in other ways the open Internet is not. For example, most data plans work only in the country where they are bought. Cross a national border and you risk “bill shock” for data use, which is charged at a much higher rate, even if you use the same company’s wireless connections. While this looks like a bug to customers, it looks like a feature to the phone companies—and also to governments that enjoy getting a slice of the action and never liked the borderless Internet anyway.

Then there is the video issue. According to Lowell McAdam, CEO and Chairman of Verizon, video already makes up for half the data traffic on his company’s wireless network, and will reach two-thirds by 2017 (http://variety.com/2013/digital/news/verizon-topper-try-a-la-carte-to-protect-cable-1200338345). Thus, the Internet is being turned into a distribution system for “content”. In terms of market power, content producers and distributors are sure to bias the growth paths of developers and their offerings, including Google and Android.

Google’s Business

Back when Google began, in the late 1990s, advertising was barely imagined as a business model. In fact, Sergey Brin and Larry Page, Google’s founders, initially were opposed to advertising. But since then, Google has become the largest company in the on-line advertising business, and advertising is the largest percentage of Google’s revenue base.

Most of that advertising is not of the old Madison Avenue kind, which mostly sent brand messages out over print and broadcast media. Most of on-line advertising today, including Google’s, is descended from direct marketing, which is descended from direct mail. It is meant to be personal, and it is guided by data provided through personal use, whether those persons like it or not. While Android does provide choices about privacy protection with each app one installs, users have little if any control over what happens to personal data that gets passed on to third parties, such as data brokers and analytics firms.

Lately the tide has been turning against privacy compromises and abuses by the on-line advertising business and the apps and services supported by it. Although Google takes a stronger moral stance on behalf of personal privacy than do many other on-line advertising companies, the fact remains that most users of Android phones and tablets are Google’s consumers rather than its customers, which means those consumers are the product being sold to Google’s actual customers, which are advertisers. From Steven Levy’s book In the Plex: “We don’t monetize the thing we create”, Andy Rubin says. “We monetize the people that use it. The more people that use our products, the more opportunity we have to advertise to them.”

Android phones also come defaulted with a pile of Google apps, placed in privileged positions in the phone’s app directory. This does not sit well with Google’s competitors. On April 9, 2013, Fairsearch, a coalition that includes Microsoft, Nokia and Oracle, filed a complaint with the European Commission against what it called “Google’s anti-competitive strategy to dominate the mobile marketplace and cement its control over consumer Internet data for on-line advertising as usage shifts to mobile”, adding “Google is using its Android mobile operating system as a ’Trojan Horse’ to deceive partners, monopolize the mobile marketplace, and control consumer data” (http://www.fairsearch.org/mobile/fairsearch-announces-complaint-in-eu-on-googles-anti-competitive-mobile-strategy). That’s a bit hyperbolic, but I believe they have a case.

For our purposes in the Linux community, the main thing at issue is bias in the evolution of Android and the apps that run on it. As long as advertising remains Google’s main business, and Android remains a Google project, it will be hard to keep a bias toward advertising’s imperatives from having an influence.

E-commerce

In 1995, my wife asked a question so profound that it has haunted me ever since: “Why can’t I take my shopping cart from one site to another?” The answer is that every e-commerce site is a silo with its own shopping cart, its own cookies for visitors, its own pile of visitor data, and its own silo’d relationships with visitors and customers. Nearly two decades have passed since then, and the situation hasn’t changed. In fact, it is now much worse. All of us doing business on the Net need to maintain up to hundreds of different login/password combinations, and our relationships with vendors are as silo’d as they ever were. The off-line world also is infected by the same urge to entrap visitors and customers, which is why it’s no coincidence that “loyalty programs” requiring that we carry around separate cards and key tags for every store chain, also began to take off in the mid-1990s.

The problem here is summed up by one of my favorite slides, by Phil Windley. It says this:

History of E-Commerce

1995: Invention of the cookie.

The End.

Cookies are a convention of client-server computing. While client-server has proven to be a handy way to build the on-line world, it forecloses countless opportunities that can be seen only from the client’s side—when the person there isn’t being a client. We still don’t have the shopping cart we can take from site to site because we can’t imagine doing that inside a world where we’re always the submissive party and never the dominant one. Or even an equal. Sure, we are free to run our own servers if we like, but that answer doesn’t help. In the world of e-commerce, we are merely clients, rather than human beings.

Far more can be done when we are free and independent than when we are captive and dependent. And that’s the rub here. In spite of all the good Android has done for us as developers and users, too much of what we can do is still trapped inside the walls of captors on which we depend, starting with Google.

We need to start thinking and working outside the boxes that phone companies, Google and client-server-based e-commerce have built around us. Can we break out of those boxes and still be on Android? I don’t have the answer, but I am working on it. Hope you are too.■

Doc Searls is Senior Editor of Linux Journal. He is also a fellow with the Berkman Center for Internet and Society at Harvard University and the Center for Information Technology and Society at UC Santa Barbara.

Send comments or feedback via http://www.linuxjournal.com/contact or to ljeditor@linuxjournal.com.

Table of Contents

COVER

CONTENTS

LINUX JOURNAL | MASTHEAD

Current_Issue.tar.gz

The Face of a Toaster

LETTERS

UPFRONT

diff -u

They Said It

One Tail Just Isn’t Enough

Non-Linux FOSS: libnotify, OS X Style

Android Poll

GIS with GRASS

EDITORS’ CHOICE

Weechat, Irssi’s Little Brother

COLUMNS

At the Forge

Unicode

Work the Shell

Cribbage: Pairs and Three of a Kinds

Hack and /

Two Pi R 2: Web Servers

The Open-Source Classroom

Prospecting for Ones and Zeros

NEW PRODUCTS

MSC Embedded Inc.’s Qseven Starter Kit

Peter Gasston’s Modern Web (No Starch Press)

ROSA Media Player

Exablox’s OneBlox

xTuple ERP

Erica Sadun and Steve Sande’s Pitch Perfect: The Art of Promoting Your App on the Web (Addison-Wesley)

6WIND’s 6WINDGate

Wind River Linux Carrier-Grade Profile

FEATURES

AIDE—Developing for Android on Android

Compatibility Test Suite—Test Your Android Implementation

Multi-Booting the Nexus 7 Tablet

INDEPTH

Worms and Linux

OPINION

21st-Century DevOps—an End to the 20th-Century Practice of Writing Static Build and Deploy Scripts

EOF

Android’s Limits