Thursday, March 31, 2011

Google paints malware bullseye on Kansas City, Kansas

Google announced on 30-Mar-2011 that Kansas City, Kansas (KCK) would be the first city to receive ultra-high-speed broadband on the company's dime.

Google has also stated that this is the first of many cities—not the only—that will receive their attention.

When Google first announced the project, they made it pretty clear that they'll be offering competitive pricing, but the pipe will be huge: gigabit fibre to the curb. The project is ambitious and potentially wonderful for those areas receiving the attention.

It will be interesting to see the fallout in KCK from the incumbents in the telco/cable duopoly. To start with, expect lawsuits. Incumbents don't like it when new players invade "their turf"; and when the city government is "in cahoots" with the new guy, there's not much room for backroom deals to quash the competition.

The second thing to expect is competition; however, I don't see that happening anywhere but within the active service boundaries. I have a friend who lived in an area with a single cable provider; he was at their mercy for Internet pricing, and DSL wasn't a viable option because of the distance to the C/O. When a second cable provider started installing their plant in his neighborhood, he tried to get some competitive pricing from the incumbent. It shouldn't come as a surprise that they were not willing to help him out: although they knew there was competition in the neighborhood, they also knew that his specific household couldn't get service from the new operator—even though the guy across the street could. Only once the new operator built out further and was able to offer service to his household was the old company interested in offering a better price. My friend opted for the new company as much out of spite as anything else.

The third thing to expect, however, is probably not on many people's radar just yet: expect KCK to become a hotbed for bot-herder interest. It's pretty logical:
  • The average Internet user—either business or personal consumers—won't have the resources or skill to safeguard system(s) from malware, especially if under persistent scrutiny and attack.
  • A fat-pipe zombie is a much more valuable asset to a bot-herder than a small-pipe zombie.
Put those together, and I'm afraid that you'll see KCK join the ranks of geographic hotspots for spam and other zombie-machine activity.

Don't get me wrong: I'm happy for KCK and how this will (hopefully) provide a positive impact on the rest of the Kansas City metro area (where I happen to live). But you can bet that the malware community will also be watching this with interest and work to find ways to take advantage of those lucky subscribers' machines—and the big, fat Google-pipes to which they're attached.

Friday, March 25, 2011

iPhone Battery Drain--Fix!

It's happened to me twice now, but this time I was able to do something about it...

My iPhone went bonkers yesterday and stared draining the battery like there was no tomorrow. It went from fully-charged to 20% in 3 hours; that's just nuts.

I did what every iOS user does in that situation: kill all the backgrounded stuff and restart the device. Usually that works pretty well. This time it didn't.

The phone was using the battery so "hard" that the device was warm to the touch. Again, that's just nuts.

So I got the thing connected to my office WiFi and started watching the syslogger for my firewall; every TCP connection would then be logged for the device.

Sure enough, I found the culprit: the Exchange ActiveSync service for my home email (yes, I run an Exchange server at home) was getting requests from the phone every couple of seconds.

Given that I had problems with my Inbox earlier this week using Thunderbird, I assumed that I had another "broken" email message in my Inbox, and moved all the messages to another folder. The inbox cleared up on my iPhone, but it was still going after the server like gangbusters.

So I deleted the EAS profile from the phone. Blissful silence from the syslogger was the result. Then I re-applied the EAS configuration profile to the phone to re-establish the EAS connection, and voila!, the phone isn't going apeshit anymore.

I'm not sure I want to tempt fate and put all the messages from my inbox back, but I am pretty happy with the result: the battery should once again last all day.

Sunday, March 13, 2011

The automation maker's nightmare: undocumented API changes.

One of my main job duties is taking the mundane and manual and figuring out a way to use technology to automate them. I've been doing this for over 10 years now, and I like to think I've gotten pretty good at it. The processes I've built (with the help of some amazingly intelligent business folk who understood the problems and were able to empower me to help them) have saved the company not just in dollars (which is easy to justify when you're liberating employees from the menial tasks of pure data-entry job responsibilities), but in reputation by driving inaccuracies out and efficiency in. It's not the case that what I write is flawless—I'm not that good, and I wouldn't trust anyone who claimed to be—but that flaws tend to show up in a way that is repeatable and (usually) correctable.

And so it goes...

One of the most powerful tools I have in my automation tool-set is Perl. The rich API that is available gives me the capability of adding lots of functionality and automation to the business environment. When it comes to getting external data into SQL databases, it's the first thing for which I reach.

I had a script that I wrote a few years ago, and while it worked well at getting records pushed into a table, it wasn't very fast. In short, I was doing a lot of work in the script that could have been done by the DB server if I'd just take the time to write a stored procedure for the "heavy lifting." But, because I was only parsing data on a handful of rows, and I was doing it no more often than every 10 minutes, it wasn't worth the effort.

That all changed a few days ago: I discovered that the script had been disabled for about 18 months, and all that data that should have been pushed into the target table had not. The data in question was the raw feed from an external data source; the script's job was to keep a rarely-used "raw data" table populated for the occasional special report. In our normal process, the raw data is transformed before being consumed by the business systems, and that hadn't been affected by the disabled script—which is why this issue hadn't come to light earlier.

But now I had one of those "occasions" where I needed the raw data for a special report, and 18 months of data was over 1 million rows to read and insert from thousands of text files.

Under the original design, here's what the script did:
  1. read a line of data
  2. parse the data into fields
  3. check the target table to see if the data existed
    1. if it didn't exist, insert it
    2. if it existed but was exactly identical, do nothing
    3. if it existed but was different, update the record
That logic worked, but it relied on 3 separate SQL statements ( a SELECT, an INSERT and an UPDATE), and two of them would necessarily run for each record being inserted. Updating the script to use a stored procedure (SP) to take care of the process would be easy; because I wanted to retain all the debugging and logging information in the original script, I would need to take some additional steps to make sure the new SP would return useful information back to the calling script. My DBMS allows the use of output parameters, so that wasn't an obstacle.

So I building the new SP. The target engine was Microsoft SQL 2005, so it looked a bit like this:

CREATE PROCEDURE dbo.usp_Archive_AddRec (
   @arg1 int, --unique identifier for the record
   @arg2 int, --additional record data
   @oldarg2 int=NULL OUTPUT --previous value of @arg2 if updated
)
AS 
SET NOCOUNT ON;
DECLARE @status int;
SET @status=0; -- returned value: 0=no changes, 1=record inserted, 2=record updated

DECLARE @oldbits TABLE (item int);

BEGIN TRY
   INSERT dbo.tbl_Archive (pk,item)
   VALUES (@arg1,@arg2);
   SET @status=1;
END TRY
BEGIN CATCH
   --most typical reason for the failure to insert
   --is an existing record; do an update instead
   BEGIN TRY
      UPDATE dbo.tbl_Archive
      SET item=@arg2
      OUTPUT DELETED.item INTO @oldbits --capture the previous value during update!
      WHERE
         pk=@arg1
         AND item != @arg2 --only update if the record is actually different
      ;
      IF @@rowcount > 0 BEGIN
         SELECT TOP 1 @oldarg2=item FROM @oldbits;
         SET @status=2;
      END;
   END TRY
   BEGIN CATCH
      SET @status=-1; --some unexpected SQL result/error
   END CATCH;
END CATCH
RETURN @status;

Well, that worked great in manual testing, so I went to update the perl code to accommodate the changes. My first roadblock was the inability to get ahold of the return value from the SP; the database driver will create an automatic parameter called @RETURNVALUE, but the DBI engine for perl doesn't have any documented way to get at it (at least, not when the SP isn't returning a dataset). I spent a while searching the Internet for a fix, but no avail. I'm sure it's possible, but in the interest of getting things done, I decided to punt and added the status as another OUTPUT argument.

I moved on to get the code running with the combination of input and output parameters, using the bind_param_inout form of parameter binding in order to get the return data.

It took a number of iterations to figure out the idiosyncrasies, but I finally had the script running on my sample dataset. At that point, I was able to turn it loose on the unprocessed data files from my original raw source; it was fantastic: The script chewed through 18 months of records in just over an hour, inserting over 1.1 million records. In comparison, the previous script ran through less than 5 months of files in around 10 hours.

Given the all-over success, I put the new script into the production automation process to take advantage of the additional performance (even on small datasets, improved efficiency is a good thing on a multiuser database system).

And immediately started getting syntax errors from the automation system.

What? I checked and rechecked, but everything was right. It tested perfectly on my development machine. I checked the versions of the Perl DBI modules: identical. I checked the SQL drivers: while they weren't identical, they were the "latest/greatest" for each platform. Okay, so that was a possible source; unfortunately, I could prove that it wasn't the problem because I could use the driver and a command-line utility and run the code without error. At that point, I knew the issue was within Perl, and a quick check showed that I was running 5.8 on my dev machine, but 5.10 on the automation host.

More troubleshooting, and I finally discovered the root cause through a Profiler run. On my dev machine running 5.8, the Perl statement I wrote and the statement that Profiler showed the DBMS receiving, they were identical:
exec dbo.usp_Archive_AddRec (
      @arg1=@p1,
      @arg2=@p2,
      @oldarg2=@p3 OUTPUT,
      @status=@p4 OUTPUT
   );
But Profiler showed me that the 5.10 production system was not sending the SQL as I wrote it; instead, it was doing some automatic edits:
exec dbo.usp_Archive_AddRec (
      @arg1=@p1,
      @arg2=@p2,
      @oldarg2=@p3 OUTPUT OUTPUT,
      @status=@p4 OUTPUT OUTPUT
   );
As near as I can tell, when the bind_param_inout DBI method is used in the 5.10 environment, it automatically updates the SQL query to convert the argument to an OUTPUT parameter. I spent a couple of hours searching for a switch or other argument to disable the behavior, but no luck. I was stuck with it.

My final solution now tests for the Perl version (using the built-in special variable $]) and adjusts the SQL statement accordingly; it now works without modification on every Perl environment in our datacenter.

Wednesday, March 9, 2011

Overcoming a CDMA smartphone's limitations

A number of pundits have posited that the Verizon iPhone wouldn't gain much traction with current GSM customers because of CDMA's designed-in inability to do voice & data simultaneously. AT&T is trying to play up that aspect of their smartphone lineup (not just the iPhone), while Verizon is playing up the "dropped call" perception that AT&T is suffering.

I'm a longtime, loyal Verizon customer. I have received great coverage and customer support the whole time I've been with them, and I'm not interested in anything that's not on Verizon. So how does one overcome the limitation with CDMA?

With WiFi.

As I understand it, this is the case with all smartphones, not just Verizon's iPhone: if you have WiFi available (and connected), all the data flows through that route, leaving the cellular connection available for calls. That may be fine for home and at the office (assuming WiFi is available), but how do you manage that when out-and-about?

Well, my solution may be a little unorthodox, but I find that it works quite well: I keep a Verizon MiFi in my car, so WiFi is readily available wherever I go.

Yeah, that's a bit crazy, but follow my reasoning, and you may agree that it's "crazy like a fox."

First off, I have a laptop and iPad that don't have cellular network access, so I've made the conscious decision to use a MiFi for them. At the time I purchased the MiFi, I also had an iPod Touch, so that made three devices that could share the one data plan. Brilliant device, the MiFi.

But now that I've traded the iPod for an iPhone, why not cancel the MiFi data plan in exchange for the lower cost of using the Personal Hotspot app? I quickly discovered that there are many disadvantages that come with that decision, most of which stem from Apple's design of the iOS operating system: when using the hotspot app, your iPhone more or less ceases to be a phone.

I don't have any experience with Android, but it may be similar: unless you root your phone, you live with lots of limitation on what it can do concurrently.

So if you decide that you're going to have that dedicated WiFi hotspot, you've just created the situation that will overcome CDMA limitations: Let your smartphone use the MiFi, too. Yeah, it's kind of redundant: using a Verizon MiFi to provide WiFi to a Verizon smartphone: either way—cellular or WiFi—the phone is using Verizon's network.

That's cool. You have access to voice and data simultaneously, even when you're out-and-about. But here's where it gets really interesting: I have no facts to back this up, but it has been my experience that the MiFi does a better job of maintaining data connections when traveling in the hinterlands and low-coverage areas. I routinely travel in places where my phone (both Blackberry and iPhone) goes into low/no data modes while the MiFi happily runs as if nothing had changed. It is my theory that when the carrier provisions a data-only device like the MiFi, it has the option to favor data over voice for the signal usage, where a phone will always be provisioned to prioritize voice over data. When in those coverage areas, I find that my CDMA phone is no better or worse than in good coverage: the data and calls still get through.

So there you have it: Get a dedicated personal hotspot, and your CDMA smartphone—not to mention any other WiFi-only devices—will always have access to data, even when you're on a call.

Monday, March 7, 2011

Blackberry vs iPhone

Executive Summary: don't give up your corporate Blackberry in favor of an iPhone, regardless of whether you're on Verizon or AT&T.

Background:
I've been a Verizon customer for as long as I can remember—over 15 years—and have been a Blackberry/RIM customer and BES+Exchange administrator for over 5 years. I've used and supported Blackberry devices going back to the 7250 (CDMA Color "click wheel"), and still believe that the "Curve 2" is the best device they've come out with for Verizon to date.

That said: for anything other than phone calls and interacting with the corporate Exchange server, the Blackberry pretty much sucks. The Java platform is slow and clunky, and something as simple as web browsing is an exercise in futility.

When the iPhone was released, I thought it was a clever toy. I had associates with them that loved them, but in many ways my Blackberry Curve was better. I had a flash for my camera. I could take motion video. I could type long messages on the thumb keyboard with a minimum of errors, and when AutoCorrect was 'invoked', it normally made sense; better yet, I had complete configuration control over AutoCorrect in the Settings application. Had the iPhone come to Verizon in its pre-iOS4 incarnations, there's no doubt that I'd have stuck with my Blackberry.

Enter the iPad.

To be sure, I agreed with many of the tech press who thought of the iPad as a bit of a toy—"an iPod Touch on steroids"—from the rumors and actual announcement by Apple in early 2010. I had no desire to buy one, and would probably still be a happy laptop-only guy had I not won one in a drawing. Of course, it was "only" the $500 16GB WiFi model, but it was enough: in short order, I realized how useful a smaller, pocket-sized version of the iPad would be, and soon had an iPod Touch to complement the iPad.

Make no mistake: for me, the iPod Touch was a miniature iPad; in no way was the iPad ever like an iPod Touch on steroids. And I still carried my Blackberry in order to "get work done."

I soon discovered the limitations of he WiFi-only lifestyle: while WiFi has become fairly ubiquitous in my town and areas where I work and live, coverage just isn't universal. One solution would be an upgrade of the iPad by replacing it with a 3G model; another would be replacing the Touch with an iPhone. Unfortunately, both solutions leave the other device "unconnected", force you into multiple data plans, or force you to leave/change carriers. My solution was to add the Verizon MiFi.

That choice turned out to be genius: not only did it answer the need I had for the two iOS devices, it also filled the gap for my laptop use. Now I didn't need to plan my on-call outings around local hotspots: the hotspot came with me. On more than one occasion, having the MiFi in the car gave me and/or my wife constant connectivity for all the devices at hand.

So this setup worked fine for the better part of a year: I'd keep my Blackberry for phone and email; the MiFi was "always on" and available in the car; use the iPod for casual gaming and web browsing; the iPad was a pure entertainment device—movies, social networking, Angry Birds HD—while still keeping a laptop and desktop for "real work". My business life was primarily supported on the business Blackberry's phone/data plan, and my personal life was primarily supported on the personal MiFi data plan.

When the Verizon iPhone finally made its appearance in 2011, I decided to take the plunge: I'd eject the Blackberry and iPod in favor of the iPhone in order to pilot the device for the company.

Results:
The iPhone on Verizon is a very capable phone. Shortly after my switch, I had to work with a vendor's tech support for an issue and spent over 4 hours on a call. This is no feat for a Verizon Blackberry, but is anecdotally known to be problematic on AT&T iPhones.

As expected, all my iOS applications were able to move from the iPod to the iPhone, but in order to match some of the corporate Blackberry functionality, I had to add a couple additional apps: While iPhone supports Exchange ActiveSync (EAS), [in my environment] it only supports synchronization of mail, contacts and calendar. I'm a regular user of Tasks and Notes, so iMExchange 2 was my fix. Later, I learned that the only way to manage out-of-office notifications "natively" is through Safari, so I added iGone to improve the experience a bit. Great: free functionality on the Blackberry can only be achieved with $4 in applications on the iPhone.

But wait; there's more: EAS, VPN, iMExchange and any other interactions with the corporate network rely on your Windows/Active Directory logon. When your corporate policy requires you to change your password on a regular basis, you must manually change it on your iOS devices to match. And you better do it quickly, or your devices could fail to logon enough times to lock you out of your account. Yet another place where the Blackberry excels (because it uses a proxy agent to access your stuff); and unlike my other conversion issues, "there's no app for that."

As a replacement for the iPod Touch, I really appreciate the additional functionality of the GPS, accelerometer and always-on connectivity (the WiFi-only devices drop their connections when in stand-by, while iPhone switches to 3G when in standby).

Other nits:
  • the Calendar app on iPhone—whether a side-effect of EAS or a "standard" iOS functionality—removes stubbed-in meeting invitations. Before I started using iPhone, meeting invites were always written to my calendar as "tentative"; they stayed that way until I chose to do something about them. With iPhone, those tentatives are removed and aren't part of my calendar at all if I don't take some sort of action.
  • Sound/vibrate on iPhone—With iPhone, you have two states for sound/vibration: "noisy" and "silent". While you have a great deal of control over the ringtones for phone calls, you are otherwise brutally limited in your control over notification sounds and/or vibration. In contrast, Profiles on the Blackberry are wonderful. Not only can you set per-contact notification styles, you can set up various sound-only, vibrate-only and sound+vibrate profiles for various message types. Further, on the newer Blackberrys, you can set up a "bedside mode" profile that is automatically invoked when some combination of "plugged in" and time rules are met. In my case, I would switch to "phone only" from 11pm to 6am when on the charger, and "vibrate only" at any other time.
  • Virtual Keyboard on iPhone (and to be fair, any touch-only device) is nowhere as reliable or responsive as the thumb-board of the Blackberry. To be fair, I think some of the Blackberry models have terrible keyboards, too, but the frequency in which I accidentally hit "delete" instead of "m" on iPhone is just appalling. And Apples insistence that no 3rd parties "muck with" the keyboard means there will never be any real improvements in it.
  • Email Management: The only place iPhone beats Blackberry for email management is support for multiple Exchange connections (starting in iOS 4); there's no way to connect to multiple BES servers using Blackberry. That said, however, the Blackberry is the hands-down champion for email management. The Blackberry "knows" that it's a messaging platform. When a new message comes in, the device can be configured to jump directly to it when unholstered or awakened from "sleep." In contrast, messaging is just another app (albiet built-in) on iPhone. When a new message comes in, you may get notification, but when you invoke the app, it returns to the last state you left it; sometimes, that requires 3-4 taps to get to the new message. In addition, with iPhone, you can't:
    • Filter messages from being forwarded to the handheld; Blackberry has a robust ruleset for managing this.
    • Select multiple messages for mark-as-read/unread (although you can move or delete multiple messages) or mark read before a given date—I used this function all the time on my BB.
    • Delete on the handheld but leave on the server.
Wrap-up:
So that's it: I've joined many of you that are in a love/hate relationship with an iPhone. The Apple hegemony is set up so that I'll likely never see my "nits" addressed, and some of them—like the EAS password issue—are deal-breakers for my recommendation as a replacement for a business Blackberry. Personally, I'm not ready to give up on iPhone and go back to a Blackberry, but I've been on one long enough to have interest in what RIM decides to do with Qnx after they get the Playbook released. If they manage to breathe new life into their devices and put some good, basic functionality back into their OS, I can see myself going back to Blackberry for my corporate life.