Tuesday, September 14, 2010

Boxee pre-order available (why would you?)

So Boxee is now availble for pre-order, according to the press-release I was emailed:


D-Link has signed up Amazon http://amzn.to/theboxeeboxbydlink (in the US) and Best Buy http://www.bestbuy.ca/boxee / Future Shop http://www.futureshop.ca/boxee (in Canada) as exclusive pre-order partners for the Boxee Box.

 

The highlights they point out:


  • it will have access to more HD content than its PC cousin


  • no need for keyboard/mouse in the living room or running a 10ft cable to connect your laptop


  • it’s beautiful, though a bit pointy in parts : )

 

Beauty is in the eye of the beholder I guess, I don't care for it myself:

201009141143

It's OK I guess, but that needs to sit next to your TV. It's only 4.5" x 4.5" x 4.6" so it's not too big, you could probably squeeze it on top of your DVR or cable box, but the area around it becomes unusable space for me. The one thing I loved: The remote. It has a basic 4-axis control pad with a 'select' button, a play/pause and what looks to be a power button. That's all run-of-the-mill stuff. The cool stuff is on the flip-side. Flip the remote over and it has a full keyboard. If you've ever tried to search on an AppleTV, Tivo or Cable remote you know how huge this is. No word so far on if it supports Hulu. The big thing for me is the price. $199 per unit. It's $100 more than the new Apple TV and $140 more than a Roku. Unless they have Hulu, I think the Boxee box is DOA....

 

Friday, January 8, 2010

My kids will never have a 'must see TV' night

There was a time when Thursday nights were 'must see TV' nights. Friends, Seinfeld and ER made a pretty compelling night of 'must see TV' as the slogan went. A significant portion of the nation would be sharing the same experience Thursday nights. I remember getting up early on a Saturday, even though it wasn't a school day because that's when the good cartoons where on. I doubt my kids will ever have that notion or experience. Between DVR/TiVo, On Demand broadcasts, Web delivery and AppleTV there's not as big a driver to sit down at a scheduled day and time and watch.


For the past few years we only had terrestrial broadcast TV and an AppleTV in the house. One time, while watching Arthur on PBS, Renee had to go to the bathroom and was jumping up and down demanding we 'pause it!', not understanding that not all TV shows were like the AppleTV version. A few months back we added Cable to the house and a DVR unit, so now we can pause live TV, further blurring the distinction between the on demand vs on schedule showings. She doesn't understand why she can't watch Zula Patrol any time she wants and has to wait until 7:30 to see it. She's convinced its something that I'm not doing for her, and not a case of it not being available On Demand or via AppleTV. Further there's only one episode to watch and when it's done it's done. With all her other shows there's always another episode, so she asks to watch another Zula Patrol, and i have to tell her no. From her perspective it's no different then saying she can only watch one Super Why. We have more, I'm just not allowing her to watch them. Except in this case, we legitimately don't have more to watch. Kind of works in my favor I guess, there's no chance I'll cave in and let her watch a 2nd one.


As more and more entertainment options become less tied to the providers schedule and less tied to the TV as the only way to watch it, the notion of a good night for TV will wither away. There will still be some notion of scheduling, but it'll be the date and time it's put on the distribution network. It probably won't be the same though. Even for me it's not quite the same. I love CBS's monday line-up, but I don't think I've watched any of them at there actual broadcast time in 2 or 3 years. I don't think TV's dead or going away. As Randall Hounsell put it, TV is still "a lean back experience." People will still want to get someplace comfy and be immersed in a world that isn't there own.


Back when CD's were the norm, but vinyl records were still around, my over-used joke used to be that my kids were going to be asking me "Dad, how do we get this big black disc into the CD player?" Now I'm not so sure they'll even remember what a CD is. Never thought the same would happen for TV. The Qwest commercial from the late-90's is finally coming true.







A tired man goes into a cheap motel in the middle of nowhere and asks about amenities. When he asks about entertainment, the girl responds "all rooms have every movie ever made in any language anytime day or night." It'll probably be 20 years after the fact, but it's coming.


Thursday, December 31, 2009

Getting there from here

I've started this rambling post four or five times. Deleted it every time. So I've decided to take a different tact. Rather than writing a long post that says everything, I'm going to write pieces of it. This way I'll actually get something written. As the Japanese proverb goes "Vision without action is a daydream. Action without vision is a nightmare.” All too often I've run into folks who are long on vision but short on action. A few times I've been in organizations that are restructuring to better cope with the current environment. The ever present 're-org'. One of those was the transformation into a service delivery organization. Which was a good idea and there was good vision behind it. The action is where the idea died and cost people their jobs.

I've had this picture hanging around for a few years. I stole it from an issue of eWeek. It was from one of those articles that is really nothing more than a advertisement in essay form.
Cci00001

This picture does a pretty good job at explaining what I mean. Although it's not the 'buzz word' it used to be, being a 'service based organization' was the goal of a lot of IT organizations. On paper it looks great. It can be an effective way to run an organization. Unfortunately the trick is getting from where you are to where you want to be. It's been all to common for organizations to 'green field' the new way of doing things and make a sweeping change to transform into the desired structure in the shortest possible time frame. And it's usually a disaster for the first two quarters. There's a lot of uncertainty on how things get done or who does them. Process bottlenecks creep up everywhere. There's inconsistency in implementation between teams. And while all of this is going on, real work needs to get done to keep the business going. After a while people start to revert to the old way of doing things or a hodge-podge in between the old and new.

 

The step that gets missed is the transition and how much transition can be achieved in one fell swoop. If you're currently a 'turmoil' or 'reactive' organization and you want to be a service-based organization, it's unrealistic to jump right to the end state. With out learning the lessons that come with being reactive, it's difficult to be proactive. If an organization doesn't have a solid proactive foundation, it can never be service based. Worse yet, there are budgetary considerations that go along with crossing over from one level to another. Software and hardware tools are often needed to achieve the desired state. Although often over looked in the planning stages, it's possible to make up that budgetary gap. Another gap that's overlooked is the people side of things. I have never seen an organization budget staff time and overhead to these types of changes. It's always expected to be done in the margins after a one or two hour 'training course' that typically just reads the new process aloud to everyone in attendance. No attention is paid to how to get the staff to the end goal. No real-world examples provided for how things should work. No governing authority to turn to for guidance. No one to find parts of the organization that our floundering in the new process/structure and pitch in and help them through it. Proud in their new organizational structure and plan, leadership pass it down the chain, with implementation left as an exercise to the reader.

 

So as I try to manage my team, I've tried to utilize some of the failure lessons I've learned. I don't make broad sweeping changes if avoidable. There always needs to be a balance of course, you don't want to make hundreds of small course corrections when a few larger ones will be as effective, but I lean towards the smaller changes. I plan in the overhead. If I'm going to add new processes or procedures to my staff's duties, I adjust time expectations accordingly. An example would be our post-mortems on outages. I wanted to change how that was done. It should be a 30 minute meeting, but because people were new to it, the first few where schedule for 60 to 90 minutes and we brought in lunch. Walk everyone through the new process a few times. Going back to the post-mortem example, we talked openly about the process and the actual problem in the same context. Giving people a new process with out a concrete example to work with leaves things to interpretation, and you'll get as many interpretations as you have staff members. By walking through it a few times with everyone they all here the same questions and my answers to those questions. It's not perfect or without flaws, but it seems to be working.

 

google's don't ask don't tell install problem...

Yeah, yeah, I know... nobody but me seems to care. Anyway, it seems google has decided to update 'Google Voice and Video" to version 1.0.19.1554 and "Google Software Update: to version 1.0.7.1306. What's interesting about the 2nd one is what happens now when you try to launch the old "Google Update" application:

Finderscreensnapz001

You're greeted with this:

Google updaterscreensnapz001

Only one option "Get More Google Software" which isn't what I asked for, I asked for Google updates. Ok, no problem, I'll just quit the app:

Google updaterscreensnapz002

Or maybe I won't since they've taken that choice (and the 'about' option so I can't tell they've updated it with out my knowledge perhaps). Oh well, lets take the only path given to me and lets "Get More Google Software"

Securityagentscreensnapz002

Oh fun! Three new windows. One expected, two not. The google web page with mac software, a dialog box because finder needs permissions, and an disconcerting 'preparing to move to trash' dialog box. I just wanted to see if I was running the latest version of google earth and now I'm given this? Worse yet, is there's not much context for the permissions request:

Securityagentscreensnapz003

As before, it's not so much about the actual changes google has made to my machines, but the lack of permission they sought from me to do it. This one is particularly troublesome for me because they don't really give me any choice but to take their path and that path isn't clearly explained in their dialog boxes.

Tuesday, July 21, 2009

My first run-in with DotHIll storage

So I'm having my first run-in with a DotHill Array (rebranded as an HP product). The HP model number is MSA2012fc but the DotHill number would be a 2730. It's your typical looking 3U 12 disk array. It has two controllers and two 4Gb uplinks per controller. Not too bad. It only does whole-disk raid sets however so it'd be a little silly to plug it into a SAN switch, but it can done. The HP web interface is pretty straight forward and has simple concepts, you create a vdisk, which is a raid set of drives and then you carve off chunks of that vdisk to present as LUNs to the hosts. Similar to the old school Clariion's, it has the notion of assigning a lun to a particular controller. So you have to manually/mentally balance your workload. It does provide some fairly comprehensive performance stats which can help in that regard and there's a command line interface w00t!. So far my results with these devices hasn't been good at all.

First off, I should point of we have two of these. One works, one doesn't. On the one that works, when I run a simple Bonnie++ test I get results like this:

Version 1.95 Sequential Output Sequential Input Random
Concurrency 1 Per Chr Block Rewrite Per Chr Block Seeks
OS RHEL 5.3 K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
size 118G 749 99 172335 42 66940 15 1518 59 199006 18 160.0 24
Latency 11964us 25145ms 3660ms 346ms 571ms 45881us
Version 1.95 Sequential Create Random Create
Create Read Delete Create Read Delete
/sec %CPU /sec %CPU /sec %CPU /sec %CPU /sec %CPU /sec %CPU
28611 66 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 13800us 108us 1157us 1328us 13us 1145us


Not very good at all, especially considering it has 4GB of cache and this is the only test running. I mean 25145ms worth of latency for the block output? Yikes!

On the second one, it seems to work ok. I can do some "dd if=/dev/random of=somefile" of arbitrary size and it happily chugs along. But if you try to run a Bonnie++ run against it you get a lot of errors:

Jul 21 10:09:36 oracle-dev-02 kernel: : exe="?" (sauid=81, hostname=?, addr=?, terminal=?)'
Jul 21 11:00:50 oracle-dev-02 kernel: sd 0:0:0:1: SCSI error: return code = 0x08000002
Jul 21 11:00:50 oracle-dev-02 kernel: sda: Current: sense key: Aborted Command
Jul 21 11:00:50 oracle-dev-02 kernel:     Add. Sense: Scsi parity error
Jul 21 11:00:50 oracle-dev-02 kernel:
Jul 21 11:00:50 oracle-dev-02 kernel: end_request: I/O error, dev sda, sector 934774511
Jul 21 11:00:50 oracle-dev-02 kernel: Buffer I/O error on device sda1, logical block 116846806
Jul 21 11:00:50 oracle-dev-02 kernel: lost page write due to I/O error on sda1
Jul 21 11:00:50 oracle-dev-02 kernel: Buffer I/O error on device sda1, logical block 116846807
Jul 21 11:00:50 oracle-dev-02 kernel: lost page write due to I/O error on sda1
Jul 21 11:00:50 oracle-dev-02 kernel: Buffer I/O error on device sda1, logical block 116846808
Jul 21 11:00:50 oracle-dev-02 kernel: lost page write due to I/O error on sda1
Jul 21 11:00:50 oracle-dev-02 kernel: Buffer I/O error on device sda1, logical block 116846809
Jul 21 11:00:50 oracle-dev-02 kernel: lost page write due to I/O error on sda1
Jul 21 11:00:50 oracle-dev-02 kernel: Buffer I/O error on device sda1, logical block 116846810
Jul 21 11:00:50 oracle-dev-02 kernel: lost page write due to I/O error on sda1
Jul 21 11:00:50 oracle-dev-02 kernel: Buffer I/O error on device sda1, logical block 116846811
Jul 21 11:00:50 oracle-dev-02 kernel: lost page write due to I/O error on sda1
Jul 21 11:00:50 oracle-dev-02 kernel: Buffer I/O error on device sda1, logical block 116846812
Jul 21 11:00:50 oracle-dev-02 kernel: lost page write due to I/O error on sda1
Jul 21 11:00:50 oracle-dev-02 kernel: Buffer I/O error on device sda1, logical block 116846813
Jul 21 11:00:50 oracle-dev-02 kernel: lost page write due to I/O error on sda1
Jul 21 11:00:50 oracle-dev-02 kernel: Buffer I/O error on device sda1, logical block 116846814
Jul 21 11:00:50 oracle-dev-02 kernel: lost page write due to I/O error on sda1
Jul 21 11:00:50 oracle-dev-02 kernel: Buffer I/O error on device sda1, logical block 116846815
Jul 21 11:00:50 oracle-dev-02 kernel: lost page write due to I/O error on sda1
Jul 21 11:00:50 oracle-dev-02 kernel: Aborting journal on device sda1.
Jul 21 11:00:50 oracle-dev-02 kernel: __journal_remove_journal_head: freeing b_committed_data
Jul 21 11:00:50 oracle-dev-02 last message repeated 5 times

Wee fun! Even better is it corrupts the disk beyond recognition. You can't even fsck it in a reasonable about of time. It's faster to reformat. We've had HP on site and opened a a case or two on this but so far have gotten no where. For me the biggest issue is I don't have enough time to sit on hold with the call center while they find someone who knows how to use linux with an MSA.

So far I've tried swapping HBA's, cables, ports, fibers. Updating all the firmware and drivers (the MSA, the HBA, OS, etc). I've tried different versions of RHEL 5.3 and 5.2. I've tried only using the HP supplied drivers, firmware and utils. All with the same results. Later this week, I'm going to give in and install win2k3 or win2k8 and run something like Bst5 or iozone and hope I can reproduce the error. Under low loads it doesn't error out. It performs poorly but doesn't error out. I hate these kinds of problems. There's obviously something wrong with the array, since one works and the other doesn't, but it passes all the diags. At some point this thing's going to end up like the printer in Office Space.

Friday, July 10, 2009

Getting back to linux...

Back in my university days I was all about Linux. My first 'machine' was a 386sx, probably 16Mhz or so and booted of a 5.25" floppy. Having to compile the kernel every time you wanted to make any kind of changes and then 'rawrite' it out of the floppies. And forget about package management (well until slackware for me...) My first 'workstation/server' that I seriously used, scuba.uwsuper.edu, was a 486DX-50 with a Cirix CPU around 1992. I think it might have had 256MB of Ram and a 80gb Segate drive (3.5" form factor no less!). I think some archive.org listings for the web pages I used to host on it are still around, although from near the end of my use of it. http://web.archive.org/web/*/scuba.uwsuper.edu Good times...


Then I moved to the DC area and started working with Sun and AIX hardware. Linux moved to a novelty/side item for me. RedHat, back when you could run it and not pay for it if you didn't want to. I'd have a 2nd PC in my office, mostly to act as my X server for working with the Sun boxes more than anything else. At Convergys and Red Cross, we had linux. A fair bit of it too, but in most cases it was never the 'core' of the product/platform offerings.


Well my current job with StreamSage is primarily a Linux shop, in particular a RedHat shop (Comcast, the corporate parent is a large RedHat customer). So it's been an interesting time getting back into the swing of things. On the one hand, I really like getting back into the linux state of mind. On the other hand, I've really come to appreciate the work that has been done in AIX and Solaris in terms of hardware management, diagnostics and configuration. I mean there are Linux equivalents in a lot of cases and a lot of it is an artifact of the hardware and software being built by the same people but boy, I miss the AIX and Solaris troubleshooting tools.


The unpredictable future and buying hardware...

At my previous clients site, they have some old Sun servers that they are upgrading to M5000's. The hardware they evaluated (and they did very rigorous testing of the hardware) was a T5240, T5440 and M5000. The M5000 was configured with 4 processors not the full 8 that are possible, which is the subject of my post. When choosing the M-series server they decided to go with the M5000 because it would have 2 free slots allowing them to add memory or processors later. So they are trying to protect themselves against a CPU utilization problem down the road by having slots to put additional capacity in. I've been down this road a few times myself. I bought the 880s and 890s with 4 procs just in case we needed the other 4 down the road. Unfortunately most of the time I never needed those slots. and wasted the rack space, power and cooling. In my current clients case they should probably go with the M4000 instead.


On a list price comparison over five years you get:


M4000 $66,380 M5000 $81,880


Maintenance (numbers are swags for platinum pre-paid for 3 years):


M4000 $17,000 M5000 $22,000


I'm typically a Veritas user, so that adds complexity. Last time I looked (a year and a half ago) the M4000 was a tier E and the M5000 was a tier H. So SF for oracle for both would be:


M4000 $4,000 M5000 $9,000


Veritas Maintenance would be somewhere around (swag, 3 years):


M4000 $2,400 M5000 $2,700


The rest a bit of a wash, and brings us to:


M4000 $89,760 M5000 $115,580.


So the M5000 which has two advantages: 4 internal drives (no value unless you're partitioning) and 2 expansion slots (potential future value) has a 28% greater price premium over the M400. The reason engineers like myself make choices like this the unpredictable future. Often times when I'm asked to spec out hardware for an application I'm given initial requirements like 12,000 total users, 300 users concurrently. And if I'm lucky some information about the resource utilization associated with each user session. Most the time it's a shot in the dark however, and have to dig around for similar usage profiles via google and try to work that into my sizing model. But that's relatively straight forward. There's some art and finesse to it but at the end of the day it usually comes down to a derivative formula of X sessions * Y-Mb-per-Session + overhead + wiggle-room = ZGb of memory. Same kind of thing for CPU and I/O. Where it gets hard is when you have to forecast the life of the machine. You're forced to try and pick a machine that will meet the needs of not only year one but years two through four or five as well. When we ask the customer what their growth rate is they'll usually shrug and give a non-answer. Or they'll give you an answer that's based directly on other non-knowable facts like "our user base will increase at the same percentage as our market share". Great. Thanks for that. It's very tempting to just go out and buy the top of the line server to ensure we never get a resource problem. Buy a tour bus when all we need is a passenger van. But when they see the sticker price of that tour bus, we're usually back to the drawing board. That's what makes machines like the M5000 or the 890 it replaced so appealing. It has room for an extra row of seats in case the number of passengers we need increases drastically. Unfortunately you have to pay extra fuel costs to hall that extra space around (maintenances) and there's the up-front acquisition costs as well.


It's all about going back to the well. The reason we over-build or infrastructure this way is because of the difficulty in going back to the well for additional funding. In my work in the non-profit space there's a real risk of that well being dry as well. For example, I could buy the M4000 and then if I have problems in year 2 or 3 I would then do a forklift upgrade to the M5000 (swap the boot disks and away I go). Easy stuff, except I have to actually buy that M5000. Which comes with lots of questions: Why didn't you buy an M5000 from the start? Why were your forecasts wrong? Where do you think we can come up with that kind of money? Collective amnesia will shift all the blame to the people who spec'ed out the system. Blame rolls down hill. It picks up mass and speed as it rolls and engineers are usually at the bottom of the hill with the operations folks (often one and the same). So we buy machines that have that 'extra reserve' built in. In high end servers you can usually turn on the additional capacity by purchasing a license key. But in the mid-to-low end range we're only offered machines with expandability. So if our forecast is off or the conditions change, we're able to bring a lower incremental cost to the table to gain additional performance and capacity. Unfortunately for me however, I have rarely needed that expanded capacity. I can only remember two examples one success, were we added two boards (4 CPUs + memory) to an 890 and one case where they didn't make the versions of the board we currently had in the server which meant we would have had to replace all the boards which would have cost almost as much as replacing the server outright.


I used to be a 'keep something in reserve' kind of engineer. Be able to put the rabbit out of the hat to meet the increased demand that we didn't know was coming. Basically pull of a Montgomery Scott to save the day. By doing so however, I have enabled the behavior that has gotten me here in the first place. By not purchasing the equipment the requirements suggest and adding some reserve "just in case" the cycle repeats itself. Now I'm not going to purchase the minimum needed to meet the requirements given to me (however flawed they may be), but I am going to start putting the decision back on the requesters and have them make the choice. In writing. With as much concurrence as can be achieved from the project team as a whole. So if I were to travel back in time to the period before the aforementioned M5000s were purchased. I would offer the M4000s instead. I would tell them you save X dollars up front. Your downside risk is you may have to replace this server if your usage or growth models are wrong. And, perhaps most importantly, get documented concurrence from the stake holders.


This has turned into a much longer post than I had originally intended... phew. Now onto my next client/project.